From erik.helin at oracle.com Mon Oct 2 12:22:37 2017 From: erik.helin at oracle.com (Erik Helin) Date: Mon, 2 Oct 2017 05:22:37 -0700 Subject: RFR: 8179387: Factor out CMS specific code from GenCollectedHeap into its own subclass In-Reply-To: <51036840-16f4-c8eb-082e-76e0a5b1534f@redhat.com> References: <47e22e86-7d7c-606f-1936-346229f39ca2@oracle.com> <9a846161-c8ac-dedf-5952-f457d546fd9a@redhat.com> <4d5e6af8-d975-7803-64c5-7295e0d56154@redhat.com> <13358626-e399-e352-1711-587416621aac@redhat.com> <27af0ad2-fe78-3536-2143-996dd42583ab@oracle.com> <4bc53aaa-b98a-8a61-73bf-d30ac3f402b8@redhat.com> <666af7f2-27e9-48c6-91e4-eaefa5289e18@redhat.com> <3ec8a6a3-5a4b-a910-f6ec-ed1c0dad4cad@oracle.com> <5417889c-5289-37cd-eb31-a2b55f70e85e@redhat.com> <088d467c-8038-60bc-1eab-b34061ad20d9@redhat.com> <7abbeec1-b353-c6e9-9827-e70f49269d50@oracle.com> <6e3b0ca0-68c9-f5b6-1cb7-ddca71df2c0e@redhat.com> <51d7f145-27e3-5bfb-f1da-decd6e19858c@oracle.com> <51036840-16f4-c8eb-082e-76e0a5b1534f@redhat.com> Message-ID: <2c1bdd7c-005c-4376-cd9f-3db739165bc2@oracle.com> On 09/29/2017 03:47 AM, Roman Kennke wrote:>>> Differential: >>> http://cr.openjdk.java.net/~rkennke/8179387/webrev.08.diff/ >>> >> >> Thanks, this is good. I don't know enough about the rest of the >> change to be a reviewer, but I think you have your reviews. >> Coleen >> >>> Full: >>> http://cr.openjdk.java.net/~rkennke/8179387/webrev.08/ >>> > > Ping? I believe this never actually went in. Is there anything missing? Sigh, sorry, I completely forgot about this patch, thanks for pinging. Yes, this looks good, and thank you for taking on this work! > Do you want me to re-base it onto the single-repo and post another webrev? If you don't mind, then yes, please rebase it on top of the consolidated repo. Then I can sponsor it for you. Thanks, Erik > Roman From rkennke at redhat.com Mon Oct 2 14:54:21 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 2 Oct 2017 16:54:21 +0200 Subject: RFR: 8179387: Factor out CMS specific code from GenCollectedHeap into its own subclass In-Reply-To: <2c1bdd7c-005c-4376-cd9f-3db739165bc2@oracle.com> References: <9a846161-c8ac-dedf-5952-f457d546fd9a@redhat.com> <4d5e6af8-d975-7803-64c5-7295e0d56154@redhat.com> <13358626-e399-e352-1711-587416621aac@redhat.com> <27af0ad2-fe78-3536-2143-996dd42583ab@oracle.com> <4bc53aaa-b98a-8a61-73bf-d30ac3f402b8@redhat.com> <666af7f2-27e9-48c6-91e4-eaefa5289e18@redhat.com> <3ec8a6a3-5a4b-a910-f6ec-ed1c0dad4cad@oracle.com> <5417889c-5289-37cd-eb31-a2b55f70e85e@redhat.com> <088d467c-8038-60bc-1eab-b34061ad20d9@redhat.com> <7abbeec1-b353-c6e9-9827-e70f49269d50@oracle.com> <6e3b0ca0-68c9-f5b6-1cb7-ddca71df2c0e@redhat.com> <51d7f145-27e3-5bfb-f1da-decd6e19858c@oracle.com> <51036840-16f4-c8eb-082e-76e0a5b1534f@redhat.com> <2c1bdd7c-005c-4376-cd9f-3db739165bc2@oracle.com> Message-ID: <7a9cdfff-bcd5-b616-f512-78adf42c70f9@redhat.com> Am 02.10.2017 um 14:22 schrieb Erik Helin: > On 09/29/2017 03:47 AM, Roman Kennke wrote:>>> Differential: >>>> http://cr.openjdk.java.net/~rkennke/8179387/webrev.08.diff/ >>>> >>> >>> Thanks, this is good.? I don't know enough about the rest of the >>> change to be a reviewer, but I think you have your reviews. >>> Coleen >>> >>>> Full: >>>> http://cr.openjdk.java.net/~rkennke/8179387/webrev.08/ >>>> >> >> Ping? I believe this never actually went in. Is there anything missing? > > Sigh, sorry, I completely forgot about this patch, thanks for pinging. > Yes, this looks good, and thank you for taking on this work! > >> Do you want me to re-base it onto the single-repo and post another >> webrev? > > If you don't mind, then yes, please rebase it on top of the > consolidated repo. Then I can sponsor it for you. Ok, here it comes: http://cr.openjdk.java.net/~rkennke/8179387/webrev.11/ I've run hotspot_gc tests again. Looking good. Thanks for reviewing and sponsoring! Roman From david.holmes at oracle.com Tue Oct 3 02:29:58 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 3 Oct 2017 12:29:58 +1000 Subject: Status of AppCDS In-Reply-To: References: Message-ID: <1e21205c-b462-5dc8-69a4-985fae6dd021@oracle.com> Moving to hotspot-runtime-dev and bcc'ing the discuss list. Hello Augusto, On 3/10/2017 7:25 AM, August Nagro wrote: > Hello, > > Both Class Data Sharing (CDS) [1] and AppCDS [2] are very interesting but > seemingly neglected features of the Java Platform, offering the ability to > reduce startup time. AppCDS is under continual development and is not "neglected" at all. > Class Data Sharing is the global cache stored in > /lib/[arch]/server/classes.jsa, and circumvents long class-loading times by > caching the JVM?s internal representation of system jars and memory-mapping > them in during startup. The feature's documentation [1] has been updated > for Java 9, but I have yet to see a JDK or JRE installation that can use it > without manually generating the archive. In every case I've encountered, > the file needs to first be created (and given appropriate access > permissions) with admin access, and then generated by `java -Xshare:dump`. Yes you have to create the shared archive if not using a JRE installer. But that can be done by whomever deploys the JRE in your environment. My understanding is that, due to the format of the archive, a given archive is only guaranteed to beusable on the machine on which it was created. > AppCDS extends the same feature to user libraries. It works well, but has > been experimental since (before?) JDK 8 and is commercial. Not sure it is still considered "experimental" but it is still evolving. It is a commercial feature in Java 9 but will not be in the future: http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html > > Java has been criticized for its slow startup times. Improvements would > enable command line utilities, the realistic use of JShell .jsh scripts > (currently takes >4 seconds for hello world on my Mac [3]), and other > applications. I'm not sure how *CDS concerts with other OpenJDK efforts > like AOT compilation, but it offers the advantage of architecture > independency and the ability to be shared across multiple JVMs. An archive can be shared across multiple JVMs on the same machine. > Is *CDS doomed for deprecation, or will it be improved? Without work to CDS > or changes to AppCDS, the current state is not very usable. *CDS has been constantly improved over the last few releases and continues to be improved for the next versions of the platform. Regards, David Holmes > Regards, > > August Nagro > > [1]: https://docs.oracle.com/javase/9/vm/class-data-sharing.htm > [2]: > http://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-31503FCE-93D0-4175-9B4F-F6A738B2F4C4 > > [3]: > Executing command `jshell helloworld.jsh`, with file containing > ``` > System.out.println("hello world"); > /exit > ``` > From david.holmes at oracle.com Tue Oct 3 02:35:52 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 3 Oct 2017 12:35:52 +1000 Subject: Status of AppCDS In-Reply-To: <1e21205c-b462-5dc8-69a4-985fae6dd021@oracle.com> References: <1e21205c-b462-5dc8-69a4-985fae6dd021@oracle.com> Message-ID: On 3/10/2017 12:29 PM, David Holmes wrote: > Moving to hotspot-runtime-dev and bcc'ing the discuss list. > > Hello Augusto, Apologies - August. David > On 3/10/2017 7:25 AM, August Nagro wrote: >> Hello, >> >> Both Class Data Sharing (CDS) [1] and AppCDS [2] are very interesting but >> seemingly neglected features of the Java Platform, offering the >> ability to >> reduce startup time. > > AppCDS is under continual development and is not "neglected" at all. > >> Class Data Sharing is the global cache stored in >> /lib/[arch]/server/classes.jsa, and circumvents long class-loading >> times by >> caching the JVM?s internal representation of system jars and >> memory-mapping >> them in during startup. The feature's documentation [1] has been updated >> for Java 9, but I have yet to see a JDK or JRE installation that can >> use it >> without manually generating the archive. In every case I've encountered, >> the file needs to first be created (and given appropriate access >> permissions) with admin access, and then generated by `java >> -Xshare:dump`. > > Yes you have to create the shared archive if not using a JRE installer. > But that can be done by whomever deploys the JRE in your environment. My > understanding is that, due to the format of the archive, a given archive > is only guaranteed to beusable on the machine on which it was created. > >> AppCDS extends the same feature to user libraries. It works well, but has >> been experimental since (before?) JDK 8 and is commercial. > > Not sure it is still considered "experimental" but it is still evolving. > It is a commercial feature in Java 9 but will not be in the future: > > http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html > >> >> Java has been criticized for its slow startup times. Improvements would >> enable command line utilities, the realistic use of JShell .jsh scripts >> (currently takes >4 seconds for hello world on my Mac [3]), and other >> applications. I'm not sure how *CDS concerts with other OpenJDK efforts >> like AOT compilation, but it offers the advantage of architecture >> independency and the ability to be shared across multiple JVMs. > > An archive can be shared across multiple JVMs on the same machine. > >> Is *CDS doomed for deprecation, or will it be improved? Without work >> to CDS >> or changes to AppCDS, the current state is not very usable. > > *CDS has been constantly improved over the last few releases and > continues to be improved for the next versions of the platform. > > Regards, > David Holmes > >> Regards, >> >> August Nagro >> >> [1]: https://docs.oracle.com/javase/9/vm/class-data-sharing.htm >> [2]: >> http://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-31503FCE-93D0-4175-9B4F-F6A738B2F4C4 >> >> >> [3]: >> Executing command `jshell helloworld.jsh`, with file containing >> ``` >> System.out.println("hello world"); >> /exit >> ``` >> From jiangli.zhou at Oracle.COM Tue Oct 3 15:51:37 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Tue, 3 Oct 2017 08:51:37 -0700 Subject: Status of AppCDS In-Reply-To: <1e21205c-b462-5dc8-69a4-985fae6dd021@oracle.com> References: <1e21205c-b462-5dc8-69a4-985fae6dd021@oracle.com> Message-ID: <5680D4A2-4857-478B-8268-937EBC9507FD@oracle.com> Hi August, Just to add some info in addition to David?s response. CDS is continuously being enhanced since JDK 8. Regarding how CDS works with AOT, the shared archive generated by CDS can coexist with the AOT library and used by JVM at runtime. Here is a very informative blog wrote by Matthew Gilliard, http://mjg123.github.io/2017/10/02/JVM-startup.html . It gives examples of how to use CDS and AOT to improve startup time. Regards, Jiangli > On Oct 2, 2017, at 7:29 PM, David Holmes wrote: > > Moving to hotspot-runtime-dev and bcc'ing the discuss list. > > Hello Augusto, > > On 3/10/2017 7:25 AM, August Nagro wrote: >> Hello, >> Both Class Data Sharing (CDS) [1] and AppCDS [2] are very interesting but >> seemingly neglected features of the Java Platform, offering the ability to >> reduce startup time. > > AppCDS is under continual development and is not "neglected" at all. > >> Class Data Sharing is the global cache stored in >> /lib/[arch]/server/classes.jsa, and circumvents long class-loading times by >> caching the JVM?s internal representation of system jars and memory-mapping >> them in during startup. The feature's documentation [1] has been updated >> for Java 9, but I have yet to see a JDK or JRE installation that can use it >> without manually generating the archive. In every case I've encountered, >> the file needs to first be created (and given appropriate access >> permissions) with admin access, and then generated by `java -Xshare:dump`. > > Yes you have to create the shared archive if not using a JRE installer. But that can be done by whomever deploys the JRE in your environment. My understanding is that, due to the format of the archive, a given archive is only guaranteed to beusable on the machine on which it was created. > >> AppCDS extends the same feature to user libraries. It works well, but has >> been experimental since (before?) JDK 8 and is commercial. > > Not sure it is still considered "experimental" but it is still evolving. It is a commercial feature in Java 9 but will not be in the future: > > http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html > >> Java has been criticized for its slow startup times. Improvements would >> enable command line utilities, the realistic use of JShell .jsh scripts >> (currently takes >4 seconds for hello world on my Mac [3]), and other >> applications. I'm not sure how *CDS concerts with other OpenJDK efforts >> like AOT compilation, but it offers the advantage of architecture >> independency and the ability to be shared across multiple JVMs. > > An archive can be shared across multiple JVMs on the same machine. > >> Is *CDS doomed for deprecation, or will it be improved? Without work to CDS >> or changes to AppCDS, the current state is not very usable. > > *CDS has been constantly improved over the last few releases and continues to be improved for the next versions of the platform. > > Regards, > David Holmes > >> Regards, >> August Nagro >> [1]: https://docs.oracle.com/javase/9/vm/class-data-sharing.htm >> [2]: >> http://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-31503FCE-93D0-4175-9B4F-F6A738B2F4C4 >> [3]: >> Executing command `jshell helloworld.jsh`, with file containing >> ``` >> System.out.println("hello world"); >> /exit >> ``` From augustnagro at gmail.com Tue Oct 3 21:03:30 2017 From: augustnagro at gmail.com (August Nagro) Date: Tue, 03 Oct 2017 21:03:30 +0000 Subject: Status of AppCDS In-Reply-To: <1e21205c-b462-5dc8-69a4-985fae6dd021@oracle.com> References: <1e21205c-b462-5dc8-69a4-985fae6dd021@oracle.com> Message-ID: > you have to create the shared archive if not using a JRE installer. I've tried `java -Xshare:dump` on a Windows and Mac machine with JRE 9 installed with the oracle installer. In both cases I had to run the following to generate the archive: ``` sudo touch /lib/[arch]/server/classes.jsa sudo chmod 777 /lib/[arch]/server/classes.jsa java -Xshare:dump ``` > Not sure it is still considered "experimental" but it is still evolving. It is a commercial feature in Java 9 but will not be in the future: That's great; I was just going off the docs for `-XX:+UseAppCDS` on the man page for java. > *CDS has been constantly improved over the last few releases andcontinues to be improved for the next versions of the platform. Good to here! - August On Mon, Oct 2, 2017 at 9:30 PM David Holmes wrote: > Moving to hotspot-runtime-dev and bcc'ing the discuss list. > > Hello Augusto, > > On 3/10/2017 7:25 AM, August Nagro wrote: > > Hello, > > > > Both Class Data Sharing (CDS) [1] and AppCDS [2] are very interesting but > > seemingly neglected features of the Java Platform, offering the ability > to > > reduce startup time. > > AppCDS is under continual development and is not "neglected" at all. > > > Class Data Sharing is the global cache stored in > > /lib/[arch]/server/classes.jsa, and circumvents long class-loading times > by > > caching the JVM?s internal representation of system jars and > memory-mapping > > them in during startup. The feature's documentation [1] has been updated > > for Java 9, but I have yet to see a JDK or JRE installation that can use > it > > without manually generating the archive. In every case I've encountered, > > the file needs to first be created (and given appropriate access > > permissions) with admin access, and then generated by `java > -Xshare:dump`. > > Yes you have to create the shared archive if not using a JRE installer. > But that can be done by whomever deploys the JRE in your environment. My > understanding is that, due to the format of the archive, a given archive > is only guaranteed to beusable on the machine on which it was created. > > > AppCDS extends the same feature to user libraries. It works well, but has > > been experimental since (before?) JDK 8 and is commercial. > > Not sure it is still considered "experimental" but it is still evolving. > It is a commercial feature in Java 9 but will not be in the future: > > http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html > > > > > Java has been criticized for its slow startup times. Improvements would > > enable command line utilities, the realistic use of JShell .jsh scripts > > (currently takes >4 seconds for hello world on my Mac [3]), and other > > applications. I'm not sure how *CDS concerts with other OpenJDK efforts > > like AOT compilation, but it offers the advantage of architecture > > independency and the ability to be shared across multiple JVMs. > > An archive can be shared across multiple JVMs on the same machine. > > > Is *CDS doomed for deprecation, or will it be improved? Without work to > CDS > > or changes to AppCDS, the current state is not very usable. > > *CDS has been constantly improved over the last few releases and > continues to be improved for the next versions of the platform. > > Regards, > David Holmes > > > Regards, > > > > August Nagro > > > > [1]: https://docs.oracle.com/javase/9/vm/class-data-sharing.htm > > [2]: > > > http://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-31503FCE-93D0-4175-9B4F-F6A738B2F4C4 > > > > [3]: > > Executing command `jshell helloworld.jsh`, with file containing > > ``` > > System.out.println("hello world"); > > /exit > > ``` > > > From augustnagro at gmail.com Tue Oct 3 21:04:37 2017 From: augustnagro at gmail.com (August Nagro) Date: Tue, 03 Oct 2017 21:04:37 +0000 Subject: Status of AppCDS In-Reply-To: <5680D4A2-4857-478B-8268-937EBC9507FD@oracle.com> References: <1e21205c-b462-5dc8-69a4-985fae6dd021@oracle.com> <5680D4A2-4857-478B-8268-937EBC9507FD@oracle.com> Message-ID: Good article, thanks for sharing. I've also played around with Graal's `native-image` tool to generate standalone binaries, and while it's the fastest (hello world in .006ms) the tool can't compile JavaFX applications. On Tue, Oct 3, 2017 at 10:51 AM Jiangli Zhou wrote: > Hi August, > > Just to add some info in addition to David?s response. CDS is continuously > being enhanced since JDK 8. > > Regarding how CDS works with AOT, the shared archive generated by CDS can > coexist with the AOT library and used by JVM at runtime. Here is a very > informative blog wrote by Matthew Gilliard, > http://mjg123.github.io/2017/10/02/JVM-startup.html. It gives examples of > how to use CDS and AOT to improve startup time. > > Regards, > Jiangli > > > On Oct 2, 2017, at 7:29 PM, David Holmes wrote: > > Moving to hotspot-runtime-dev and bcc'ing the discuss list. > > Hello Augusto, > > On 3/10/2017 7:25 AM, August Nagro wrote: > > Hello, > Both Class Data Sharing (CDS) [1] and AppCDS [2] are very interesting but > seemingly neglected features of the Java Platform, offering the ability to > reduce startup time. > > > AppCDS is under continual development and is not "neglected" at all. > > Class Data Sharing is the global cache stored in > /lib/[arch]/server/classes.jsa, and circumvents long class-loading times by > caching the JVM?s internal representation of system jars and memory-mapping > them in during startup. The feature's documentation [1] has been updated > for Java 9, but I have yet to see a JDK or JRE installation that can use it > without manually generating the archive. In every case I've encountered, > the file needs to first be created (and given appropriate access > permissions) with admin access, and then generated by `java -Xshare:dump`. > > > Yes you have to create the shared archive if not using a JRE installer. > But that can be done by whomever deploys the JRE in your environment. My > understanding is that, due to the format of the archive, a given archive is > only guaranteed to beusable on the machine on which it was created. > > AppCDS extends the same feature to user libraries. It works well, but has > been experimental since (before?) JDK 8 and is commercial. > > > Not sure it is still considered "experimental" but it is still evolving. > It is a commercial feature in Java 9 but will not be in the future: > > http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html > > Java has been criticized for its slow startup times. Improvements would > enable command line utilities, the realistic use of JShell .jsh scripts > (currently takes >4 seconds for hello world on my Mac [3]), and other > applications. I'm not sure how *CDS concerts with other OpenJDK efforts > like AOT compilation, but it offers the advantage of architecture > independency and the ability to be shared across multiple JVMs. > > > An archive can be shared across multiple JVMs on the same machine. > > Is *CDS doomed for deprecation, or will it be improved? Without work to CDS > or changes to AppCDS, the current state is not very usable. > > > *CDS has been constantly improved over the last few releases and continues > to be improved for the next versions of the platform. > > Regards, > David Holmes > > Regards, > August Nagro > [1]: https://docs.oracle.com/javase/9/vm/class-data-sharing.htm > [2]: > > http://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-31503FCE-93D0-4175-9B4F-F6A738B2F4C4 > [3]: > Executing command `jshell helloworld.jsh`, with file containing > ``` > System.out.println("hello world"); > /exit > ``` > > > From david.holmes at oracle.com Tue Oct 3 21:07:35 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 4 Oct 2017 07:07:35 +1000 Subject: Status of AppCDS In-Reply-To: References: <1e21205c-b462-5dc8-69a4-985fae6dd021@oracle.com> Message-ID: <17bbb98d-26d3-b92b-cd25-ad3cb85f4dcc@oracle.com> On 4/10/2017 6:16 AM, August Nagro wrote: > > you have to create the shared archive if not using a JRE installer. > > I've tried `java -Xshare:dump` on a Windows and Mac machine with JRE 9 > installed with the oracle installer. In both cases I had to run the > following to generate the archive: > ``` > sudo touch /lib/[arch]/server/classes.jsa > sudo chmod 777 /lib/[arch]/server/classes.jsa > java -Xshare:dump > ``` Of course you have to have permission to write the shared archive into the default location. That's why you would normally do this as part of deloying the JRE on your machine (eg as part of the installer process on Windows). There is also an option to specify a different archive location using -XX:+UnlockDiagnosticVMoptions -XX:SharedArchiveFile=xxx David ----- > > Not sure it is still considered "experimental" but it is still evolving. > It is a commercial feature in Java 9 but will not be in the future: > > That's great; I was just going off the docs for `-XX:+UseAppCDS` on the > man page for java. > > > *CDS has been constantly improved over the last few releases > andcontinues to be improved for the next versions of the platform. > > Good to here! > > Thanks, > > - August > > On Mon, Oct 2, 2017 at 9:30 PM David Holmes > wrote: > > Moving to hotspot-runtime-dev and bcc'ing the discuss list. > > Hello Augusto, > > On 3/10/2017 7:25 AM, August Nagro wrote: > > Hello, > > > > Both Class Data Sharing (CDS) [1] and AppCDS [2] are very > interesting but > > seemingly neglected features of the Java Platform, offering the > ability to > > reduce startup time. > > AppCDS is under continual development and is not "neglected" at all. > > > Class Data Sharing is the global cache stored in > > /lib/[arch]/server/classes.jsa, and circumvents long > class-loading times by > > caching the JVM?s internal representation of system jars and > memory-mapping > > them in during startup. The feature's documentation [1] has been > updated > > for Java 9, but I have yet to see a JDK or JRE installation that > can use it > > without manually generating the archive. In every case I've > encountered, > > the file needs to first be created (and given appropriate access > > permissions) with admin access, and then generated by `java > -Xshare:dump`. > > Yes you have to create the shared archive if not using a JRE installer. > But that can be done by whomever deploys the JRE in your environment. My > understanding is that, due to the format of the archive, a given archive > is only guaranteed to beusable on the machine on which it was created. > > > AppCDS extends the same feature to user libraries. It works well, > but has > > been experimental since (before?) JDK 8 and is commercial. > > Not sure it is still considered "experimental" but it is still evolving. > It is a commercial feature in Java 9 but will not be in the future: > > http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html > > > > > Java has been criticized for its slow startup times. Improvements > would > > enable command line utilities, the realistic use of JShell .jsh > scripts > > (currently takes >4 seconds for hello world on my Mac [3]), and other > > applications. I'm not sure how *CDS concerts with other OpenJDK > efforts > > like AOT compilation, but it offers the advantage of architecture > > independency and the ability to be shared across multiple JVMs. > > An archive can be shared across multiple JVMs on the same machine. > > > Is *CDS doomed for deprecation, or will it be improved? Without > work to CDS > > or changes to AppCDS, the current state is not very usable. > > *CDS has been constantly improved over the last few releases and > continues to be improved for the next versions of the platform. > > Regards, > David Holmes > > > Regards, > > > > August Nagro > > > > [1]: https://docs.oracle.com/javase/9/vm/class-data-sharing.htm > > [2]: > > > http://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-31503FCE-93D0-4175-9B4F-F6A738B2F4C4 > > > > [3]: > > Executing command `jshell helloworld.jsh`, with file containing > > ``` > > System.out.println("hello world"); > > /exit > > ``` > > > From calvin.cheung at oracle.com Tue Oct 3 21:26:04 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 03 Oct 2017 14:26:04 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() Message-ID: <59D4006C.3010000@oracle.com> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 The open code change for this fix involves adding the _java_platform_loader to the SystemDictionary class. webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ Testing: JPRT hs-tier3 on linux-x64, macosx, solaris-sparcv9, windows-x64. thanks, Calvin From augustnagro at gmail.com Tue Oct 3 21:35:29 2017 From: augustnagro at gmail.com (August Nagro) Date: Tue, 03 Oct 2017 21:35:29 +0000 Subject: Status of AppCDS In-Reply-To: <17bbb98d-26d3-b92b-cd25-ad3cb85f4dcc@oracle.com> References: <1e21205c-b462-5dc8-69a4-985fae6dd021@oracle.com> <17bbb98d-26d3-b92b-cd25-ad3cb85f4dcc@oracle.com> Message-ID: Ah, my assumption was that the archive would be created by default by the installer. On Tue, Oct 3, 2017 at 4:07 PM David Holmes wrote: > On 4/10/2017 6:16 AM, August Nagro wrote: > > > you have to create the shared archive if not using a JRE installer. > > > > I've tried `java -Xshare:dump` on a Windows and Mac machine with JRE 9 > > installed with the oracle installer. In both cases I had to run the > > following to generate the archive: > > ``` > > sudo touch /lib/[arch]/server/classes.jsa > > sudo chmod 777 /lib/[arch]/server/classes.jsa > > java -Xshare:dump > > ``` > > Of course you have to have permission to write the shared archive into > the default location. That's why you would normally do this as part of > deloying the JRE on your machine (eg as part of the installer process on > Windows). > > There is also an option to specify a different archive location using > -XX:+UnlockDiagnosticVMoptions -XX:SharedArchiveFile=xxx > > David > ----- > > > > Not sure it is still considered "experimental" but it is still > evolving. > > It is a commercial feature in Java 9 but will not be in the future: > > > > That's great; I was just going off the docs for `-XX:+UseAppCDS` on the > > man page for java. > > > > > *CDS has been constantly improved over the last few releases > > andcontinues to be improved for the next versions of the platform. > > > > Good to here! > > > > Thanks, > > > > - August > > > > On Mon, Oct 2, 2017 at 9:30 PM David Holmes > > wrote: > > > > Moving to hotspot-runtime-dev and bcc'ing the discuss list. > > > > Hello Augusto, > > > > On 3/10/2017 7:25 AM, August Nagro wrote: > > > Hello, > > > > > > Both Class Data Sharing (CDS) [1] and AppCDS [2] are very > > interesting but > > > seemingly neglected features of the Java Platform, offering the > > ability to > > > reduce startup time. > > > > AppCDS is under continual development and is not "neglected" at all. > > > > > Class Data Sharing is the global cache stored in > > > /lib/[arch]/server/classes.jsa, and circumvents long > > class-loading times by > > > caching the JVM?s internal representation of system jars and > > memory-mapping > > > them in during startup. The feature's documentation [1] has been > > updated > > > for Java 9, but I have yet to see a JDK or JRE installation that > > can use it > > > without manually generating the archive. In every case I've > > encountered, > > > the file needs to first be created (and given appropriate access > > > permissions) with admin access, and then generated by `java > > -Xshare:dump`. > > > > Yes you have to create the shared archive if not using a JRE > installer. > > But that can be done by whomever deploys the JRE in your > environment. My > > understanding is that, due to the format of the archive, a given > archive > > is only guaranteed to beusable on the machine on which it was > created. > > > > > AppCDS extends the same feature to user libraries. It works well, > > but has > > > been experimental since (before?) JDK 8 and is commercial. > > > > Not sure it is still considered "experimental" but it is still > evolving. > > It is a commercial feature in Java 9 but will not be in the future: > > > > > http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html > > > > > > > > Java has been criticized for its slow startup times. Improvements > > would > > > enable command line utilities, the realistic use of JShell .jsh > > scripts > > > (currently takes >4 seconds for hello world on my Mac [3]), and > other > > > applications. I'm not sure how *CDS concerts with other OpenJDK > > efforts > > > like AOT compilation, but it offers the advantage of architecture > > > independency and the ability to be shared across multiple JVMs. > > > > An archive can be shared across multiple JVMs on the same machine. > > > > > Is *CDS doomed for deprecation, or will it be improved? Without > > work to CDS > > > or changes to AppCDS, the current state is not very usable. > > > > *CDS has been constantly improved over the last few releases and > > continues to be improved for the next versions of the platform. > > > > Regards, > > David Holmes > > > > > Regards, > > > > > > August Nagro > > > > > > [1]: https://docs.oracle.com/javase/9/vm/class-data-sharing.htm > > > [2]: > > > > > > http://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-31503FCE-93D0-4175-9B4F-F6A738B2F4C4 > > > > > > [3]: > > > Executing command `jshell helloworld.jsh`, with file containing > > > ``` > > > System.out.println("hello world"); > > > /exit > > > ``` > > > > > > From igor.ignatyev at oracle.com Tue Oct 3 23:29:43 2017 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Tue, 3 Oct 2017 16:29:43 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: <59CD73AC.3080207@oracle.com> References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> <5F386D9A-CA05-4732-8F68-F493DD2E8E99@oracle.com> <59CD73AC.3080207@oracle.com> Message-ID: Hi Misha, 1st of all, thank you for working on this very important piece. it generally looks great, although I have a number of mostly stylistic comments. http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html > 29 * @requires (docker.support == "true") 1. although all properties in requires context are strings, jtreg does support boolean, so this line can be simplified to '@requires (docker.support' > 31 * @modules java.base/jdk.internal.misc > 32 * @modules java.management > 33 * jdk.jartool/sun.tools.jar 2. could you please use one style? either add @modules to all lines or leave it only at L#31 > 35 * @run main DockerBasicTest 3. do we have to run DockerBasicTest in JDK under test? can't it be run as @run driver? > 46 if (!DockerTestUtils.canTestDocker()) > 47 return; 4. it's been proven to be less error prone to have { } even for one-line blocks, could you please add them here? > 72 .addDockerOpts("--volume", System.getProperty("test.classes") + ":/test-classes/"); 5. as you already depend on jdk.test.lib, it might be better to reuse Utils::TEST_CLASSES > 74 DockerTestUtils.dockerRunJava(opts). > 75 shouldHaveExitValue(0).shouldContain("Hello Docker"); 6. lines shouldn't end w/ '.', it should be moved to next line and preferable be alined w/ 1st '.' in prev line. I have seen in other files as well. I'd also like to see exact one method call per line. http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html > 47 // Diagnostics: set to true to enable more diagnostic info > 48 private static final boolean DEBUG = false; 1. I'd prefer to always have all available diagnostic info. is there any problem w/ having DEBUG=true? > 129 Files.createDirectories(jdkDstDir); > ... > 133 Files.walkFileTree(jdkSrcDir, new CopyFileVisitor(jdkSrcDir, jdkDstDir)); 2. since jdkDstDir is supposed to be empty, you can use Files::copy method to copy the dir. > 126 Path jdkSrcDir = Paths.get(System.getProperty("test.jdk")); 3. again, as you already depend on jdk.test.lib, it might be better to reuse Utils::TEST_JDK > 149 public static void > 150 buildDockerImage(String imageName, Path dockerfile, Path buildDir) throws Exception { > 213 public static OutputAnalyzer execute(boolean retainChildStdout, String... command) > 214 throws Exception { > 230 public static OutputAnalyzer execute(boolean retainChildStdout, List command) > 231 throws Exception { 4. when you wrap a line, you should use double indentation, otherwise one can misread it, in last two examples I read throws Exception as 1st statement of a method. 5. your 'execute(boolean, String...)' creates a list from 'command' array and calls 'execute(boolean, List)', which create an array from 'command' list and pass it to ProcessBuilder::new. should we make 'execute(boolean, List)' call 'execute(boolean, String...)' and move all the logic to the latter? also, there are easier ways to get a list from an arrays -- j.u.Arrays.asList(T...) or j.u.List.of(T...) Thanks, -- Igor > On Sep 28, 2017, at 3:11 PM, Mikhailo Seledtsov wrote: > > Here is the updated webrev: > http://cr.openjdk.java.net/~mseledtsov/8181592.02/ > > > Leonid, George - thank you for your comments. > In addition to addressing your feedback, I also did: > - implemented @requires docker.support > - added dockerRunJava() method and associated data structure DockerRunOptions, for running Java tests inside > docker environment, and to account for java opts, test java opts, docker opts, classes and class params > - added a simple HelloWorld test case that runs HelloWorld inside a container > - ran jtreg with extra Java options, make sure they are added correctly to the docker run command > - added docker image cleanup after testing is done > > > Please review. > > Misha > > > On 9/27/17, 11:00 AM, mikhailo wrote: >> Leonid, >> >> Thank you for review and constructive feedback. See my comment in line. >> >> >> On 09/26/2017 11:19 AM, Leonid Mesnik wrote: >>> Misha >>> >>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>> Copyright is incorrect, need to updated it for GPL. >> Fixed >>> >>> The Hotspot is Oracle VM name only so test might fail for OpenJDK. I think you need to fix this check. >> I see. I fixed this by using Platform.vmName which should be correct in all cases. I double-checked with OpenJDK also. >>> >>> The requires checks only that test is executed only on the 64-bit linux. Does it make a sense to introduce more docker-specific check? >> I agree this is a better way. I will do some prototyping; if such check is feasible and efficient in at requires then I will add it. >>> >>> >>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/Dockerfile-BasicTest.html >>> Could you please explain why oraclelinux 7.0 is used as a base image for test. >> I have upgraded to Oracle Linux 7.2. If we have specific requirement I will change it to that. If we have requirements in the future to support multiple OS, I can add Dockerfile generation. >> For this basic sanity tests I think this should suffice. >>> >>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html >>> The content looks fine. >>> >>> I don?t see anything to clean up docker images on the system. Could you please explain how tests are going to cleanup images. >> To clean up containers I will add "--rm" to the 'docker run' command. This should ensure that container data is removed after container stops. >> As for the image - I use the same image name. The image will stay in the local registry unless manually removed. I should probably do 'docker rmi' at the end of the test to clean this up. >> >> >> Once I implement these changes I will send the updated webrev. >> >> Thank you, >> Misha >>> >>> Leonid >>> >>> >>>> On Sep 21, 2017, at 5:58 PM, mikhailo > wrote: >>>> >>>> Please review this initial drop of Docker test utils and a sanity test. This change lays ground >>>> for further test development and test utils improvement in this area. >>>> >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ >>>> Testing: >>>> - run this test on machine with Docker enabled - works >>>> - run this test on Linux-x64 with no Docker engine or Docker disabled - test skipped (as expected) >>>> - run this test on automated system - in progress >>>> >>>> >>>> Thank you, >>>> Misha >>>> >>> >> From ioi.lam at oracle.com Wed Oct 4 05:38:09 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 3 Oct 2017 22:38:09 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D4006C.3010000@oracle.com> References: <59D4006C.3010000@oracle.com> Message-ID: <9aa58a27-edc1-ae76-5ae8-541a9d9c94b0@oracle.com> Looks good. Thanks! - Ioi On 10/3/17 2:26 PM, Calvin Cheung wrote: > bug: https://bugs.openjdk.java.net/browse/JDK-8185694 > > The open code change for this fix involves adding the > _java_platform_loader to the SystemDictionary class. > > webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ > > Testing: > ??? JPRT > ??? hs-tier3 on linux-x64, macosx, solaris-sparcv9, windows-x64. > > thanks, > Calvin From david.holmes at oracle.com Wed Oct 4 06:36:39 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 4 Oct 2017 16:36:39 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D4006C.3010000@oracle.com> References: <59D4006C.3010000@oracle.com> Message-ID: <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> Hi Calvin, On 4/10/2017 7:26 AM, Calvin Cheung wrote: > bug: https://bugs.openjdk.java.net/browse/JDK-8185694 > > The open code change for this fix involves adding the > _java_platform_loader to the SystemDictionary class. > > webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ I think you should be calling getPlatformClassLoader rather than poking at the parent field of the system loader. That way you don't have to worry about whether or not the system loader has been customized by the user. Thanks, David > Testing: > ??? JPRT > ??? hs-tier3 on linux-x64, macosx, solaris-sparcv9, windows-x64. > > thanks, > Calvin From yasuenag at gmail.com Wed Oct 4 12:37:32 2017 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Wed, 4 Oct 2017 21:37:32 +0900 Subject: PING: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: <19df096b-3243-1ac0-3d3a-e955c63c534d@oracle.com> <53fb4559-aeed-da1a-67fe-6cb50fd8e9ce@gmail.com> Message-ID: <3bab2e13-47d4-95bf-28aa-40bda4065b68@gmail.com> PING: Could you review and sponsor it? > http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ Thanks, Yasumasa On 2017/09/27 12:05, Yasumasa Suenaga wrote: > Hi all, > > I added a testcase for this issue in new webrev: > > http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ > > > Thanks, > > Yasumasa > > > > 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >> Hi all, >> >> I uploaded webrev for this issue against jdk10/hs. >> Could you review it? >> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >> >> >> I cannot access JPRT. So I need a sponsor. >> >> >> Thanks, >> >> Yasumasa >> >> >> >> 2017-09-21 3:11 GMT+09:00 Man Cao : >>> Thank Yasumasa and Stefan for the responses. >>> >>> Good to know that the patch is not blocked due to breaking some internal >>> invariants/assumptions, but just due to its P5 status. >>> Is it possible to push it to P4? >>> >>> -Man >>> >>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>> wrote: >>>> >>>> Hi, >>>> >>>> (CC'ed hotspot-runtime-dev) >>>> >>>>> I think the reason is that this bug is a P5. The compressed class space >>>>> belongs to the runtime code, so you might get more traction for this on the >>>>> hotspot-runtime-dev list. >>>> >>>> >>>> I will send review request against jdk10/master or jdk10/hs after repos >>>> are opened. >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> >>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>> >>>>> Hi Man, >>>>> >>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>> >>>>>> Hi Yasumasa, Stefan, >>>>>> >>>>>> Do you have any thoughts on why this patch has been pending for 2+ >>>>>> years? This patch could really save us from some annoying issues since we >>>>>> are automatically monitoring hsperfdata counters. >>>>> >>>>> >>>>> I think the reason is that this bug is a P5. The compressed class space >>>>> belongs to the runtime code, so you might get more traction for this on the >>>>> hotspot-runtime-dev list. >>>>> >>>>> StefanK >>>>> >>>>>> >>>>>> -Man >>>>>> >>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>> > wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I wonder if there is any recent update on the patch for JDK-8087291. >>>>>> Is it possible to push this patch into JDK9? Except for its low >>>>>> priority (P5), >>>>>> is there any complication that prevents this patch getting approved >>>>>> (for example, some JVM logic requires CompressedClassSpaceSize to be >>>>>> 1GB by default)? >>>>>> >>>>>> I work in the Java Platform Team at Google. We have encountered >>>>>> annoying issues that the hsperfdata counter >>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>> a too large value (about 1GB) even if user sets >>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the confusing 1GB >>>>>> memory reserved by metaspace, >>>>>> regardless of MaxMetaspaceSize value. The root cause for these >>>>>> issues is that CompressedClassSpaceSize is not automatically capped >>>>>> by MaxMetaspaceSize >>>>>> during VM initialization, and this patch seems fix the root cause. >>>>>> (I'm aware that even after this patch, the reserved size could still >>>>>> be up to 2*MaxMetaspaceSize, >>>>>> but it is better than the current situation.) >>>>>> >>>>>> Thanks, >>>>>> Man >>>>>> >>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>> >>>>>> Thank you for your comment! >>>>>> > Try running a debug JVM with your patch with this command >>>>>> line. >>>>>> > >>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>> >>>>>> It works on fastdebug VM. >>>>>> Please review again. >>>>>> >>>>>> Thanks, >>>>>> Yasumasa >>>>>> >>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>> > Yasumasa, >>>>>> > >>>>>> > Try running a debug JVM with your patch with this command >>>>>> line. >>>>>> > >>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>> > >>>>>> > On a linux system I get this when I build with your patch. >>>>>> > >>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>> >> # To suppress the following error report, specify this >>>>>> argument >>>>>> >> # after -XX: or in .hotspotrc: >>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>> >> # >>>>>> >> # A fatal error has been detected by the Java Runtime >>>>>> Environment: >>>>>> >> # >>>>>> >> # Internal Error >>>>>> >> >>>>>> >>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>> >> # assert(size > MediumChunk || size > ClassMediumChunk) >>>>>> failed: Not a >>>>>> >> humongous chunk >>>>>> >> # >>>>>> > >>>>>> > >>>>>> > Jon >>>>>> > >>>>>> > >>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>> >> I want to continue to discuss about CompressedClassSpace and >>>>>> MaxMetaspace in this (RFR) thread. >>>>>> >> >>>>>> >> >>>>>> >>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>> >>>>>> >>>>>> >>>> Should I resize CompressedClassSpaceSize than to show >>>>>> error message? >>>>>> >>> If you add slightly better heuristics for the setup of the >>>>>> CompressedClassSpaceSize flag, for example lowering the >>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is set, then it >>>>>> might be less likely that you'll hit the OutOfMemoryError when >>>>>> the system is set up with strict overcommit settings. >>>>>> >> >>>>>> >> I've uploaded new webrev: >>>>>> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>> >>>>>> >> >>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>> CompressedClassSpaceSize, and >>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>> >> >>>>>> >> I add to check CompressedClassSpaceSize in >>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>> >> If InitialBootClassLoaderMetaspaceSize is greater than >>>>>> MaxMetaspaceSize, >>>>>> >> VM will fail with error message. >>>>>> >> >>>>>> >> InitialBootClassLoaderMetaspaceSize will be set to >>>>>> MaxMetaspaceSize >>>>>> >> when UseCompressedClassPointers is not set in >>>>>> Metaspace::ergo_initialize(). >>>>>> >> >>>>>> >> >>>>>> >> Thanks, >>>>>> >> >>>>>> >> Yasumasa >>>>>> >> >>>>>> >>>>>> >>> From martin.doerr at sap.com Wed Oct 4 14:22:19 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Wed, 4 Oct 2017 14:22:19 +0000 Subject: RFR(S): 8188773: PPC64 and s390: Fix UseMembar and enable ShareVtableStubs Message-ID: <70280b00f01e47f8be4cecc940b4065c@sap.com> Hi, can somebody please review this small PPC64 and s390 change? http://cr.openjdk.java.net/~mdoerr/8188773_ppc64_s390_UseMembar_ShareVtableStubs/webrev.00/ It fixes UseMembar and enables ShareVtableStubs on these 2 platforms: On PPC64, we currently generate unnecessary fence instructions which make performance even worse. On s390, the signal handler needs a fix in order to work with -XX:+UseMember. In addition, switch on ShareVtableStubs in order to reduce code cache usage. Modern processors don't seem to benefit from having it off it since the predictors have become better. Best regards, Martin From goetz.lindenmaier at sap.com Wed Oct 4 14:40:29 2017 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Wed, 4 Oct 2017 14:40:29 +0000 Subject: RFR(S): 8188773: PPC64 and s390: Fix UseMembar and enable ShareVtableStubs In-Reply-To: <70280b00f01e47f8be4cecc940b4065c@sap.com> References: <70280b00f01e47f8be4cecc940b4065c@sap.com> Message-ID: <271195a54bf949afaa11cc9db698b5de@sap.com> Hi Martin, the change looks good. If you don't mind you could fix Oracle Copyright in os_linux_s390.cpp It needs to be 2016, 2017. This was already broken before your change. Best regards, Goetz. > -----Original Message----- > From: Doerr, Martin > Sent: Mittwoch, 4. Oktober 2017 16:22 > To: hotspot-runtime-dev at openjdk.java.net; Lindenmaier, Goetz > > Subject: RFR(S): 8188773: PPC64 and s390: Fix UseMembar and enable > ShareVtableStubs > > Hi, > > > > can somebody please review this small PPC64 and s390 change? > > http://cr.openjdk.java.net/~mdoerr/8188773_ppc64_s390_UseMembar_Sh > areVtableStubs/webrev.00/ > > > > It fixes UseMembar and enables ShareVtableStubs on these 2 platforms: > > On PPC64, we currently generate unnecessary fence instructions which make > performance even worse. > > On s390, the signal handler needs a fix in order to work with - > XX:+UseMember. > > In addition, switch on ShareVtableStubs in order to reduce code cache usage. > Modern processors don't seem to benefit from having it off it since the > predictors have become better. > > > > Best regards, > > Martin > > From goetz.lindenmaier at sap.com Wed Oct 4 14:58:36 2017 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Wed, 4 Oct 2017 14:58:36 +0000 Subject: RFR(s): 8187547: PPC64: icache invalidation is incorrect in some places In-Reply-To: <5aac8f0a-b106-b7c7-8974-c4509c821d40@azul.com> References: <07b0d758-40eb-dc1f-e25b-49e031849744@azul.com> <1b20a156de344954a0ffe0bc88a07d6e@sap.com> <107e53d21d06483db1b01d0dd91c9cc9@sap.com> <5aac8f0a-b106-b7c7-8974-c4509c821d40@azul.com> Message-ID: <2b86eafec5d8404697be81a741c245a0@sap.com> HI Anton, I pushed the change the other day, but obviously I forgot to set you as author. Sorry for that. Feel free to account it to you by referencing this email. Best regards, Goetz > -----Original Message----- > From: Anton Kozlov [mailto:akozlov at azul.com] > Sent: Dienstag, 19. September 2017 13:44 > To: Lindenmaier, Goetz ; Doerr, Martin > ; ppc-aix-port-dev at openjdk.java.net; hotspot- > runtime-dev at openjdk.java.net > Subject: Re: RFR(s): 8187547: PPC64: icache invalidation is incorrect in some > places > > Hi, Goetz, > > thanks for rewiew! > > I made suggested changes in the patch and preparied webrev for jdk10- > master > > http://cr.openjdk.java.net/~akozlov/8187547/webrev.03/ > > > I assume you are allowed to contribute to openJdk being with Azul who > > have signed the OCA? > > Yes, Azul signed OCA and I'm working here. > > Thanks, > Anton > > On 15.09.2017 13:27, Lindenmaier, Goetz wrote: > > Hi Anton, > > > > thanks for fixing this issue. Looks good. > > > > Nevertheless I would find it more readable if the patch_* functions > > would just return the address of the first instruction. > > The code in nativeInst_ppc then would read > > > > if > (MacroAssembler::get_address_of_calculate_address_from_global_toc_at( > addr, cb->content_begin()) != > > (address)data) { > > address inst2_addr = addr; > > const address inst1_addr = > > > MacroAssembler::patch_calculate_address_from_global_toc_at(inst2_addr, > cb->content_begin(), > > (address)data); > > ICache::ppc64_flush_icache_bytes(inst1_addr, inst2_addr - inst1_addr + > BytesPerInstWord); > > } > > > > which I can read much more easily. > > Similar for patch_set_narrow_oop. > > You could add > > // Returns address of first instruction in sequence. > > in macroAssembler_ppc.hpp. > > > > Please update the Oracle copyright in nativeInst_ppc.cpp. > > > > Also, jdk10/master is available. Could you please prepare the webrev > against > > that repo? > > > > I assume you are allowed to contribute to openJdk being with Azul who > > have signed the OCA? > > > > Best regards, > > Goetz. > > > > > > > > > >> -----Original Message----- > >> From: ppc-aix-port-dev [mailto:ppc-aix-port-dev- > >> bounces at openjdk.java.net] On Behalf Of Anton Kozlov > >> Sent: Donnerstag, 14. September 2017 20:26 > >> To: Doerr, Martin ; ppc-aix-port- > >> dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net > >> Subject: Re: RFR(s): 8187547: PPC64: icache invalidation is incorrect in > some > >> places > >> > >> Martin, > >> > >> thanks for review! > >> > >> I tried to keep knowledge of where relocation is pointing to in > >> macroAssembler. > >> But if simplier code is preferable, of course I'm agree with this. > >> > >> Webrev with suggestion applied: > >> http://cr.openjdk.java.net/~akozlov/8187547/webrev.02 > >> > >> I'm not very confident with the process for now (when repo closed), so > >> please do what should be done, I'll try to assist as much as I can. > >> > >> Backport should be easy, as the patch applies to jdk8u as well > >> > >> Thanks, > >> Anton > >> > >> On 14.09.2017 18:35, Doerr, Martin wrote: > >>> Hi Anton, > >>> > >>> thank you very much for providing a fix. Looks correct. > >>> > >>> In your current version, other_insn_offset is always negative. > >>> I'd prefer to make it always positive and simplify the usage like: > >>> assert(other_insn_offset > 0, "first instruction must be found"); > >>> start = addr - other_insn_offset; > >>> range = BytesPerInstWord + other_insn_offset; > >>> This would be better readable. Would you agree? > >>> > >>> At the moment, jdk10 repos are temporarily closed, but we can sponsor > >> the change when it's open again and after a 2nd review. > >>> Backports will also need to get addressed. > >>> > >>> Please note that ppc-aix-port-dev is not appropriate for reviews because > >> the PPC64 port is part of the main repos. Therefore, I've added hotspot- > >> runtime-dev. > >>> > >>> Best regards, > >>> Martin > >>> > >>> > >>> -----Original Message----- > >>> From: ppc-aix-port-dev [mailto:ppc-aix-port-dev- > >> bounces at openjdk.java.net] On Behalf Of Anton Kozlov > >>> Sent: Donnerstag, 14. September 2017 15:06 > >>> To: ppc-aix-port-dev at openjdk.java.net > >>> Subject: RFR(s): 8187547: PPC64: icache invalidation is incorrect in some > >> places > >>> > >>> Hi, All! > >>> > >>> Icache invalidation range calculation in > NativeMovConstReg::set_data_plain > >> and NativeMovConstReg::set_narrow_oop is incorrect and could cause > VM > >> crash: > >>> > >>> https://bugs.openjdk.java.net/browse/JDK-8187547 > >>> > >>> I suppose the root is in mismatch of supposed and actual return values > of > >> MacroAssembler::patch_set_narrow_oop and > >> MacroAssembler::patch_calculate_address_from_global_toc_at. > >>> These functions takes address of the middle of sequence and expected > to > >> return first instruction offset (negative by current implementation). > Instead > >> of this, they return `-offset == abs(offset)` and offset to `data` > respectively. > >>> > >>> Supposed fix: http://cr.openjdk.java.net/~akozlov/8187547/webrev.01/ > >>> > >>> Thanks, > >>> Anton > >>> From martin.doerr at sap.com Wed Oct 4 15:11:52 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Wed, 4 Oct 2017 15:11:52 +0000 Subject: RFR(S): 8188773: PPC64 and s390: Fix UseMembar and enable ShareVtableStubs In-Reply-To: <271195a54bf949afaa11cc9db698b5de@sap.com> References: <70280b00f01e47f8be4cecc940b4065c@sap.com> <271195a54bf949afaa11cc9db698b5de@sap.com> Message-ID: Hi G?tz, thanks for the review. I have fixed the copyright. Best regards, Martin -----Original Message----- From: Lindenmaier, Goetz Sent: Mittwoch, 4. Oktober 2017 16:40 To: Doerr, Martin ; hotspot-runtime-dev at openjdk.java.net Subject: RE: RFR(S): 8188773: PPC64 and s390: Fix UseMembar and enable ShareVtableStubs Hi Martin, the change looks good. If you don't mind you could fix Oracle Copyright in os_linux_s390.cpp It needs to be 2016, 2017. This was already broken before your change. Best regards, Goetz. > -----Original Message----- > From: Doerr, Martin > Sent: Mittwoch, 4. Oktober 2017 16:22 > To: hotspot-runtime-dev at openjdk.java.net; Lindenmaier, Goetz > > Subject: RFR(S): 8188773: PPC64 and s390: Fix UseMembar and enable > ShareVtableStubs > > Hi, > > > > can somebody please review this small PPC64 and s390 change? > > http://cr.openjdk.java.net/~mdoerr/8188773_ppc64_s390_UseMembar_Sh > areVtableStubs/webrev.00/ > > > > It fixes UseMembar and enables ShareVtableStubs on these 2 platforms: > > On PPC64, we currently generate unnecessary fence instructions which make > performance even worse. > > On s390, the signal handler needs a fix in order to work with - > XX:+UseMember. > > In addition, switch on ShareVtableStubs in order to reduce code cache usage. > Modern processors don't seem to benefit from having it off it since the > predictors have become better. > > > > Best regards, > > Martin > > From mandy.chung at oracle.com Wed Oct 4 15:29:59 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 4 Oct 2017 08:29:59 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> Message-ID: On 10/3/17 11:36 PM, David Holmes wrote: > Hi Calvin, > > On 4/10/2017 7:26 AM, Calvin Cheung wrote: >> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 >> >> The open code change for this fix involves adding the >> _java_platform_loader to the SystemDictionary class. >> >> webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ > > I think you should be calling getPlatformClassLoader rather than > poking at the parent field of the system loader. That way you don't > have to worry about whether or not the system loader has been > customized by the user. Agree.? ClassLoader::getPlatformClassLoader is the right way to get the platform class loader. Mandy From calvin.cheung at oracle.com Wed Oct 4 17:10:34 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 04 Oct 2017 10:10:34 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <9aa58a27-edc1-ae76-5ae8-541a9d9c94b0@oracle.com> References: <59D4006C.3010000@oracle.com> <9aa58a27-edc1-ae76-5ae8-541a9d9c94b0@oracle.com> Message-ID: <59D5160A.6070209@oracle.com> Thanks - Ioi. Calvin On 10/3/17, 10:38 PM, Ioi Lam wrote: > Looks good. Thanks! > > - Ioi > > > On 10/3/17 2:26 PM, Calvin Cheung wrote: >> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 >> >> The open code change for this fix involves adding the >> _java_platform_loader to the SystemDictionary class. >> >> webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ >> >> Testing: >> JPRT >> hs-tier3 on linux-x64, macosx, solaris-sparcv9, windows-x64. >> >> thanks, >> Calvin > From calvin.cheung at oracle.com Wed Oct 4 17:12:34 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 04 Oct 2017 10:12:34 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> Message-ID: <59D51682.1060301@oracle.com> Mandy, David, Thanks for the review. I've made the change based on your suggestion: http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ thanks, Calvin On 10/4/17, 8:29 AM, mandy chung wrote: > > > On 10/3/17 11:36 PM, David Holmes wrote: >> Hi Calvin, >> >> On 4/10/2017 7:26 AM, Calvin Cheung wrote: >>> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 >>> >>> The open code change for this fix involves adding the >>> _java_platform_loader to the SystemDictionary class. >>> >>> webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ >> >> I think you should be calling getPlatformClassLoader rather than >> poking at the parent field of the system loader. That way you don't >> have to worry about whether or not the system loader has been >> customized by the user. > Agree. ClassLoader::getPlatformClassLoader is the right way to get > the platform class loader. > > Mandy From mandy.chung at oracle.com Wed Oct 4 17:45:14 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 4 Oct 2017 10:45:14 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D51682.1060301@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> Message-ID: <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> On 10/4/17 10:12 AM, Calvin Cheung wrote: > Mandy, David, > > Thanks for the review. > > I've made the change based on your suggestion: > http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ systemDictionary.cpp SystemDictionary::roots_oops_do and oops_do have to be updated to traverse during GC.? Is this new java_platform_loader function used anywhere? Currently SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass is preloaded.? Shouldn't this be removed?? What about jdk_internal_loader_ClassLoaders_AppClassLoader? thread.cpp 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR?? Should it simply use CHECK_JNI_ERR as in other places? Mandy From mandy.chung at oracle.com Wed Oct 4 18:12:04 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 4 Oct 2017 11:12:04 -0700 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks Message-ID: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> This patch separates the JNI `FindClass` issue from the review thread for JDK-8188052 [1] into a different issue. webrev: http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/index.html This patch changes `FindClass` to specify the class loading context used for the load and unload hook. `FindClass` when called from JNI_OnUnload is changed to use the system class loader as the context. This is an incompatible behavioral change and this bumps the version so that a native library can determine the new behavior if desire.? This change proposes to defines JNI_VERSION_10 and I expect this name will be revisited along with the version string for the proposed new release cadence [2]. CSR:? https://bugs.openjdk.java.net/browse/JDK-8188069 Thanks David for the suggested spec wordings. thanks Mandy [1] http://mail.openjdk.java.net/pipermail/core-libs-dev/2017-September/049292.html [2] http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html From leonid.mesnik at oracle.com Wed Oct 4 18:19:36 2017 From: leonid.mesnik at oracle.com (Leonid Mesnik) Date: Wed, 4 Oct 2017 11:19:36 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: <59CD73AC.3080207@oracle.com> References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> <5F386D9A-CA05-4732-8F68-F493DD2E8E99@oracle.com> <59CD73AC.3080207@oracle.com> Message-ID: <661D3614-44C0-45C5-AE04-5AE1AF42C739@oracle.com> Hi Overall looks good. There few comments: http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/jtreg-ext/requires/VMProps.java.sdiff.html 307 System.err.println("dockerSupprt() threw exception: " + e); Missed ?o? in Support. 316 p.waitFor(); I think that it is needed to guard execution with timeout. I am not sure that jtreg is going to run properties with timeout. http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html 28 * @requires (sun.arch.data.model != "32") & (os.family == "linux") Is it possible to check this inside docker.support and use only ?docker.support? property? http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/lib/jdk/test/lib/containers/docker/DockerRunOptions.java.html 44 public DockerRunOptions(){} Why this empty constructor is needed? Does it make also to make this class immutable i.e. init all properties in constructors and have them read-only. Leonid > On Sep 28, 2017, at 3:11 PM, Mikhailo Seledtsov wrote: > > Here is the updated webrev: > http://cr.openjdk.java.net/~mseledtsov/8181592.02/ > > > Leonid, George - thank you for your comments. > In addition to addressing your feedback, I also did: > - implemented @requires docker.support > - added dockerRunJava() method and associated data structure DockerRunOptions, for running Java tests inside > docker environment, and to account for java opts, test java opts, docker opts, classes and class params > - added a simple HelloWorld test case that runs HelloWorld inside a container > - ran jtreg with extra Java options, make sure they are added correctly to the docker run command > - added docker image cleanup after testing is done > > > Please review. > > Misha > > > On 9/27/17, 11:00 AM, mikhailo wrote: >> Leonid, >> >> Thank you for review and constructive feedback. See my comment in line. >> >> >> On 09/26/2017 11:19 AM, Leonid Mesnik wrote: >>> Misha >>> >>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>> Copyright is incorrect, need to updated it for GPL. >> Fixed >>> >>> The Hotspot is Oracle VM name only so test might fail for OpenJDK. I think you need to fix this check. >> I see. I fixed this by using Platform.vmName which should be correct in all cases. I double-checked with OpenJDK also. >>> >>> The requires checks only that test is executed only on the 64-bit linux. Does it make a sense to introduce more docker-specific check? >> I agree this is a better way. I will do some prototyping; if such check is feasible and efficient in at requires then I will add it. >>> >>> >>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/Dockerfile-BasicTest.html >>> Could you please explain why oraclelinux 7.0 is used as a base image for test. >> I have upgraded to Oracle Linux 7.2. If we have specific requirement I will change it to that. If we have requirements in the future to support multiple OS, I can add Dockerfile generation. >> For this basic sanity tests I think this should suffice. >>> >>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html >>> The content looks fine. >>> >>> I don?t see anything to clean up docker images on the system. Could you please explain how tests are going to cleanup images. >> To clean up containers I will add "--rm" to the 'docker run' command. This should ensure that container data is removed after container stops. >> As for the image - I use the same image name. The image will stay in the local registry unless manually removed. I should probably do 'docker rmi' at the end of the test to clean this up. >> >> >> Once I implement these changes I will send the updated webrev. >> >> Thank you, >> Misha >>> >>> Leonid >>> >>> >>>> On Sep 21, 2017, at 5:58 PM, mikhailo > wrote: >>>> >>>> Please review this initial drop of Docker test utils and a sanity test. This change lays ground >>>> for further test development and test utils improvement in this area. >>>> >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ >>>> Testing: >>>> - run this test on machine with Docker enabled - works >>>> - run this test on Linux-x64 with no Docker engine or Docker disabled - test skipped (as expected) >>>> - run this test on automated system - in progress >>>> >>>> >>>> Thank you, >>>> Misha >>>> >>> >> From david.holmes at oracle.com Wed Oct 4 21:41:39 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 5 Oct 2017 07:41:39 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D51682.1060301@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> Message-ID: <34d47bd2-d331-ba7c-89be-d9733f404462@oracle.com> Hi Calvin, On 5/10/2017 3:12 AM, Calvin Cheung wrote: > Mandy, David, > > Thanks for the review. > > I've made the change based on your suggestion: > http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ share/classfile/systemDictionary.cpp There's an existing bug that this local is not used: 132 Klass* system_klass = WK_KLASS(ClassLoader_klass); (and the variable is poorly named). Now that you need to use WK_KLASS(ClassLoader_klass) twice you should instead use the local. --- As Mandy alludes these: 177 // Returns true if the passed class loader is the builtin application class loader 178 // or a custom system class loader. A customer system class loader can be 179 // specified via -Djava.system.class.loader. 180 bool SystemDictionary::is_system_class_loader(oop class_loader) { 181 if (class_loader == NULL) { 182 return false; 183 } 184 return (class_loader->klass() == SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() || 185 class_loader == _java_system_loader); 186 } 187 188 // Returns true if the passed class loader is the platform class loader. 189 bool SystemDictionary::is_platform_class_loader(oop class_loader) { 190 if (class_loader == NULL) { 191 return false; 192 } 193 return (class_loader->klass() == SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); 194 } would seem to be checking the wrong things - they should just be checking java_system_loader() and java_platform_loader() shouldn't they? In the case of is_system_class_loader() it seems odd that it checks a specific hard-wired class as well as the defined _java_system_loader ?? Thanks, David ----- > thanks, > Calvin > > On 10/4/17, 8:29 AM, mandy chung wrote: >> >> >> On 10/3/17 11:36 PM, David Holmes wrote: >>> Hi Calvin, >>> >>> On 4/10/2017 7:26 AM, Calvin Cheung wrote: >>>> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 >>>> >>>> The open code change for this fix involves adding the >>>> _java_platform_loader to the SystemDictionary class. >>>> >>>> webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ >>> >>> I think you should be calling getPlatformClassLoader rather than >>> poking at the parent field of the system loader. That way you don't >>> have to worry about whether or not the system loader has been >>> customized by the user. >> Agree.? ClassLoader::getPlatformClassLoader is the right way to get >> the platform class loader. >> >> Mandy From calvin.cheung at oracle.com Wed Oct 4 22:22:42 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 04 Oct 2017 15:22:42 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <34d47bd2-d331-ba7c-89be-d9733f404462@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <34d47bd2-d331-ba7c-89be-d9733f404462@oracle.com> Message-ID: <59D55F32.7010906@oracle.com> On 10/4/17, 2:41 PM, David Holmes wrote: > Hi Calvin, > > On 5/10/2017 3:12 AM, Calvin Cheung wrote: >> Mandy, David, >> >> Thanks for the review. >> >> I've made the change based on your suggestion: >> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ > > share/classfile/systemDictionary.cpp > > There's an existing bug that this local is not used: > > 132 Klass* system_klass = WK_KLASS(ClassLoader_klass); > > (and the variable is poorly named). Now that you need to use > WK_KLASS(ClassLoader_klass) twice you should instead use the local. How about changing the local variable name to class_loader_klass and using it at lines 135 and 143? > > --- > > As Mandy alludes these: > > 177 // Returns true if the passed class loader is the builtin > application class loader > 178 // or a custom system class loader. A customer system class > loader can be > 179 // specified via -Djava.system.class.loader. > 180 bool SystemDictionary::is_system_class_loader(oop class_loader) { > 181 if (class_loader == NULL) { > 182 return false; > 183 } > 184 return (class_loader->klass() == > SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() > || > 185 class_loader == _java_system_loader); > 186 } > 187 > 188 // Returns true if the passed class loader is the platform class > loader. > 189 bool SystemDictionary::is_platform_class_loader(oop class_loader) { > 190 if (class_loader == NULL) { > 191 return false; > 192 } > 193 return (class_loader->klass() == > SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); > 194 } > > would seem to be checking the wrong things - they should just be > checking java_system_loader() and java_platform_loader() shouldn't they? Do you mean lines 184 and 185 could become the following? return (class_loader == _java_system_loader); and line 193 could become the following? return (class_loader == _java_platform_loader); > > In the case of is_system_class_loader() it seems odd that it checks a > specific hard-wired class as well as the defined _java_system_loader ?? Not sure. That code has been there for a long time. Also, both SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() and SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() are being used in the closed source for JavaCalls. thanks, Calvin > > Thanks, > David > ----- > > > >> thanks, >> Calvin >> >> On 10/4/17, 8:29 AM, mandy chung wrote: >>> >>> >>> On 10/3/17 11:36 PM, David Holmes wrote: >>>> Hi Calvin, >>>> >>>> On 4/10/2017 7:26 AM, Calvin Cheung wrote: >>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 >>>>> >>>>> The open code change for this fix involves adding the >>>>> _java_platform_loader to the SystemDictionary class. >>>>> >>>>> webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ >>>> >>>> I think you should be calling getPlatformClassLoader rather than >>>> poking at the parent field of the system loader. That way you don't >>>> have to worry about whether or not the system loader has been >>>> customized by the user. >>> Agree. ClassLoader::getPlatformClassLoader is the right way to get >>> the platform class loader. >>> >>> Mandy From calvin.cheung at oracle.com Wed Oct 4 22:32:29 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 04 Oct 2017 15:32:29 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> Message-ID: <59D5617D.4050509@oracle.com> On 10/4/17, 10:45 AM, mandy chung wrote: > > > On 10/4/17 10:12 AM, Calvin Cheung wrote: >> Mandy, David, >> >> Thanks for the review. >> >> I've made the change based on your suggestion: >> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ > systemDictionary.cpp > > SystemDictionary::roots_oops_do and oops_do have to be updated to > traverse during GC. It is being handled in the following line of code in roots_oops_do(): CDS_ONLY(SystemDictionaryShared::roots_oops_do(strong);) The implementation is in closed source. > Is this new java_platform_loader function used anywhere? Yes, it is being used in closed source. > > Currently > SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass > is preloaded. Shouldn't this be removed? What about > jdk_internal_loader_ClassLoaders_AppClassLoader? They're being used in lines 184 and 193 in systemDictionary.cpp and also in closed source. > > thread.cpp > > 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); > > What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? > Should it simply use CHECK_JNI_ERR as in other places? They are the same, in utilities/exceptions.hpp: #define CHECK_JNI_ERR CHECK_(JNI_ERR) and it expands to the following: __the_thread__); if ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return (-1); (void)(0 I can change it to CHECK_JNI_ERR. thanks, Calvin > > Mandy From jiangli.zhou at Oracle.COM Wed Oct 4 23:12:11 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Wed, 4 Oct 2017 16:12:11 -0700 Subject: RFR: 8174986: CDS archived java heap region may not compatible with AOT Message-ID: <45A25AD9-D558-4F2C-8C64-F6D25BFDD8D2@oracle.com> Hi, Please review the following enhancement that always sets the narrow klass encoding shift to be LogKlassAlignmentInBytes when CDS is enabled. This is done so the archived java heap regions (the shared string space and adjustable archive heap space) can coexist with AOT code, because AOT uses LogKlassAlignmentInBytes for narrow klass encode shift when generating the precompiled code. I also took the opportunity to clean up Meatspace::global_initialize(). I moved the block of code that initializes the shared spaces and meatspace at CDS runtime into a new function, MetaspaceShared::initialize_runtime_shared_and_meta_spaces(). I also added more comments so it is easier to understand the logic. webrev: http://cr.openjdk.java.net/~jiangli/8174986/webrev.00/ RFE: https://bugs.openjdk.java.net/browse/JDK-8174986?filter=14921 Tested with all CDS/AppCDS tests on linux x64. Thanks, Jiangli From mandy.chung at oracle.com Wed Oct 4 23:24:15 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 4 Oct 2017 16:24:15 -0700 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> Message-ID: <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> Updated webrev: http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.01/ JNI FindClass change has been separated from this patch [1]. I made further clean up to the NativeLibrary implementation and replaced the use of Vector/Stack.? I also added a native test to verify the native library can be unloaded and reloaded. Summary: The patch replaces the ClassLoader use of finalizer with phantom reference, specifically Cleaner, for unloading native libraries.? It registers the class loader for cleanup only if it's not a builtin class loader which will never be unloaded. thanks Mandy [1] http://mail.openjdk.java.net/pipermail/core-libs-dev/2017-October/049389.html From david.holmes at oracle.com Wed Oct 4 23:30:52 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 5 Oct 2017 09:30:52 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D55F32.7010906@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <34d47bd2-d331-ba7c-89be-d9733f404462@oracle.com> <59D55F32.7010906@oracle.com> Message-ID: <35aaace0-baa2-aa22-b0a9-cd16bff95631@oracle.com> On 5/10/2017 8:22 AM, Calvin Cheung wrote: > On 10/4/17, 2:41 PM, David Holmes wrote: >> Hi Calvin, >> >> On 5/10/2017 3:12 AM, Calvin Cheung wrote: >>> Mandy, David, >>> >>> Thanks for the review. >>> >>> I've made the change based on your suggestion: >>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ >> >> share/classfile/systemDictionary.cpp >> >> There's an existing bug that this local is not used: >> >> 132?? Klass* system_klass = WK_KLASS(ClassLoader_klass); >> >> (and the variable is poorly named). Now that you need to use >> WK_KLASS(ClassLoader_klass) twice you should instead use the local. > How about changing the local variable name to class_loader_klass and > using it at lines 135 and 143? Yep. >> >> --- >> >> As Mandy alludes these: >> >> 177 // Returns true if the passed class loader is the builtin >> application class loader >> ?178 // or a custom system class loader. A customer system class >> loader can be >> ?179 // specified via -Djava.system.class.loader. >> ?180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >> ?181?? if (class_loader == NULL) { >> ?182???? return false; >> ?183?? } >> ?184?? return (class_loader->klass() == >> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >> || >> ?185?????????? class_loader == _java_system_loader); >> ?186 } >> ?187 >> ?188 // Returns true if the passed class loader is the platform class >> loader. >> ?189 bool SystemDictionary::is_platform_class_loader(oop class_loader) { >> ?190?? if (class_loader == NULL) { >> ?191???? return false; >> ?192?? } >> ?193?? return (class_loader->klass() == >> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); >> >> ?194 } >> >> would seem to be checking the wrong things - they should just be >> checking java_system_loader() and java_platform_loader() shouldn't they? > Do you mean lines 184 and 185 could become the following? > > ??? return (class_loader == _java_system_loader); > > and line 193 could become the following? > > ??? return (class_loader == _java_platform_loader); Yes - exactly. >> >> In the case of is_system_class_loader() it seems odd that it checks a >> specific hard-wired class as well as the defined _java_system_loader ?? > Not sure. That code has been there for a long time. > > Also, both > SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() and > SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() > are being used in the closed source for JavaCalls. Unless you really need to determine that the loader is of an actual type, you should only be querying java_system_loader() and java_platform_loader(). Thanks, David ----- > thanks, > Calvin >> >> Thanks, >> David >> ----- >> >> >> >>> thanks, >>> Calvin >>> >>> On 10/4/17, 8:29 AM, mandy chung wrote: >>>> >>>> >>>> On 10/3/17 11:36 PM, David Holmes wrote: >>>>> Hi Calvin, >>>>> >>>>> On 4/10/2017 7:26 AM, Calvin Cheung wrote: >>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 >>>>>> >>>>>> The open code change for this fix involves adding the >>>>>> _java_platform_loader to the SystemDictionary class. >>>>>> >>>>>> webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ >>>>> >>>>> I think you should be calling getPlatformClassLoader rather than >>>>> poking at the parent field of the system loader. That way you don't >>>>> have to worry about whether or not the system loader has been >>>>> customized by the user. >>>> Agree.? ClassLoader::getPlatformClassLoader is the right way to get >>>> the platform class loader. >>>> >>>> Mandy From mikhailo.seledtsov at oracle.com Thu Oct 5 00:27:46 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Wed, 4 Oct 2017 17:27:46 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> <5F386D9A-CA05-4732-8F68-F493DD2E8E99@oracle.com> <59CD73AC.3080207@oracle.com> Message-ID: <962734d4-6802-cb25-58b8-ec6c8c0bf940@oracle.com> Hi Igor, ? Thank you for review. Please see my comments in line. On 10/03/2017 04:29 PM, Igor Ignatyev wrote: > Hi Misha, > > 1st of all, thank you for working on this very important piece. it > generally looks great, although I have a number of mostly stylistic > comments. > > http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html > >> 29 * @requires (docker.support == "true") > 1. although all properties in requires context are strings, jtreg does > support boolean, so this line can be simplified to '@requires > (docker.support' Fixed >> 31 * @modules java.base/jdk.internal.misc >> 32 * @modules java.management >> 33 * jdk.jartool/sun.tools.jar > 2. could you please use one style? either add @modules to all lines or > leave it only at L#31 Fixed > >> 35 * @run main DockerBasicTest > 3. do we have to run DockerBasicTest in JDK under test? can't it be > run as @run driver? Fixed >> 46 if (!DockerTestUtils.canTestDocker()) >> 47 return; > 4. it's been proven to be less error prone to have { } even for > one-line blocks, could you please add them here? OK, updated the changes. Will also use this style from now on. > >> 72 .addDockerOpts("--volume", System.getProperty("test.classes") + ":/test-classes/"); > 5. as you already depend on jdk.test.lib, it might be better to > reuse?Utils::TEST_CLASSES Fixed >> 74 DockerTestUtils.dockerRunJava(opts). >> 75 shouldHaveExitValue(0).shouldContain("Hello Docker"); > 6. lines shouldn't end w/ '.', it should be moved to next line and > preferable be alined w/ 1st '.' in prev line. I have seen in other > files as well. > I'd also like to see exact one method call per line. OK. Updated this and other similar lines to have one statement per line. > > > http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html > >> 47 // Diagnostics: set to true to enable more diagnostic info >> 48 private static final boolean DEBUG = false; > 1. I'd prefer to always have all available diagnostic info. is there > any problem w/ having DEBUG=true? OK, I have removed the DEBUG, and make sure I always output the diagnostic info that was conditional on this DEBUG flag. This also lead to removal of "printChildOutput" parameter for execute(), which further simplified the code. >> 129 Files.createDirectories(jdkDstDir); >> ... >> 133 Files.walkFileTree(jdkSrcDir, new CopyFileVisitor(jdkSrcDir, jdkDstDir)); > 2. since?jdkDstDir is supposed to be empty, you can use Files::copy > method to copy the dir. Looks like files.copy does not copy the full tree recursively, so I still need a CopyFileVisitor. > >> 126 Path jdkSrcDir = Paths.get(System.getProperty("test.jdk")); > 3. again, as you already depend on jdk.test.lib, it might be better to > reuse?Utils::TEST_JDK Fixed > >> 149 public static void >> 150 buildDockerImage(String imageName, Path dockerfile, Path buildDir) throws Exception { >> 213 public static OutputAnalyzer execute(boolean retainChildStdout, String... command) >> 214 throws Exception { >> 230 public static OutputAnalyzer execute(boolean retainChildStdout, List command) >> 231 throws Exception { > 4. when you wrap a line, you should use double indentation, otherwise > one can misread it, in last two examples I read throws Exception as > 1st statement of a method. Fixed > > 5. your 'execute(boolean, String...)' creates a list from 'command' > array and calls 'execute(boolean, List)', which create an > array from 'command' list and pass it to ProcessBuilder::new. should > we make 'execute(boolean, List)' call 'execute(boolean, > String...)' and move all the logic to the latter? > > also, there are easier ways to get a list from an arrays -- > j.u.Arrays.asList(T...) or j.u.List.of(T...) > Fixed. I have also corrected/updated JavaDoc comments. I will post the updated webrev once I address comments from Leonid. Thank you, Misha > Thanks, > -- Igor > > >> On Sep 28, 2017, at 3:11 PM, Mikhailo Seledtsov >> > > wrote: >> >> Here is the updated webrev: >> http://cr.openjdk.java.net/~mseledtsov/8181592.02/ >> >> >> >> Leonid, George - thank you for your comments. >> In addition to addressing your feedback, I also did: >> ?- implemented @requires docker.support >> ?- added dockerRunJava() method and associated data structure >> DockerRunOptions, for running Java tests inside >> ????docker environment, and to account for java opts, test java opts, >> docker opts, classes and class params >> ?- added a simple HelloWorld test case that runs HelloWorld inside a >> container >> ?- ran jtreg with extra Java options, make sure they are added >> correctly to the docker run command >> ?- added docker image cleanup after testing is done >> >> >> Please review. >> >> Misha >> >> >> On 9/27/17, 11:00 AM, mikhailo wrote: >>> Leonid, >>> >>> Thank you for review and constructive feedback. See my comment in line. >>> >>> >>> On 09/26/2017 11:19 AM, Leonid Mesnik wrote: >>>> Misha >>>> >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>>> >>>> >>>> >>>> Copyright is incorrect, need to updated it for GPL. >>> Fixed >>>> >>>> The Hotspot is Oracle VM name only so test might fail for OpenJDK. >>>> I think you need to fix this check. >>> I see. I fixed this by using Platform.vmName which should be correct >>> in all cases. I double-checked with OpenJDK also. >>>> >>>> The requires checks only that test is executed only on the 64-bit >>>> linux. Does it make a sense to introduce more docker-specific check? >>> I agree this is a better way. I will do some prototyping; if such >>> check is feasible and efficient in at requires then I will add it. >>>> >>>> >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/Dockerfile-BasicTest.html >>>> >>>> >>>> >>>> Could you please explain why oraclelinux 7.0 is used as a base >>>> image for test. >>> I have upgraded to Oracle Linux 7.2. If we have specific requirement >>> I will change it to that. If we have requirements in the future to >>> support multiple OS, I can add Dockerfile generation. >>> For this basic sanity tests I think this should suffice. >>>> >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html >>>> >>>> >>>> >>>> The content looks fine. >>>> >>>> I don?t see anything to clean up docker images on the system. Could >>>> you please explain how tests are going to cleanup images. >>> To clean up containers I will add "--rm" to the 'docker run' >>> command. This should ensure that container data is removed after >>> container stops. >>> As for the image - I use the same image name. The image will stay in >>> the local registry unless manually removed. I should probably do >>> 'docker rmi' at the end of the test to clean this up. >>> >>> >>> Once I implement these changes I will send the updated webrev. >>> >>> Thank you, >>> Misha >>>> >>>> Leonid >>>> >>>> >>>>> On Sep 21, 2017, at 5:58 PM, mikhailo >>>>> >>>> >>>>> > wrote: >>>>> >>>>> Please review this initial drop of Docker test utils and a sanity >>>>> test. This change lays ground >>>>> for further test development and test utils improvement in this area. >>>>> >>>>> ???JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >>>>> ???Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ >>>>> >>>>> >>>>> ???Testing: >>>>> ??????- run this test on machine with Docker enabled - works >>>>> ??????- run this test on Linux-x64 with no Docker engine or Docker >>>>> disabled - test skipped (as expected) >>>>> ??????- run this test on automated system - in progress >>>>> >>>>> >>>>> Thank you, >>>>> Misha >>>>> >>>> >>> > From david.holmes at oracle.com Thu Oct 5 00:44:32 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 5 Oct 2017 10:44:32 +1000 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> Message-ID: <18eeaeed-164a-1d02-1546-93cf77c0d771@oracle.com> Hi Mandy, On 5/10/2017 4:12 AM, mandy chung wrote: > This patch separates the JNI `FindClass` issue from the review thread > for JDK-8188052 [1] into a different issue. > > webrev: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/index.html src/hotspot/share/prims/jni.cpp Okay ... so by nulling fromClass in the classloader during finalization (soon to be moved to the Cleaner) you can now distinguish between the OnLoad case and the OnUnload case, within FindClass - a comment to clarify that would be good, please. However you still have: 407 if (loader.is_null() && but you deleted the initialization of loader: - 404 loader = Handle(THREAD, k->class_loader()); so it will by default be null. I suppose checking the loader is only a potential optimization as the name of the class will be uniquely determined anyway. But the code should be cleaned up. --- Otherwise all seems fine. Thanks, David > > This patch changes `FindClass` to specify the class loading context used > for the load and unload hook. `FindClass` when called from JNI_OnUnload > is changed to use the system class loader as the context. This is an > incompatible behavioral change and this bumps the version so that a > native library can determine the new behavior if desire.? This change > proposes to defines JNI_VERSION_10 and I expect this name will be > revisited along with the version string for the proposed new release > cadence [2]. > > CSR:? https://bugs.openjdk.java.net/browse/JDK-8188069 > > Thanks David for the suggested spec wordings. > > thanks > Mandy > [1] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2017-September/049292.html > > [2] > http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html From mikhailo.seledtsov at oracle.com Thu Oct 5 00:52:50 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Wed, 4 Oct 2017 17:52:50 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: <661D3614-44C0-45C5-AE04-5AE1AF42C739@oracle.com> References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> <5F386D9A-CA05-4732-8F68-F493DD2E8E99@oracle.com> <59CD73AC.3080207@oracle.com> <661D3614-44C0-45C5-AE04-5AE1AF42C739@oracle.com> Message-ID: Hi Leonid, ? Thank you for review. Please see my comments inline: On 10/04/2017 11:19 AM, Leonid Mesnik wrote: > Hi > > Overall looks good. There few comments: > http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/jtreg-ext/requires/VMProps.java.sdiff.html > > > ?307 ? ? ? ? ? ? System.err.println("dockerSupprt() threw exception: " > + e); > Missed ?o? in Support. > Fixed > 316 ? ? ? ? p.waitFor(); > I think that it is needed to guard execution with timeout. I am not > sure that jtreg is going to run properties with timeout. Fixed. I added 10 seconds for timeout, which should be sufficient. IMO, if it takes more than 10 sec to invoke this basic command then something is wrong, and the docker tests should not be run. > > http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html > > 28 * @requires (sun.arch.data.model != "32") & (os.family == "linux") > Is it possible to check this inside docker.support and use only > ?docker.support? property? Added. > > http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/lib/jdk/test/lib/containers/docker/DockerRunOptions.java.html > > 44 public DockerRunOptions(){} > Why this empty constructor is needed? Fixed - removed. > Does it make also to make this class immutable i.e. init all > properties in constructors and have them read-only. The idea of this class is that fields can be modified by the user. The default constructor fills out values for most common use case; when user decided to modify these options to specify different test run conditions they should be able to modify it. I could surround values by getters or setters, or make every new set return a new copy of the object (an immutable pattern), but I did not wish to complicate the code. The uses of this class a fairly simple, and are unlikely to result in data races. The test create runOptions object, specifies/modifies fields if necessary, then passes to DockerTestUtils for running docker container with these options. I hope you do not mind if I leave it as is. Thank you for review, Misha > > Leonid > >> On Sep 28, 2017, at 3:11 PM, Mikhailo Seledtsov >> > > wrote: >> >> Here is the updated webrev: >> http://cr.openjdk.java.net/~mseledtsov/8181592.02/ >> >> >> >> Leonid, George - thank you for your comments. >> In addition to addressing your feedback, I also did: >> ?- implemented @requires docker.support >> ?- added dockerRunJava() method and associated data structure >> DockerRunOptions, for running Java tests inside >> ????docker environment, and to account for java opts, test java opts, >> docker opts, classes and class params >> ?- added a simple HelloWorld test case that runs HelloWorld inside a >> container >> ?- ran jtreg with extra Java options, make sure they are added >> correctly to the docker run command >> ?- added docker image cleanup after testing is done >> >> >> Please review. >> >> Misha >> >> >> On 9/27/17, 11:00 AM, mikhailo wrote: >>> Leonid, >>> >>> Thank you for review and constructive feedback. See my comment in line. >>> >>> >>> On 09/26/2017 11:19 AM, Leonid Mesnik wrote: >>>> Misha >>>> >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>>> >>>> >>>> >>>> Copyright is incorrect, need to updated it for GPL. >>> Fixed >>>> >>>> The Hotspot is Oracle VM name only so test might fail for OpenJDK. >>>> I think you need to fix this check. >>> I see. I fixed this by using Platform.vmName which should be correct >>> in all cases. I double-checked with OpenJDK also. >>>> >>>> The requires checks only that test is executed only on the 64-bit >>>> linux. Does it make a sense to introduce more docker-specific check? >>> I agree this is a better way. I will do some prototyping; if such >>> check is feasible and efficient in at requires then I will add it. >>>> >>>> >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/Dockerfile-BasicTest.html >>>> >>>> >>>> >>>> Could you please explain why oraclelinux 7.0 is used as a base >>>> image for test. >>> I have upgraded to Oracle Linux 7.2. If we have specific requirement >>> I will change it to that. If we have requirements in the future to >>> support multiple OS, I can add Dockerfile generation. >>> For this basic sanity tests I think this should suffice. >>>> >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html >>>> >>>> >>>> >>>> The content looks fine. >>>> >>>> I don?t see anything to clean up docker images on the system. Could >>>> you please explain how tests are going to cleanup images. >>> To clean up containers I will add "--rm" to the 'docker run' >>> command. This should ensure that container data is removed after >>> container stops. >>> As for the image - I use the same image name. The image will stay in >>> the local registry unless manually removed. I should probably do >>> 'docker rmi' at the end of the test to clean this up. >>> >>> >>> Once I implement these changes I will send the updated webrev. >>> >>> Thank you, >>> Misha >>>> >>>> Leonid >>>> >>>> >>>>> On Sep 21, 2017, at 5:58 PM, mikhailo >>>>> >>>> >>>>> > wrote: >>>>> >>>>> Please review this initial drop of Docker test utils and a sanity >>>>> test. This change lays ground >>>>> for further test development and test utils improvement in this area. >>>>> >>>>> ???JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >>>>> ???Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ >>>>> >>>>> >>>>> ???Testing: >>>>> ??????- run this test on machine with Docker enabled - works >>>>> ??????- run this test on Linux-x64 with no Docker engine or Docker >>>>> disabled - test skipped (as expected) >>>>> ??????- run this test on automated system - in progress >>>>> >>>>> >>>>> Thank you, >>>>> Misha >>>>> >>>> >>> > From mikhailo.seledtsov at oracle.com Thu Oct 5 01:08:46 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Wed, 4 Oct 2017 18:08:46 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> <5F386D9A-CA05-4732-8F68-F493DD2E8E99@oracle.com> <59CD73AC.3080207@oracle.com> <661D3614-44C0-45C5-AE04-5AE1AF42C739@oracle.com> Message-ID: <92e4360f-b278-0805-9c98-42e1cd98ee6d@oracle.com> Here is the updated webrev that addresses feedback from Igor and Leonid: http://cr.openjdk.java.net/~mseledtsov/8181592.03/ Thank you, Misha On 10/04/2017 05:52 PM, mikhailo wrote: > Hi Leonid, > > ? Thank you for review. Please see my comments inline: > > > On 10/04/2017 11:19 AM, Leonid Mesnik wrote: >> Hi >> >> Overall looks good. There few comments: >> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/jtreg-ext/requires/VMProps.java.sdiff.html >> >> >> >> ?307 ? ? ? ? ? ? System.err.println("dockerSupprt() threw exception: >> " + e); >> Missed ?o? in Support. >> > Fixed >> 316 ? ? ? ? p.waitFor(); >> I think that it is needed to guard execution with timeout. I am not >> sure that jtreg is going to run properties with timeout. > Fixed. I added 10 seconds for timeout, which should be sufficient. > IMO, if it takes more than 10 sec to invoke this basic command then > something is wrong, and the docker tests should not be run. >> >> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >> >> >> 28? * @requires (sun.arch.data.model != "32") & (os.family == "linux") >> Is it possible to check this inside docker.support and use only >> ?docker.support? property? > Added. >> >> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/lib/jdk/test/lib/containers/docker/DockerRunOptions.java.html >> >> >> 44???? public DockerRunOptions(){} >> Why this empty constructor is needed? > Fixed - removed. >> Does it make also to make this class immutable i.e. init all >> properties in constructors and have them read-only. > The idea of this class is that fields can be modified by the user. The > default constructor fills out values for most common use case; when > user decided to modify these options to specify different test run > conditions they should be able to modify it. > > I could surround values by getters or setters, or make every new set > return a new copy of the object (an immutable pattern), but I did not > wish to complicate the code. The uses of this class a fairly simple, > and are unlikely to result in data races. The test create runOptions > object, specifies/modifies fields if necessary, then passes to > DockerTestUtils for running docker container with these options. > > I hope you do not mind if I leave it as is. > > > Thank you for review, > Misha >> >> Leonid >> >>> On Sep 28, 2017, at 3:11 PM, Mikhailo Seledtsov >>> >> > wrote: >>> >>> Here is the updated webrev: >>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/ >>> >>> >>> >>> Leonid, George - thank you for your comments. >>> In addition to addressing your feedback, I also did: >>> ?- implemented @requires docker.support >>> ?- added dockerRunJava() method and associated data structure >>> DockerRunOptions, for running Java tests inside >>> ????docker environment, and to account for java opts, test java >>> opts, docker opts, classes and class params >>> ?- added a simple HelloWorld test case that runs HelloWorld inside a >>> container >>> ?- ran jtreg with extra Java options, make sure they are added >>> correctly to the docker run command >>> ?- added docker image cleanup after testing is done >>> >>> >>> Please review. >>> >>> Misha >>> >>> >>> On 9/27/17, 11:00 AM, mikhailo wrote: >>>> Leonid, >>>> >>>> Thank you for review and constructive feedback. See my comment in >>>> line. >>>> >>>> >>>> On 09/26/2017 11:19 AM, Leonid Mesnik wrote: >>>>> Misha >>>>> >>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>>>> >>>>> >>>>> >>>>> Copyright is incorrect, need to updated it for GPL. >>>> Fixed >>>>> >>>>> The Hotspot is Oracle VM name only so test might fail for OpenJDK. >>>>> I think you need to fix this check. >>>> I see. I fixed this by using Platform.vmName which should be >>>> correct in all cases. I double-checked with OpenJDK also. >>>>> >>>>> The requires checks only that test is executed only on the 64-bit >>>>> linux. Does it make a sense to introduce more docker-specific check? >>>> I agree this is a better way. I will do some prototyping; if such >>>> check is feasible and efficient in at requires then I will add it. >>>>> >>>>> >>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/Dockerfile-BasicTest.html >>>>> >>>>> >>>>> >>>>> Could you please explain why oraclelinux 7.0 is used as a base >>>>> image for test. >>>> I have upgraded to Oracle Linux 7.2. If we have specific >>>> requirement I will change it to that. If we have requirements in >>>> the future to support multiple OS, I can add Dockerfile generation. >>>> For this basic sanity tests I think this should suffice. >>>>> >>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html >>>>> >>>>> >>>>> >>>>> The content looks fine. >>>>> >>>>> I don?t see anything to clean up docker images on the system. >>>>> Could you please explain how tests are going to cleanup images. >>>> To clean up containers I will add "--rm" to the 'docker run' >>>> command. This should ensure that container data is removed after >>>> container stops. >>>> As for the image - I use the same image name. The image will stay >>>> in the local registry unless manually removed. I should probably do >>>> 'docker rmi' at the end of the test to clean this up. >>>> >>>> >>>> Once I implement these changes I will send the updated webrev. >>>> >>>> Thank you, >>>> Misha >>>>> >>>>> Leonid >>>>> >>>>> >>>>>> On Sep 21, 2017, at 5:58 PM, mikhailo >>>>>> >>>>> >>>>>> > wrote: >>>>>> >>>>>> Please review this initial drop of Docker test utils and a sanity >>>>>> test. This change lays ground >>>>>> for further test development and test utils improvement in this >>>>>> area. >>>>>> >>>>>> ???JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >>>>>> ???Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ >>>>>> >>>>>> >>>>>> ???Testing: >>>>>> ??????- run this test on machine with Docker enabled - works >>>>>> ??????- run this test on Linux-x64 with no Docker engine or >>>>>> Docker disabled - test skipped (as expected) >>>>>> ??????- run this test on automated system - in progress >>>>>> >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>> >>>>> >>>> >> > From david.holmes at oracle.com Thu Oct 5 01:21:27 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 5 Oct 2017 11:21:27 +1000 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> Message-ID: <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> Hi Mandy On 5/10/2017 9:24 AM, mandy chung wrote: > Updated webrev: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.01/ > > JNI FindClass change has been separated from this patch [1]. I made > further clean up to the NativeLibrary implementation and replaced the > use of Vector/Stack.? I also added a native test to verify the native > library can be unloaded and reloaded. > > Summary: The patch replaces the ClassLoader use of finalizer with > phantom reference, specifically Cleaner, for unloading native > libraries.? It registers the class loader for cleanup only if it's not a > builtin class loader which will never be unloaded. src/java.base/share/classes/java/lang/ClassLoader.java This comment is not a sentence: 2442 /* If the library is being loaded (must be by the same thread, 2443 * because Runtime.load and Runtime.loadLibrary are 2444 * synchronous). Overall the changes seem fine, but I do have a concern about the use of ConcurrentHashMap. This would seem to be a significant change in the initialization order - and I'm wondering how libjava itself actually gets loaded ?? --- I assume the tests will be updated to use JNI_VERSION_10, assuming you push the FindClass changes first? libnativeLibraryTest.c: Not sure I see the point in the FindClass("java/lang/ClassLoader") as if it fails to find it then surely it already failed to find "java/lang/Error" and you'll possibly crash trying to throw. NativeLibraryTest.java: Not clear how/if this actually verifies any unloading actually takes place? Also if the OnUnload throws an error and that happens in the Cleaner thread, how would that exception result in the test reporting a failure ?? I think OnUnload has to update some state stored in the main test class so that it can confirm the unloading occurred successfully, or not. Thanks, David > thanks > Mandy > [1] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2017-October/049389.html > > From mandy.chung at oracle.com Thu Oct 5 02:09:26 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 4 Oct 2017 19:09:26 -0700 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: <18eeaeed-164a-1d02-1546-93cf77c0d771@oracle.com> References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> <18eeaeed-164a-1d02-1546-93cf77c0d771@oracle.com> Message-ID: <90f72a9f-14cc-9c4c-4499-9ff6b48ce3e2@oracle.com> On 10/4/17 5:44 PM, David Holmes wrote: > Hi Mandy, > > On 5/10/2017 4:12 AM, mandy chung wrote: >> This patch separates the JNI `FindClass` issue from the review thread >> for JDK-8188052 [1] into a different issue. >> >> webrev: >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/index.html > > > src/hotspot/share/prims/jni.cpp > > Okay ... so by nulling fromClass in the classloader during > finalization (soon to be moved to the Cleaner) you can now distinguish > between the OnLoad case and the OnUnload case, within FindClass - a > comment to clarify that would be good, please. Added. > However you still have: > > ?407???? if (loader.is_null() && > > but you deleted the initialization of loader: > > - 404???? loader = Handle(THREAD, k->class_loader()); > > so it will by default be null. I suppose checking the loader is only a > potential optimization as the name of the class will be uniquely > determined anyway. But the code should be cleaned up. > Good catch.?? Yes it's an optimization to avoid making the Java call unnecessary. Updated: http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/src/hotspot/share/prims/jni.cpp.sdiff.html Mandy From david.holmes at oracle.com Thu Oct 5 02:50:45 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 5 Oct 2017 12:50:45 +1000 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: <90f72a9f-14cc-9c4c-4499-9ff6b48ce3e2@oracle.com> References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> <18eeaeed-164a-1d02-1546-93cf77c0d771@oracle.com> <90f72a9f-14cc-9c4c-4499-9ff6b48ce3e2@oracle.com> Message-ID: Looks good. Thanks, David On 5/10/2017 12:09 PM, mandy chung wrote: > > > On 10/4/17 5:44 PM, David Holmes wrote: >> Hi Mandy, >> >> On 5/10/2017 4:12 AM, mandy chung wrote: >>> This patch separates the JNI `FindClass` issue from the review thread >>> for JDK-8188052 [1] into a different issue. >>> >>> webrev: >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/index.html >> >> >> >> src/hotspot/share/prims/jni.cpp >> >> Okay ... so by nulling fromClass in the classloader during >> finalization (soon to be moved to the Cleaner) you can now distinguish >> between the OnLoad case and the OnUnload case, within FindClass - a >> comment to clarify that would be good, please. > > Added. > >> However you still have: >> >> ?407???? if (loader.is_null() && >> >> but you deleted the initialization of loader: >> >> - 404???? loader = Handle(THREAD, k->class_loader()); >> >> so it will by default be null. I suppose checking the loader is only a >> potential optimization as the name of the class will be uniquely >> determined anyway. But the code should be cleaned up. >> > > Good catch.?? Yes it's an optimization to avoid making the Java call > unnecessary. > > Updated: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/src/hotspot/share/prims/jni.cpp.sdiff.html > > > Mandy > From calvin.cheung at oracle.com Thu Oct 5 04:45:27 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 04 Oct 2017 21:45:27 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <35aaace0-baa2-aa22-b0a9-cd16bff95631@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <34d47bd2-d331-ba7c-89be-d9733f404462@oracle.com> <59D55F32.7010906@oracle.com> <35aaace0-baa2-aa22-b0a9-cd16bff95631@oracle.com> Message-ID: <59D5B8E7.8070307@oracle.com> On 10/4/17, 4:30 PM, David Holmes wrote: > On 5/10/2017 8:22 AM, Calvin Cheung wrote: >> On 10/4/17, 2:41 PM, David Holmes wrote: >>> Hi Calvin, >>> >>> On 5/10/2017 3:12 AM, Calvin Cheung wrote: >>>> Mandy, David, >>>> >>>> Thanks for the review. >>>> >>>> I've made the change based on your suggestion: >>>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ >>> >>> share/classfile/systemDictionary.cpp >>> >>> There's an existing bug that this local is not used: >>> >>> 132 Klass* system_klass = WK_KLASS(ClassLoader_klass); >>> >>> (and the variable is poorly named). Now that you need to use >>> WK_KLASS(ClassLoader_klass) twice you should instead use the local. >> How about changing the local variable name to class_loader_klass and >> using it at lines 135 and 143? > > Yep. I've made this change. Updated webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.02/ It also contains the change in threads.cpp - from CHECK_(JNI_ERR) to CHECK_JNI_ERR. > >>> >>> --- >>> >>> As Mandy alludes these: >>> >>> 177 // Returns true if the passed class loader is the builtin >>> application class loader >>> 178 // or a custom system class loader. A customer system class >>> loader can be >>> 179 // specified via -Djava.system.class.loader. >>> 180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>> 181 if (class_loader == NULL) { >>> 182 return false; >>> 183 } >>> 184 return (class_loader->klass() == >>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>> || >>> 185 class_loader == _java_system_loader); >>> 186 } >>> 187 >>> 188 // Returns true if the passed class loader is the platform >>> class loader. >>> 189 bool SystemDictionary::is_platform_class_loader(oop >>> class_loader) { >>> 190 if (class_loader == NULL) { >>> 191 return false; >>> 192 } >>> 193 return (class_loader->klass() == >>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); >>> >>> 194 } >>> >>> would seem to be checking the wrong things - they should just be >>> checking java_system_loader() and java_platform_loader() shouldn't >>> they? >> Do you mean lines 184 and 185 could become the following? >> >> return (class_loader == _java_system_loader); After a closer look at the comment, I think lines 184 and 185 are correct. // Returns true if the passed class loader is the builtin application class loader // or a custom system class loader. I'm afraid something may break and would like to leave it alone for now. >> >> and line 193 could become the following? >> >> return (class_loader == _java_platform_loader); I can't change this one either; I tried it and resulting in the following build error: Creating interim jimage Error occurred during initialization of boot layer java.lang.LayerInstantiationException: Class loader (instance of): jdk/internal/loader/ClassLoaders$PlatformClassLoader tried to define prohibited package name: java.sql thanks, Calvin > > Yes - exactly. > >>> >>> In the case of is_system_class_loader() it seems odd that it checks >>> a specific hard-wired class as well as the defined >>> _java_system_loader ?? >> Not sure. That code has been there for a long time. >> >> Also, both >> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >> and >> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() >> are being used in the closed source for JavaCalls. > > Unless you really need to determine that the loader is of an actual > type, you should only be querying java_system_loader() and > java_platform_loader(). > > Thanks, > David > ----- > >> thanks, >> Calvin >>> >>> Thanks, >>> David >>> ----- >>> >>> >>> >>>> thanks, >>>> Calvin >>>> >>>> On 10/4/17, 8:29 AM, mandy chung wrote: >>>>> >>>>> >>>>> On 10/3/17 11:36 PM, David Holmes wrote: >>>>>> Hi Calvin, >>>>>> >>>>>> On 4/10/2017 7:26 AM, Calvin Cheung wrote: >>>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 >>>>>>> >>>>>>> The open code change for this fix involves adding the >>>>>>> _java_platform_loader to the SystemDictionary class. >>>>>>> >>>>>>> webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ >>>>>> >>>>>> I think you should be calling getPlatformClassLoader rather than >>>>>> poking at the parent field of the system loader. That way you >>>>>> don't have to worry about whether or not the system loader has >>>>>> been customized by the user. >>>>> Agree. ClassLoader::getPlatformClassLoader is the right way to >>>>> get the platform class loader. >>>>> >>>>> Mandy From david.holmes at oracle.com Thu Oct 5 05:02:33 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 5 Oct 2017 15:02:33 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D5B8E7.8070307@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <34d47bd2-d331-ba7c-89be-d9733f404462@oracle.com> <59D55F32.7010906@oracle.com> <35aaace0-baa2-aa22-b0a9-cd16bff95631@oracle.com> <59D5B8E7.8070307@oracle.com> Message-ID: Hi Calvin, On 5/10/2017 2:45 PM, Calvin Cheung wrote: > On 10/4/17, 4:30 PM, David Holmes wrote: >> On 5/10/2017 8:22 AM, Calvin Cheung wrote: >>> On 10/4/17, 2:41 PM, David Holmes wrote: >>>> Hi Calvin, >>>> >>>> On 5/10/2017 3:12 AM, Calvin Cheung wrote: >>>>> Mandy, David, >>>>> >>>>> Thanks for the review. >>>>> >>>>> I've made the change based on your suggestion: >>>>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ >>>> >>>> share/classfile/systemDictionary.cpp >>>> >>>> There's an existing bug that this local is not used: >>>> >>>> 132?? Klass* system_klass = WK_KLASS(ClassLoader_klass); >>>> >>>> (and the variable is poorly named). Now that you need to use >>>> WK_KLASS(ClassLoader_klass) twice you should instead use the local. >>> How about changing the local variable name to class_loader_klass and >>> using it at lines 135 and 143? >> >> Yep. > I've made this change. > Updated webrev: > http://cr.openjdk.java.net/~ccheung/8185694/webrev.02/ > > It also contains the change in threads.cpp - from CHECK_(JNI_ERR) to > CHECK_JNI_ERR. >> >>>> >>>> --- >>>> >>>> As Mandy alludes these: >>>> >>>> 177 // Returns true if the passed class loader is the builtin >>>> application class loader >>>> ?178 // or a custom system class loader. A customer system class >>>> loader can be >>>> ?179 // specified via -Djava.system.class.loader. >>>> ?180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>>> ?181?? if (class_loader == NULL) { >>>> ?182???? return false; >>>> ?183?? } >>>> ?184?? return (class_loader->klass() == >>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>> || >>>> ?185?????????? class_loader == _java_system_loader); >>>> ?186 } >>>> ?187 >>>> ?188 // Returns true if the passed class loader is the platform >>>> class loader. >>>> ?189 bool SystemDictionary::is_platform_class_loader(oop >>>> class_loader) { >>>> ?190?? if (class_loader == NULL) { >>>> ?191???? return false; >>>> ?192?? } >>>> ?193?? return (class_loader->klass() == >>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); >>>> >>>> ?194 } >>>> >>>> would seem to be checking the wrong things - they should just be >>>> checking java_system_loader() and java_platform_loader() shouldn't >>>> they? >>> Do you mean lines 184 and 185 could become the following? >>> >>> ???? return (class_loader == _java_system_loader); > After a closer look at the comment, I think lines 184 and 185 are correct. > > // Returns true if the passed class loader is the builtin application > class loader > // or a custom system class loader. > > I'm afraid something may break and would like to leave it alone for now. It needs to be looked at - at a minimum file a follow up bug. Both conditions imply the loader IS the system (aka application) class loader. >>> >>> and line 193 could become the following? >>> >>> ???? return (class_loader == _java_platform_loader); > I can't change this one either; I tried it and resulting in the > following build error: > > Creating interim jimage > Error occurred during initialization of boot layer > java.lang.LayerInstantiationException: Class loader (instance of): > jdk/internal/loader/ClassLoaders$PlatformClassLoader tried to define > prohibited package name: java.sql Something very weird going on there that needs to be resolved before these changes are pushed IMHO. Sorry but there's obviously more happening here than meets the eye. David > thanks, > Calvin >> >> Yes - exactly. >> >>>> >>>> In the case of is_system_class_loader() it seems odd that it checks >>>> a specific hard-wired class as well as the defined >>>> _java_system_loader ?? >>> Not sure. That code has been there for a long time. >>> >>> Also, both >>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>> and >>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() >>> are being used in the closed source for JavaCalls. >> >> Unless you really need to determine that the loader is of an actual >> type, you should only be querying java_system_loader() and >> java_platform_loader(). >> >> Thanks, >> David >> ----- >> >>> thanks, >>> Calvin >>>> >>>> Thanks, >>>> David >>>> ----- >>>> >>>> >>>> >>>>> thanks, >>>>> Calvin >>>>> >>>>> On 10/4/17, 8:29 AM, mandy chung wrote: >>>>>> >>>>>> >>>>>> On 10/3/17 11:36 PM, David Holmes wrote: >>>>>>> Hi Calvin, >>>>>>> >>>>>>> On 4/10/2017 7:26 AM, Calvin Cheung wrote: >>>>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 >>>>>>>> >>>>>>>> The open code change for this fix involves adding the >>>>>>>> _java_platform_loader to the SystemDictionary class. >>>>>>>> >>>>>>>> webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ >>>>>>> >>>>>>> I think you should be calling getPlatformClassLoader rather than >>>>>>> poking at the parent field of the system loader. That way you >>>>>>> don't have to worry about whether or not the system loader has >>>>>>> been customized by the user. >>>>>> Agree.? ClassLoader::getPlatformClassLoader is the right way to >>>>>> get the platform class loader. >>>>>> >>>>>> Mandy From mandy.chung at oracle.com Thu Oct 5 05:14:13 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 4 Oct 2017 22:14:13 -0700 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> Message-ID: On 10/4/17 6:21 PM, David Holmes wrote: > Hi Mandy > > On 5/10/2017 9:24 AM, mandy chung wrote: >> Updated webrev: >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.01/ >> >> JNI FindClass change has been separated from this patch [1]. I made >> further clean up to the NativeLibrary implementation and replaced the >> use of Vector/Stack.? I also added a native test to verify the native >> library can be unloaded and reloaded. >> >> Summary: The patch replaces the ClassLoader use of finalizer with >> phantom reference, specifically Cleaner, for unloading native >> libraries.? It registers the class loader for cleanup only if it's >> not a builtin class loader which will never be unloaded. > > src/java.base/share/classes/java/lang/ClassLoader.java > > This comment is not a sentence: > > 2442???????????????? /* If the library is being loaded (must be by the > same thread, > 2443????????????????? * because Runtime.load and Runtime.loadLibrary are > 2444????????????????? * synchronous). > This is an existing comment.? I rewrite it in the updated webrev. > Overall the changes seem fine, but I do have a concern about the use > of ConcurrentHashMap. This would seem to be a significant change in > the initialization order - ConcurrentHashMap is referenced in ClassLoader but actually not loaded until later.? So it does change the initialization order. ? I now change the implementation to lazily create CHM.? This saves the footprint when a class loader does not load any native library. > and I'm wondering how libjava itself actually gets loaded ?? > ?os::native_java_library() which is called by JDK_Version_init() at VM creation time. > --- > > I assume the tests will be updated to use JNI_VERSION_10, assuming you > push the FindClass changes first? JDK-8188052 will be pushed first to jdk10/hs.? This fix will have to wait until JDK-8188052 is integrated to jdk10/master.?? I will update the test to use JNI_VERSION_10 before I push. > > libnativeLibraryTest.c: > > Not sure I see the point in the FindClass("java/lang/ClassLoader") as > if it fails to find it then surely it already failed to find > "java/lang/Error" and you'll possibly crash trying to throw. > I agree that should be cleaned up.? I took out FindClass("java/lang/ClassLoader") case. > NativeLibraryTest.java: > > Not clear how/if this actually verifies any unloading actually takes > place? If the native library is not unloaded, p.Test. will fail when reloading the native library.? I think this is adequate. I added further checks to verify the number of times the native library is unloaded as you suggest below. > Also if the OnUnload throws an error and that happens in the Cleaner > thread, how would that exception result in the test reporting a > failure ?? > Good point.? Cleaner ignores all exceptions.? I change it to be a fatal error. > I think OnUnload has to update some state stored in the main test > class so that it can confirm the unloading occurred successfully, or not. > I added this per your suggestion to enhance the verification. Updated webrev: http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.02/ Mandy From david.holmes at oracle.com Thu Oct 5 06:06:15 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 5 Oct 2017 16:06:15 +1000 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> Message-ID: Hi Mandy, On 5/10/2017 3:14 PM, mandy chung wrote: > On 10/4/17 6:21 PM, David Holmes wrote: >> Hi Mandy >> >> On 5/10/2017 9:24 AM, mandy chung wrote: >>> Updated webrev: >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.01/ >>> >>> JNI FindClass change has been separated from this patch [1]. I made >>> further clean up to the NativeLibrary implementation and replaced the >>> use of Vector/Stack.? I also added a native test to verify the native >>> library can be unloaded and reloaded. >>> >>> Summary: The patch replaces the ClassLoader use of finalizer with >>> phantom reference, specifically Cleaner, for unloading native >>> libraries.? It registers the class loader for cleanup only if it's >>> not a builtin class loader which will never be unloaded. >> >> src/java.base/share/classes/java/lang/ClassLoader.java >> >> This comment is not a sentence: >> >> 2442???????????????? /* If the library is being loaded (must be by the >> same thread, >> 2443????????????????? * because Runtime.load and Runtime.loadLibrary are >> 2444????????????????? * synchronous). >> > This is an existing comment.? I rewrite it in the updated webrev. Sorry I didn't notice that was refactored code. Thanks for the rewrite. >> Overall the changes seem fine, but I do have a concern about the use >> of ConcurrentHashMap. This would seem to be a significant change in >> the initialization order - > > ConcurrentHashMap is referenced in ClassLoader but actually not loaded It was loaded in clinit: 2689 private static Map systemNativeLibraries 2690 = new ConcurrentHashMap<>(); which was my concern. > until later.? So it does change the initialization order. ? I now change > the implementation to lazily create CHM.? This saves the footprint when > a class loader does not load any native library. Okay. But now I'm wondering about whether systemNativeLibraries/nativeLibraries need to be volatile? Just looking in ClassLoader it would seem yes, but if the only paths to loading a library are through Runtime, and those methods are synchronized on the Runtime instance, then everything is serialized anyway. Are there other paths that might allow concurrent access? (If the synchronization in Runtime guarantees only a single library can ever be loaded at a time across all classloaders, then you don't even need the synchronization on the loadedLibraryNames ??). >> and I'm wondering how libjava itself actually gets loaded ?? >> > > ?os::native_java_library() which is called by JDK_Version_init() at VM > creation time. Okay a different mechanism - and presumably no JNI_OnLoad function to execute :) >> --- >> >> I assume the tests will be updated to use JNI_VERSION_10, assuming you >> push the FindClass changes first? > > JDK-8188052 will be pushed first to jdk10/hs.? This fix will have to > wait until JDK-8188052 is integrated to jdk10/master.?? I will update > the test to use JNI_VERSION_10 before I push. Ok. >> >> libnativeLibraryTest.c: >> >> Not sure I see the point in the FindClass("java/lang/ClassLoader") as >> if it fails to find it then surely it already failed to find >> "java/lang/Error" and you'll possibly crash trying to throw. >> > I agree that should be cleaned up.? I took out > FindClass("java/lang/ClassLoader") case. So is: 68 excls = (*env)->FindClass(env, "java/lang/Error"); 69 if (excls == NULL) { 70 (*env)->FatalError(env, "Error class not found"); 71 } just a sanity check that you can in fact load a class? 75 (*env)->FatalError(env, "Error class not found"); That's the wrong error message. >> NativeLibraryTest.java: >> >> Not clear how/if this actually verifies any unloading actually takes >> place? > > If the native library is not unloaded, p.Test. will fail when > reloading the native library.? I think this is adequate. I added further Ah I hadn't realized that part. > checks to verify the number of times the native library is unloaded as > you suggest below. > >> Also if the OnUnload throws an error and that happens in the Cleaner >> thread, how would that exception result in the test reporting a >> failure ?? >> > > Good point.? Cleaner ignores all exceptions.? I change it to be a fatal > error. >> I think OnUnload has to update some state stored in the main test >> class so that it can confirm the unloading occurred successfully, or not. >> > > I added this per your suggestion to enhance the verification. Okay. Not sure why you needed to introduce q.Bridge - I find the test logic rather hard to follow, not clear who calls what method from where and when. I was thinking of a simple public int loadedCount; public int unloadedCount; that onLoad/OnUnload would increment. Thanks, David ----- > Updated webrev: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.02/ > > Mandy > From ioi.lam at oracle.com Thu Oct 5 16:30:46 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 5 Oct 2017 09:30:46 -0700 Subject: RFR: 8174986: CDS archived java heap region may not compatible with AOT In-Reply-To: <45A25AD9-D558-4F2C-8C64-F6D25BFDD8D2@oracle.com> References: <45A25AD9-D558-4F2C-8C64-F6D25BFDD8D2@oracle.com> Message-ID: <7a049844-4c67-3d52-d6d0-b0449efd58e8@oracle.com> Hi Jiangli: The changes look good, and thank you for doing the clean up. I agree that MetaspaceShared::initialize_runtime_shared_and_meta_spaces() is a better location for this functionality that inside metaspace.cpp. Just a small typo: 306?? // Set narrow_klass_shift to be LogKlassAlignmentInBytes. This is consisten -> consistent Thanks - Ioi On 10/4/17 4:12 PM, Jiangli Zhou wrote: > Hi, > > Please review the following enhancement that always sets the narrow klass encoding shift to be LogKlassAlignmentInBytes when CDS is enabled. This is done so the archived java heap regions (the shared string space and adjustable archive heap space) can coexist with AOT code, because AOT uses LogKlassAlignmentInBytes for narrow klass encode shift when generating the precompiled code. > > I also took the opportunity to clean up Meatspace::global_initialize(). I moved the block of code that initializes the shared spaces and meatspace at CDS runtime into a new function, MetaspaceShared::initialize_runtime_shared_and_meta_spaces(). I also added more comments so it is easier to understand the logic. > > webrev: http://cr.openjdk.java.net/~jiangli/8174986/webrev.00/ > RFE: https://bugs.openjdk.java.net/browse/JDK-8174986?filter=14921 > > Tested with all CDS/AppCDS tests on linux x64. > > Thanks, > Jiangli From jiangli.zhou at oracle.com Thu Oct 5 16:48:01 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Thu, 5 Oct 2017 09:48:01 -0700 Subject: RFR: 8174986: CDS archived java heap region may not compatible with AOT In-Reply-To: <7a049844-4c67-3d52-d6d0-b0449efd58e8@oracle.com> References: <45A25AD9-D558-4F2C-8C64-F6D25BFDD8D2@oracle.com> <7a049844-4c67-3d52-d6d0-b0449efd58e8@oracle.com> Message-ID: Thank for the review, Ioi! I?ll fix the typo. Jiangli > On Oct 5, 2017, at 9:30 AM, Ioi Lam wrote: > > Hi Jiangli: > > The changes look good, and thank you for doing the clean up. I agree that MetaspaceShared::initialize_runtime_shared_and_meta_spaces() is a better location for this functionality that inside metaspace.cpp. > > Just a small typo: > > 306 // Set narrow_klass_shift to be LogKlassAlignmentInBytes. This is consisten > > -> consistent > > Thanks > > - Ioi > > > On 10/4/17 4:12 PM, Jiangli Zhou wrote: >> Hi, >> >> Please review the following enhancement that always sets the narrow klass encoding shift to be LogKlassAlignmentInBytes when CDS is enabled. This is done so the archived java heap regions (the shared string space and adjustable archive heap space) can coexist with AOT code, because AOT uses LogKlassAlignmentInBytes for narrow klass encode shift when generating the precompiled code. >> >> I also took the opportunity to clean up Meatspace::global_initialize(). I moved the block of code that initializes the shared spaces and meatspace at CDS runtime into a new function, MetaspaceShared::initialize_runtime_shared_and_meta_spaces(). I also added more comments so it is easier to understand the logic. >> >> webrev: http://cr.openjdk.java.net/~jiangli/8174986/webrev.00/ > >> RFE: https://bugs.openjdk.java.net/browse/JDK-8174986?filter=14921 > >> >> Tested with all CDS/AppCDS tests on linux x64. >> >> Thanks, >> Jiangli From jiangli.zhou at Oracle.COM Thu Oct 5 17:38:38 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Thu, 5 Oct 2017 10:38:38 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D5617D.4050509@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> Message-ID: <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> Hi Calvin, > On Oct 4, 2017, at 3:32 PM, Calvin Cheung wrote: > > > > On 10/4/17, 10:45 AM, mandy chung wrote: >> >> >> On 10/4/17 10:12 AM, Calvin Cheung wrote: >>> Mandy, David, >>> >>> Thanks for the review. >>> >>> I've made the change based on your suggestion: >>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ >> systemDictionary.cpp >> >> SystemDictionary::roots_oops_do and oops_do have to be updated to traverse during GC. > It is being handled in the following line of code in roots_oops_do(): > CDS_ONLY(SystemDictionaryShared::roots_oops_do(strong);) Since you moved _java_platform_loader to systemDictionary.hpp, please handle that in SystemDictionary::roots_oops_do() and SystemDictionary::oops_do(). > > The implementation is in closed source. Please clean up the closed code to remove those. Thanks, Jiangli >> Is this new java_platform_loader function used anywhere? > Yes, it is being used in closed source. >> >> Currently SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass is preloaded. Shouldn't this be removed? What about jdk_internal_loader_ClassLoaders_AppClassLoader? > They're being used in lines 184 and 193 in systemDictionary.cpp and also in closed source. >> >> thread.cpp >> >> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >> >> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other places? > They are the same, in utilities/exceptions.hpp: > #define CHECK_JNI_ERR CHECK_(JNI_ERR) > > and it expands to the following: > __the_thread__); if ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return (-1); (void)(0 > > I can change it to CHECK_JNI_ERR. > > thanks, > Calvin >> >> Mandy From jiangli.zhou at Oracle.COM Thu Oct 5 18:02:12 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Thu, 5 Oct 2017 11:02:12 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <34d47bd2-d331-ba7c-89be-d9733f404462@oracle.com> <59D55F32.7010906@oracle.com> <35aaace0-baa2-aa22-b0a9-cd16bff95631@oracle.com> <59D5B8E7.8070307@oracle.com> Message-ID: <92884DA2-48F0-4B5C-A27F-AEF1D86B7A80@oracle.com> > On Oct 4, 2017, at 10:02 PM, David Holmes wrote: > > Hi Calvin, > > On 5/10/2017 2:45 PM, Calvin Cheung wrote: >> On 10/4/17, 4:30 PM, David Holmes wrote: >>> On 5/10/2017 8:22 AM, Calvin Cheung wrote: >>>> On 10/4/17, 2:41 PM, David Holmes wrote: >>>>> Hi Calvin, >>>>> >>>>> On 5/10/2017 3:12 AM, Calvin Cheung wrote: >>>>>> Mandy, David, >>>>>> >>>>>> Thanks for the review. >>>>>> >>>>>> I've made the change based on your suggestion: >>>>>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ >>>>> >>>>> share/classfile/systemDictionary.cpp >>>>> >>>>> There's an existing bug that this local is not used: >>>>> >>>>> 132 Klass* system_klass = WK_KLASS(ClassLoader_klass); >>>>> >>>>> (and the variable is poorly named). Now that you need to use WK_KLASS(ClassLoader_klass) twice you should instead use the local. >>>> How about changing the local variable name to class_loader_klass and using it at lines 135 and 143? >>> >>> Yep. >> I've made this change. >> Updated webrev: >> http://cr.openjdk.java.net/~ccheung/8185694/webrev.02/ >> It also contains the change in threads.cpp - from CHECK_(JNI_ERR) to CHECK_JNI_ERR. >>> >>>>> >>>>> --- >>>>> >>>>> As Mandy alludes these: >>>>> >>>>> 177 // Returns true if the passed class loader is the builtin application class loader >>>>> 178 // or a custom system class loader. A customer system class loader can be >>>>> 179 // specified via -Djava.system.class.loader. >>>>> 180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>>>> 181 if (class_loader == NULL) { >>>>> 182 return false; >>>>> 183 } >>>>> 184 return (class_loader->klass() == SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() || >>>>> 185 class_loader == _java_system_loader); >>>>> 186 } >>>>> 187 >>>>> 188 // Returns true if the passed class loader is the platform class loader. >>>>> 189 bool SystemDictionary::is_platform_class_loader(oop class_loader) { >>>>> 190 if (class_loader == NULL) { >>>>> 191 return false; >>>>> 192 } >>>>> 193 return (class_loader->klass() == SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); >>>>> 194 } >>>>> >>>>> would seem to be checking the wrong things - they should just be checking java_system_loader() and java_platform_loader() shouldn't they? >>>> Do you mean lines 184 and 185 could become the following? >>>> >>>> return (class_loader == _java_system_loader); >> After a closer look at the comment, I think lines 184 and 185 are correct. >> // Returns true if the passed class loader is the builtin application class loader >> // or a custom system class loader. >> I'm afraid something may break and would like to leave it alone for now. > > It needs to be looked at - at a minimum file a follow up bug. Both conditions imply the loader IS the system (aka application) class loader. I agree with David. The checks for SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() and SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() look strange. They should be investigated and cleaned up. > >>>> >>>> and line 193 could become the following? >>>> >>>> return (class_loader == _java_platform_loader); >> I can't change this one either; I tried it and resulting in the following build error: >> Creating interim jimage >> Error occurred during initialization of boot layer >> java.lang.LayerInstantiationException: Class loader (instance of): jdk/internal/loader/ClassLoaders$PlatformClassLoader tried to define prohibited package name: java.sql > > Something very weird going on there that needs to be resolved before these changes are pushed IMHO. Sorry but there's obviously more happening here than meets the eye. The following in Modules::define_module() is related. Calvin, maybe the _java_platform_loader is not yet set up by the time this code path is entered? // Only modules defined to either the boot or platform class loader, can define a "java/" package. if (!h_loader.is_null() && !SystemDictionary::is_platform_class_loader(h_loader()) && (strncmp(package_name, JAVAPKG, JAVAPKG_LEN) == 0 && (package_name[JAVAPKG_LEN] == '/' || package_name[JAVAPKG_LEN] == '\0'))) { const char* class_loader_name = SystemDictionary::loader_name(h_loader()); size_t pkg_len = strlen(package_name); char* pkg_name = NEW_RESOURCE_ARRAY_IN_THREAD(THREAD, char, pkg_len); strncpy(pkg_name, package_name, pkg_len); StringUtils::replace_no_expand(pkg_name, "/", "."); const char* msg_text1 = "Class loader (instance of): "; const char* msg_text2 = " tried to define prohibited package name: "; size_t len = strlen(msg_text1) + strlen(class_loader_name) + strlen(msg_text2) + pkg_len + 1; char* message = NEW_RESOURCE_ARRAY_IN_THREAD(THREAD, char, len); jio_snprintf(message, len, "%s%s%s%s", msg_text1, class_loader_name, msg_text2, pkg_name); THROW_MSG(vmSymbols::java_lang_IllegalArgumentException(), message); } Thanks, Jiangli > > David > >> thanks, >> Calvin >>> >>> Yes - exactly. >>> >>>>> >>>>> In the case of is_system_class_loader() it seems odd that it checks a specific hard-wired class as well as the defined _java_system_loader ?? >>>> Not sure. That code has been there for a long time. >>>> >>>> Also, both SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() and SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() are being used in the closed source for JavaCalls. >>> >>> Unless you really need to determine that the loader is of an actual type, you should only be querying java_system_loader() and java_platform_loader(). >>> >>> Thanks, >>> David >>> ----- >>> >>>> thanks, >>>> Calvin >>>>> >>>>> Thanks, >>>>> David >>>>> ----- >>>>> >>>>> >>>>> >>>>>> thanks, >>>>>> Calvin >>>>>> >>>>>> On 10/4/17, 8:29 AM, mandy chung wrote: >>>>>>> >>>>>>> >>>>>>> On 10/3/17 11:36 PM, David Holmes wrote: >>>>>>>> Hi Calvin, >>>>>>>> >>>>>>>> On 4/10/2017 7:26 AM, Calvin Cheung wrote: >>>>>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 >>>>>>>>> >>>>>>>>> The open code change for this fix involves adding the _java_platform_loader to the SystemDictionary class. >>>>>>>>> >>>>>>>>> webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ >>>>>>>> >>>>>>>> I think you should be calling getPlatformClassLoader rather than poking at the parent field of the system loader. That way you don't have to worry about whether or not the system loader has been customized by the user. >>>>>>> Agree. ClassLoader::getPlatformClassLoader is the right way to get the platform class loader. >>>>>>> >>>>>>> Mandy From coleen.phillimore at oracle.com Thu Oct 5 18:15:38 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 5 Oct 2017 14:15:38 -0400 Subject: RFR (S) 8178870: instrumentation.retransformClasses cause coredump Message-ID: <286a1baa-1b89-6cbb-fea2-ae3a8fe4a328@oracle.com> Summary: Don't double-free cached class bytes on redefinition loading failure. See bug for more description.? Tested with test/jdk/java/lang/instrument, test/hotspot/jtreg/runtime/RedefineTests, internal CDS tests, internal nsk.jvmti tests, and added test. open webrev at http://cr.openjdk.java.net/~coleenp/8178870.01/webrev bug link https://bugs.openjdk.java.net/browse/JDK-8178870 Thanks, Coleen From calvin.cheung at oracle.com Thu Oct 5 19:26:49 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 05 Oct 2017 12:26:49 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <92884DA2-48F0-4B5C-A27F-AEF1D86B7A80@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <34d47bd2-d331-ba7c-89be-d9733f404462@oracle.com> <59D55F32.7010906@oracle.com> <35aaace0-baa2-aa22-b0a9-cd16bff95631@oracle.com> <59D5B8E7.8070307@oracle.com> <92884DA2-48F0-4B5C-A27F-AEF1D86B7A80@oracle.com> Message-ID: <59D68779.7050603@oracle.com> On 10/5/17, 11:02 AM, Jiangli Zhou wrote: > >> On Oct 4, 2017, at 10:02 PM, David Holmes > > wrote: >> >> Hi Calvin, >> >> On 5/10/2017 2:45 PM, Calvin Cheung wrote: >>> On 10/4/17, 4:30 PM, David Holmes wrote: >>>> On 5/10/2017 8:22 AM, Calvin Cheung wrote: >>>>> On 10/4/17, 2:41 PM, David Holmes wrote: >>>>>> Hi Calvin, >>>>>> >>>>>> On 5/10/2017 3:12 AM, Calvin Cheung wrote: >>>>>>> Mandy, David, >>>>>>> >>>>>>> Thanks for the review. >>>>>>> >>>>>>> I've made the change based on your suggestion: >>>>>>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ >>>>>>> >>>>>> >>>>>> share/classfile/systemDictionary.cpp >>>>>> >>>>>> There's an existing bug that this local is not used: >>>>>> >>>>>> 132 Klass* system_klass = WK_KLASS(ClassLoader_klass); >>>>>> >>>>>> (and the variable is poorly named). Now that you need to use >>>>>> WK_KLASS(ClassLoader_klass) twice you should instead use the local. >>>>> How about changing the local variable name to class_loader_klass >>>>> and using it at lines 135 and 143? >>>> >>>> Yep. >>> I've made this change. >>> Updated webrev: >>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.02/ >>> >>> It also contains the change in threads.cpp - from CHECK_(JNI_ERR) to >>> CHECK_JNI_ERR. >>>> >>>>>> >>>>>> --- >>>>>> >>>>>> As Mandy alludes these: >>>>>> >>>>>> 177 // Returns true if the passed class loader is the builtin >>>>>> application class loader >>>>>> 178 // or a custom system class loader. A customer system class >>>>>> loader can be >>>>>> 179 // specified via -Djava.system.class.loader. >>>>>> 180 bool SystemDictionary::is_system_class_loader(oop >>>>>> class_loader) { >>>>>> 181 if (class_loader == NULL) { >>>>>> 182 return false; >>>>>> 183 } >>>>>> 184 return (class_loader->klass() == >>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>>> || >>>>>> 185 class_loader == _java_system_loader); >>>>>> 186 } >>>>>> 187 >>>>>> 188 // Returns true if the passed class loader is the platform >>>>>> class loader. >>>>>> 189 bool SystemDictionary::is_platform_class_loader(oop >>>>>> class_loader) { >>>>>> 190 if (class_loader == NULL) { >>>>>> 191 return false; >>>>>> 192 } >>>>>> 193 return (class_loader->klass() == >>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); >>>>>> >>>>>> 194 } >>>>>> >>>>>> would seem to be checking the wrong things - they should just be >>>>>> checking java_system_loader() and java_platform_loader() >>>>>> shouldn't they? >>>>> Do you mean lines 184 and 185 could become the following? >>>>> >>>>> return (class_loader == _java_system_loader); >>> After a closer look at the comment, I think lines 184 and 185 are >>> correct. >>> // Returns true if the passed class loader is the builtin >>> application class loader >>> // or a custom system class loader. >>> I'm afraid something may break and would like to leave it alone for now. >> >> It needs to be looked at - at a minimum file a follow up bug. Both >> conditions imply the loader IS the system (aka application) class loader. > > I agree with David. The checks for > SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() > and > SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() > look strange. They should be investigated and cleaned up. I've tried simplifying the is_system_class_loader() as mentioned above and it seems to work but more testing is required. > >> >>>>> >>>>> and line 193 could become the following? >>>>> >>>>> return (class_loader == _java_platform_loader); >>> I can't change this one either; I tried it and resulting in the >>> following build error: >>> Creating interim jimage >>> Error occurred during initialization of boot layer >>> java.lang.LayerInstantiationException: Class loader (instance of): >>> jdk/internal/loader/ClassLoaders$PlatformClassLoader tried to define >>> prohibited package name: java.sql >> >> Something very weird going on there that needs to be resolved before >> these changes are pushed IMHO. Sorry but there's obviously more >> happening here than meets the eye. > > The following in Modules::define_module() is related. Calvin, maybe > the _java_platform_loader is not yet set up by the time this code path > is entered? Thanks for the hint. The problem is that the following code was called before SystemDictionary::compute_java_loaders(). I'm experimenting moving the SystemDictionary::compute_java_loaders() up during vm init prior to call_initPhase2() like the following: // cache the system and platform class loaders SystemDictionary::compute_java_loaders(CHECK_JNI_ERR); // This will initialize the module system. Only java.base classes can be // loaded until phase 2 completes call_initPhase2(CHECK_JNI_ERR); thanks, Calvin > > // Only modules defined to either the boot or platform class loader, > can define a "java/" package. > if (!h_loader.is_null() && > !SystemDictionary::is_platform_class_loader(h_loader()) && > (strncmp(package_name, JAVAPKG, JAVAPKG_LEN) == 0 && > (package_name[JAVAPKG_LEN] == '/' || > package_name[JAVAPKG_LEN] == '\0'))) { > const char* class_loader_name = SystemDictionary::loader_name(h_loader()); > size_t pkg_len = strlen(package_name); > char* pkg_name = NEW_RESOURCE_ARRAY_IN_THREAD(THREAD, char, pkg_len); > strncpy(pkg_name, package_name, pkg_len); > StringUtils::replace_no_expand(pkg_name, "/", "."); > constchar* msg_text1 = "Class loader (instance of): "; > constchar* msg_text2 = " tried to define prohibited package name: "; > size_t len = strlen(msg_text1) + strlen(class_loader_name) + > strlen(msg_text2) + pkg_len + 1; > char* message = NEW_RESOURCE_ARRAY_IN_THREAD(THREAD, char, len); > jio_snprintf(message, len, "%s%s%s%s", msg_text1, > class_loader_name, msg_text2, pkg_name); > THROW_MSG(vmSymbols::java_lang_IllegalArgumentException(), message); > } > > Thanks, > Jiangli > >> >> David >> >>> thanks, >>> Calvin >>>> >>>> Yes - exactly. >>>> >>>>>> >>>>>> In the case of is_system_class_loader() it seems odd that it >>>>>> checks a specific hard-wired class as well as the defined >>>>>> _java_system_loader ?? >>>>> Not sure. That code has been there for a long time. >>>>> >>>>> Also, both >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>> and >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() >>>>> are being used in the closed source for JavaCalls. >>>> >>>> Unless you really need to determine that the loader is of an actual >>>> type, you should only be querying java_system_loader() and >>>> java_platform_loader(). >>>> >>>> Thanks, >>>> David >>>> ----- >>>> >>>>> thanks, >>>>> Calvin >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> ----- >>>>>> >>>>>> >>>>>> >>>>>>> thanks, >>>>>>> Calvin >>>>>>> >>>>>>> On 10/4/17, 8:29 AM, mandy chung wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 10/3/17 11:36 PM, David Holmes wrote: >>>>>>>>> Hi Calvin, >>>>>>>>> >>>>>>>>> On 4/10/2017 7:26 AM, Calvin Cheung wrote: >>>>>>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8185694 >>>>>>>>>> >>>>>>>>>> The open code change for this fix involves adding the >>>>>>>>>> _java_platform_loader to the SystemDictionary class. >>>>>>>>>> >>>>>>>>>> webrev: >>>>>>>>>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.00/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> I think you should be calling getPlatformClassLoader rather >>>>>>>>> than poking at the parent field of the system loader. That way >>>>>>>>> you don't have to worry about whether or not the system loader >>>>>>>>> has been customized by the user. >>>>>>>> Agree. ClassLoader::getPlatformClassLoader is the right way to >>>>>>>> get the platform class loader. >>>>>>>> >>>>>>>> Mandy > From coleen.phillimore at oracle.com Thu Oct 5 19:28:38 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 5 Oct 2017 15:28:38 -0400 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> Message-ID: http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/src/hotspot/share/prims/jni.cpp.udiff.html The type Klass has a class_loader() and protection_domain() accessors so it is unnecessary to cast to InstanceKlass.? We went through and removed many of these unnecessary InstanceKlass::casts. 407 if (loader.is_null() && I agree with David's comment that this line will always evaluate to true so should be cleaned up also. 413 thread); 414 if (HAS_PENDING_EXCEPTION) { 415 Handle ex(thread, thread->pending_exception()); 416 CLEAR_PENDING_EXCEPTION; 417 THROW_HANDLE_0(ex); 418 } This is strange too.? Why doesn't this just pass CHECK_NULL at line 413, which is the same thing as checking for and clearing and throwing the pending exception? Thanks, Coleen On 10/4/17 2:12 PM, mandy chung wrote: > This patch separates the JNI `FindClass` issue from the review thread > for JDK-8188052 [1] into a different issue. > > webrev: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/index.html > > This patch changes `FindClass` to specify the class loading context > used for the load and unload hook. `FindClass` when called from > JNI_OnUnload is changed to use the system class loader as the context. > This is an incompatible behavioral change and this bumps the version > so that a native library can determine the new behavior if desire.? > This change proposes to defines JNI_VERSION_10 and I expect this name > will be revisited along with the version string for the proposed new > release cadence [2]. > > CSR:? https://bugs.openjdk.java.net/browse/JDK-8188069 > > Thanks David for the suggested spec wordings. > > thanks > Mandy > [1] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2017-September/049292.html > [2] > http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html From coleen.phillimore at oracle.com Thu Oct 5 20:13:39 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 5 Oct 2017 16:13:39 -0400 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> Message-ID: <7f970d16-19de-7729-48a4-0653a6d61f7e@oracle.com> I should say, besides these style comments this change looks good. Coleen On 10/5/17 3:28 PM, coleen.phillimore at oracle.com wrote: > > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/src/hotspot/share/prims/jni.cpp.udiff.html > > > The type Klass has a class_loader() and protection_domain() accessors > so it is unnecessary to cast to InstanceKlass.? We went through and > removed many of these unnecessary InstanceKlass::casts. > > 407???? if (loader.is_null() && > > I agree with David's comment that this line will always evaluate to > true so should be cleaned up also. > > 413????????????????????????????? thread); > ?414?????? if (HAS_PENDING_EXCEPTION) { > ?415???????? Handle ex(thread, thread->pending_exception()); > ?416???????? CLEAR_PENDING_EXCEPTION; > ?417???????? THROW_HANDLE_0(ex); > ?418?????? } > > > This is strange too.? Why doesn't this just pass CHECK_NULL at line > 413, which is the same thing as checking for and clearing and throwing > the pending exception? > > Thanks, > Coleen > > > On 10/4/17 2:12 PM, mandy chung wrote: >> This patch separates the JNI `FindClass` issue from the review thread >> for JDK-8188052 [1] into a different issue. >> >> webrev: >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/index.html >> >> This patch changes `FindClass` to specify the class loading context >> used for the load and unload hook. `FindClass` when called from >> JNI_OnUnload is changed to use the system class loader as the >> context. This is an incompatible behavioral change and this bumps the >> version so that a native library can determine the new behavior if >> desire.? This change proposes to defines JNI_VERSION_10 and I expect >> this name will be revisited along with the version string for the >> proposed new release cadence [2]. >> >> CSR:? https://bugs.openjdk.java.net/browse/JDK-8188069 >> >> Thanks David for the suggested spec wordings. >> >> thanks >> Mandy >> [1] >> http://mail.openjdk.java.net/pipermail/core-libs-dev/2017-September/049292.html >> [2] >> http://mail.openjdk.java.net/pipermail/discuss/2017-September/004281.html > From coleen.phillimore at oracle.com Thu Oct 5 20:32:42 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 5 Oct 2017 16:32:42 -0400 Subject: RFR (S) 8178870: instrumentation.retransformClasses cause coredump In-Reply-To: References: <286a1baa-1b89-6cbb-fea2-ae3a8fe4a328@oracle.com> Message-ID: <55a246ee-a9b7-8cd6-1826-385308f61218@oracle.com> Thanks Serguei for reviewing and discussion about this bug. On 10/5/17 4:19 PM, serguei.spitsyn at oracle.com wrote: > Hi Coleen, > > The fix look good. > - if (the_class->get_cached_class_file_bytes() == 0) { > + if (the_class->get_cached_class_file() == 0) { > . . . > - else if (scratch_class->get_cached_class_file_bytes() != > - the_class->get_cached_class_file_bytes()) { > + else if (scratch_class->get_cached_class_file() != > + the_class->get_cached_class_file()) { > Thank you for additionally fixing the part above! > I was surprised that the original code > usedget_cached_class_file_bytes() instead ofget_cached_class_file() . > > Nice test! Thank you! > > A couple of minor comments. > > http://cr.openjdk.java.net/~coleenp/8178870.01/webrev/test/hotspot/jtreg/runtime/RedefineTests/RedefineDoubleDelete.java.html > 47 " int faa() { System.out.println(\"baa\"); return 2; }" + > ?? A typo: baa => faa > I want the redefined guy to say baa. > > http://cr.openjdk.java.net/~coleenp/8178870.01/webrev/test/hotspot/jtreg/runtime/RedefineTests/libRedefineDoubleDelete.c.html > 78 int pos = 0; > 79 int found = 0; > 80 int i; > 81 for (i = 0; i < newClassDataLen; i++) { > 82 newClassData[i] = class_data[i]; > 83 // Rewrite oo in class to aa > 84 if (class_data[i] == 'o') { > 85 if (i == pos + 1) { > 86 newClassData[i-1] = 'a'; > 87 newClassData[i] = 'a'; > 88 } else { > 89 pos = i; > 90 } > 91 } > 92 } > ? The local 'found' is not used. > ? The fragment can be simplified a little bit: > for (i = 0; i < newClassDataLen; i++) { > newClassData[i] = class_data[i]; > // Rewrite oo in class to aa > if (i > 0 && class_data[i] == 'o' && class_data[i-1] == 'o') { > newClassData[i] = newClassData[i-1] = 'a'; > } > } > Yes, this is better.?? I've made this change: ??? int i; ??? for (i = 0; i < newClassDataLen; i++) { ??????? newClassData[i] = class_data[i]; ??????? // Rewrite oo in class to aa ??????? if (i > 0 && class_data[i] == 'o' && class_data[i-1] == 'o') { ??????????? newClassData[i] = newClassData[i-1] = 'a'; ??????? } ??? } It's been so long since I've written C code, I wasn't expecting that the for loop couldn't declare int i. Thanks!! Coleen > Thanks, > Serguei > > > On 10/5/17 11:15, coleen.phillimore at oracle.com wrote: >> Summary: Don't double-free cached class bytes on redefinition loading >> failure. >> >> See bug for more description.? Tested with >> test/jdk/java/lang/instrument, >> test/hotspot/jtreg/runtime/RedefineTests, internal CDS tests, >> internal nsk.jvmti tests, and added test. >> >> open webrev at http://cr.openjdk.java.net/~coleenp/8178870.01/webrev >> bug link https://bugs.openjdk.java.net/browse/JDK-8178870 >> >> Thanks, >> Coleen > From coleen.phillimore at oracle.com Thu Oct 5 20:54:13 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 5 Oct 2017 16:54:13 -0400 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> Message-ID: <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> On 10/5/17 1:38 PM, Jiangli Zhou wrote: > Hi Calvin, > >> On Oct 4, 2017, at 3:32 PM, Calvin Cheung wrote: >> >> >> >> On 10/4/17, 10:45 AM, mandy chung wrote: >>> >>> On 10/4/17 10:12 AM, Calvin Cheung wrote: >>>> Mandy, David, >>>> >>>> Thanks for the review. >>>> >>>> I've made the change based on your suggestion: >>>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ >>> systemDictionary.cpp >>> >>> SystemDictionary::roots_oops_do and oops_do have to be updated to traverse during GC. >> It is being handled in the following line of code in roots_oops_do(): >> CDS_ONLY(SystemDictionaryShared::roots_oops_do(strong);) > Since you moved _java_platform_loader to systemDictionary.hpp, please handle that in SystemDictionary::roots_oops_do() and SystemDictionary::oops_do(). Yes, agree with Jiangli.? This was a good find, Mandy. http://cr.openjdk.java.net/~ccheung/8185694/webrev.02/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html *+ Klass* class_loader_klass = WK_KLASS(ClassLoader_klass);* This should be: ??? InstanceKlass* class_loader_klass = SystemDictionary::ClassLoader_klass(); The WK X macro is inappropriate to use here.?? I'm surprised it's not #undef at this point. > I agree with David. The checks for > SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() > and > SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() > look strange. They should be investigated and cleaned up. > I've tried simplifying the is_system_class_loader() as mentioned above > and it seems to work but more testing is required. This would be nice if this could be simplified as suggested. So if you use -Djava.system.loader=myLoader on the command line, setting _java_system_loader, then does that mean that the classes loaded by SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() are not in the system loader?? ie. they can be unloaded?? What is the result of the call to SystemDictionary::is_system_class_loader() used for??? I guess same question applies to the platform class loader. thanks, Coleen > >> The implementation is in closed source. > Please clean up the closed code to remove those. > > > Thanks, > > Jiangli > >>> Is this new java_platform_loader function used anywhere? >> Yes, it is being used in closed source. >>> Currently SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass is preloaded. Shouldn't this be removed? What about jdk_internal_loader_ClassLoaders_AppClassLoader? >> They're being used in lines 184 and 193 in systemDictionary.cpp and also in closed source. >>> thread.cpp >>> >>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>> >>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other places? >> They are the same, in utilities/exceptions.hpp: >> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >> >> and it expands to the following: >> __the_thread__); if ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return (-1); (void)(0 >> >> I can change it to CHECK_JNI_ERR. >> >> thanks, >> Calvin >>> Mandy From mandy.chung at oracle.com Thu Oct 5 21:56:39 2017 From: mandy.chung at oracle.com (mandy chung) Date: Thu, 5 Oct 2017 14:56:39 -0700 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: <7f970d16-19de-7729-48a4-0653a6d61f7e@oracle.com> References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> <7f970d16-19de-7729-48a4-0653a6d61f7e@oracle.com> Message-ID: <7fb4775b-0b1a-2c47-6742-f3dd6301f98c@oracle.com> On 10/5/17 1:13 PM, coleen.phillimore at oracle.com wrote: > > I should say, besides these style comments this change looks good. > Coleen > > On 10/5/17 3:28 PM, coleen.phillimore at oracle.com wrote: >> >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/src/hotspot/share/prims/jni.cpp.udiff.html >> >> >> The type Klass has a class_loader() and protection_domain() accessors >> so it is unnecessary to cast to InstanceKlass.? We went through and >> removed many of these unnecessary InstanceKlass::casts. >> That's from existing code which may not be touched for a long time. Since I'm on this file, I can clean this up. >> 407???? if (loader.is_null() && >> - if (loader.is_null() && + if (k->class_loader() == NULL && Already fixed per David's comment. >> I agree with David's comment that this line will always evaluate to >> true so should be cleaned up also. >> >> 413????????????????????????????? thread); >> ?414?????? if (HAS_PENDING_EXCEPTION) { >> ?415???????? Handle ex(thread, thread->pending_exception()); >> ?416???????? CLEAR_PENDING_EXCEPTION; >> ?417???????? THROW_HANDLE_0(ex); >> ?418?????? } >> >> >> This is strange too.? Why doesn't this just pass CHECK_NULL at line >> 413, which is the same thing as checking for and clearing and >> throwing the pending exception? >> This is existing code.?? I removed these lines and pass CHECK_NULL to call_static. How does this version look? http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/src/hotspot/share/prims/jni.cpp.sdiff.html One other thing I notice: it only sets the protection_domain to the caller class when FindClass is called from JNI_OnLoad/JNI_OnUnload. For other caller class, it uses null protection_domain. I would expect it should use the caller's protection domain? to look up a class.? I think it's a bug, isn't? Mandy From mandy.chung at oracle.com Thu Oct 5 22:24:22 2017 From: mandy.chung at oracle.com (mandy chung) Date: Thu, 5 Oct 2017 15:24:22 -0700 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> Message-ID: <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> Hi David, This version only modifies ClassLoader.java and the new test: http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.03/ On 10/4/17 11:06 PM, David Holmes wrote: > Hi Mandy, > >>> Overall the changes seem fine, but I do have a concern about the use >>> of ConcurrentHashMap. This would seem to be a significant change in >>> the initialization order - >> >> ConcurrentHashMap is referenced in ClassLoader but actually not loaded I missed the first sentence "I misread the source previously that CHM is already loaded at clinit time.? But I figure that ....".?? So I fixed it to lazily initialize it when the first native library is loaded. > > It was loaded in clinit: > > 2689???? private static Map systemNativeLibraries > 2690???????? = new ConcurrentHashMap<>(); > > which was my concern. I got that.? Thanks for bringing this up. > >> until later.? So it does change the initialization order. ? I now >> change the implementation to lazily create CHM.? This saves the >> footprint when a class loader does not load any native library. > > Okay. But now I'm wondering about whether > systemNativeLibraries/nativeLibraries need to be volatile? It's read for JNI native method binding (ClassLoader::findNative method) and therefore I used double-locking idiom instead to avoid any performance impact.?? I simplified this further.? The native libraries map is created with the loadedLibraryNames lock. > Just looking in ClassLoader it would seem yes, but if the only paths > to loading a library are through Runtime, and those methods are > synchronized on the Runtime instance, then everything is serialized > anyway. Are there other paths that might allow concurrent access? (If > the synchronization in Runtime guarantees only a single library can > ever be loaded at a time across all classloaders, then you don't even > need the synchronization on the loadedLibraryNames ??). That's a good point.? This code including the synchronization on loadedLibraryNames was written long ago.? I prefer to file a JBS issue to study the locking more closely. >>> >>> libnativeLibraryTest.c: >>> >>> Not sure I see the point in the FindClass("java/lang/ClassLoader") >>> as if it fails to find it then surely it already failed to find >>> "java/lang/Error" and you'll possibly crash trying to throw. >>> >> I agree that should be cleaned up.? I took out >> FindClass("java/lang/ClassLoader") case. > > So is: > > ? 68???? excls = (*env)->FindClass(env, "java/lang/Error"); > ? 69???? if (excls == NULL) { > ? 70???????? (*env)->FatalError(env, "Error class not found"); > ? 71???? } > > just a sanity check that you can in fact load a class? > ?75???????? (*env)->FatalError(env, "Error class not found"); > > That's the wrong error message. Thanks for catching this.? I revised the test and fixed the above. > > Okay. Not sure why you needed to introduce q.Bridge - I agree it's not needed any more.? Initially I was trying to have a method to return a count.? But later modified to a different method. > I find the test logic rather hard to follow, not clear who calls what > method from where and when. I was thinking of a simple > > public int loadedCount; > public int unloadedCount; > > that onLoad/OnUnload would increment. Well the test intends to verify if the native library can be reloaded (i.e. it has to be unloaded before reloading; otherwise UnsatifisedLinkError will be thrown).?? It has a native count method and it would fail if it is not loaded. I revise the test a little.? p.Test.run() loads the native library. The test checks if the native count method returns the correct value (i.e. the native library is loaded and OnLoad is invoked).? Calling run() again should get ULE since it can't be reloaded if already loaded. I keep the unloadedCount which is not strictly necessary but it is an additional sanity check.? I added more comment and hope it is clearer now. Mandy From david.holmes at oracle.com Fri Oct 6 00:57:01 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Oct 2017 10:57:01 +1000 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: <7fb4775b-0b1a-2c47-6742-f3dd6301f98c@oracle.com> References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> <7f970d16-19de-7729-48a4-0653a6d61f7e@oracle.com> <7fb4775b-0b1a-2c47-6742-f3dd6301f98c@oracle.com> Message-ID: On 6/10/2017 7:56 AM, mandy chung wrote: > On 10/5/17 1:13 PM, coleen.phillimore at oracle.com wrote: >> >> I should say, besides these style comments this change looks good. >> Coleen >> >> On 10/5/17 3:28 PM, coleen.phillimore at oracle.com wrote: >>> >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/src/hotspot/share/prims/jni.cpp.udiff.html >>> >>> >>> The type Klass has a class_loader() and protection_domain() accessors >>> so it is unnecessary to cast to InstanceKlass.? We went through and >>> removed many of these unnecessary InstanceKlass::casts. >>> > > That's from existing code which may not be touched for a long time. > Since I'm on this file, I can clean this up. >>> 407???? if (loader.is_null() && >>> > > - if (loader.is_null() && > + if (k->class_loader() == NULL && Already fixed per David's comment. > >>> I agree with David's comment that this line will always evaluate to >>> true so should be cleaned up also. >>> >>> 413????????????????????????????? thread); >>> ?414?????? if (HAS_PENDING_EXCEPTION) { >>> ?415???????? Handle ex(thread, thread->pending_exception()); >>> ?416???????? CLEAR_PENDING_EXCEPTION; >>> ?417???????? THROW_HANDLE_0(ex); >>> ?418?????? } >>> >>> >>> This is strange too.? Why doesn't this just pass CHECK_NULL at line >>> 413, which is the same thing as checking for and clearing and >>> throwing the pending exception? >>> > > This is existing code.?? I removed these lines and pass CHECK_NULL to > call_static. Which, by the way, highlights the existing incorrect use of '0' instead of NULL in this code. > How does this version look? > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/src/hotspot/share/prims/jni.cpp.sdiff.html > > > One other thing I notice: it only sets the protection_domain to the > caller class when FindClass is called from JNI_OnLoad/JNI_OnUnload. > > For other caller class, it uses null protection_domain. I would expect > it should use the caller's protection domain? to look up a class.? I > think it's a bug, isn't? Neither Class.forName nor ClassLoader.loadClass take ProtectionDomains as arguments, so I think the "normal" behaviour of FindClass to not use a PD is correct. That of course begs the question as to why the OnLoad/OnUnload contexts are special. I don't have an answer but can do some research. Thanks, David > Mandy > From calvin.cheung at oracle.com Fri Oct 6 00:59:12 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 05 Oct 2017 17:59:12 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> Message-ID: <59D6D560.4020804@oracle.com> On 10/5/17, 1:54 PM, coleen.phillimore at oracle.com wrote: > > > On 10/5/17 1:38 PM, Jiangli Zhou wrote: >> Hi Calvin, >> >>> On Oct 4, 2017, at 3:32 PM, Calvin Cheung >>> wrote: >>> >>> >>> >>> On 10/4/17, 10:45 AM, mandy chung wrote: >>>> >>>> On 10/4/17 10:12 AM, Calvin Cheung wrote: >>>>> Mandy, David, >>>>> >>>>> Thanks for the review. >>>>> >>>>> I've made the change based on your suggestion: >>>>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ >>>> systemDictionary.cpp >>>> >>>> SystemDictionary::roots_oops_do and oops_do have to be updated to >>>> traverse during GC. >>> It is being handled in the following line of code in roots_oops_do(): >>> CDS_ONLY(SystemDictionaryShared::roots_oops_do(strong);) >> Since you moved _java_platform_loader to systemDictionary.hpp, please >> handle that in SystemDictionary::roots_oops_do() and >> SystemDictionary::oops_do(). > > Yes, agree with Jiangli. This was a good find, Mandy. > > http://cr.openjdk.java.net/~ccheung/8185694/webrev.02/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html > > > *+ Klass* class_loader_klass = WK_KLASS(ClassLoader_klass);* > > This should be: > > InstanceKlass* class_loader_klass = > SystemDictionary::ClassLoader_klass(); > > The WK X macro is inappropriate to use here. I'm surprised it's not > #undef at this point. I've modified the code to using SystemDictionary::ClassLoader_klass() but I don't need the local variable because I'm creating the _java_system_loader and _java_platform_loader separately in 2 different functions. It is because the _java_platform_loader is required for module system initialization in call_initPhase2. It would be too early to create the _java_system_loader before call_initPhase2 because in call_initPhase3, it will check if there is a custom system loader defined via the -Djava.system.loader property. > >> I agree with David. The checks for >> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >> and >> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() >> look strange. They should be investigated and cleaned up. >> I've tried simplifying the is_system_class_loader() as mentioned >> above and it seems to work but more testing is required. > > This would be nice if this could be simplified as suggested. Yes, the simplification is possible. I also updated roots_oops_do() and oops_do() for the _java_platform_loader. updated webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.03/ > > So if you use -Djava.system.loader=myLoader on the command line, > setting _java_system_loader, then does that mean that the classes > loaded by > SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() > are not in the system loader? ie. they can be unloaded? What is the > result of the call to SystemDictionary::is_system_class_loader() used > for? I guess same question applies to the platform class loader. Below are couple of usages of is_system_class_loader() in ClassLoaderData: // Returns true if this class loader data is for the system class loader. bool ClassLoaderData::is_system_class_loader_data() const { return SystemDictionary::is_system_class_loader(class_loader()); } // Returns true if this class loader data is one of the 3 builtin // (boot, application/system or platform) class loaders. Note, the // builtin loaders are not freed by a GC. bool ClassLoaderData::is_builtin_class_loader_data() const { return (is_the_null_class_loader_data() || SystemDictionary::is_system_class_loader(class_loader()) || SystemDictionary::is_platform_class_loader(class_loader())); } I ran tests using custom loader and class unload tests like the following and they passed: open/test/jdk/java/lang/ClassLoader/CustomSystemLoader/InitSystemLoaderTest.java open/test/jdk/java/lang/System/ open/test/hotspot/jtreg/runtime/ClassUnload thanks, Calvin > > thanks, > Coleen > >> >>> The implementation is in closed source. >> Please clean up the closed code to remove those. >> >> >> Thanks, >> >> Jiangli >> >>>> Is this new java_platform_loader function used anywhere? >>> Yes, it is being used in closed source. >>>> Currently >>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>> is preloaded. Shouldn't this be removed? What about >>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>> They're being used in lines 184 and 193 in systemDictionary.cpp and >>> also in closed source. >>>> thread.cpp >>>> >>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>> >>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? >>>> Should it simply use CHECK_JNI_ERR as in other places? >>> They are the same, in utilities/exceptions.hpp: >>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>> >>> and it expands to the following: >>> __the_thread__); if >>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return >>> (-1); (void)(0 >>> >>> I can change it to CHECK_JNI_ERR. >>> >>> thanks, >>> Calvin >>>> Mandy > From coleen.phillimore at oracle.com Fri Oct 6 01:15:25 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 5 Oct 2017 21:15:25 -0400 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D6D560.4020804@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <59D6D560.4020804@oracle.com> Message-ID: <0cbf19d5-abc7-a97c-9dec-0731bcc8f570@oracle.com> On 10/5/17 8:59 PM, Calvin Cheung wrote: > > > On 10/5/17, 1:54 PM, coleen.phillimore at oracle.com wrote: >> >> >> On 10/5/17 1:38 PM, Jiangli Zhou wrote: >>> Hi Calvin, >>> >>>> On Oct 4, 2017, at 3:32 PM, Calvin Cheung >>>> wrote: >>>> >>>> >>>> >>>> On 10/4/17, 10:45 AM, mandy chung wrote: >>>>> >>>>> On 10/4/17 10:12 AM, Calvin Cheung wrote: >>>>>> Mandy, David, >>>>>> >>>>>> Thanks for the review. >>>>>> >>>>>> I've made the change based on your suggestion: >>>>>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.01/ >>>>> systemDictionary.cpp >>>>> >>>>> SystemDictionary::roots_oops_do and oops_do have to be updated to >>>>> traverse during GC. >>>> It is being handled in the following line of code in roots_oops_do(): >>>> ? CDS_ONLY(SystemDictionaryShared::roots_oops_do(strong);) >>> Since you moved _java_platform_loader to systemDictionary.hpp, >>> please handle that in SystemDictionary::roots_oops_do() and >>> SystemDictionary::oops_do(). >> >> Yes, agree with Jiangli.? This was a good find, Mandy. >> >> http://cr.openjdk.java.net/~ccheung/8185694/webrev.02/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html >> >> >> *+ Klass* class_loader_klass = WK_KLASS(ClassLoader_klass);* >> >> This should be: >> >> ??? InstanceKlass* class_loader_klass = >> SystemDictionary::ClassLoader_klass(); >> >> The WK X macro is inappropriate to use here.?? I'm surprised it's not >> #undef at this point. > I've modified the code to using SystemDictionary::ClassLoader_klass() > but I don't need the local variable because I'm creating the > _java_system_loader and _java_platform_loader separately in 2 > different functions. > > It is because the _java_platform_loader is required for module system > initialization in call_initPhase2. > It would be too early to create the _java_system_loader before > call_initPhase2 because in call_initPhase3, it will check if there is > a custom system loader defined via the -Djava.system.loader property. > >> >>> I agree with David. The checks for >>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>> and >>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass() >>> look strange. They should be investigated and cleaned up. >>> I've tried simplifying the is_system_class_loader() as mentioned >>> above and it seems to work but more testing is required. >> >> This would be nice if this could be simplified as suggested. > Yes, the simplification is possible. I also updated roots_oops_do() > and oops_do() for the _java_platform_loader. > > updated webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.03/ >> >> So if you use -Djava.system.loader=myLoader on the command line, >> setting _java_system_loader, then does that mean that the classes >> loaded by >> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >> are not in the system loader?? ie. they can be unloaded?? What is the >> result of the call to SystemDictionary::is_system_class_loader() used >> for??? I guess same question applies to the platform class loader. > Below are couple of usages of is_system_class_loader() in > ClassLoaderData: > // Returns true if this class loader data is for the system class loader. > bool ClassLoaderData::is_system_class_loader_data() const { > ? return SystemDictionary::is_system_class_loader(class_loader()); > } > > // Returns true if this class loader data is one of the 3 builtin > // (boot, application/system or platform) class loaders. Note, the > // builtin loaders are not freed by a GC. > bool ClassLoaderData::is_builtin_class_loader_data() const { > ? return (is_the_null_class_loader_data() || > ????????? SystemDictionary::is_system_class_loader(class_loader()) || > SystemDictionary::is_platform_class_loader(class_loader())); > } > > I ran tests using custom loader and class unload tests like the > following and they passed: > > open/test/jdk/java/lang/ClassLoader/CustomSystemLoader/InitSystemLoaderTest.java > > open/test/jdk/java/lang/System/ > open/test/hotspot/jtreg/runtime/ClassUnload Great.?? This change looks good.? A couple of small things: In http://cr.openjdk.java.net/~ccheung/8185694/webrev.03/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html *if (class_loader == NULL) {* *return false;* *}* You probably don't have to check for NULL if you're simply checking for equality.?? You may want to assert that compute_java_{system,platform}_loader do not return NULL.? I think that is an invariant.?? I don't need to see another version and I'm glad you made this clearer. I thought these were used by class unloading and determining that modules on the reads/exports list don't need to be walked.? If you run the test/hotspot/jtreg/runtime/modules tests, that would be good but they'll run under JPRT. Thanks, Coleen > > thanks, > Calvin >> >> thanks, >> Coleen >> >>> >>>> The implementation is in closed source. >>> Please clean up the closed code to remove those. >>> >>> >>> Thanks, >>> >>> Jiangli >>> >>>>> Is this new java_platform_loader function used anywhere? >>>> Yes, it is being used in closed source. >>>>> Currently >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>> is preloaded.? Shouldn't this be removed?? What about >>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>> They're being used in lines 184 and 193 in systemDictionary.cpp and >>>> also in closed source. >>>>> thread.cpp >>>>> >>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>> >>>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR?? >>>>> Should it simply use CHECK_JNI_ERR as in other places? >>>> They are the same, in utilities/exceptions.hpp: >>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>> >>>> and it expands to the following: >>>> __the_thread__); if >>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return >>>> (-1); (void)(0 >>>> >>>> I can change it to CHECK_JNI_ERR. >>>> >>>> thanks, >>>> Calvin >>>>> Mandy >> From david.holmes at oracle.com Fri Oct 6 01:15:52 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Oct 2017 11:15:52 +1000 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> <7f970d16-19de-7729-48a4-0653a6d61f7e@oracle.com> <7fb4775b-0b1a-2c47-6742-f3dd6301f98c@oracle.com> Message-ID: <320f5310-230c-874a-5cf6-d2b5c0b7c2e0@oracle.com> Hi Mandy, On 6/10/2017 10:57 AM, David Holmes wrote: > On 6/10/2017 7:56 AM, mandy chung wrote: >> On 10/5/17 1:13 PM, coleen.phillimore at oracle.com wrote: >>> >>> I should say, besides these style comments this change looks good. >>> Coleen >>> >>> On 10/5/17 3:28 PM, coleen.phillimore at oracle.com wrote: >>>> >>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/src/hotspot/share/prims/jni.cpp.udiff.html >>>> >>>> >>>> The type Klass has a class_loader() and protection_domain() >>>> accessors so it is unnecessary to cast to InstanceKlass.? We went >>>> through and removed many of these unnecessary InstanceKlass::casts. >>>> >> >> That's from existing code which may not be touched for a long time. >> Since I'm on this file, I can clean this up. >>>> 407???? if (loader.is_null() && >>>> >> >> - if (loader.is_null() && >> + if (k->class_loader() == NULL && Already fixed per David's comment. >> >>>> I agree with David's comment that this line will always evaluate to >>>> true so should be cleaned up also. >>>> >>>> 413????????????????????????????? thread); >>>> ?414?????? if (HAS_PENDING_EXCEPTION) { >>>> ?415???????? Handle ex(thread, thread->pending_exception()); >>>> ?416???????? CLEAR_PENDING_EXCEPTION; >>>> ?417???????? THROW_HANDLE_0(ex); >>>> ?418?????? } >>>> >>>> >>>> This is strange too.? Why doesn't this just pass CHECK_NULL at line >>>> 413, which is the same thing as checking for and clearing and >>>> throwing the pending exception? >>>> >> >> This is existing code.?? I removed these lines and pass CHECK_NULL to >> call_static. > > Which, by the way, highlights the existing incorrect use of '0' instead > of NULL in this code. > >> How does this version look? >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/src/hotspot/share/prims/jni.cpp.sdiff.html >> >> >> One other thing I notice: it only sets the protection_domain to the >> caller class when FindClass is called from JNI_OnLoad/JNI_OnUnload. >> >> For other caller class, it uses null protection_domain. I would expect >> it should use the caller's protection domain? to look up a class.? I >> think it's a bug, isn't? > > Neither Class.forName nor ClassLoader.loadClass take ProtectionDomains > as arguments, so I think the "normal" behaviour of FindClass to not use > a PD is correct. That of course begs the question as to why the > OnLoad/OnUnload contexts are special. I don't have an answer but can do > some research. The PD change relates to a very old security bug: https://bugs.openjdk.java.net/browse/JDK-4288452 (not public) Can't say any more other than "it ain't broken, so don't 'fix' it". Cheers, David > Thanks, > David > >> Mandy >> From david.holmes at oracle.com Fri Oct 6 01:33:55 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Oct 2017 11:33:55 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> Message-ID: <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> Hi Coleen, Calvin, On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: > So if you use -Djava.system.loader=myLoader on the command line, setting > _java_system_loader, then does that mean that the classes loaded by > SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() are > not in the system loader?? ie. they can be unloaded?? What is the result > of the call to SystemDictionary::is_system_class_loader() used for??? I > guess same question applies to the platform class loader. The classloading delegation hierarchy (as of JDK 9) is: - boot loader (native) - platform loader (built-in) - system (aka application) loader (built-in) If the user specifies a custom class for the system loader then it is loaded by an instance of the default system loader and becomes a fourth level in the hierarchy, and it is then technically the "system loader". None of these loaders, or the classes they load can be unloaded. Which is presumably why the code checks both: 180 bool SystemDictionary::is_system_class_loader(oop class_loader) { 181 if (class_loader == NULL) { 182 return false; 183 } 184 return (class_loader->klass() == SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() || 185 class_loader == _java_system_loader); 186 } because we need to treat both of these instances as the "system loader" from a VM perspective? The same does not apply to the platform loader. David ----- > thanks, > Coleen > >> >>> The implementation is in closed source. >> Please clean up the closed code to remove those. >> >> >> Thanks, >> >> Jiangli >> >>>> Is this new java_platform_loader function used anywhere? >>> Yes, it is being used in closed source. >>>> Currently >>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>> is preloaded.? Shouldn't this be removed?? What about >>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>> They're being used in lines 184 and 193 in systemDictionary.cpp and >>> also in closed source. >>>> thread.cpp >>>> >>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>> >>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? >>>> Should it simply use CHECK_JNI_ERR as in other places? >>> They are the same, in utilities/exceptions.hpp: >>> #define CHECK_JNI_ERR??????????????????????????? CHECK_(JNI_ERR) >>> >>> and it expands to the following: >>> __the_thread__); if >>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return >>> (-1); (void)(0 >>> >>> I can change it to CHECK_JNI_ERR. >>> >>> thanks, >>> Calvin >>>> Mandy > From coleen.phillimore at oracle.com Fri Oct 6 01:35:40 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 5 Oct 2017 21:35:40 -0400 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: <7fb4775b-0b1a-2c47-6742-f3dd6301f98c@oracle.com> References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> <7f970d16-19de-7729-48a4-0653a6d61f7e@oracle.com> <7fb4775b-0b1a-2c47-6742-f3dd6301f98c@oracle.com> Message-ID: Hi Mandy, Version 02 looks fine.? David already commented on the inconsistency we saw with protection domain. thanks, Coleen On 10/5/17 5:56 PM, mandy chung wrote: > On 10/5/17 1:13 PM, coleen.phillimore at oracle.com wrote: >> >> I should say, besides these style comments this change looks good. >> Coleen >> >> On 10/5/17 3:28 PM, coleen.phillimore at oracle.com wrote: >>> >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/src/hotspot/share/prims/jni.cpp.udiff.html >>> >>> >>> The type Klass has a class_loader() and protection_domain() >>> accessors so it is unnecessary to cast to InstanceKlass.? We went >>> through and removed many of these unnecessary InstanceKlass::casts. >>> > > That's from existing code which may not be touched for a long time.? > Since I'm on this file, I can clean this up. >>> 407???? if (loader.is_null() && >>> > > - if (loader.is_null() && > + if (k->class_loader() == NULL && Already fixed per David's comment. >>> I agree with David's comment that this line will always evaluate to >>> true so should be cleaned up also. >>> >>> 413????????????????????????????? thread); >>> ?414?????? if (HAS_PENDING_EXCEPTION) { >>> ?415???????? Handle ex(thread, thread->pending_exception()); >>> ?416???????? CLEAR_PENDING_EXCEPTION; >>> ?417???????? THROW_HANDLE_0(ex); >>> ?418?????? } >>> >>> >>> This is strange too.? Why doesn't this just pass CHECK_NULL at line >>> 413, which is the same thing as checking for and clearing and >>> throwing the pending exception? >>> > > This is existing code.?? I removed these lines and pass CHECK_NULL to > call_static. > > How does this version look? > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/src/hotspot/share/prims/jni.cpp.sdiff.html > > One other thing I notice: it only sets the protection_domain to the > caller class when FindClass is called from JNI_OnLoad/JNI_OnUnload. > > For other caller class, it uses null protection_domain. I would expect > it should use the caller's protection domain? to look up a class.? I > think it's a bug, isn't? > > Mandy > From coleen.phillimore at oracle.com Fri Oct 6 01:38:11 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 5 Oct 2017 21:38:11 -0400 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> Message-ID: On 10/5/17 9:33 PM, David Holmes wrote: > Hi Coleen, Calvin, > > On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >> So if you use -Djava.system.loader=myLoader on the command line, >> setting _java_system_loader, then does that mean that the classes >> loaded by >> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >> are not in the system loader?? ie. they can be unloaded?? What is the >> result of the call to SystemDictionary::is_system_class_loader() used >> for??? I guess same question applies to the platform class loader. > > The classloading delegation hierarchy (as of JDK 9) is: > - boot loader (native) > ? - platform loader (built-in) > ??? - system (aka application) loader (built-in) > > If the user specifies a custom class for the system loader then it is > loaded by an instance of the default system loader and becomes a > fourth level in the hierarchy, and it is then technically the "system > loader". None of these loaders, or the classes they load can be unloaded. > > Which is presumably why the code checks both: > > ?180 bool SystemDictionary::is_system_class_loader(oop class_loader) { > ?181?? if (class_loader == NULL) { > ?182???? return false; > ?183?? } > ?184?? return (class_loader->klass() == > SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() > || > ?185?????????? class_loader == _java_system_loader); > ?186 } > Okay then this code shouldn't change. > because we need to treat both of these instances as the "system > loader" from a VM perspective? The same does not apply to the platform > loader. I don't think we have good tests for this. Coleen > > David > ----- > >> thanks, >> Coleen >> >>> >>>> The implementation is in closed source. >>> Please clean up the closed code to remove those. >>> >>> >>> Thanks, >>> >>> Jiangli >>> >>>>> Is this new java_platform_loader function used anywhere? >>>> Yes, it is being used in closed source. >>>>> Currently >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>> is preloaded.? Shouldn't this be removed?? What about >>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>> They're being used in lines 184 and 193 in systemDictionary.cpp and >>>> also in closed source. >>>>> thread.cpp >>>>> >>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>> >>>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR?? >>>>> Should it simply use CHECK_JNI_ERR as in other places? >>>> They are the same, in utilities/exceptions.hpp: >>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>> >>>> and it expands to the following: >>>> __the_thread__); if >>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return >>>> (-1); (void)(0 >>>> >>>> I can change it to CHECK_JNI_ERR. >>>> >>>> thanks, >>>> Calvin >>>>> Mandy >> From jiangli.zhou at oracle.com Fri Oct 6 01:39:40 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Thu, 5 Oct 2017 18:39:40 -0700 Subject: RFR (S) 8178870: instrumentation.retransformClasses cause coredump In-Reply-To: <286a1baa-1b89-6cbb-fea2-ae3a8fe4a328@oracle.com> References: <286a1baa-1b89-6cbb-fea2-ae3a8fe4a328@oracle.com> Message-ID: <2B16AE11-E336-42AC-B643-7C5A90AA2069@oracle.com> Hi Coleen, Not an expert in JVMTI area. The bug analysis, fix and test look good to me! Thanks, Jiangli > On Oct 5, 2017, at 11:15 AM, coleen.phillimore at oracle.com wrote: > > Summary: Don't double-free cached class bytes on redefinition loading failure. > > See bug for more description. Tested with test/jdk/java/lang/instrument, test/hotspot/jtreg/runtime/RedefineTests, internal CDS tests, internal nsk.jvmti tests, and added test. > > open webrev at http://cr.openjdk.java.net/~coleenp/8178870.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8178870 > > Thanks, > Coleen From david.holmes at oracle.com Fri Oct 6 01:43:55 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Oct 2017 11:43:55 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D6D560.4020804@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <59D6D560.4020804@oracle.com> Message-ID: <7f4cf10c-7a8d-34ff-2719-9082b0bdf1e9@oracle.com> On 6/10/2017 10:59 AM, Calvin Cheung wrote: > updated webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.03/ src/hotspot/share/classfile/systemDictionary.hpp // Returns default system loader Actually returns the system loader whether default or not. Unclear if you might actually want two queries: one for default and one for actual (which has the default as its parent) // Returns default platform loader You don't need to say "default" as there is only ever one platform loader. --- share/classfile/systemDictionary.cpp 186 return (class_loader == _java_system_loader); I think you need to restore this to checking both the field or whether an instance of the default system loader class - as per previous email - to handle the custom system loader case. The explicit NULL checks will be needed if these can be called before the platform/system loaders have been determined. I suspect they can given the complex initialization process we have with modules. Thanks, David From david.holmes at oracle.com Fri Oct 6 02:16:39 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Oct 2017 12:16:39 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> Message-ID: On 6/10/2017 11:38 AM, coleen.phillimore at oracle.com wrote: > On 10/5/17 9:33 PM, David Holmes wrote: >> Hi Coleen, Calvin, >> >> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>> So if you use -Djava.system.loader=myLoader on the command line, >>> setting _java_system_loader, then does that mean that the classes >>> loaded by >>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>> are not in the system loader?? ie. they can be unloaded?? What is the >>> result of the call to SystemDictionary::is_system_class_loader() used >>> for??? I guess same question applies to the platform class loader. >> >> The classloading delegation hierarchy (as of JDK 9) is: >> - boot loader (native) >> ? - platform loader (built-in) >> ??? - system (aka application) loader (built-in) >> >> If the user specifies a custom class for the system loader then it is >> loaded by an instance of the default system loader and becomes a >> fourth level in the hierarchy, and it is then technically the "system >> loader". None of these loaders, or the classes they load can be unloaded. >> >> Which is presumably why the code checks both: >> >> ?180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >> ?181?? if (class_loader == NULL) { >> ?182???? return false; >> ?183?? } >> ?184?? return (class_loader->klass() == >> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >> || >> ?185?????????? class_loader == _java_system_loader); >> ?186 } >> > > Okay then this code shouldn't change. Right. >> because we need to treat both of these instances as the "system >> loader" from a VM perspective? The same does not apply to the platform >> loader. > > I don't think we have good tests for this. There are a few tests that define a custom system loader: ./jdk/java/lang/System/LoggerFinder/ ./jdk/java/lang/instrument/CustomSystemLoader/ ./jdk/java/lang/ClassLoader/securityManager/ClassLoaderTest.java ./jdk/sun/security/util/Resources/customSysClassLoader/BootMessages.java ./hotspot/jtreg/runtime/modules/ModuleStress/ModuleStress.java ./jdk/java/lang/ClassLoader/CustomSystemLoader/InitSystemLoaderTest.java but the last one seems to be the extent of actually testing it and it is minimal: a check that the system loader is custom loader instance, and that the delegation grandparent is the platform loader. David > Coleen >> >> David >> ----- >> >>> thanks, >>> Coleen >>> >>>> >>>>> The implementation is in closed source. >>>> Please clean up the closed code to remove those. >>>> >>>> >>>> Thanks, >>>> >>>> Jiangli >>>> >>>>>> Is this new java_platform_loader function used anywhere? >>>>> Yes, it is being used in closed source. >>>>>> Currently >>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>> is preloaded.? Shouldn't this be removed?? What about >>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>> They're being used in lines 184 and 193 in systemDictionary.cpp and >>>>> also in closed source. >>>>>> thread.cpp >>>>>> >>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>> >>>>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? >>>>>> Should it simply use CHECK_JNI_ERR as in other places? >>>>> They are the same, in utilities/exceptions.hpp: >>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>> >>>>> and it expands to the following: >>>>> __the_thread__); if >>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return >>>>> (-1); (void)(0 >>>>> >>>>> I can change it to CHECK_JNI_ERR. >>>>> >>>>> thanks, >>>>> Calvin >>>>>> Mandy >>> > From david.holmes at oracle.com Fri Oct 6 02:22:19 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Oct 2017 12:22:19 +1000 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: <320f5310-230c-874a-5cf6-d2b5c0b7c2e0@oracle.com> References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> <7f970d16-19de-7729-48a4-0653a6d61f7e@oracle.com> <7fb4775b-0b1a-2c47-6742-f3dd6301f98c@oracle.com> <320f5310-230c-874a-5cf6-d2b5c0b7c2e0@oracle.com> Message-ID: <70579656-9ef7-59d1-2062-0467488d7e22@oracle.com> Forgot to add webrev.02 looks fine to me too. Thanks, David ----- On 6/10/2017 11:15 AM, David Holmes wrote: > Hi Mandy, > > On 6/10/2017 10:57 AM, David Holmes wrote: >> On 6/10/2017 7:56 AM, mandy chung wrote: >>> On 10/5/17 1:13 PM, coleen.phillimore at oracle.com wrote: >>>> >>>> I should say, besides these style comments this change looks good. >>>> Coleen >>>> >>>> On 10/5/17 3:28 PM, coleen.phillimore at oracle.com wrote: >>>>> >>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.00/src/hotspot/share/prims/jni.cpp.udiff.html >>>>> >>>>> >>>>> The type Klass has a class_loader() and protection_domain() >>>>> accessors so it is unnecessary to cast to InstanceKlass.? We went >>>>> through and removed many of these unnecessary InstanceKlass::casts. >>>>> >>> >>> That's from existing code which may not be touched for a long time. >>> Since I'm on this file, I can clean this up. >>>>> 407???? if (loader.is_null() && >>>>> >>> >>> - if (loader.is_null() && >>> + if (k->class_loader() == NULL && Already fixed per David's comment. >>> >>>>> I agree with David's comment that this line will always evaluate to >>>>> true so should be cleaned up also. >>>>> >>>>> 413????????????????????????????? thread); >>>>> ?414?????? if (HAS_PENDING_EXCEPTION) { >>>>> ?415???????? Handle ex(thread, thread->pending_exception()); >>>>> ?416???????? CLEAR_PENDING_EXCEPTION; >>>>> ?417???????? THROW_HANDLE_0(ex); >>>>> ?418?????? } >>>>> >>>>> >>>>> This is strange too.? Why doesn't this just pass CHECK_NULL at line >>>>> 413, which is the same thing as checking for and clearing and >>>>> throwing the pending exception? >>>>> >>> >>> This is existing code.?? I removed these lines and pass CHECK_NULL to >>> call_static. >> >> Which, by the way, highlights the existing incorrect use of '0' >> instead of NULL in this code. >> >>> How does this version look? >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/src/hotspot/share/prims/jni.cpp.sdiff.html >>> >>> >>> One other thing I notice: it only sets the protection_domain to the >>> caller class when FindClass is called from JNI_OnLoad/JNI_OnUnload. >>> >>> For other caller class, it uses null protection_domain. I would >>> expect it should use the caller's protection domain? to look up a >>> class.? I think it's a bug, isn't? >> >> Neither Class.forName nor ClassLoader.loadClass take ProtectionDomains >> as arguments, so I think the "normal" behaviour of FindClass to not >> use a PD is correct. That of course begs the question as to why the >> OnLoad/OnUnload contexts are special. I don't have an answer but can >> do some research. > > The PD change relates to a very old security bug: > > https://bugs.openjdk.java.net/browse/JDK-4288452 (not public) > > Can't say any more other than "it ain't broken, so don't 'fix' it". > > Cheers, > David > >> Thanks, >> David >> >>> Mandy >>> From mandy.chung at oracle.com Fri Oct 6 02:29:05 2017 From: mandy.chung at oracle.com (mandy chung) Date: Thu, 5 Oct 2017 19:29:05 -0700 Subject: Review Request JDK-8188052: JNI_FindClass needs to specify the class loading context used for library lifecycle hooks In-Reply-To: References: <20f86e9c-06b2-a6e5-273d-8536d62e9d64@oracle.com> <7f970d16-19de-7729-48a4-0653a6d61f7e@oracle.com> <7fb4775b-0b1a-2c47-6742-f3dd6301f98c@oracle.com> Message-ID: Thanks for the review, David and Coleen. Mandy On 10/5/17 6:35 PM, coleen.phillimore at oracle.com wrote: > > Hi Mandy, > Version 02 looks fine.? David already commented on the inconsistency > we saw with protection domain. > thanks, > Coleen From david.holmes at oracle.com Fri Oct 6 02:48:43 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Oct 2017 12:48:43 +1000 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> Message-ID: <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> Hi Mandy, On 6/10/2017 8:24 AM, mandy chung wrote: > Hi David, > > This version only modifies ClassLoader.java and the new test: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.03/ This seems mostly fine now - thanks for simplifying and clarifying the test operation. I agree we can look at the overall locking strategy separately. Regarding the need for volatile to make the double-checked locking correct ... I can't quite convince myself that any thread that tries to execute a native method and so calls back to findNative from the VM, must have already gone through the code that ensured the native library maps have been initialized. I think they must, but I can't quite convince myself of it. :( Unless you can dispel my doubts I think we need to make the fields volatile to be strictly safe. That said I'm not sure lazy initialization is really worthwhile in the first place. Is the CHM creation very costly? Should we be sizing them explicitly? (More like the default Vector size used previously?) Thanks, David ----- > On 10/4/17 11:06 PM, David Holmes wrote: >> Hi Mandy, >> >>>> Overall the changes seem fine, but I do have a concern about the use >>>> of ConcurrentHashMap. This would seem to be a significant change in >>>> the initialization order - >>> >>> ConcurrentHashMap is referenced in ClassLoader but actually not loaded > > I missed the first sentence "I misread the source previously that CHM is > already loaded at clinit time.? But I figure that ....".?? So I fixed it > to lazily initialize it when the first native library is loaded. >> >> It was loaded in clinit: >> >> 2689???? private static Map systemNativeLibraries >> 2690???????? = new ConcurrentHashMap<>(); >> >> which was my concern. > > I got that.? Thanks for bringing this up. >> >>> until later.? So it does change the initialization order. ? I now >>> change the implementation to lazily create CHM.? This saves the >>> footprint when a class loader does not load any native library. >> >> Okay. But now I'm wondering about whether >> systemNativeLibraries/nativeLibraries need to be volatile? > > It's read for JNI native method binding (ClassLoader::findNative method) > and therefore I used double-locking idiom instead to avoid any > performance impact.?? I simplified this further.? The native libraries > map is created with the loadedLibraryNames lock. > >> Just looking in ClassLoader it would seem yes, but if the only paths >> to loading a library are through Runtime, and those methods are >> synchronized on the Runtime instance, then everything is serialized >> anyway. Are there other paths that might allow concurrent access? (If >> the synchronization in Runtime guarantees only a single library can >> ever be loaded at a time across all classloaders, then you don't even >> need the synchronization on the loadedLibraryNames ??). > > That's a good point.? This code including the synchronization on > loadedLibraryNames was written long ago.? I prefer to file a JBS issue > to study the locking more closely. > > >>>> >>>> libnativeLibraryTest.c: >>>> >>>> Not sure I see the point in the FindClass("java/lang/ClassLoader") >>>> as if it fails to find it then surely it already failed to find >>>> "java/lang/Error" and you'll possibly crash trying to throw. >>>> >>> I agree that should be cleaned up.? I took out >>> FindClass("java/lang/ClassLoader") case. >> >> So is: >> >> ? 68???? excls = (*env)->FindClass(env, "java/lang/Error"); >> ? 69???? if (excls == NULL) { >> ? 70???????? (*env)->FatalError(env, "Error class not found"); >> ? 71???? } >> >> just a sanity check that you can in fact load a class? >> ?75???????? (*env)->FatalError(env, "Error class not found"); >> >> That's the wrong error message. > > Thanks for catching this.? I revised the test and fixed the above. > >> >> Okay. Not sure why you needed to introduce q.Bridge - > > I agree it's not needed any more.? Initially I was trying to have a > method to return a count.? But later modified to a different method. > >> I find the test logic rather hard to follow, not clear who calls what >> method from where and when. I was thinking of a simple >> >> public int loadedCount; >> public int unloadedCount; >> >> that onLoad/OnUnload would increment. > Well the test intends to verify if the native library can be reloaded > (i.e. it has to be unloaded before reloading; otherwise > UnsatifisedLinkError will be thrown).?? It has a native count method and > it would fail if it is not loaded. > > I revise the test a little.? p.Test.run() loads the native library. The > test checks if the native count method returns the correct value (i.e. > the native library is loaded and OnLoad is invoked).? Calling run() > again should get ULE since it can't be reloaded if already loaded. > > I keep the unloadedCount which is not strictly necessary but it is an > additional sanity check.? I added more comment and hope it is clearer now. > > > Mandy From coleen.phillimore at oracle.com Fri Oct 6 02:56:21 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 5 Oct 2017 22:56:21 -0400 Subject: RFR (S) 8178870: instrumentation.retransformClasses cause coredump In-Reply-To: <2B16AE11-E336-42AC-B643-7C5A90AA2069@oracle.com> References: <286a1baa-1b89-6cbb-fea2-ae3a8fe4a328@oracle.com> <2B16AE11-E336-42AC-B643-7C5A90AA2069@oracle.com> Message-ID: <78413dab-10a7-a29b-bb71-7b96423b3499@oracle.com> Thank you for the code review! I had to fix the .c file slightly for Solaris because in solaris C you still can't declare variables before statements.? It's a minor change.? I also added -lc to the makefile for solaris because the other .c files had it. http://cr.openjdk.java.net/~coleenp/8178870.02/webrev/index.html Thanks, Coleen On 10/5/17 9:39 PM, Jiangli Zhou wrote: > Hi Coleen, > > Not an expert in JVMTI area. The bug analysis, fix and test look good to me! > > Thanks, > Jiangli > >> On Oct 5, 2017, at 11:15 AM, coleen.phillimore at oracle.com wrote: >> >> Summary: Don't double-free cached class bytes on redefinition loading failure. >> >> See bug for more description. Tested with test/jdk/java/lang/instrument, test/hotspot/jtreg/runtime/RedefineTests, internal CDS tests, internal nsk.jvmti tests, and added test. >> >> open webrev at http://cr.openjdk.java.net/~coleenp/8178870.01/webrev >> bug link https://bugs.openjdk.java.net/browse/JDK-8178870 >> >> Thanks, >> Coleen From david.holmes at oracle.com Fri Oct 6 04:47:00 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Oct 2017 14:47:00 +1000 Subject: RFR: 8185529: JCK api/java_lang/Object/WaitTests failed with jdk10/hs nightly Message-ID: Bug is not public - sorry. webrev: http://cr.openjdk.java.net/~dholmes/8185529/webrev/ Also patch inline below. Problem: o.wait(Long.MAX_VALUE) returns immediately Cause: arithmetic overflow gives a time in the past so the timeout expires immediately This was introduced by the fix for: https://bugs.openjdk.java.net/browse/JDK-8174231 "Factor out and share PlatformEvent and Parker code for POSIX systems." while we go to great lengths to ensure we avoid overflows inside the time calculation functions, I overlooked the fact that we now first convert a millisecond value into nanoseconds, which can overflow. Fix: limit the millis value to the existing hard limit of MAX_SECS*MILLIUNITS ms. That can not overflow when converted to nanos. Thanks, David ----- --- old/src/hotspot/os/posix/os_posix.cpp 2017-10-06 00:34:31.595159338 -0400 +++ new/src/hotspot/os/posix/os_posix.cpp 2017-10-06 00:34:29.443037883 -0400 @@ -1770,6 +1770,12 @@ if (v == 0) { // Do this the hard way by blocking ... struct timespec abst; + // We have to watch for overflow when converting millis to nanos, + // but if millis is that large then we will end up limiting to + // MAX_SECS anyway, so just do that here. + if (millis / MILLIUNITS > MAX_SECS) { + millis = jlong(MAX_SECS) * MILLIUNITS; + } to_abstime(&abst, millis * (NANOUNITS / MILLIUNITS), false); From calvin.cheung at oracle.com Fri Oct 6 05:28:12 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 05 Oct 2017 22:28:12 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> Message-ID: <59D7146C.3010206@oracle.com> On 10/5/17, 6:33 PM, David Holmes wrote: > Hi Coleen, Calvin, > > On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >> So if you use -Djava.system.loader=myLoader on the command line, >> setting _java_system_loader, then does that mean that the classes >> loaded by >> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >> are not in the system loader? ie. they can be unloaded? What is the >> result of the call to SystemDictionary::is_system_class_loader() used >> for? I guess same question applies to the platform class loader. > > The classloading delegation hierarchy (as of JDK 9) is: > - boot loader (native) > - platform loader (built-in) > - system (aka application) loader (built-in) > > If the user specifies a custom class for the system loader then it is > loaded by an instance of the default system loader and becomes a > fourth level in the hierarchy, and it is then technically the "system > loader". None of these loaders, or the classes they load can be unloaded. > > Which is presumably why the code checks both: > > 180 bool SystemDictionary::is_system_class_loader(oop class_loader) { > 181 if (class_loader == NULL) { > 182 return false; > 183 } > 184 return (class_loader->klass() == > SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() > || > 185 class_loader == _java_system_loader); > 186 } > > because we need to treat both of these instances as the "system > loader" from a VM perspective? The same does not apply to the platform > loader. We're obtaining the _java_system_loader after initPhase3 even before this change. Roughly, the calling sequence of initPhase3 is as follows: call_initPhase3() -> ClassLoader.initPhase3() -> ClassLoader.initSystemClassLoader() which contains the following code: String cn = System.getProperty("java.system.class.loader"); if (cn != null) { try { // custom class loader is only supported to be loaded from unnamed module Constructor ctor = Class.forName(cn, false, builtinLoader) .getDeclaredConstructor(ClassLoader.class); scl = (ClassLoader) ctor.newInstance(builtinLoader); } catch (Exception e) { throw new Error(e); } } else { scl = builtinLoader; } return scl; So initSystemClassLoader() will either return the built-in system loader or a custom loader if it exists. We use the getSystemClassLoader API to obtain the _java_system_loader: public static ClassLoader getSystemClassLoader() { switch (VM.initLevel()) { case 0: case 1: case 2: // the system class loader is the built-in app class loader during startup return getBuiltinAppClassLoader(); case 3: String msg = "getSystemClassLoader should only be called after VM booted"; throw new InternalError(msg); case 4: // system fully initialized assert VM.isBooted() && scl != null; SecurityManager sm = System.getSecurityManager(); if (sm != null) { checkClassLoaderPermission(scl, Reflection.getCallerClass()); } return scl; default: throw new InternalError("should not reach here"); } } So the _java_system_loader will either be the built-in system loader or a custom loader. (case 4 in the above) I don't quite understand why the check in line 184 is required? Is it for checking if a given class_loader is the same type (like an instanceof) as the built-in system loader? thanks, Calvin > > David > ----- > >> thanks, >> Coleen >> >>> >>>> The implementation is in closed source. >>> Please clean up the closed code to remove those. >>> >>> >>> Thanks, >>> >>> Jiangli >>> >>>>> Is this new java_platform_loader function used anywhere? >>>> Yes, it is being used in closed source. >>>>> Currently >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>> is preloaded. Shouldn't this be removed? What about >>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>> They're being used in lines 184 and 193 in systemDictionary.cpp and >>>> also in closed source. >>>>> thread.cpp >>>>> >>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>> >>>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? >>>>> Should it simply use CHECK_JNI_ERR as in other places? >>>> They are the same, in utilities/exceptions.hpp: >>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>> >>>> and it expands to the following: >>>> __the_thread__); if >>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return >>>> (-1); (void)(0 >>>> >>>> I can change it to CHECK_JNI_ERR. >>>> >>>> thanks, >>>> Calvin >>>>> Mandy >> From david.holmes at oracle.com Fri Oct 6 05:38:39 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Oct 2017 15:38:39 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D7146C.3010206@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> Message-ID: On 6/10/2017 3:28 PM, Calvin Cheung wrote: > On 10/5/17, 6:33 PM, David Holmes wrote: >> Hi Coleen, Calvin, >> >> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>> So if you use -Djava.system.loader=myLoader on the command line, >>> setting _java_system_loader, then does that mean that the classes >>> loaded by >>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>> are not in the system loader?? ie. they can be unloaded?? What is the >>> result of the call to SystemDictionary::is_system_class_loader() used >>> for??? I guess same question applies to the platform class loader. >> >> The classloading delegation hierarchy (as of JDK 9) is: >> - boot loader (native) >> ? - platform loader (built-in) >> ??? - system (aka application) loader (built-in) >> >> If the user specifies a custom class for the system loader then it is >> loaded by an instance of the default system loader and becomes a >> fourth level in the hierarchy, and it is then technically the "system >> loader". None of these loaders, or the classes they load can be unloaded. >> >> Which is presumably why the code checks both: >> >> ?180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >> ?181?? if (class_loader == NULL) { >> ?182???? return false; >> ?183?? } >> ?184?? return (class_loader->klass() == >> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >> || >> ?185?????????? class_loader == _java_system_loader); >> ?186 } >> >> because we need to treat both of these instances as the "system >> loader" from a VM perspective? The same does not apply to the platform >> loader. > We're obtaining the _java_system_loader after initPhase3 even before > this change. Roughly, the calling sequence of initPhase3 is as follows: > > call_initPhase3() > ??? -> ClassLoader.initPhase3() > ??????? -> ClassLoader.initSystemClassLoader()? which contains the > following code: > > ??????? String cn = System.getProperty("java.system.class.loader"); > ??????? if (cn != null) { > ??????????? try { > ??????????????? // custom class loader is only supported to be loaded > from unnamed module > ??????????????? Constructor ctor = Class.forName(cn, false, > builtinLoader) > .getDeclaredConstructor(ClassLoader.class); > ??????????????? scl = (ClassLoader) ctor.newInstance(builtinLoader); > ??????????? } catch (Exception e) { > ??????????????? throw new Error(e); > ??????????? } > ??????? } else { > ??????????? scl = builtinLoader; > ??????? } > ??????? return scl; > > ??????? So initSystemClassLoader() will either return the built-in > system loader or a custom loader if it exists. > > We use the getSystemClassLoader API to obtain the _java_system_loader: > > ??? public static ClassLoader getSystemClassLoader() { > ??????? switch (VM.initLevel()) { > ??????????? case 0: > ??????????? case 1: > ??????????? case 2: > ??????????????? // the system class loader is the built-in app class > loader during startup > ??????????????? return getBuiltinAppClassLoader(); > ??????????? case 3: > ??????????????? String msg = "getSystemClassLoader should only be > called after VM booted"; > ??????????????? throw new InternalError(msg); > ??????????? case 4: > ??????????????? // system fully initialized > ??????????????? assert VM.isBooted() && scl != null; > ??????????????? SecurityManager sm = System.getSecurityManager(); > ??????????????? if (sm != null) { > ??????????????????? checkClassLoaderPermission(scl, > Reflection.getCallerClass()); > ??????????????? } > ??????????????? return scl; > ??????????? default: > ??????????????? throw new InternalError("should not reach here"); > ??????? } > ??? } > > ??? So the _java_system_loader will either be the built-in system > loader or a custom loader. (case 4 in the above) > > ??? I don't quite understand why the check in line 184 is required? > ??? Is it for checking if a given class_loader is the same type (like > an instanceof) as the built-in system loader? I believe it is checking if the loader is the built-in default system loader, both to account for the case where/if SystemDictionary::is_system_class_loader is called prior to initPhase3 completing; and also to account for encountering the default-built-in loader when the custom system loader delegates to it. I'd have to examine every call path to SystemDictionary::is_system_class_loader to check all the details. David ----- > thanks, > Calvin >> >> David >> ----- >> >>> thanks, >>> Coleen >>> >>>> >>>>> The implementation is in closed source. >>>> Please clean up the closed code to remove those. >>>> >>>> >>>> Thanks, >>>> >>>> Jiangli >>>> >>>>>> Is this new java_platform_loader function used anywhere? >>>>> Yes, it is being used in closed source. >>>>>> Currently >>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>> is preloaded.? Shouldn't this be removed?? What about >>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>> They're being used in lines 184 and 193 in systemDictionary.cpp and >>>>> also in closed source. >>>>>> thread.cpp >>>>>> >>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>> >>>>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? >>>>>> Should it simply use CHECK_JNI_ERR as in other places? >>>>> They are the same, in utilities/exceptions.hpp: >>>>> #define CHECK_JNI_ERR??????????????????????????? CHECK_(JNI_ERR) >>>>> >>>>> and it expands to the following: >>>>> __the_thread__); if >>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return >>>>> (-1); (void)(0 >>>>> >>>>> I can change it to CHECK_JNI_ERR. >>>>> >>>>> thanks, >>>>> Calvin >>>>>> Mandy >>> From calvin.cheung at oracle.com Fri Oct 6 05:40:42 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 05 Oct 2017 22:40:42 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <7f4cf10c-7a8d-34ff-2719-9082b0bdf1e9@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <59D6D560.4020804@oracle.com> <7f4cf10c-7a8d-34ff-2719-9082b0bdf1e9@oracle.com> Message-ID: <59D7175A.3030309@oracle.com> On 10/5/17, 6:43 PM, David Holmes wrote: > On 6/10/2017 10:59 AM, Calvin Cheung wrote: >> updated webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.03/ > > src/hotspot/share/classfile/systemDictionary.hpp > > // Returns default system loader > > Actually returns the system loader whether default or not. I'll update the comment. > Unclear if you might actually want two queries: one for default and > one for actual (which has the default as its parent) I can file another bug if we need the default one. > > // Returns default platform loader > > You don't need to say "default" as there is only ever one platform > loader. I'll update the comment. > > --- > > share/classfile/systemDictionary.cpp > > 186 return (class_loader == _java_system_loader); > > I think you need to restore this to checking both the field or whether > an instance of the default system loader class - as per previous email > - to handle the custom system loader case. I don't mind restoring the check but I've run many tests including most of the ones mentioned in your email and didn't see any failures. > > The explicit NULL checks will be needed if these can be called before > the platform/system loaders have been determined. I suspect they can > given the complex initialization process we have with modules. So far I haven't run into any issues yet. thanks, Calvin > > Thanks, > David From david.holmes at oracle.com Fri Oct 6 06:05:15 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 6 Oct 2017 16:05:15 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D7175A.3030309@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <59D6D560.4020804@oracle.com> <7f4cf10c-7a8d-34ff-2719-9082b0bdf1e9@oracle.com> <59D7175A.3030309@oracle.com> Message-ID: <751a4ea2-638a-86f3-c710-af0f3a9dadd8@oracle.com> On 6/10/2017 3:40 PM, Calvin Cheung wrote: > On 10/5/17, 6:43 PM, David Holmes wrote: >> On 6/10/2017 10:59 AM, Calvin Cheung wrote: >>> updated webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.03/ >> >> src/hotspot/share/classfile/systemDictionary.hpp >> >> // Returns default system loader >> >> Actually returns the system loader whether default or not. > I'll update the comment. >> Unclear if you might actually want two queries: one for default and >> one for actual (which has the default as its parent) > I can file another bug if we need the default one. >> >> // Returns default platform loader >> >> You don't need to say "default" as there is only ever one platform >> loader. > I'll update the comment. >> >> --- >> >> share/classfile/systemDictionary.cpp >> >> ?186?? return (class_loader == _java_system_loader); >> >> I think you need to restore this to checking both the field or whether >> an instance of the default system loader class - as per previous email >> - to handle the custom system loader case. > I don't mind restoring the check but I've run many tests including most > of the ones mentioned in your email and didn't see any failures. As Coleen indicated we don't really seem to test this area extensively. But again I'd have to look at all the code paths to see where we might have exposure to problems. >> >> The explicit NULL checks will be needed if these can be called before >> the platform/system loaders have been determined. I suspect they can >> given the complex initialization process we have with modules. > So far I haven't run into any issues yet. With the NULL checks removed? IF so, same answer as above. Thanks, David > thanks, > Calvin >> >> Thanks, >> David From calvin.cheung at oracle.com Fri Oct 6 06:48:14 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 05 Oct 2017 23:48:14 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <751a4ea2-638a-86f3-c710-af0f3a9dadd8@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <59D6D560.4020804@oracle.com> <7f4cf10c-7a8d-34ff-2719-9082b0bdf1e9@oracle.com> <59D7175A.3030309@oracle.com> <751a4ea2-638a-86f3-c710-af0f3a9dadd8@oracle.com> Message-ID: <59D7272E.7000702@oracle.com> On 10/5/17, 11:05 PM, David Holmes wrote: > On 6/10/2017 3:40 PM, Calvin Cheung wrote: >> On 10/5/17, 6:43 PM, David Holmes wrote: >>> On 6/10/2017 10:59 AM, Calvin Cheung wrote: >>>> updated webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.03/ >>> >>> src/hotspot/share/classfile/systemDictionary.hpp >>> >>> // Returns default system loader >>> >>> Actually returns the system loader whether default or not. >> I'll update the comment. >>> Unclear if you might actually want two queries: one for default and >>> one for actual (which has the default as its parent) >> I can file another bug if we need the default one. >>> >>> // Returns default platform loader >>> >>> You don't need to say "default" as there is only ever one platform >>> loader. >> I'll update the comment. >>> >>> --- >>> >>> share/classfile/systemDictionary.cpp >>> >>> 186 return (class_loader == _java_system_loader); >>> >>> I think you need to restore this to checking both the field or >>> whether an instance of the default system loader class - as per >>> previous email - to handle the custom system loader case. >> I don't mind restoring the check but I've run many tests including >> most of the ones mentioned in your email and didn't see any failures. > > As Coleen indicated we don't really seem to test this area > extensively. But again I'd have to look at all the code paths to see > where we might have exposure to problems. > >>> >>> The explicit NULL checks will be needed if these can be called >>> before the platform/system loaders have been determined. I suspect >>> they can given the complex initialization process we have with modules. >> So far I haven't run into any issues yet. > > With the NULL checks removed? IF so, same answer as above. That was with the NULL check. Without the NULL check, the following test failed: ./hotspot/jtreg/runtime/modules/ModuleStress/ModuleStress.java I'll restore SystemDictionary::is_system_class_loader() to its original form. I'll send updated webrev after more testing. thanks, Calvin > > Thanks, > David > >> thanks, >> Calvin >>> >>> Thanks, >>> David From daniel.daugherty at oracle.com Fri Oct 6 13:58:01 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 6 Oct 2017 07:58:01 -0600 Subject: RFR: 8185529: JCK api/java_lang/Object/WaitTests failed with jdk10/hs nightly In-Reply-To: References: Message-ID: <71c0d821-2cff-4c81-3b10-423daefcf25e@oracle.com> Thumbs up! Dan On 10/5/17 10:47 PM, David Holmes wrote: > Bug is not public - sorry. > > webrev: http://cr.openjdk.java.net/~dholmes/8185529/webrev/ > > Also patch inline below. > > Problem: o.wait(Long.MAX_VALUE) returns immediately > > Cause: arithmetic overflow gives a time in the past so the timeout > expires immediately > > This was introduced by the fix for: > > https://bugs.openjdk.java.net/browse/JDK-8174231 "Factor out and share > PlatformEvent and Parker code for POSIX systems." > > while we go to great lengths to ensure we avoid overflows inside the > time calculation functions, I overlooked the fact that we now first > convert a millisecond value into nanoseconds, which can overflow. > > Fix: limit the millis value to the existing hard limit of > MAX_SECS*MILLIUNITS ms. That can not overflow when converted to nanos. > > Thanks, > David > ----- > > --- old/src/hotspot/os/posix/os_posix.cpp??? 2017-10-06 > 00:34:31.595159338 -0400 > +++ new/src/hotspot/os/posix/os_posix.cpp??? 2017-10-06 > 00:34:29.443037883 -0400 > @@ -1770,6 +1770,12 @@ > > ?? if (v == 0) { // Do this the hard way by blocking ... > ???? struct timespec abst; > +??? // We have to watch for overflow when converting millis to nanos, > +??? // but if millis is that large then we will end up limiting to > +??? // MAX_SECS anyway, so just do that here. > +??? if (millis / MILLIUNITS > MAX_SECS) { > +????? millis = jlong(MAX_SECS) * MILLIUNITS; > +??? } > ???? to_abstime(&abst, millis * (NANOUNITS / MILLIUNITS), false); From calvin.cheung at oracle.com Fri Oct 6 16:03:24 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Fri, 06 Oct 2017 09:03:24 -0700 Subject: RFR: 8174986: CDS archived java heap region may not compatible with AOT In-Reply-To: <7a049844-4c67-3d52-d6d0-b0449efd58e8@oracle.com> References: <45A25AD9-D558-4F2C-8C64-F6D25BFDD8D2@oracle.com> <7a049844-4c67-3d52-d6d0-b0449efd58e8@oracle.com> Message-ID: <59D7A94C.8070509@oracle.com> Hi Jiangli, The changes look good. thanks, Calvin On 10/5/17, 9:30 AM, Ioi Lam wrote: > Hi Jiangli: > > The changes look good, and thank you for doing the clean up. I agree > that MetaspaceShared::initialize_runtime_shared_and_meta_spaces() is a > better location for this functionality that inside metaspace.cpp. > > Just a small typo: > > 306 // Set narrow_klass_shift to be LogKlassAlignmentInBytes. This > is consisten > > -> consistent > > Thanks > > - Ioi > > > On 10/4/17 4:12 PM, Jiangli Zhou wrote: >> Hi, >> >> Please review the following enhancement that always sets the narrow >> klass encoding shift to be LogKlassAlignmentInBytes when CDS is >> enabled. This is done so the archived java heap regions (the shared >> string space and adjustable archive heap space) can coexist with AOT >> code, because AOT uses LogKlassAlignmentInBytes for narrow klass >> encode shift when generating the precompiled code. >> >> I also took the opportunity to clean up >> Meatspace::global_initialize(). I moved the block of code that >> initializes the shared spaces and meatspace at CDS runtime into a new >> function, >> MetaspaceShared::initialize_runtime_shared_and_meta_spaces(). I also >> added more comments so it is easier to understand the logic. >> >> webrev: http://cr.openjdk.java.net/~jiangli/8174986/webrev.00/ >> >> RFE: https://bugs.openjdk.java.net/browse/JDK-8174986?filter=14921 >> >> >> Tested with all CDS/AppCDS tests on linux x64. >> >> Thanks, >> Jiangli > From mandy.chung at oracle.com Fri Oct 6 17:18:54 2017 From: mandy.chung at oracle.com (mandy chung) Date: Fri, 6 Oct 2017 10:18:54 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59D6D560.4020804@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <59D6D560.4020804@oracle.com> Message-ID: On 10/5/17 5:59 PM, Calvin Cheung wrote: > I've modified the code to using SystemDictionary::ClassLoader_klass() > but I don't need the local variable because I'm creating the > _java_system_loader and _java_platform_loader separately in 2 > different functions. > > It is because the _java_platform_loader is required for module system > initialization in call_initPhase2. > It would be too early to create the _java_system_loader before > call_initPhase2 because in call_initPhase3, it will check if there is > a custom system loader defined via the -Djava.system.loader property. Where in the VM that accesses _java_platform_loader before initPhase? The module system initialization implementation is in the library. I don't see how the VM will need to access it before call_initPhase2. Mandy From calvin.cheung at oracle.com Fri Oct 6 17:33:35 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Fri, 06 Oct 2017 10:33:35 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <59D6D560.4020804@oracle.com> Message-ID: <59D7BE6F.9050000@oracle.com> On 10/6/17, 10:18 AM, mandy chung wrote: > > > On 10/5/17 5:59 PM, Calvin Cheung wrote: >> I've modified the code to using SystemDictionary::ClassLoader_klass() >> but I don't need the local variable because I'm creating the >> _java_system_loader and _java_platform_loader separately in 2 >> different functions. >> >> It is because the _java_platform_loader is required for module system >> initialization in call_initPhase2. >> It would be too early to create the _java_system_loader before >> call_initPhase2 because in call_initPhase3, it will check if there is >> a custom system loader defined via the -Djava.system.loader property. > Where in the VM that accesses _java_platform_loader before initPhase? > > The module system initialization implementation is in the library. I > don't see how the VM will need to access it before call_initPhase2. During initPhase2, the java library calls the JVM_DefineModule() API which in turn calls the Modules::define_module() where it requires the _java_platform_loader: if (!h_loader.is_null() && !*SystemDictionary::is_platform_class_loader(h_loader())* && (strncmp(package_name, JAVAPKG, JAVAPKG_LEN) == 0 && (package_name[JAVAPKG_LEN] == '/' || package_name[JAVAPKG_LEN] == '\0'))) { thanks, Calvin From jiangli.zhou at Oracle.COM Fri Oct 6 18:10:32 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Fri, 6 Oct 2017 11:10:32 -0700 Subject: RFR (S) 8178870: instrumentation.retransformClasses cause coredump In-Reply-To: <78413dab-10a7-a29b-bb71-7b96423b3499@oracle.com> References: <286a1baa-1b89-6cbb-fea2-ae3a8fe4a328@oracle.com> <2B16AE11-E336-42AC-B643-7C5A90AA2069@oracle.com> <78413dab-10a7-a29b-bb71-7b96423b3499@oracle.com> Message-ID: Hi Coleen, The update looks correct. I also ran your test locally on linux-64 and verified it passed with your fix. Thanks, Jiangli > On Oct 5, 2017, at 7:56 PM, coleen.phillimore at oracle.com wrote: > > Thank you for the code review! > > I had to fix the .c file slightly for Solaris because in solaris C you still can't declare variables before statements. It's a minor change. I also added -lc to the makefile for solaris because the other .c files had it. > > http://cr.openjdk.java.net/~coleenp/8178870.02/webrev/index.html > > Thanks, > Coleen > > On 10/5/17 9:39 PM, Jiangli Zhou wrote: >> Hi Coleen, >> >> Not an expert in JVMTI area. The bug analysis, fix and test look good to me! >> >> Thanks, >> Jiangli >> >>> On Oct 5, 2017, at 11:15 AM, coleen.phillimore at oracle.com wrote: >>> >>> Summary: Don't double-free cached class bytes on redefinition loading failure. >>> >>> See bug for more description. Tested with test/jdk/java/lang/instrument, test/hotspot/jtreg/runtime/RedefineTests, internal CDS tests, internal nsk.jvmti tests, and added test. >>> >>> open webrev at http://cr.openjdk.java.net/~coleenp/8178870.01/webrev >>> bug link https://bugs.openjdk.java.net/browse/JDK-8178870 >>> >>> Thanks, >>> Coleen > From coleen.phillimore at oracle.com Fri Oct 6 18:17:11 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 6 Oct 2017 14:17:11 -0400 Subject: RFR (S) 8178870: instrumentation.retransformClasses cause coredump In-Reply-To: References: <286a1baa-1b89-6cbb-fea2-ae3a8fe4a328@oracle.com> <2B16AE11-E336-42AC-B643-7C5A90AA2069@oracle.com> <78413dab-10a7-a29b-bb71-7b96423b3499@oracle.com> Message-ID: Thank you, Jiangli for the last look. Coleen On 10/6/17 2:10 PM, Jiangli Zhou wrote: > Hi Coleen, > > The update looks correct. I also ran your test locally on linux-64 and verified it passed with your fix. > > Thanks, > Jiangli > >> On Oct 5, 2017, at 7:56 PM, coleen.phillimore at oracle.com wrote: >> >> Thank you for the code review! >> >> I had to fix the .c file slightly for Solaris because in solaris C you still can't declare variables before statements. It's a minor change. I also added -lc to the makefile for solaris because the other .c files had it. >> >> http://cr.openjdk.java.net/~coleenp/8178870.02/webrev/index.html >> >> Thanks, >> Coleen >> >> On 10/5/17 9:39 PM, Jiangli Zhou wrote: >>> Hi Coleen, >>> >>> Not an expert in JVMTI area. The bug analysis, fix and test look good to me! >>> >>> Thanks, >>> Jiangli >>> >>>> On Oct 5, 2017, at 11:15 AM, coleen.phillimore at oracle.com wrote: >>>> >>>> Summary: Don't double-free cached class bytes on redefinition loading failure. >>>> >>>> See bug for more description. Tested with test/jdk/java/lang/instrument, test/hotspot/jtreg/runtime/RedefineTests, internal CDS tests, internal nsk.jvmti tests, and added test. >>>> >>>> open webrev at http://cr.openjdk.java.net/~coleenp/8178870.01/webrev >>>> bug link https://bugs.openjdk.java.net/browse/JDK-8178870 >>>> >>>> Thanks, >>>> Coleen From mandy.chung at oracle.com Fri Oct 6 19:37:55 2017 From: mandy.chung at oracle.com (mandy chung) Date: Fri, 6 Oct 2017 12:37:55 -0700 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> Message-ID: <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> Hi David, Thanks for raising the lazy initialization issue.? The recent version of ClassLoader.java in webrev.03 is uploaded in place: http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.03/ The native libraries map is now created lazily with synchronization.??? I keep the lazy initialization that will save to create a CHM as many custom class loaders don't have native code.? I think it's a good saving.?? In addition, if we iniitialize the static systemNativeLibraries at time, it may want to avoid using CHM as it changes the class initialization order. thanks Mandy From serguei.spitsyn at oracle.com Fri Oct 6 20:07:20 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 6 Oct 2017 13:07:20 -0700 Subject: RFR (S) 8178870: instrumentation.retransformClasses cause coredump In-Reply-To: <78413dab-10a7-a29b-bb71-7b96423b3499@oracle.com> References: <286a1baa-1b89-6cbb-fea2-ae3a8fe4a328@oracle.com> <2B16AE11-E336-42AC-B643-7C5A90AA2069@oracle.com> <78413dab-10a7-a29b-bb71-7b96423b3499@oracle.com> Message-ID: <2d79aaf8-ffda-0ac3-dfa4-f1582efa0451@oracle.com> Hi Coleen, Looks good. Thanks, Serguei On 10/5/17 19:56, coleen.phillimore at oracle.com wrote: > Thank you for the code review! > > I had to fix the .c file slightly for Solaris because in solaris C you > still can't declare variables before statements.? It's a minor > change.? I also added -lc to the makefile for solaris because the > other .c files had it. > > http://cr.openjdk.java.net/~coleenp/8178870.02/webrev/index.html > > Thanks, > Coleen > > On 10/5/17 9:39 PM, Jiangli Zhou wrote: >> Hi Coleen, >> >> Not an expert in JVMTI area. The bug analysis, fix and test look good >> to me! >> >> Thanks, >> Jiangli >> >>> On Oct 5, 2017, at 11:15 AM, coleen.phillimore at oracle.com wrote: >>> >>> Summary: Don't double-free cached class bytes on redefinition >>> loading failure. >>> >>> See bug for more description.? Tested with >>> test/jdk/java/lang/instrument, >>> test/hotspot/jtreg/runtime/RedefineTests, internal CDS tests, >>> internal nsk.jvmti tests, and added test. >>> >>> open webrev at http://cr.openjdk.java.net/~coleenp/8178870.01/webrev >>> bug link https://bugs.openjdk.java.net/browse/JDK-8178870 >>> >>> Thanks, >>> Coleen > From david.holmes at oracle.com Fri Oct 6 23:28:41 2017 From: david.holmes at oracle.com (David Holmes) Date: Sat, 7 Oct 2017 09:28:41 +1000 Subject: RFR: 8185529: JCK api/java_lang/Object/WaitTests failed with jdk10/hs nightly In-Reply-To: <71c0d821-2cff-4c81-3b10-423daefcf25e@oracle.com> References: <71c0d821-2cff-4c81-3b10-423daefcf25e@oracle.com> Message-ID: <70fbc83d-61dd-779e-d580-eb432a96f767@oracle.com> Thanks Dan! David On 6/10/2017 11:58 PM, Daniel D. Daugherty wrote: > Thumbs up! > > Dan > > > On 10/5/17 10:47 PM, David Holmes wrote: >> Bug is not public - sorry. >> >> webrev: http://cr.openjdk.java.net/~dholmes/8185529/webrev/ >> >> Also patch inline below. >> >> Problem: o.wait(Long.MAX_VALUE) returns immediately >> >> Cause: arithmetic overflow gives a time in the past so the timeout >> expires immediately >> >> This was introduced by the fix for: >> >> https://bugs.openjdk.java.net/browse/JDK-8174231 "Factor out and share >> PlatformEvent and Parker code for POSIX systems." >> >> while we go to great lengths to ensure we avoid overflows inside the >> time calculation functions, I overlooked the fact that we now first >> convert a millisecond value into nanoseconds, which can overflow. >> >> Fix: limit the millis value to the existing hard limit of >> MAX_SECS*MILLIUNITS ms. That can not overflow when converted to nanos. >> >> Thanks, >> David >> ----- >> >> --- old/src/hotspot/os/posix/os_posix.cpp??? 2017-10-06 >> 00:34:31.595159338 -0400 >> +++ new/src/hotspot/os/posix/os_posix.cpp??? 2017-10-06 >> 00:34:29.443037883 -0400 >> @@ -1770,6 +1770,12 @@ >> >> ?? if (v == 0) { // Do this the hard way by blocking ... >> ???? struct timespec abst; >> +??? // We have to watch for overflow when converting millis to nanos, >> +??? // but if millis is that large then we will end up limiting to >> +??? // MAX_SECS anyway, so just do that here. >> +??? if (millis / MILLIUNITS > MAX_SECS) { >> +????? millis = jlong(MAX_SECS) * MILLIUNITS; >> +??? } >> ???? to_abstime(&abst, millis * (NANOUNITS / MILLIUNITS), false); > From david.holmes at oracle.com Mon Oct 9 01:14:40 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 9 Oct 2017 11:14:40 +1000 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> Message-ID: Hi Mandy, This all looks fine to me now. Thanks, David On 7/10/2017 5:37 AM, mandy chung wrote: > Hi David, > > Thanks for raising the lazy initialization issue.? The recent version of > ClassLoader.java in webrev.03 is uploaded in place: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.03/ > > The native libraries map is now created lazily with synchronization. I > keep the lazy initialization that will save to create a CHM as many > custom class loaders don't have native code.? I think it's a good > saving.?? In addition, if we iniitialize the static > systemNativeLibraries at time, it may want to avoid using CHM > as it changes the class initialization order. > > thanks > Mandy From david.holmes at oracle.com Mon Oct 9 01:35:33 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 9 Oct 2017 11:35:33 +1000 Subject: RFR: 8185529: JCK api/java_lang/Object/WaitTests failed with jdk10/hs nightly In-Reply-To: <70fbc83d-61dd-779e-d580-eb432a96f767@oracle.com> References: <71c0d821-2cff-4c81-3b10-423daefcf25e@oracle.com> <70fbc83d-61dd-779e-d580-eb432a96f767@oracle.com> Message-ID: Can I get a second review please. Thanks, David On 7/10/2017 9:28 AM, David Holmes wrote: > Thanks Dan! > > David > > On 6/10/2017 11:58 PM, Daniel D. Daugherty wrote: >> Thumbs up! >> >> Dan >> >> >> On 10/5/17 10:47 PM, David Holmes wrote: >>> Bug is not public - sorry. >>> >>> webrev: http://cr.openjdk.java.net/~dholmes/8185529/webrev/ >>> >>> Also patch inline below. >>> >>> Problem: o.wait(Long.MAX_VALUE) returns immediately >>> >>> Cause: arithmetic overflow gives a time in the past so the timeout >>> expires immediately >>> >>> This was introduced by the fix for: >>> >>> https://bugs.openjdk.java.net/browse/JDK-8174231 "Factor out and >>> share PlatformEvent and Parker code for POSIX systems." >>> >>> while we go to great lengths to ensure we avoid overflows inside the >>> time calculation functions, I overlooked the fact that we now first >>> convert a millisecond value into nanoseconds, which can overflow. >>> >>> Fix: limit the millis value to the existing hard limit of >>> MAX_SECS*MILLIUNITS ms. That can not overflow when converted to nanos. >>> >>> Thanks, >>> David >>> ----- >>> >>> --- old/src/hotspot/os/posix/os_posix.cpp??? 2017-10-06 >>> 00:34:31.595159338 -0400 >>> +++ new/src/hotspot/os/posix/os_posix.cpp??? 2017-10-06 >>> 00:34:29.443037883 -0400 >>> @@ -1770,6 +1770,12 @@ >>> >>> ?? if (v == 0) { // Do this the hard way by blocking ... >>> ???? struct timespec abst; >>> +??? // We have to watch for overflow when converting millis to nanos, >>> +??? // but if millis is that large then we will end up limiting to >>> +??? // MAX_SECS anyway, so just do that here. >>> +??? if (millis / MILLIUNITS > MAX_SECS) { >>> +????? millis = jlong(MAX_SECS) * MILLIUNITS; >>> +??? } >>> ???? to_abstime(&abst, millis * (NANOUNITS / MILLIUNITS), false); >> From calvin.cheung at oracle.com Mon Oct 9 03:26:26 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Sun, 08 Oct 2017 20:26:26 -0700 Subject: RFR: 8185529: JCK api/java_lang/Object/WaitTests failed with jdk10/hs nightly In-Reply-To: References: <71c0d821-2cff-4c81-3b10-423daefcf25e@oracle.com> <70fbc83d-61dd-779e-d580-eb432a96f767@oracle.com> Message-ID: <59DAEC62.1060200@oracle.com> Hi David, I've tried your fix and the test passed with it. 'r'eviewed. thanks, Calvin On 10/8/17, 6:35 PM, David Holmes wrote: > Can I get a second review please. > > Thanks, > David > > On 7/10/2017 9:28 AM, David Holmes wrote: >> Thanks Dan! >> >> David >> >> On 6/10/2017 11:58 PM, Daniel D. Daugherty wrote: >>> Thumbs up! >>> >>> Dan >>> >>> >>> On 10/5/17 10:47 PM, David Holmes wrote: >>>> Bug is not public - sorry. >>>> >>>> webrev: http://cr.openjdk.java.net/~dholmes/8185529/webrev/ >>>> >>>> Also patch inline below. >>>> >>>> Problem: o.wait(Long.MAX_VALUE) returns immediately >>>> >>>> Cause: arithmetic overflow gives a time in the past so the timeout >>>> expires immediately >>>> >>>> This was introduced by the fix for: >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8174231 "Factor out and >>>> share PlatformEvent and Parker code for POSIX systems." >>>> >>>> while we go to great lengths to ensure we avoid overflows inside >>>> the time calculation functions, I overlooked the fact that we now >>>> first convert a millisecond value into nanoseconds, which can >>>> overflow. >>>> >>>> Fix: limit the millis value to the existing hard limit of >>>> MAX_SECS*MILLIUNITS ms. That can not overflow when converted to nanos. >>>> >>>> Thanks, >>>> David >>>> ----- >>>> >>>> --- old/src/hotspot/os/posix/os_posix.cpp 2017-10-06 >>>> 00:34:31.595159338 -0400 >>>> +++ new/src/hotspot/os/posix/os_posix.cpp 2017-10-06 >>>> 00:34:29.443037883 -0400 >>>> @@ -1770,6 +1770,12 @@ >>>> >>>> if (v == 0) { // Do this the hard way by blocking ... >>>> struct timespec abst; >>>> + // We have to watch for overflow when converting millis to nanos, >>>> + // but if millis is that large then we will end up limiting to >>>> + // MAX_SECS anyway, so just do that here. >>>> + if (millis / MILLIUNITS > MAX_SECS) { >>>> + millis = jlong(MAX_SECS) * MILLIUNITS; >>>> + } >>>> to_abstime(&abst, millis * (NANOUNITS / MILLIUNITS), false); >>> From david.holmes at oracle.com Mon Oct 9 05:00:50 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 9 Oct 2017 15:00:50 +1000 Subject: RFR: 8185529: JCK api/java_lang/Object/WaitTests failed with jdk10/hs nightly In-Reply-To: <59DAEC62.1060200@oracle.com> References: <71c0d821-2cff-4c81-3b10-423daefcf25e@oracle.com> <70fbc83d-61dd-779e-d580-eb432a96f767@oracle.com> <59DAEC62.1060200@oracle.com> Message-ID: <6cbde01c-7194-5cf8-c550-eaacf00e4bf0@oracle.com> Thanks Calvin! David On 9/10/2017 1:26 PM, Calvin Cheung wrote: > Hi David, > > I've tried your fix and the test passed with it. > 'r'eviewed. > > thanks, > Calvin > > On 10/8/17, 6:35 PM, David Holmes wrote: >> Can I get a second review please. >> >> Thanks, >> David >> >> On 7/10/2017 9:28 AM, David Holmes wrote: >>> Thanks Dan! >>> >>> David >>> >>> On 6/10/2017 11:58 PM, Daniel D. Daugherty wrote: >>>> Thumbs up! >>>> >>>> Dan >>>> >>>> >>>> On 10/5/17 10:47 PM, David Holmes wrote: >>>>> Bug is not public - sorry. >>>>> >>>>> webrev: http://cr.openjdk.java.net/~dholmes/8185529/webrev/ >>>>> >>>>> Also patch inline below. >>>>> >>>>> Problem: o.wait(Long.MAX_VALUE) returns immediately >>>>> >>>>> Cause: arithmetic overflow gives a time in the past so the timeout >>>>> expires immediately >>>>> >>>>> This was introduced by the fix for: >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-8174231 "Factor out and >>>>> share PlatformEvent and Parker code for POSIX systems." >>>>> >>>>> while we go to great lengths to ensure we avoid overflows inside >>>>> the time calculation functions, I overlooked the fact that we now >>>>> first convert a millisecond value into nanoseconds, which can >>>>> overflow. >>>>> >>>>> Fix: limit the millis value to the existing hard limit of >>>>> MAX_SECS*MILLIUNITS ms. That can not overflow when converted to nanos. >>>>> >>>>> Thanks, >>>>> David >>>>> ----- >>>>> >>>>> --- old/src/hotspot/os/posix/os_posix.cpp??? 2017-10-06 >>>>> 00:34:31.595159338 -0400 >>>>> +++ new/src/hotspot/os/posix/os_posix.cpp??? 2017-10-06 >>>>> 00:34:29.443037883 -0400 >>>>> @@ -1770,6 +1770,12 @@ >>>>> >>>>> ?? if (v == 0) { // Do this the hard way by blocking ... >>>>> ???? struct timespec abst; >>>>> +??? // We have to watch for overflow when converting millis to nanos, >>>>> +??? // but if millis is that large then we will end up limiting to >>>>> +??? // MAX_SECS anyway, so just do that here. >>>>> +??? if (millis / MILLIUNITS > MAX_SECS) { >>>>> +????? millis = jlong(MAX_SECS) * MILLIUNITS; >>>>> +??? } >>>>> ???? to_abstime(&abst, millis * (NANOUNITS / MILLIUNITS), false); >>>> From Alan.Bateman at oracle.com Mon Oct 9 08:20:46 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 9 Oct 2017 09:20:46 +0100 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> Message-ID: On 06/10/2017 20:37, mandy chung wrote: > : > > The native libraries map is now created lazily with > synchronization.??? I keep the lazy initialization that will save to > create a CHM as many custom class loaders don't have native code.? I > think it's a good saving.?? In addition, if we iniitialize the static > systemNativeLibraries at time, it may want to avoid using CHM > as it changes the class initialization order. > Alternatively change nativeLibraries and systemNativeLibraries to volatile so the synchronization is only needed to initialize them. Otherwise this version (webrev.03) looks good to me. -Alan From peter.levart at gmail.com Mon Oct 9 10:47:31 2017 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 9 Oct 2017 12:47:31 +0200 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> Message-ID: <8f1ec049-52ad-9fec-ca86-57c32e76514e@gmail.com> On 10/09/2017 10:20 AM, Alan Bateman wrote: > On 06/10/2017 20:37, mandy chung wrote: >> : >> >> The native libraries map is now created lazily with >> synchronization.??? I keep the lazy initialization that will save to >> create a CHM as many custom class loaders don't have native code.? I >> think it's a good saving.?? In addition, if we iniitialize the static >> systemNativeLibraries at time, it may want to avoid using >> CHM as it changes the class initialization order. >> > Alternatively change nativeLibraries and systemNativeLibraries to > volatile so the synchronization is only needed to initialize them. > Otherwise this version (webrev.03) looks good to me. > > -Alan Yes Mandy, you could use volatile fields + double checked locking for initialization. In addition, the initializers to 'null' value are not needed / are a waste of instructions (the default is guaranteed by JLS): 2695???? // Native libraries belonging to system classes. 2696???? private static Map systemNativeLibraries = null; 2697 2698???? // Native libraries associated with the class loader. 2699???? private Map nativeLibraries = null; Regards, Peter From coleen.phillimore at oracle.com Mon Oct 9 13:15:23 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 9 Oct 2017 09:15:23 -0400 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: <19df096b-3243-1ac0-3d3a-e955c63c534d@oracle.com> <53fb4559-aeed-da1a-67fe-6cb50fd8e9ce@gmail.com> Message-ID: Hi, this looks reasonable but I would prefer that the code in arguments.cpp call something in metaspace.cpp, like check_metaspace_sizes() so that you don't have to tell arguments what the MediumChunk size is. I will sponsor this for you. thanks, Coleen On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: > Hi all, > > I added a testcase for this issue in new webrev: > > http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ > > > Thanks, > > Yasumasa > > > > 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >> Hi all, >> >> I uploaded webrev for this issue against jdk10/hs. >> Could you review it? >> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >> >> >> I cannot access JPRT. So I need a sponsor. >> >> >> Thanks, >> >> Yasumasa >> >> >> >> 2017-09-21 3:11 GMT+09:00 Man Cao : >>> Thank Yasumasa and Stefan for the responses. >>> >>> Good to know that the patch is not blocked due to breaking some internal >>> invariants/assumptions, but just due to its P5 status. >>> Is it possible to push it to P4? >>> >>> -Man >>> >>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>> wrote: >>>> Hi, >>>> >>>> (CC'ed hotspot-runtime-dev) >>>> >>>>> I think the reason is that this bug is a P5. The compressed class space >>>>> belongs to the runtime code, so you might get more traction for this on the >>>>> hotspot-runtime-dev list. >>>> >>>> I will send review request against jdk10/master or jdk10/hs after repos >>>> are opened. >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> >>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>> Hi Man, >>>>> >>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>> Hi Yasumasa, Stefan, >>>>>> >>>>>> Do you have any thoughts on why this patch has been pending for 2+ >>>>>> years? This patch could really save us from some annoying issues since we >>>>>> are automatically monitoring hsperfdata counters. >>>>> >>>>> I think the reason is that this bug is a P5. The compressed class space >>>>> belongs to the runtime code, so you might get more traction for this on the >>>>> hotspot-runtime-dev list. >>>>> >>>>> StefanK >>>>> >>>>>> -Man >>>>>> >>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>> > wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I wonder if there is any recent update on the patch for JDK-8087291. >>>>>> Is it possible to push this patch into JDK9? Except for its low >>>>>> priority (P5), >>>>>> is there any complication that prevents this patch getting approved >>>>>> (for example, some JVM logic requires CompressedClassSpaceSize to be >>>>>> 1GB by default)? >>>>>> >>>>>> I work in the Java Platform Team at Google. We have encountered >>>>>> annoying issues that the hsperfdata counter >>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>> a too large value (about 1GB) even if user sets >>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the confusing 1GB >>>>>> memory reserved by metaspace, >>>>>> regardless of MaxMetaspaceSize value. The root cause for these >>>>>> issues is that CompressedClassSpaceSize is not automatically capped >>>>>> by MaxMetaspaceSize >>>>>> during VM initialization, and this patch seems fix the root cause. >>>>>> (I'm aware that even after this patch, the reserved size could still >>>>>> be up to 2*MaxMetaspaceSize, >>>>>> but it is better than the current situation.) >>>>>> >>>>>> Thanks, >>>>>> Man >>>>>> >>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>> >>>>>> Thank you for your comment! >>>>>> > Try running a debug JVM with your patch with this command >>>>>> line. >>>>>> > >>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>> >>>>>> It works on fastdebug VM. >>>>>> Please review again. >>>>>> >>>>>> Thanks, >>>>>> Yasumasa >>>>>> >>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>> > Yasumasa, >>>>>> > >>>>>> > Try running a debug JVM with your patch with this command >>>>>> line. >>>>>> > >>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>> > >>>>>> > On a linux system I get this when I build with your patch. >>>>>> > >>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>> >> # To suppress the following error report, specify this >>>>>> argument >>>>>> >> # after -XX: or in .hotspotrc: >>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>> >> # >>>>>> >> # A fatal error has been detected by the Java Runtime >>>>>> Environment: >>>>>> >> # >>>>>> >> # Internal Error >>>>>> >> >>>>>> >>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>> >> # assert(size > MediumChunk || size > ClassMediumChunk) >>>>>> failed: Not a >>>>>> >> humongous chunk >>>>>> >> # >>>>>> > >>>>>> > >>>>>> > Jon >>>>>> > >>>>>> > >>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>> >> I want to continue to discuss about CompressedClassSpace and >>>>>> MaxMetaspace in this (RFR) thread. >>>>>> >> >>>>>> >> >>>>>> >>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>> >>>>>> >>>>>> >>>> Should I resize CompressedClassSpaceSize than to show >>>>>> error message? >>>>>> >>> If you add slightly better heuristics for the setup of the >>>>>> CompressedClassSpaceSize flag, for example lowering the >>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is set, then it >>>>>> might be less likely that you'll hit the OutOfMemoryError when >>>>>> the system is set up with strict overcommit settings. >>>>>> >> >>>>>> >> I've uploaded new webrev: >>>>>> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>> >>>>>> >> >>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>> CompressedClassSpaceSize, and >>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>> >> >>>>>> >> I add to check CompressedClassSpaceSize in >>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>> >> If InitialBootClassLoaderMetaspaceSize is greater than >>>>>> MaxMetaspaceSize, >>>>>> >> VM will fail with error message. >>>>>> >> >>>>>> >> InitialBootClassLoaderMetaspaceSize will be set to >>>>>> MaxMetaspaceSize >>>>>> >> when UseCompressedClassPointers is not set in >>>>>> Metaspace::ergo_initialize(). >>>>>> >> >>>>>> >> >>>>>> >> Thanks, >>>>>> >> >>>>>> >> Yasumasa >>>>>> >> >>>>>> >>>>>> From mandy.chung at oracle.com Mon Oct 9 16:39:04 2017 From: mandy.chung at oracle.com (mandy chung) Date: Mon, 9 Oct 2017 09:39:04 -0700 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <8f1ec049-52ad-9fec-ca86-57c32e76514e@gmail.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> <8f1ec049-52ad-9fec-ca86-57c32e76514e@gmail.com> Message-ID: <6bf283ec-186f-6bb4-a5df-f4ba946f4ea2@oracle.com> On 10/9/17 3:47 AM, Peter Levart wrote: > > > On 10/09/2017 10:20 AM, Alan Bateman wrote: >> On 06/10/2017 20:37, mandy chung wrote: >>> : >>> >>> The native libraries map is now created lazily with >>> synchronization.??? I keep the lazy initialization that will save to >>> create a CHM as many custom class loaders don't have native code.? I >>> think it's a good saving.?? In addition, if we iniitialize the >>> static systemNativeLibraries at time, it may want to avoid >>> using CHM as it changes the class initialization order. >>> >> Alternatively change nativeLibraries and systemNativeLibraries to >> volatile so the synchronization is only needed to initialize them. >> Otherwise this version (webrev.03) looks good to me. >> >> -Alan > > Yes Mandy, you could use volatile fields + double checked locking for > initialization. Since this might affect JNI binding, I have avoided changing it to volatile in this patch until we get some performance number (I might be overly conversative).? I prefer to follow up this together in the lock cleanup RFE that David suggests. > In addition, the initializers to 'null' value are not needed / are a > waste of instructions (the default is guaranteed by JLS): > > 2695???? // Native libraries belonging to system classes. > 2696???? private static Map > systemNativeLibraries = null; > 2697 > 2698???? // Native libraries associated with the class loader. > 2699???? private Map nativeLibraries = null; sure. Mandy From leonid.mesnik at oracle.com Mon Oct 9 17:41:19 2017 From: leonid.mesnik at oracle.com (Leonid Mesnik) Date: Mon, 9 Oct 2017 10:41:19 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> <5F386D9A-CA05-4732-8F68-F493DD2E8E99@oracle.com> <59CD73AC.3080207@oracle.com> <661D3614-44C0-45C5-AE04-5AE1AF42C739@oracle.com> Message-ID: Hi Thank you for fixes and explanations. Looks good for me now. Leonid > On Oct 4, 2017, at 5:52 PM, mikhailo wrote: > > Hi Leonid, > > Thank you for review. Please see my comments inline: > > On 10/04/2017 11:19 AM, Leonid Mesnik wrote: >> Hi >> >> Overall looks good. There few comments: >> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/jtreg-ext/requires/VMProps.java.sdiff.html >> >> 307 System.err.println("dockerSupprt() threw exception: " + e); >> Missed ?o? in Support. >> > Fixed >> 316 p.waitFor(); >> I think that it is needed to guard execution with timeout. I am not sure that jtreg is going to run properties with timeout. > Fixed. I added 10 seconds for timeout, which should be sufficient. > IMO, if it takes more than 10 sec to invoke this basic command then something is wrong, and the docker tests should not be run. >> >> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >> 28 * @requires (sun.arch.data.model != "32") & (os.family == "linux") >> Is it possible to check this inside docker.support and use only ?docker.support? property? > Added. >> >> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/lib/jdk/test/lib/containers/docker/DockerRunOptions.java.html >> 44 public DockerRunOptions(){} >> Why this empty constructor is needed? > Fixed - removed. >> Does it make also to make this class immutable i.e. init all properties in constructors and have them read-only. > The idea of this class is that fields can be modified by the user. The default constructor fills out values for most common use case; when user decided to modify these options to specify different test run conditions they should be able to modify it. > > I could surround values by getters or setters, or make every new set return a new copy of the object (an immutable pattern), but I did not wish to complicate the code. The uses of this class a fairly simple, and are unlikely to result in data races. The test create runOptions object, specifies/modifies fields if necessary, then passes to DockerTestUtils for running docker container with these options. > > I hope you do not mind if I leave it as is. > > > Thank you for review, > Misha >> >> Leonid >> >>> On Sep 28, 2017, at 3:11 PM, Mikhailo Seledtsov > wrote: >>> >>> Here is the updated webrev: >>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/ >>> >>> >>> Leonid, George - thank you for your comments. >>> In addition to addressing your feedback, I also did: >>> - implemented @requires docker.support >>> - added dockerRunJava() method and associated data structure DockerRunOptions, for running Java tests inside >>> docker environment, and to account for java opts, test java opts, docker opts, classes and class params >>> - added a simple HelloWorld test case that runs HelloWorld inside a container >>> - ran jtreg with extra Java options, make sure they are added correctly to the docker run command >>> - added docker image cleanup after testing is done >>> >>> >>> Please review. >>> >>> Misha >>> >>> >>> On 9/27/17, 11:00 AM, mikhailo wrote: >>>> Leonid, >>>> >>>> Thank you for review and constructive feedback. See my comment in line. >>>> >>>> >>>> On 09/26/2017 11:19 AM, Leonid Mesnik wrote: >>>>> Misha >>>>> >>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html > >>>>> Copyright is incorrect, need to updated it for GPL. >>>> Fixed >>>>> >>>>> The Hotspot is Oracle VM name only so test might fail for OpenJDK. I think you need to fix this check. >>>> I see. I fixed this by using Platform.vmName which should be correct in all cases. I double-checked with OpenJDK also. >>>>> >>>>> The requires checks only that test is executed only on the 64-bit linux. Does it make a sense to introduce more docker-specific check? >>>> I agree this is a better way. I will do some prototyping; if such check is feasible and efficient in at requires then I will add it. >>>>> >>>>> >>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/Dockerfile-BasicTest.html > >>>>> Could you please explain why oraclelinux 7.0 is used as a base image for test. >>>> I have upgraded to Oracle Linux 7.2. If we have specific requirement I will change it to that. If we have requirements in the future to support multiple OS, I can add Dockerfile generation. >>>> For this basic sanity tests I think this should suffice. >>>>> >>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html > >>>>> The content looks fine. >>>>> >>>>> I don?t see anything to clean up docker images on the system. Could you please explain how tests are going to cleanup images. >>>> To clean up containers I will add "--rm" to the 'docker run' command. This should ensure that container data is removed after container stops. >>>> As for the image - I use the same image name. The image will stay in the local registry unless manually removed. I should probably do 'docker rmi' at the end of the test to clean this up. >>>> >>>> >>>> Once I implement these changes I will send the updated webrev. >>>> >>>> Thank you, >>>> Misha >>>>> >>>>> Leonid >>>>> >>>>> >>>>>> On Sep 21, 2017, at 5:58 PM, mikhailo >> wrote: >>>>>> >>>>>> Please review this initial drop of Docker test utils and a sanity test. This change lays ground >>>>>> for further test development and test utils improvement in this area. >>>>>> >>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >>>>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ > >>>>>> Testing: >>>>>> - run this test on machine with Docker enabled - works >>>>>> - run this test on Linux-x64 with no Docker engine or Docker disabled - test skipped (as expected) >>>>>> - run this test on automated system - in progress >>>>>> >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>> >>>>> >>>> >> > From daniel.daugherty at oracle.com Mon Oct 9 19:41:24 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 9 Oct 2017 13:41:24 -0600 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle Message-ID: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> Greetings, We have a (eXtra Large) fix for the following bug: 8167108 inconsistent handling of SR_lock can lead to crashes https://bugs.openjdk.java.net/browse/JDK-8167108 This fix adds a Safe Memory Reclamation (SMR) mechanism based on Hazard Pointers to manage JavaThread lifecycle. Here's a PDF for the internal wiki that we've been using to describe and track the work on this project: http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf Dan has noticed that the indenting is wrong in some of the code quotes in the PDF that are not present in the internal wiki. We don't have a solution for that problem yet. Here's the webrev for current JDK10 version of this fix: http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full This fix has been run through many rounds of JPRT and Mach5 tier[2-5] testing, additional stress testing on Dan's Solaris X64 server, and additional testing on Erik and Robbin's machines. We welcome comments, suggestions and feedback. Daniel Daugherty Erik Osterlund Robbin Ehn From igor.ignatyev at oracle.com Mon Oct 9 19:57:12 2017 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Mon, 9 Oct 2017 12:57:12 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: <92e4360f-b278-0805-9c98-42e1cd98ee6d@oracle.com> References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> <5F386D9A-CA05-4732-8F68-F493DD2E8E99@oracle.com> <59CD73AC.3080207@oracle.com> <661D3614-44C0-45C5-AE04-5AE1AF42C739@oracle.com> <92e4360f-b278-0805-9c98-42e1cd98ee6d@oracle.com> Message-ID: Hi Misha, since 'docker.support' is true only if we are on linux-x64, you can remove L#28 (@requires (sun.arch.data.model != "32") & (os.family == "linux")) from DockerBasicTest. the rest looks good to me. no need to send a new webrev. Thanks, -- Igor > On Oct 4, 2017, at 6:08 PM, mikhailo wrote: > > Here is the updated webrev that addresses feedback from Igor and Leonid: > > http://cr.openjdk.java.net/~mseledtsov/8181592.03/ > > > Thank you, > > Misha > > > On 10/04/2017 05:52 PM, mikhailo wrote: >> Hi Leonid, >> >> Thank you for review. Please see my comments inline: >> >> >> On 10/04/2017 11:19 AM, Leonid Mesnik wrote: >>> Hi >>> >>> Overall looks good. There few comments: >>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/jtreg-ext/requires/VMProps.java.sdiff.html >>> >>> 307 System.err.println("dockerSupprt() threw exception: " + e); >>> Missed ?o? in Support. >>> >> Fixed >>> 316 p.waitFor(); >>> I think that it is needed to guard execution with timeout. I am not sure that jtreg is going to run properties with timeout. >> Fixed. I added 10 seconds for timeout, which should be sufficient. >> IMO, if it takes more than 10 sec to invoke this basic command then something is wrong, and the docker tests should not be run. >>> >>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>> 28 * @requires (sun.arch.data.model != "32") & (os.family == "linux") >>> Is it possible to check this inside docker.support and use only ?docker.support? property? >> Added. >>> >>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/lib/jdk/test/lib/containers/docker/DockerRunOptions.java.html >>> 44 public DockerRunOptions(){} >>> Why this empty constructor is needed? >> Fixed - removed. >>> Does it make also to make this class immutable i.e. init all properties in constructors and have them read-only. >> The idea of this class is that fields can be modified by the user. The default constructor fills out values for most common use case; when user decided to modify these options to specify different test run conditions they should be able to modify it. >> >> I could surround values by getters or setters, or make every new set return a new copy of the object (an immutable pattern), but I did not wish to complicate the code. The uses of this class a fairly simple, and are unlikely to result in data races. The test create runOptions object, specifies/modifies fields if necessary, then passes to DockerTestUtils for running docker container with these options. >> >> I hope you do not mind if I leave it as is. >> >> >> Thank you for review, >> Misha >>> >>> Leonid >>> >>>> On Sep 28, 2017, at 3:11 PM, Mikhailo Seledtsov > wrote: >>>> >>>> Here is the updated webrev: >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/ >>>> >>>> >>>> Leonid, George - thank you for your comments. >>>> In addition to addressing your feedback, I also did: >>>> - implemented @requires docker.support >>>> - added dockerRunJava() method and associated data structure DockerRunOptions, for running Java tests inside >>>> docker environment, and to account for java opts, test java opts, docker opts, classes and class params >>>> - added a simple HelloWorld test case that runs HelloWorld inside a container >>>> - ran jtreg with extra Java options, make sure they are added correctly to the docker run command >>>> - added docker image cleanup after testing is done >>>> >>>> >>>> Please review. >>>> >>>> Misha >>>> >>>> >>>> On 9/27/17, 11:00 AM, mikhailo wrote: >>>>> Leonid, >>>>> >>>>> Thank you for review and constructive feedback. See my comment in line. >>>>> >>>>> >>>>> On 09/26/2017 11:19 AM, Leonid Mesnik wrote: >>>>>> Misha >>>>>> >>>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>>>>> Copyright is incorrect, need to updated it for GPL. >>>>> Fixed >>>>>> >>>>>> The Hotspot is Oracle VM name only so test might fail for OpenJDK. I think you need to fix this check. >>>>> I see. I fixed this by using Platform.vmName which should be correct in all cases. I double-checked with OpenJDK also. >>>>>> >>>>>> The requires checks only that test is executed only on the 64-bit linux. Does it make a sense to introduce more docker-specific check? >>>>> I agree this is a better way. I will do some prototyping; if such check is feasible and efficient in at requires then I will add it. >>>>>> >>>>>> >>>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/Dockerfile-BasicTest.html >>>>>> Could you please explain why oraclelinux 7.0 is used as a base image for test. >>>>> I have upgraded to Oracle Linux 7.2. If we have specific requirement I will change it to that. If we have requirements in the future to support multiple OS, I can add Dockerfile generation. >>>>> For this basic sanity tests I think this should suffice. >>>>>> >>>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html >>>>>> The content looks fine. >>>>>> >>>>>> I don?t see anything to clean up docker images on the system. Could you please explain how tests are going to cleanup images. >>>>> To clean up containers I will add "--rm" to the 'docker run' command. This should ensure that container data is removed after container stops. >>>>> As for the image - I use the same image name. The image will stay in the local registry unless manually removed. I should probably do 'docker rmi' at the end of the test to clean this up. >>>>> >>>>> >>>>> Once I implement these changes I will send the updated webrev. >>>>> >>>>> Thank you, >>>>> Misha >>>>>> >>>>>> Leonid >>>>>> >>>>>> >>>>>>> On Sep 21, 2017, at 5:58 PM, mikhailo > wrote: >>>>>>> >>>>>>> Please review this initial drop of Docker test utils and a sanity test. This change lays ground >>>>>>> for further test development and test utils improvement in this area. >>>>>>> >>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >>>>>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ >>>>>>> Testing: >>>>>>> - run this test on machine with Docker enabled - works >>>>>>> - run this test on Linux-x64 with no Docker engine or Docker disabled - test skipped (as expected) >>>>>>> - run this test on automated system - in progress >>>>>>> >>>>>>> >>>>>>> Thank you, >>>>>>> Misha >>>>>>> >>>>>> >>>>> >>> >> > From mikhailo.seledtsov at oracle.com Mon Oct 9 20:02:55 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Mon, 9 Oct 2017 13:02:55 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> <5F386D9A-CA05-4732-8F68-F493DD2E8E99@oracle.com> <59CD73AC.3080207@oracle.com> <661D3614-44C0-45C5-AE04-5AE1AF42C739@oracle.com> <92e4360f-b278-0805-9c98-42e1cd98ee6d@oracle.com> Message-ID: <54de81f0-ebc8-791e-b78e-361441676f22@oracle.com> Thank you for review, Misha On 10/09/2017 12:57 PM, Igor Ignatyev wrote: > Hi Misha, > > since 'docker.support' is true only if we are on linux-x64, you can remove L#28 (@requires (sun.arch.data.model != "32") & (os.family == "linux")) from DockerBasicTest. the rest looks good to me. > no need to send a new webrev. Fixed > > Thanks, > -- Igor >> On Oct 4, 2017, at 6:08 PM, mikhailo wrote: >> >> Here is the updated webrev that addresses feedback from Igor and Leonid: >> >> http://cr.openjdk.java.net/~mseledtsov/8181592.03/ >> >> >> Thank you, >> >> Misha >> >> >> On 10/04/2017 05:52 PM, mikhailo wrote: >>> Hi Leonid, >>> >>> Thank you for review. Please see my comments inline: >>> >>> >>> On 10/04/2017 11:19 AM, Leonid Mesnik wrote: >>>> Hi >>>> >>>> Overall looks good. There few comments: >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/jtreg-ext/requires/VMProps.java.sdiff.html >>>> >>>> 307 System.err.println("dockerSupprt() threw exception: " + e); >>>> Missed ?o? in Support. >>>> >>> Fixed >>>> 316 p.waitFor(); >>>> I think that it is needed to guard execution with timeout. I am not sure that jtreg is going to run properties with timeout. >>> Fixed. I added 10 seconds for timeout, which should be sufficient. >>> IMO, if it takes more than 10 sec to invoke this basic command then something is wrong, and the docker tests should not be run. >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>>> 28 * @requires (sun.arch.data.model != "32") & (os.family == "linux") >>>> Is it possible to check this inside docker.support and use only ?docker.support? property? >>> Added. >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/lib/jdk/test/lib/containers/docker/DockerRunOptions.java.html >>>> 44 public DockerRunOptions(){} >>>> Why this empty constructor is needed? >>> Fixed - removed. >>>> Does it make also to make this class immutable i.e. init all properties in constructors and have them read-only. >>> The idea of this class is that fields can be modified by the user. The default constructor fills out values for most common use case; when user decided to modify these options to specify different test run conditions they should be able to modify it. >>> >>> I could surround values by getters or setters, or make every new set return a new copy of the object (an immutable pattern), but I did not wish to complicate the code. The uses of this class a fairly simple, and are unlikely to result in data races. The test create runOptions object, specifies/modifies fields if necessary, then passes to DockerTestUtils for running docker container with these options. >>> >>> I hope you do not mind if I leave it as is. >>> >>> >>> Thank you for review, >>> Misha >>>> Leonid >>>> >>>>> On Sep 28, 2017, at 3:11 PM, Mikhailo Seledtsov > wrote: >>>>> >>>>> Here is the updated webrev: >>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/ >>>>> >>>>> >>>>> Leonid, George - thank you for your comments. >>>>> In addition to addressing your feedback, I also did: >>>>> - implemented @requires docker.support >>>>> - added dockerRunJava() method and associated data structure DockerRunOptions, for running Java tests inside >>>>> docker environment, and to account for java opts, test java opts, docker opts, classes and class params >>>>> - added a simple HelloWorld test case that runs HelloWorld inside a container >>>>> - ran jtreg with extra Java options, make sure they are added correctly to the docker run command >>>>> - added docker image cleanup after testing is done >>>>> >>>>> >>>>> Please review. >>>>> >>>>> Misha >>>>> >>>>> >>>>> On 9/27/17, 11:00 AM, mikhailo wrote: >>>>>> Leonid, >>>>>> >>>>>> Thank you for review and constructive feedback. See my comment in line. >>>>>> >>>>>> >>>>>> On 09/26/2017 11:19 AM, Leonid Mesnik wrote: >>>>>>> Misha >>>>>>> >>>>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>>>>>> Copyright is incorrect, need to updated it for GPL. >>>>>> Fixed >>>>>>> The Hotspot is Oracle VM name only so test might fail for OpenJDK. I think you need to fix this check. >>>>>> I see. I fixed this by using Platform.vmName which should be correct in all cases. I double-checked with OpenJDK also. >>>>>>> The requires checks only that test is executed only on the 64-bit linux. Does it make a sense to introduce more docker-specific check? >>>>>> I agree this is a better way. I will do some prototyping; if such check is feasible and efficient in at requires then I will add it. >>>>>>> >>>>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/Dockerfile-BasicTest.html >>>>>>> Could you please explain why oraclelinux 7.0 is used as a base image for test. >>>>>> I have upgraded to Oracle Linux 7.2. If we have specific requirement I will change it to that. If we have requirements in the future to support multiple OS, I can add Dockerfile generation. >>>>>> For this basic sanity tests I think this should suffice. >>>>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html >>>>>>> The content looks fine. >>>>>>> >>>>>>> I don?t see anything to clean up docker images on the system. Could you please explain how tests are going to cleanup images. >>>>>> To clean up containers I will add "--rm" to the 'docker run' command. This should ensure that container data is removed after container stops. >>>>>> As for the image - I use the same image name. The image will stay in the local registry unless manually removed. I should probably do 'docker rmi' at the end of the test to clean this up. >>>>>> >>>>>> >>>>>> Once I implement these changes I will send the updated webrev. >>>>>> >>>>>> Thank you, >>>>>> Misha >>>>>>> Leonid >>>>>>> >>>>>>> >>>>>>>> On Sep 21, 2017, at 5:58 PM, mikhailo > wrote: >>>>>>>> >>>>>>>> Please review this initial drop of Docker test utils and a sanity test. This change lays ground >>>>>>>> for further test development and test utils improvement in this area. >>>>>>>> >>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >>>>>>>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ >>>>>>>> Testing: >>>>>>>> - run this test on machine with Docker enabled - works >>>>>>>> - run this test on Linux-x64 with no Docker engine or Docker disabled - test skipped (as expected) >>>>>>>> - run this test on automated system - in progress >>>>>>>> >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Misha >>>>>>>> From mikhailo.seledtsov at oracle.com Mon Oct 9 20:03:10 2017 From: mikhailo.seledtsov at oracle.com (mikhailo) Date: Mon, 9 Oct 2017 13:03:10 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> <5F386D9A-CA05-4732-8F68-F493DD2E8E99@oracle.com> <59CD73AC.3080207@oracle.com> <661D3614-44C0-45C5-AE04-5AE1AF42C739@oracle.com> Message-ID: <9a438f03-f4e5-5856-f229-cfe9b4284c56@oracle.com> Thank you for review, Misha On 10/09/2017 10:41 AM, Leonid Mesnik wrote: > Hi > > Thank you for fixes and explanations. Looks good for me now. > > Leonid >> On Oct 4, 2017, at 5:52 PM, mikhailo > > wrote: >> >> Hi Leonid, >> >> ? Thank you for review. Please see my comments inline: >> >> >> On 10/04/2017 11:19 AM, Leonid Mesnik wrote: >>> Hi >>> >>> Overall looks good. There few comments: >>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/jtreg-ext/requires/VMProps.java.sdiff.html >>> >>> >>> ?307 System.err.println("dockerSupprt() threw exception: " + e); >>> Missed ?o? in Support. >>> >> Fixed >>> 316 ? ? ? ? p.waitFor(); >>> I think that it is needed to guard execution with timeout. I am not >>> sure that jtreg is going to run properties with timeout. >> Fixed. I added 10 seconds for timeout, which should be sufficient. >> IMO, if it takes more than 10 sec to invoke this basic command then >> something is wrong, and the docker tests should not be run. >>> >>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>> >>> 28 * @requires (sun.arch.data.model != "32") & (os.family == "linux") >>> Is it possible to check this inside docker.support and use only >>> ?docker.support? property? >> Added. >>> >>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/test/lib/jdk/test/lib/containers/docker/DockerRunOptions.java.html >>> >>> 44 public DockerRunOptions(){} >>> Why this empty constructor is needed? >> Fixed - removed. >>> Does it make also to make this class immutable i.e. init all >>> properties in constructors and have them read-only. >> The idea of this class is that fields can be modified by the user. >> The default constructor fills out values for most common use case; >> when user decided to modify these options to specify different test >> run conditions they should be able to modify it. >> >> I could surround values by getters or setters, or make every new set >> return a new copy of the object (an immutable pattern), but I did not >> wish to complicate the code. The uses of this class a fairly simple, >> and are unlikely to result in data races. The test create runOptions >> object, specifies/modifies fields if necessary, then passes to >> DockerTestUtils for running docker container with these options. >> >> I hope you do not mind if I leave it as is. >> >> >> Thank you for review, >> Misha >>> >>> Leonid >>> >>>> On Sep 28, 2017, at 3:11 PM, Mikhailo Seledtsov >>>> >>> > wrote: >>>> >>>> Here is the updated webrev: >>>> http://cr.openjdk.java.net/~mseledtsov/8181592.02/ >>>> >>>> >>>> >>>> Leonid, George - thank you for your comments. >>>> In addition to addressing your feedback, I also did: >>>> ?- implemented @requires docker.support >>>> ?- added dockerRunJava() method and associated data structure >>>> DockerRunOptions, for running Java tests inside >>>> ????docker environment, and to account for java opts, test java >>>> opts, docker opts, classes and class params >>>> ?- added a simple HelloWorld test case that runs HelloWorld inside >>>> a container >>>> ?- ran jtreg with extra Java options, make sure they are added >>>> correctly to the docker run command >>>> ?- added docker image cleanup after testing is done >>>> >>>> >>>> Please review. >>>> >>>> Misha >>>> >>>> >>>> On 9/27/17, 11:00 AM, mikhailo wrote: >>>>> Leonid, >>>>> >>>>> Thank you for review and constructive feedback. See my comment in >>>>> line. >>>>> >>>>> >>>>> On 09/26/2017 11:19 AM, Leonid Mesnik wrote: >>>>>> Misha >>>>>> >>>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/DockerBasicTest.java.html >>>>>> >>>>>> >>>>>> >>>>>> Copyright is incorrect, need to updated it for GPL. >>>>> Fixed >>>>>> >>>>>> The Hotspot is Oracle VM name only so test might fail for >>>>>> OpenJDK. I think you need to fix this check. >>>>> I see. I fixed this by using Platform.vmName which should be >>>>> correct in all cases. I double-checked with OpenJDK also. >>>>>> >>>>>> The requires checks only that test is executed only on the 64-bit >>>>>> linux. Does it make a sense to introduce more docker-specific check? >>>>> I agree this is a better way. I will do some prototyping; if such >>>>> check is feasible and efficient in at requires then I will add it. >>>>>> >>>>>> >>>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/hotspot/jtreg/runtime/containers/docker/Dockerfile-BasicTest.html >>>>>> >>>>>> >>>>>> >>>>>> Could you please explain why oraclelinux 7.0 is used as a base >>>>>> image for test. >>>>> I have upgraded to Oracle Linux 7.2. If we have specific >>>>> requirement I will change it to that. If we have requirements in >>>>> the future to support multiple OS, I can add Dockerfile generation. >>>>> For this basic sanity tests I think this should suffice. >>>>>> >>>>>> http://cr.openjdk.java.net/~mseledtsov/8181592.00/test/lib/jdk/test/lib/containers/docker/DockerTestUtils.java.html >>>>>> >>>>>> >>>>>> >>>>>> The content looks fine. >>>>>> >>>>>> I don?t see anything to clean up docker images on the system. >>>>>> Could you please explain how tests are going to cleanup images. >>>>> To clean up containers I will add "--rm" to the 'docker run' >>>>> command. This should ensure that container data is removed after >>>>> container stops. >>>>> As for the image - I use the same image name. The image will stay >>>>> in the local registry unless manually removed. I should probably >>>>> do 'docker rmi' at the end of the test to clean this up. >>>>> >>>>> >>>>> Once I implement these changes I will send the updated webrev. >>>>> >>>>> Thank you, >>>>> Misha >>>>>> >>>>>> Leonid >>>>>> >>>>>> >>>>>>> On Sep 21, 2017, at 5:58 PM, mikhailo >>>>>>> >>>>>> >>>>>>> > wrote: >>>>>>> >>>>>>> Please review this initial drop of Docker test utils and a >>>>>>> sanity test. This change lays ground >>>>>>> for further test development and test utils improvement in this >>>>>>> area. >>>>>>> >>>>>>> ???JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >>>>>>> ???Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ >>>>>>> >>>>>>> >>>>>>> ???Testing: >>>>>>> ??????- run this test on machine with Docker enabled - works >>>>>>> ??????- run this test on Linux-x64 with no Docker engine or >>>>>>> Docker disabled - test skipped (as expected) >>>>>>> ??????- run this test on automated system - in progress >>>>>>> >>>>>>> >>>>>>> Thank you, >>>>>>> Misha >>>>>>> >>>>>> >>>>> >>> >> > From mandy.chung at oracle.com Mon Oct 9 20:17:36 2017 From: mandy.chung at oracle.com (mandy chung) Date: Mon, 9 Oct 2017 13:17:36 -0700 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> Message-ID: <57e0b49a-bed6-25c2-7d01-b87f08d82a75@oracle.com> David, Peter, Alan The VM has a fast path to search the symbol from libjava.so first for bootstrap loader.? That was the case I mostly concern about performance and it's not impacted by this change.?? Also I consulted with Claes on the performance impact.?? I took your suggestion and made systemNativeLibraries and nativeLibraries volatile. Updated webrev: http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.04 thanks Mandy On 10/9/17 1:20 AM, Alan Bateman wrote: > Alternatively change nativeLibraries and systemNativeLibraries to > volatile so the synchronization is only needed to initialize them. > Otherwise this version (webrev.03) looks good to me. > > -Alan From manc at google.com Mon Oct 9 21:07:52 2017 From: manc at google.com (Man Cao) Date: Mon, 9 Oct 2017 14:07:52 -0700 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: <19df096b-3243-1ac0-3d3a-e955c63c534d@oracle.com> <53fb4559-aeed-da1a-67fe-6cb50fd8e9ce@gmail.com> Message-ID: Thanks for you both updating and sponsoring the webrev! We plan to back-port it to our JDK9 once it is submitted. -Man On Mon, Oct 9, 2017 at 6:15 AM, wrote: > Hi, this looks reasonable but I would prefer that the code in > arguments.cpp call something in metaspace.cpp, like check_metaspace_sizes() > so that you don't have to tell arguments what the MediumChunk size is. > > I will sponsor this for you. > > thanks, > Coleen > > > On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: > >> Hi all, >> >> I added a testcase for this issue in new webrev: >> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >> >> >> Thanks, >> >> Yasumasa >> >> >> >> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >> >>> Hi all, >>> >>> I uploaded webrev for this issue against jdk10/hs. >>> Could you review it? >>> >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>> >>> >>> I cannot access JPRT. So I need a sponsor. >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> >>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>> >>>> Thank Yasumasa and Stefan for the responses. >>>> >>>> Good to know that the patch is not blocked due to breaking some internal >>>> invariants/assumptions, but just due to its P5 status. >>>> Is it possible to push it to P4? >>>> >>>> -Man >>>> >>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> (CC'ed hotspot-runtime-dev) >>>>> >>>>> I think the reason is that this bug is a P5. The compressed class space >>>>>> belongs to the runtime code, so you might get more traction for this >>>>>> on the >>>>>> hotspot-runtime-dev list. >>>>>> >>>>> >>>>> I will send review request against jdk10/master or jdk10/hs after repos >>>>> are opened. >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> >>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>> >>>>>> Hi Man, >>>>>> >>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>> >>>>>>> Hi Yasumasa, Stefan, >>>>>>> >>>>>>> Do you have any thoughts on why this patch has been pending for 2+ >>>>>>> years? This patch could really save us from some annoying issues >>>>>>> since we >>>>>>> are automatically monitoring hsperfdata counters. >>>>>>> >>>>>> >>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>> space >>>>>> belongs to the runtime code, so you might get more traction for this >>>>>> on the >>>>>> hotspot-runtime-dev list. >>>>>> >>>>>> StefanK >>>>>> >>>>>> -Man >>>>>>> >>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>> > wrote: >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I wonder if there is any recent update on the patch for >>>>>>> JDK-8087291. >>>>>>> Is it possible to push this patch into JDK9? Except for its low >>>>>>> priority (P5), >>>>>>> is there any complication that prevents this patch getting >>>>>>> approved >>>>>>> (for example, some JVM logic requires CompressedClassSpaceSize >>>>>>> to be >>>>>>> 1GB by default)? >>>>>>> >>>>>>> I work in the Java Platform Team at Google. We have encountered >>>>>>> annoying issues that the hsperfdata counter >>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>> a too large value (about 1GB) even if user sets >>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>> confusing 1GB >>>>>>> memory reserved by metaspace, >>>>>>> regardless of MaxMetaspaceSize value. The root cause for these >>>>>>> issues is that CompressedClassSpaceSize is not automatically >>>>>>> capped >>>>>>> by MaxMetaspaceSize >>>>>>> during VM initialization, and this patch seems fix the root >>>>>>> cause. >>>>>>> (I'm aware that even after this patch, the reserved size could >>>>>>> still >>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>> but it is better than the current situation.) >>>>>>> >>>>>>> Thanks, >>>>>>> Man >>>>>>> >>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>> >>>>>>> Thank you for your comment! >>>>>>> > Try running a debug JVM with your patch with this command >>>>>>> line. >>>>>>> > >>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>> >>>>>> /> >>>>>>> It works on fastdebug VM. >>>>>>> Please review again. >>>>>>> >>>>>>> Thanks, >>>>>>> Yasumasa >>>>>>> >>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>> > Yasumasa, >>>>>>> > >>>>>>> > Try running a debug JVM with your patch with this command >>>>>>> line. >>>>>>> > >>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>> > >>>>>>> > On a linux system I get this when I build with your >>>>>>> patch. >>>>>>> > >>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>> >> # To suppress the following error report, specify this >>>>>>> argument >>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>> >> # >>>>>>> >> # A fatal error has been detected by the Java Runtime >>>>>>> Environment: >>>>>>> >> # >>>>>>> >> # Internal Error >>>>>>> >> >>>>>>> >>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/ >>>>>>> metaspace.cpp:2324), >>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>> >> # assert(size > MediumChunk || size > ClassMediumChunk) >>>>>>> failed: Not a >>>>>>> >> humongous chunk >>>>>>> >> # >>>>>>> > >>>>>>> > >>>>>>> > Jon >>>>>>> > >>>>>>> > >>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>> >> I want to continue to discuss about >>>>>>> CompressedClassSpace and >>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>> >> >>>>>>> >> >>>>>>> >>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-J >>>>>>> une/013873.html >>>>>>> >>>>>>> >>>>>> June/013873.html> >>>>>>> >>>> Should I resize CompressedClassSpaceSize than to show >>>>>>> error message? >>>>>>> >>> If you add slightly better heuristics for the setup of >>>>>>> the >>>>>>> CompressedClassSpaceSize flag, for example lowering the >>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is set, then >>>>>>> it >>>>>>> might be less likely that you'll hit the OutOfMemoryError >>>>>>> when >>>>>>> the system is set up with strict overcommit settings. >>>>>>> >> >>>>>>> >> I've uploaded new webrev: >>>>>>> >> http://cr.openjdk.java.net/~ys >>>>>>> uenaga/JDK-8087291/webrev.01/ >>>>>>> >>>>>> /> >>>>>>> >> >>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>> CompressedClassSpaceSize, and >>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>> >> >>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>> >> If InitialBootClassLoaderMetaspaceSize is greater than >>>>>>> MaxMetaspaceSize, >>>>>>> >> VM will fail with error message. >>>>>>> >> >>>>>>> >> InitialBootClassLoaderMetaspaceSize will be set to >>>>>>> MaxMetaspaceSize >>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>> Metaspace::ergo_initialize(). >>>>>>> >> >>>>>>> >> >>>>>>> >> Thanks, >>>>>>> >> >>>>>>> >> Yasumasa >>>>>>> >> >>>>>>> >>>>>>> >>>>>>> > From daniel.daugherty at oracle.com Mon Oct 9 21:23:23 2017 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 9 Oct 2017 15:23:23 -0600 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> Message-ID: <546f3f48-47cf-73d1-30b1-b388418ae0bf@oracle.com> Many thanks to the folks that reviewed this internally and provided much appreciated feedback: - Daniel Daugherty - David Holmes - Erik Osterlund - Jerry Thornbrugh - Karen Kinnear - Kim Barrett - Robbin Ehn - Serguei Spitsyn - Stefan Karlson Since there are three contributing authors, we have been reviewing (and arguing over) each other's code. It has been an adventure! Dan, Erik, and Robbin On 10/9/17 1:41 PM, Daniel D. Daugherty wrote: > Greetings, > > We have a (eXtra Large) fix for the following bug: > > 8167108 inconsistent handling of SR_lock can lead to crashes > https://bugs.openjdk.java.net/browse/JDK-8167108 > > This fix adds a Safe Memory Reclamation (SMR) mechanism based on > Hazard Pointers to manage JavaThread lifecycle. > > Here's a PDF for the internal wiki that we've been using to describe > and track the work on this project: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf > > > Dan has noticed that the indenting is wrong in some of the code quotes > in the PDF that are not present in the internal wiki. We don't have a > solution for that problem yet. > > Here's the webrev for current JDK10 version of this fix: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full > > This fix has been run through many rounds of JPRT and Mach5 tier[2-5] > testing, additional stress testing on Dan's Solaris X64 server, and > additional testing on Erik and Robbin's machines. > > We welcome comments, suggestions and feedback. > > Daniel Daugherty > Erik Osterlund > Robbin Ehn > From coleen.phillimore at oracle.com Tue Oct 10 02:36:15 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 9 Oct 2017 22:36:15 -0400 Subject: [10] RFR: [AOT] assert(false) failed: DEBUG MESSAGE: InterpreterMacroAssembler::call_VM_base: last_sp != NULL In-Reply-To: <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> References: <8eb949bd-fa14-2722-5eaa-21a0a0c95b26@oracle.com> <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> Message-ID: <0a232ef3-8e2b-3025-14ba-eaf8c6b409fe@oracle.com> This seems ok to me with Jamsheed's explanation. Thanks, Coleen On 9/14/17 2:54 AM, Dean Long wrote: > It looks like you accidentally dropped > hotspot-compiler-dev at openjdk.java.net when you added runtime. > > dl > > > On 9/13/2017 11:21 PM, jamsheed wrote: >> (adding runtime list for inputs) >> >> On Monday 11 September 2017 11:43 PM, jamsheed wrote: >>> brief desc: special handling of Object. in >>> TemplateInterpreter::deopt_reexecute_entry >>> >>> required last_sp to be reset explicitly in normal return path >>> >>> address TemplateInterpreter::deopt_reexecute_entry(Method* method, >>> address bcp) { >>> ? assert(method->contains(bcp), "just checkin'"); >>> ? Bytecodes::Code code?? = Bytecodes::java_code_at(method, bcp); >>> ? if (code == Bytecodes::_return) { >>> ??? // This is used for deopt during registration of finalizers >>> ??? // during Object..? We simply need to resume execution at >>> ??? // the standard return vtos bytecode to pop the frame normally. >>> ??? // reexecuting the real bytecode would cause double registration >>> ??? // of the finalizable object. >>> ??? return _normal_table.entry(Bytecodes::_return).entry(vtos); >> >> last_sp ! = null not an issue for this case, so i skip the assert in >> debug build >> >> http://cr.openjdk.java.net/~jcm/8168712/webrev.01/ >> >> Please review. >> >> Best Regards, >> Jamsheed >> >> >> >> >> > From yasuenag at gmail.com Tue Oct 10 03:09:41 2017 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Tue, 10 Oct 2017 12:09:41 +0900 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize Message-ID: Hi Coleen, >> I will sponsor this for you. Thanks! >> Hi, this looks reasonable but I would prefer that the code in >> arguments.cpp call something in metaspace.cpp, like check_metaspace_sizes() >> so that you don't have to tell arguments what the MediumChunk size is. I added new function calculate_min_metaspace_size() in metaspace.cpp to get minimum metaspace size, and uses return value from this function in metaspace initialization and metaspace size check. http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ Could you review again? Thanks, Yasumasa 2017-10-10 6:07 GMT+09:00 Man Cao : > Thanks for you both updating and sponsoring the webrev! > We plan to back-port it to our JDK9 once it is submitted. > > -Man > > On Mon, Oct 9, 2017 at 6:15 AM, wrote: >> >> Hi, this looks reasonable but I would prefer that the code in >> arguments.cpp call something in metaspace.cpp, like check_metaspace_sizes() >> so that you don't have to tell arguments what the MediumChunk size is. >> >> I will sponsor this for you. >> >> thanks, >> Coleen >> >> >> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>> >>> Hi all, >>> >>> I added a testcase for this issue in new webrev: >>> >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> >>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >>>> >>>> Hi all, >>>> >>>> I uploaded webrev for this issue against jdk10/hs. >>>> Could you review it? >>>> >>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>> >>>> >>>> I cannot access JPRT. So I need a sponsor. >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> >>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>> >>>>> Thank Yasumasa and Stefan for the responses. >>>>> >>>>> Good to know that the patch is not blocked due to breaking some >>>>> internal >>>>> invariants/assumptions, but just due to its P5 status. >>>>> Is it possible to push it to P4? >>>>> >>>>> -Man >>>>> >>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> (CC'ed hotspot-runtime-dev) >>>>>> >>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>> space >>>>>>> belongs to the runtime code, so you might get more traction for this >>>>>>> on the >>>>>>> hotspot-runtime-dev list. >>>>>> >>>>>> >>>>>> I will send review request against jdk10/master or jdk10/hs after >>>>>> repos >>>>>> are opened. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Yasumasa >>>>>> >>>>>> >>>>>> >>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>> >>>>>>> Hi Man, >>>>>>> >>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>> >>>>>>>> Hi Yasumasa, Stefan, >>>>>>>> >>>>>>>> Do you have any thoughts on why this patch has been pending for 2+ >>>>>>>> years? This patch could really save us from some annoying issues >>>>>>>> since we >>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>> >>>>>>> >>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>> space >>>>>>> belongs to the runtime code, so you might get more traction for this >>>>>>> on the >>>>>>> hotspot-runtime-dev list. >>>>>>> >>>>>>> StefanK >>>>>>> >>>>>>>> -Man >>>>>>>> >>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>> > wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I wonder if there is any recent update on the patch for >>>>>>>> JDK-8087291. >>>>>>>> Is it possible to push this patch into JDK9? Except for its low >>>>>>>> priority (P5), >>>>>>>> is there any complication that prevents this patch getting >>>>>>>> approved >>>>>>>> (for example, some JVM logic requires CompressedClassSpaceSize >>>>>>>> to be >>>>>>>> 1GB by default)? >>>>>>>> >>>>>>>> I work in the Java Platform Team at Google. We have encountered >>>>>>>> annoying issues that the hsperfdata counter >>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>>> confusing 1GB >>>>>>>> memory reserved by metaspace, >>>>>>>> regardless of MaxMetaspaceSize value. The root cause for these >>>>>>>> issues is that CompressedClassSpaceSize is not automatically >>>>>>>> capped >>>>>>>> by MaxMetaspaceSize >>>>>>>> during VM initialization, and this patch seems fix the root >>>>>>>> cause. >>>>>>>> (I'm aware that even after this patch, the reserved size could >>>>>>>> still >>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>> but it is better than the current situation.) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Man >>>>>>>> >>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>> >>>>>>>> Thank you for your comment! >>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>> command >>>>>>>> line. >>>>>>>> > >>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>> >>>>>>>> >>>>>>>> It works on fastdebug VM. >>>>>>>> Please review again. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Yasumasa >>>>>>>> >>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>> > Yasumasa, >>>>>>>> > >>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>> command >>>>>>>> line. >>>>>>>> > >>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>> > >>>>>>>> > On a linux system I get this when I build with your >>>>>>>> patch. >>>>>>>> > >>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>> >> # To suppress the following error report, specify this >>>>>>>> argument >>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>> >> # >>>>>>>> >> # A fatal error has been detected by the Java Runtime >>>>>>>> Environment: >>>>>>>> >> # >>>>>>>> >> # Internal Error >>>>>>>> >> >>>>>>>> >>>>>>>> >>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>> ClassMediumChunk) >>>>>>>> failed: Not a >>>>>>>> >> humongous chunk >>>>>>>> >> # >>>>>>>> > >>>>>>>> > >>>>>>>> > Jon >>>>>>>> > >>>>>>>> > >>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>>> >> I want to continue to discuss about >>>>>>>> CompressedClassSpace and >>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>> >> >>>>>>>> >> >>>>>>>> >>>>>>>> >>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>> Should I resize CompressedClassSpaceSize than to show >>>>>>>> error message? >>>>>>>> >>> If you add slightly better heuristics for the setup of >>>>>>>> the >>>>>>>> CompressedClassSpaceSize flag, for example lowering the >>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is set, then >>>>>>>> it >>>>>>>> might be less likely that you'll hit the OutOfMemoryError >>>>>>>> when >>>>>>>> the system is set up with strict overcommit settings. >>>>>>>> >> >>>>>>>> >> I've uploaded new webrev: >>>>>>>> >> >>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>> >>>>>>>> >>>>>>>> >> >>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>> CompressedClassSpaceSize, and >>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>> >> >>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is greater than >>>>>>>> MaxMetaspaceSize, >>>>>>>> >> VM will fail with error message. >>>>>>>> >> >>>>>>>> >> InitialBootClassLoaderMetaspaceSize will be set to >>>>>>>> MaxMetaspaceSize >>>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>>> Metaspace::ergo_initialize(). >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> Thanks, >>>>>>>> >> >>>>>>>> >> Yasumasa >>>>>>>> >> >>>>>>>> >>>>>>>> >> > From david.holmes at oracle.com Tue Oct 10 03:27:54 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 10 Oct 2017 13:27:54 +1000 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <57e0b49a-bed6-25c2-7d01-b87f08d82a75@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> <57e0b49a-bed6-25c2-7d01-b87f08d82a75@oracle.com> Message-ID: On 10/10/2017 6:17 AM, mandy chung wrote: > David, Peter, Alan > > The VM has a fast path to search the symbol from libjava.so first for > bootstrap loader.? That was the case I mostly concern about performance > and it's not impacted by this change.?? Also I consulted with Claes on > the performance impact.?? I took your suggestion and made > systemNativeLibraries and nativeLibraries volatile. > > Updated webrev: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.04 Looks good. Thanks, David ----- > thanks > Mandy > > On 10/9/17 1:20 AM, Alan Bateman wrote: >> Alternatively change nativeLibraries and systemNativeLibraries to >> volatile so the synchronization is only needed to initialize them. >> Otherwise this version (webrev.03) looks good to me. >> >> -Alan > From jamsheed.c.m at oracle.com Tue Oct 10 06:05:37 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Tue, 10 Oct 2017 11:35:37 +0530 Subject: [10] RFR: [AOT] assert(false) failed: DEBUG MESSAGE: InterpreterMacroAssembler::call_VM_base: last_sp != NULL In-Reply-To: <0a232ef3-8e2b-3025-14ba-eaf8c6b409fe@oracle.com> References: <8eb949bd-fa14-2722-5eaa-21a0a0c95b26@oracle.com> <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> <0a232ef3-8e2b-3025-14ba-eaf8c6b409fe@oracle.com> Message-ID: <0df5882b-b94c-569c-bed8-33f502e7fd8d@oracle.com> Thanks for the review, Coleen Best regards, Jamsheed On Tuesday 10 October 2017 08:06 AM, coleen.phillimore at oracle.com wrote: > > This seems ok to me with Jamsheed's explanation. > Thanks, > Coleen > > On 9/14/17 2:54 AM, Dean Long wrote: >> It looks like you accidentally dropped >> hotspot-compiler-dev at openjdk.java.net when you added runtime. >> >> dl >> >> >> On 9/13/2017 11:21 PM, jamsheed wrote: >>> (adding runtime list for inputs) >>> >>> On Monday 11 September 2017 11:43 PM, jamsheed wrote: >>>> brief desc: special handling of Object. in >>>> TemplateInterpreter::deopt_reexecute_entry >>>> >>>> required last_sp to be reset explicitly in normal return path >>>> >>>> address TemplateInterpreter::deopt_reexecute_entry(Method* method, >>>> address bcp) { >>>> ? assert(method->contains(bcp), "just checkin'"); >>>> ? Bytecodes::Code code?? = Bytecodes::java_code_at(method, bcp); >>>> ? if (code == Bytecodes::_return) { >>>> ??? // This is used for deopt during registration of finalizers >>>> ??? // during Object..? We simply need to resume execution at >>>> ??? // the standard return vtos bytecode to pop the frame normally. >>>> ??? // reexecuting the real bytecode would cause double registration >>>> ??? // of the finalizable object. >>>> ??? return _normal_table.entry(Bytecodes::_return).entry(vtos); >>> >>> last_sp ! = null not an issue for this case, so i skip the assert in >>> debug build >>> >>> http://cr.openjdk.java.net/~jcm/8168712/webrev.01/ >>> >>> Please review. >>> >>> Best Regards, >>> Jamsheed >>> >>> >>> >>> >>> >> > From Alan.Bateman at oracle.com Tue Oct 10 08:44:19 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 10 Oct 2017 09:44:19 +0100 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <57e0b49a-bed6-25c2-7d01-b87f08d82a75@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> <57e0b49a-bed6-25c2-7d01-b87f08d82a75@oracle.com> Message-ID: <16afc01d-016e-b77c-58b9-cefcdf62a03d@oracle.com> On 09/10/2017 21:17, mandy chung wrote: > David, Peter, Alan > > The VM has a fast path to search the symbol from libjava.so first for > bootstrap loader.? That was the case I mostly concern about > performance and it's not impacted by this change.?? Also I consulted > with Claes on the performance impact.?? I took your suggestion and > made systemNativeLibraries and nativeLibraries volatile. > > Updated webrev: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.04 This update looks good to me. -Alan From peter.levart at gmail.com Tue Oct 10 09:29:04 2017 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 10 Oct 2017 11:29:04 +0200 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <57e0b49a-bed6-25c2-7d01-b87f08d82a75@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> <57e0b49a-bed6-25c2-7d01-b87f08d82a75@oracle.com> Message-ID: <376a0876-d94b-8775-873c-bfd741614273@gmail.com> On 10/09/2017 10:17 PM, mandy chung wrote: > David, Peter, Alan > > The VM has a fast path to search the symbol from libjava.so first for > bootstrap loader.? That was the case I mostly concern about > performance and it's not impacted by this change.?? Also I consulted > with Claes on the performance impact.?? I took your suggestion and > made systemNativeLibraries and nativeLibraries volatile. > > Updated webrev: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.04 > Looks good now. Just one question (for a possible follow-up future optimization): 2674???? private static long findNative(ClassLoader loader, String name) { 2675???????? Map libs = 2676???????????? loader != null ? loader.nativeLibraries() : systemNativeLibraries(); 2677???????? if (libs.isEmpty()) 2678???????????? return 0; 2679 2680???????? // the native libraries map may be updated in another thread 2681???????? // when a native library is being loaded.? No symbol will be 2682???????? // searched from it yet. 2683???????? for (NativeLibrary lib : libs.values()) { 2684???????????? long entry = lib.findEntry(name); 2685???????????? if (entry != 0) return entry; 2686???????? } 2687???????? return 0; 2688???? } Now that (system)nativeLibraries is a Map, is it still necessary to iterate it and call lib.findEntry(name) on each NativeLibrary until the one that returns a non-zero entry or would it be semantically equivalent to 1st look-up the Map by the 'name' key to get the correct NativeLibrary? Regards, Peter From coleen.phillimore at oracle.com Tue Oct 10 12:02:31 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 10 Oct 2017 08:02:31 -0400 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: Message-ID: Hi Yasumasa,? I like this better.? I will sponsor it.? I think it only needs one reviewer, unless there were more?? Please send me the result of hg export. thanks, Coleen On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: > Hi Coleen, > >>> I will sponsor this for you. > Thanks! > > >>> Hi, this looks reasonable but I would prefer that the code in >>> arguments.cpp call something in metaspace.cpp, like check_metaspace_sizes() >>> so that you don't have to tell arguments what the MediumChunk size is. > I added new function calculate_min_metaspace_size() in metaspace.cpp > to get minimum metaspace size, and uses return value from this > function in metaspace initialization and metaspace size check. > > http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ > > Could you review again? > > > Thanks, > > Yasumasa > > > 2017-10-10 6:07 GMT+09:00 Man Cao : >> Thanks for you both updating and sponsoring the webrev! >> We plan to back-port it to our JDK9 once it is submitted. >> >> -Man >> >> On Mon, Oct 9, 2017 at 6:15 AM, wrote: >>> Hi, this looks reasonable but I would prefer that the code in >>> arguments.cpp call something in metaspace.cpp, like check_metaspace_sizes() >>> so that you don't have to tell arguments what the MediumChunk size is. >>> >>> I will sponsor this for you. >>> >>> thanks, >>> Coleen >>> >>> >>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>> Hi all, >>>> >>>> I added a testcase for this issue in new webrev: >>>> >>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> >>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >>>>> Hi all, >>>>> >>>>> I uploaded webrev for this issue against jdk10/hs. >>>>> Could you review it? >>>>> >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>>> >>>>> >>>>> I cannot access JPRT. So I need a sponsor. >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> >>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>> Thank Yasumasa and Stefan for the responses. >>>>>> >>>>>> Good to know that the patch is not blocked due to breaking some >>>>>> internal >>>>>> invariants/assumptions, but just due to its P5 status. >>>>>> Is it possible to push it to P4? >>>>>> >>>>>> -Man >>>>>> >>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>> wrote: >>>>>>> Hi, >>>>>>> >>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>> >>>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>>> space >>>>>>>> belongs to the runtime code, so you might get more traction for this >>>>>>>> on the >>>>>>>> hotspot-runtime-dev list. >>>>>>> >>>>>>> I will send review request against jdk10/master or jdk10/hs after >>>>>>> repos >>>>>>> are opened. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Yasumasa >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>> Hi Man, >>>>>>>> >>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>> >>>>>>>>> Do you have any thoughts on why this patch has been pending for 2+ >>>>>>>>> years? This patch could really save us from some annoying issues >>>>>>>>> since we >>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>> >>>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>>> space >>>>>>>> belongs to the runtime code, so you might get more traction for this >>>>>>>> on the >>>>>>>> hotspot-runtime-dev list. >>>>>>>> >>>>>>>> StefanK >>>>>>>> >>>>>>>>> -Man >>>>>>>>> >>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>> > wrote: >>>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I wonder if there is any recent update on the patch for >>>>>>>>> JDK-8087291. >>>>>>>>> Is it possible to push this patch into JDK9? Except for its low >>>>>>>>> priority (P5), >>>>>>>>> is there any complication that prevents this patch getting >>>>>>>>> approved >>>>>>>>> (for example, some JVM logic requires CompressedClassSpaceSize >>>>>>>>> to be >>>>>>>>> 1GB by default)? >>>>>>>>> >>>>>>>>> I work in the Java Platform Team at Google. We have encountered >>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>>>> confusing 1GB >>>>>>>>> memory reserved by metaspace, >>>>>>>>> regardless of MaxMetaspaceSize value. The root cause for these >>>>>>>>> issues is that CompressedClassSpaceSize is not automatically >>>>>>>>> capped >>>>>>>>> by MaxMetaspaceSize >>>>>>>>> during VM initialization, and this patch seems fix the root >>>>>>>>> cause. >>>>>>>>> (I'm aware that even after this patch, the reserved size could >>>>>>>>> still >>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>> but it is better than the current situation.) >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Man >>>>>>>>> >>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>> >>>>>>>>> Thank you for your comment! >>>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>>> command >>>>>>>>> line. >>>>>>>>> > >>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>> >>>>>>>>> >>>>>>>>> It works on fastdebug VM. >>>>>>>>> Please review again. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Yasumasa >>>>>>>>> >>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>> > Yasumasa, >>>>>>>>> > >>>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>>> command >>>>>>>>> line. >>>>>>>>> > >>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>> > >>>>>>>>> > On a linux system I get this when I build with your >>>>>>>>> patch. >>>>>>>>> > >>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>> >> # To suppress the following error report, specify this >>>>>>>>> argument >>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>> >> # >>>>>>>>> >> # A fatal error has been detected by the Java Runtime >>>>>>>>> Environment: >>>>>>>>> >> # >>>>>>>>> >> # Internal Error >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>> ClassMediumChunk) >>>>>>>>> failed: Not a >>>>>>>>> >> humongous chunk >>>>>>>>> >> # >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Jon >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>>>> >> I want to continue to discuss about >>>>>>>>> CompressedClassSpace and >>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>> Should I resize CompressedClassSpaceSize than to show >>>>>>>>> error message? >>>>>>>>> >>> If you add slightly better heuristics for the setup of >>>>>>>>> the >>>>>>>>> CompressedClassSpaceSize flag, for example lowering the >>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is set, then >>>>>>>>> it >>>>>>>>> might be less likely that you'll hit the OutOfMemoryError >>>>>>>>> when >>>>>>>>> the system is set up with strict overcommit settings. >>>>>>>>> >> >>>>>>>>> >> I've uploaded new webrev: >>>>>>>>> >> >>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >> >>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>> >> >>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is greater than >>>>>>>>> MaxMetaspaceSize, >>>>>>>>> >> VM will fail with error message. >>>>>>>>> >> >>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will be set to >>>>>>>>> MaxMetaspaceSize >>>>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> Thanks, >>>>>>>>> >> >>>>>>>>> >> Yasumasa >>>>>>>>> >> >>>>>>>>> >>>>>>>>> From mandy.chung at oracle.com Tue Oct 10 15:00:37 2017 From: mandy.chung at oracle.com (mandy chung) Date: Tue, 10 Oct 2017 08:00:37 -0700 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <376a0876-d94b-8775-873c-bfd741614273@gmail.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> <57e0b49a-bed6-25c2-7d01-b87f08d82a75@oracle.com> <376a0876-d94b-8775-873c-bfd741614273@gmail.com> Message-ID: <96cf434b-aa72-158f-515f-089a63c25c4d@oracle.com> On 10/10/17 2:29 AM, Peter Levart wrote: > > On 10/09/2017 10:17 PM, mandy chung wrote: >> David, Peter, Alan >> >> The VM has a fast path to search the symbol from libjava.so first for >> bootstrap loader.? That was the case I mostly concern about >> performance and it's not impacted by this change.?? Also I consulted >> with Claes on the performance impact.?? I took your suggestion and >> made systemNativeLibraries and nativeLibraries volatile. >> >> Updated webrev: >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8164512/webrev.04 >> > > Looks good now. Just one question (for a possible follow-up future > optimization): > > 2674???? private static long findNative(ClassLoader loader, String name) { > 2675???????? Map libs = > 2676???????????? loader != null ? loader.nativeLibraries() : > systemNativeLibraries(); > 2677???????? if (libs.isEmpty()) > 2678???????????? return 0; > 2679 > 2680???????? // the native libraries map may be updated in another thread > 2681???????? // when a native library is being loaded.? No symbol will be > 2682???????? // searched from it yet. > 2683???????? for (NativeLibrary lib : libs.values()) { > 2684???????????? long entry = lib.findEntry(name); > 2685???????????? if (entry != 0) return entry; > 2686???????? } > 2687???????? return 0; > 2688???? } > > Now that (system)nativeLibraries is a Map, is it still necessary to > iterate it and call lib.findEntry(name) on each NativeLibrary until > the one that returns a non-zero entry or would it be semantically > equivalent to 1st look-up the Map by the 'name' key to get the correct > NativeLibrary? > The name parameter is the symbol name but not the library name. It's confusing and therefore I renamed NativeLibrary::find to NativeLibrary::findEntry.?? I could rename the name parameter to entryName to make it clearer. Mandy From vladimir.kozlov at oracle.com Tue Oct 10 15:11:43 2017 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Tue, 10 Oct 2017 08:11:43 -0700 Subject: [10] RFR: [AOT] assert(false) failed: DEBUG MESSAGE: InterpreterMacroAssembler::call_VM_base: last_sp != NULL In-Reply-To: <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> References: <8eb949bd-fa14-2722-5eaa-21a0a0c95b26@oracle.com> <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> Message-ID: Why you added !defined(AARCH64) in templateTable_arm.cpp? Is only 32-bit affected? Thanks, Vladimir On 9/13/17 11:54 PM, Dean Long wrote: > It looks like you accidentally dropped hotspot-compiler-dev at openjdk.java.net when you added runtime. > > dl > > > On 9/13/2017 11:21 PM, jamsheed wrote: >> (adding runtime list for inputs) >> >> On Monday 11 September 2017 11:43 PM, jamsheed wrote: >>> brief desc: special handling of Object. in TemplateInterpreter::deopt_reexecute_entry >>> >>> required last_sp to be reset explicitly in normal return path >>> >>> address TemplateInterpreter::deopt_reexecute_entry(Method* method, address bcp) { >>> ? assert(method->contains(bcp), "just checkin'"); >>> ? Bytecodes::Code code?? = Bytecodes::java_code_at(method, bcp); >>> ? if (code == Bytecodes::_return) { >>> ??? // This is used for deopt during registration of finalizers >>> ??? // during Object..? We simply need to resume execution at >>> ??? // the standard return vtos bytecode to pop the frame normally. >>> ??? // reexecuting the real bytecode would cause double registration >>> ??? // of the finalizable object. >>> ??? return _normal_table.entry(Bytecodes::_return).entry(vtos); >> >> last_sp ! = null not an issue for this case, so i skip the assert in debug build >> >> http://cr.openjdk.java.net/~jcm/8168712/webrev.01/ >> >> Please review. >> >> Best Regards, >> Jamsheed >> >> >> >> >> > From calvin.cheung at oracle.com Tue Oct 10 18:53:37 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 10 Oct 2017 11:53:37 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> Message-ID: <59DD1731.9090006@oracle.com> I ran into some runtime issue when creating the _java_platform_loader before initPhase2. I've filed the following to track the above issue: https://bugs.openjdk.java.net/browse/JDK-8189120 I'm going with the fix similar to version.02 - creating the system and platform loaders after initPhase3. updated webrev: http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ thanks, Calvin On 10/5/17, 10:38 PM, David Holmes wrote: > On 6/10/2017 3:28 PM, Calvin Cheung wrote: >> On 10/5/17, 6:33 PM, David Holmes wrote: >>> Hi Coleen, Calvin, >>> >>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>> So if you use -Djava.system.loader=myLoader on the command line, >>>> setting _java_system_loader, then does that mean that the classes >>>> loaded by >>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>> are not in the system loader? ie. they can be unloaded? What is >>>> the result of the call to >>>> SystemDictionary::is_system_class_loader() used for? I guess same >>>> question applies to the platform class loader. >>> >>> The classloading delegation hierarchy (as of JDK 9) is: >>> - boot loader (native) >>> - platform loader (built-in) >>> - system (aka application) loader (built-in) >>> >>> If the user specifies a custom class for the system loader then it >>> is loaded by an instance of the default system loader and becomes a >>> fourth level in the hierarchy, and it is then technically the >>> "system loader". None of these loaders, or the classes they load can >>> be unloaded. >>> >>> Which is presumably why the code checks both: >>> >>> 180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>> 181 if (class_loader == NULL) { >>> 182 return false; >>> 183 } >>> 184 return (class_loader->klass() == >>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>> || >>> 185 class_loader == _java_system_loader); >>> 186 } >>> >>> because we need to treat both of these instances as the "system >>> loader" from a VM perspective? The same does not apply to the >>> platform loader. >> We're obtaining the _java_system_loader after initPhase3 even before >> this change. Roughly, the calling sequence of initPhase3 is as follows: >> >> call_initPhase3() >> -> ClassLoader.initPhase3() >> -> ClassLoader.initSystemClassLoader() which contains the >> following code: >> >> String cn = System.getProperty("java.system.class.loader"); >> if (cn != null) { >> try { >> // custom class loader is only supported to be >> loaded from unnamed module >> Constructor ctor = Class.forName(cn, false, >> builtinLoader) >> .getDeclaredConstructor(ClassLoader.class); >> scl = (ClassLoader) ctor.newInstance(builtinLoader); >> } catch (Exception e) { >> throw new Error(e); >> } >> } else { >> scl = builtinLoader; >> } >> return scl; >> >> So initSystemClassLoader() will either return the built-in >> system loader or a custom loader if it exists. >> >> We use the getSystemClassLoader API to obtain the _java_system_loader: >> >> public static ClassLoader getSystemClassLoader() { >> switch (VM.initLevel()) { >> case 0: >> case 1: >> case 2: >> // the system class loader is the built-in app class >> loader during startup >> return getBuiltinAppClassLoader(); >> case 3: >> String msg = "getSystemClassLoader should only be >> called after VM booted"; >> throw new InternalError(msg); >> case 4: >> // system fully initialized >> assert VM.isBooted() && scl != null; >> SecurityManager sm = System.getSecurityManager(); >> if (sm != null) { >> checkClassLoaderPermission(scl, >> Reflection.getCallerClass()); >> } >> return scl; >> default: >> throw new InternalError("should not reach here"); >> } >> } >> >> So the _java_system_loader will either be the built-in system >> loader or a custom loader. (case 4 in the above) >> >> I don't quite understand why the check in line 184 is required? >> Is it for checking if a given class_loader is the same type >> (like an instanceof) as the built-in system loader? > > I believe it is checking if the loader is the built-in default system > loader, both to account for the case where/if > SystemDictionary::is_system_class_loader is called prior to initPhase3 > completing; and also to account for encountering the default-built-in > loader when the custom system loader delegates to it. > > I'd have to examine every call path to > SystemDictionary::is_system_class_loader to check all the details. > > David > ----- > >> thanks, >> Calvin >>> >>> David >>> ----- >>> >>>> thanks, >>>> Coleen >>>> >>>>> >>>>>> The implementation is in closed source. >>>>> Please clean up the closed code to remove those. >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Jiangli >>>>> >>>>>>> Is this new java_platform_loader function used anywhere? >>>>>> Yes, it is being used in closed source. >>>>>>> Currently >>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>>> is preloaded. Shouldn't this be removed? What about >>>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp >>>>>> and also in closed source. >>>>>>> thread.cpp >>>>>>> >>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>> >>>>>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? >>>>>>> Should it simply use CHECK_JNI_ERR as in other places? >>>>>> They are the same, in utilities/exceptions.hpp: >>>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>>> >>>>>> and it expands to the following: >>>>>> __the_thread__); if >>>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) >>>>>> return (-1); (void)(0 >>>>>> >>>>>> I can change it to CHECK_JNI_ERR. >>>>>> >>>>>> thanks, >>>>>> Calvin >>>>>>> Mandy >>>> From ioi.lam at oracle.com Tue Oct 10 20:02:11 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 10 Oct 2017 13:02:11 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59DD1731.9090006@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> Message-ID: <9e0154a8-2318-cd4d-d438-17cb417d9f3f@oracle.com> Looks good. Thanks! - Ioi On 10/10/17 11:53 AM, Calvin Cheung wrote: > I ran into some runtime issue when creating the _java_platform_loader > before initPhase2. > I've filed the following to track the above issue: > ??? https://bugs.openjdk.java.net/browse/JDK-8189120 > > I'm going with the fix similar to version.02 - creating the system and > platform loaders after initPhase3. > updated webrev: > ??? http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ > > thanks, > Calvin > > On 10/5/17, 10:38 PM, David Holmes wrote: >> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>> Hi Coleen, Calvin, >>>> >>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>>> So if you use -Djava.system.loader=myLoader on the command line, >>>>> setting _java_system_loader, then does that mean that the classes >>>>> loaded by >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>> are not in the system loader?? ie. they can be unloaded? What is >>>>> the result of the call to >>>>> SystemDictionary::is_system_class_loader() used for??? I guess >>>>> same question applies to the platform class loader. >>>> >>>> The classloading delegation hierarchy (as of JDK 9) is: >>>> - boot loader (native) >>>> ? - platform loader (built-in) >>>> ??? - system (aka application) loader (built-in) >>>> >>>> If the user specifies a custom class for the system loader then it >>>> is loaded by an instance of the default system loader and becomes a >>>> fourth level in the hierarchy, and it is then technically the >>>> "system loader". None of these loaders, or the classes they load >>>> can be unloaded. >>>> >>>> Which is presumably why the code checks both: >>>> >>>> ?180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>>> ?181?? if (class_loader == NULL) { >>>> ?182???? return false; >>>> ?183?? } >>>> ?184?? return (class_loader->klass() == >>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>> || >>>> ?185?????????? class_loader == _java_system_loader); >>>> ?186 } >>>> >>>> because we need to treat both of these instances as the "system >>>> loader" from a VM perspective? The same does not apply to the >>>> platform loader. >>> We're obtaining the _java_system_loader after initPhase3 even before >>> this change. Roughly, the calling sequence of initPhase3 is as follows: >>> >>> call_initPhase3() >>> ???? -> ClassLoader.initPhase3() >>> ???????? -> ClassLoader.initSystemClassLoader()? which contains the >>> following code: >>> >>> ???????? String cn = System.getProperty("java.system.class.loader"); >>> ???????? if (cn != null) { >>> ???????????? try { >>> ???????????????? // custom class loader is only supported to be >>> loaded from unnamed module >>> ???????????????? Constructor ctor = Class.forName(cn, false, >>> builtinLoader) >>> .getDeclaredConstructor(ClassLoader.class); >>> ???????????????? scl = (ClassLoader) ctor.newInstance(builtinLoader); >>> ???????????? } catch (Exception e) { >>> ???????????????? throw new Error(e); >>> ???????????? } >>> ???????? } else { >>> ???????????? scl = builtinLoader; >>> ???????? } >>> ???????? return scl; >>> >>> ???????? So initSystemClassLoader() will either return the built-in >>> system loader or a custom loader if it exists. >>> >>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>> >>> ???? public static ClassLoader getSystemClassLoader() { >>> ???????? switch (VM.initLevel()) { >>> ???????????? case 0: >>> ???????????? case 1: >>> ???????????? case 2: >>> ???????????????? // the system class loader is the built-in app >>> class loader during startup >>> ???????????????? return getBuiltinAppClassLoader(); >>> ???????????? case 3: >>> ???????????????? String msg = "getSystemClassLoader should only be >>> called after VM booted"; >>> ???????????????? throw new InternalError(msg); >>> ???????????? case 4: >>> ???????????????? // system fully initialized >>> ???????????????? assert VM.isBooted() && scl != null; >>> ???????????????? SecurityManager sm = System.getSecurityManager(); >>> ???????????????? if (sm != null) { >>> ???????????????????? checkClassLoaderPermission(scl, >>> Reflection.getCallerClass()); >>> ???????????????? } >>> ???????????????? return scl; >>> ???????????? default: >>> ???????????????? throw new InternalError("should not reach here"); >>> ???????? } >>> ???? } >>> >>> ???? So the _java_system_loader will either be the built-in system >>> loader or a custom loader. (case 4 in the above) >>> >>> ???? I don't quite understand why the check in line 184 is required? >>> ???? Is it for checking if a given class_loader is the same type >>> (like an instanceof) as the built-in system loader? >> >> I believe it is checking if the loader is the built-in default system >> loader, both to account for the case where/if >> SystemDictionary::is_system_class_loader is called prior to >> initPhase3 completing; and also to account for encountering the >> default-built-in loader when the custom system loader delegates to it. >> >> I'd have to examine every call path to >> SystemDictionary::is_system_class_loader to check all the details. >> >> David >> ----- >> >>> thanks, >>> Calvin >>>> >>>> David >>>> ----- >>>> >>>>> thanks, >>>>> Coleen >>>>> >>>>>> >>>>>>> The implementation is in closed source. >>>>>> Please clean up the closed code to remove those. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Jiangli >>>>>> >>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>> Yes, it is being used in closed source. >>>>>>>> Currently >>>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>>>> is preloaded.? Shouldn't this be removed?? What about >>>>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp >>>>>>> and also in closed source. >>>>>>>> thread.cpp >>>>>>>> >>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>> >>>>>>>> What is the difference between CHECK_(JNI_ERR) vs >>>>>>>> CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other >>>>>>>> places? >>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>>>> >>>>>>> and it expands to the following: >>>>>>> __the_thread__); if >>>>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) >>>>>>> return (-1); (void)(0 >>>>>>> >>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>> >>>>>>> thanks, >>>>>>> Calvin >>>>>>>> Mandy >>>>> From coleen.phillimore at oracle.com Tue Oct 10 20:21:34 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 10 Oct 2017 16:21:34 -0400 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59DD1731.9090006@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> Message-ID: <43ca20cd-0d38-4c5d-dfbb-baee9507e0d9@oracle.com> This looks good to me. Coleen On 10/10/17 2:53 PM, Calvin Cheung wrote: > I ran into some runtime issue when creating the _java_platform_loader > before initPhase2. > I've filed the following to track the above issue: > ??? https://bugs.openjdk.java.net/browse/JDK-8189120 > > I'm going with the fix similar to version.02 - creating the system and > platform loaders after initPhase3. > updated webrev: > ??? http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ > > thanks, > Calvin > > On 10/5/17, 10:38 PM, David Holmes wrote: >> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>> Hi Coleen, Calvin, >>>> >>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>>> So if you use -Djava.system.loader=myLoader on the command line, >>>>> setting _java_system_loader, then does that mean that the classes >>>>> loaded by >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>> are not in the system loader?? ie. they can be unloaded? What is >>>>> the result of the call to >>>>> SystemDictionary::is_system_class_loader() used for??? I guess >>>>> same question applies to the platform class loader. >>>> >>>> The classloading delegation hierarchy (as of JDK 9) is: >>>> - boot loader (native) >>>> ? - platform loader (built-in) >>>> ??? - system (aka application) loader (built-in) >>>> >>>> If the user specifies a custom class for the system loader then it >>>> is loaded by an instance of the default system loader and becomes a >>>> fourth level in the hierarchy, and it is then technically the >>>> "system loader". None of these loaders, or the classes they load >>>> can be unloaded. >>>> >>>> Which is presumably why the code checks both: >>>> >>>> ?180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>>> ?181?? if (class_loader == NULL) { >>>> ?182???? return false; >>>> ?183?? } >>>> ?184?? return (class_loader->klass() == >>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>> || >>>> ?185?????????? class_loader == _java_system_loader); >>>> ?186 } >>>> >>>> because we need to treat both of these instances as the "system >>>> loader" from a VM perspective? The same does not apply to the >>>> platform loader. >>> We're obtaining the _java_system_loader after initPhase3 even before >>> this change. Roughly, the calling sequence of initPhase3 is as follows: >>> >>> call_initPhase3() >>> ???? -> ClassLoader.initPhase3() >>> ???????? -> ClassLoader.initSystemClassLoader()? which contains the >>> following code: >>> >>> ???????? String cn = System.getProperty("java.system.class.loader"); >>> ???????? if (cn != null) { >>> ???????????? try { >>> ???????????????? // custom class loader is only supported to be >>> loaded from unnamed module >>> ???????????????? Constructor ctor = Class.forName(cn, false, >>> builtinLoader) >>> .getDeclaredConstructor(ClassLoader.class); >>> ???????????????? scl = (ClassLoader) ctor.newInstance(builtinLoader); >>> ???????????? } catch (Exception e) { >>> ???????????????? throw new Error(e); >>> ???????????? } >>> ???????? } else { >>> ???????????? scl = builtinLoader; >>> ???????? } >>> ???????? return scl; >>> >>> ???????? So initSystemClassLoader() will either return the built-in >>> system loader or a custom loader if it exists. >>> >>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>> >>> ???? public static ClassLoader getSystemClassLoader() { >>> ???????? switch (VM.initLevel()) { >>> ???????????? case 0: >>> ???????????? case 1: >>> ???????????? case 2: >>> ???????????????? // the system class loader is the built-in app >>> class loader during startup >>> ???????????????? return getBuiltinAppClassLoader(); >>> ???????????? case 3: >>> ???????????????? String msg = "getSystemClassLoader should only be >>> called after VM booted"; >>> ???????????????? throw new InternalError(msg); >>> ???????????? case 4: >>> ???????????????? // system fully initialized >>> ???????????????? assert VM.isBooted() && scl != null; >>> ???????????????? SecurityManager sm = System.getSecurityManager(); >>> ???????????????? if (sm != null) { >>> ???????????????????? checkClassLoaderPermission(scl, >>> Reflection.getCallerClass()); >>> ???????????????? } >>> ???????????????? return scl; >>> ???????????? default: >>> ???????????????? throw new InternalError("should not reach here"); >>> ???????? } >>> ???? } >>> >>> ???? So the _java_system_loader will either be the built-in system >>> loader or a custom loader. (case 4 in the above) >>> >>> ???? I don't quite understand why the check in line 184 is required? >>> ???? Is it for checking if a given class_loader is the same type >>> (like an instanceof) as the built-in system loader? >> >> I believe it is checking if the loader is the built-in default system >> loader, both to account for the case where/if >> SystemDictionary::is_system_class_loader is called prior to >> initPhase3 completing; and also to account for encountering the >> default-built-in loader when the custom system loader delegates to it. >> >> I'd have to examine every call path to >> SystemDictionary::is_system_class_loader to check all the details. >> >> David >> ----- >> >>> thanks, >>> Calvin >>>> >>>> David >>>> ----- >>>> >>>>> thanks, >>>>> Coleen >>>>> >>>>>> >>>>>>> The implementation is in closed source. >>>>>> Please clean up the closed code to remove those. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Jiangli >>>>>> >>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>> Yes, it is being used in closed source. >>>>>>>> Currently >>>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>>>> is preloaded.? Shouldn't this be removed?? What about >>>>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp >>>>>>> and also in closed source. >>>>>>>> thread.cpp >>>>>>>> >>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>> >>>>>>>> What is the difference between CHECK_(JNI_ERR) vs >>>>>>>> CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other >>>>>>>> places? >>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>>>> >>>>>>> and it expands to the following: >>>>>>> __the_thread__); if >>>>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) >>>>>>> return (-1); (void)(0 >>>>>>> >>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>> >>>>>>> thanks, >>>>>>> Calvin >>>>>>>> Mandy >>>>> From calvin.cheung at oracle.com Tue Oct 10 20:38:33 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 10 Oct 2017 13:38:33 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <9e0154a8-2318-cd4d-d438-17cb417d9f3f@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> <9e0154a8-2318-cd4d-d438-17cb417d9f3f@oracle.com> Message-ID: <59DD2FC9.5070203@oracle.com> Thanks Ioi. Calvin On 10/10/17, 1:02 PM, Ioi Lam wrote: > Looks good. Thanks! > > - Ioi > > > On 10/10/17 11:53 AM, Calvin Cheung wrote: >> I ran into some runtime issue when creating the _java_platform_loader >> before initPhase2. >> I've filed the following to track the above issue: >> https://bugs.openjdk.java.net/browse/JDK-8189120 >> >> I'm going with the fix similar to version.02 - creating the system >> and platform loaders after initPhase3. >> updated webrev: >> http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ >> >> thanks, >> Calvin >> >> On 10/5/17, 10:38 PM, David Holmes wrote: >>> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>>> Hi Coleen, Calvin, >>>>> >>>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>>>> So if you use -Djava.system.loader=myLoader on the command line, >>>>>> setting _java_system_loader, then does that mean that the classes >>>>>> loaded by >>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>>> are not in the system loader? ie. they can be unloaded? What is >>>>>> the result of the call to >>>>>> SystemDictionary::is_system_class_loader() used for? I guess >>>>>> same question applies to the platform class loader. >>>>> >>>>> The classloading delegation hierarchy (as of JDK 9) is: >>>>> - boot loader (native) >>>>> - platform loader (built-in) >>>>> - system (aka application) loader (built-in) >>>>> >>>>> If the user specifies a custom class for the system loader then it >>>>> is loaded by an instance of the default system loader and becomes >>>>> a fourth level in the hierarchy, and it is then technically the >>>>> "system loader". None of these loaders, or the classes they load >>>>> can be unloaded. >>>>> >>>>> Which is presumably why the code checks both: >>>>> >>>>> 180 bool SystemDictionary::is_system_class_loader(oop >>>>> class_loader) { >>>>> 181 if (class_loader == NULL) { >>>>> 182 return false; >>>>> 183 } >>>>> 184 return (class_loader->klass() == >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>> || >>>>> 185 class_loader == _java_system_loader); >>>>> 186 } >>>>> >>>>> because we need to treat both of these instances as the "system >>>>> loader" from a VM perspective? The same does not apply to the >>>>> platform loader. >>>> We're obtaining the _java_system_loader after initPhase3 even >>>> before this change. Roughly, the calling sequence of initPhase3 is >>>> as follows: >>>> >>>> call_initPhase3() >>>> -> ClassLoader.initPhase3() >>>> -> ClassLoader.initSystemClassLoader() which contains the >>>> following code: >>>> >>>> String cn = System.getProperty("java.system.class.loader"); >>>> if (cn != null) { >>>> try { >>>> // custom class loader is only supported to be >>>> loaded from unnamed module >>>> Constructor ctor = Class.forName(cn, false, >>>> builtinLoader) >>>> .getDeclaredConstructor(ClassLoader.class); >>>> scl = (ClassLoader) ctor.newInstance(builtinLoader); >>>> } catch (Exception e) { >>>> throw new Error(e); >>>> } >>>> } else { >>>> scl = builtinLoader; >>>> } >>>> return scl; >>>> >>>> So initSystemClassLoader() will either return the built-in >>>> system loader or a custom loader if it exists. >>>> >>>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>>> >>>> public static ClassLoader getSystemClassLoader() { >>>> switch (VM.initLevel()) { >>>> case 0: >>>> case 1: >>>> case 2: >>>> // the system class loader is the built-in app >>>> class loader during startup >>>> return getBuiltinAppClassLoader(); >>>> case 3: >>>> String msg = "getSystemClassLoader should only be >>>> called after VM booted"; >>>> throw new InternalError(msg); >>>> case 4: >>>> // system fully initialized >>>> assert VM.isBooted() && scl != null; >>>> SecurityManager sm = System.getSecurityManager(); >>>> if (sm != null) { >>>> checkClassLoaderPermission(scl, >>>> Reflection.getCallerClass()); >>>> } >>>> return scl; >>>> default: >>>> throw new InternalError("should not reach here"); >>>> } >>>> } >>>> >>>> So the _java_system_loader will either be the built-in system >>>> loader or a custom loader. (case 4 in the above) >>>> >>>> I don't quite understand why the check in line 184 is required? >>>> Is it for checking if a given class_loader is the same type >>>> (like an instanceof) as the built-in system loader? >>> >>> I believe it is checking if the loader is the built-in default >>> system loader, both to account for the case where/if >>> SystemDictionary::is_system_class_loader is called prior to >>> initPhase3 completing; and also to account for encountering the >>> default-built-in loader when the custom system loader delegates to it. >>> >>> I'd have to examine every call path to >>> SystemDictionary::is_system_class_loader to check all the details. >>> >>> David >>> ----- >>> >>>> thanks, >>>> Calvin >>>>> >>>>> David >>>>> ----- >>>>> >>>>>> thanks, >>>>>> Coleen >>>>>> >>>>>>> >>>>>>>> The implementation is in closed source. >>>>>>> Please clean up the closed code to remove those. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Jiangli >>>>>>> >>>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>>> Yes, it is being used in closed source. >>>>>>>>> Currently >>>>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>>>>> is preloaded. Shouldn't this be removed? What about >>>>>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp >>>>>>>> and also in closed source. >>>>>>>>> thread.cpp >>>>>>>>> >>>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>>> >>>>>>>>> What is the difference between CHECK_(JNI_ERR) vs >>>>>>>>> CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other >>>>>>>>> places? >>>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>>>>> >>>>>>>> and it expands to the following: >>>>>>>> __the_thread__); if >>>>>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) >>>>>>>> return (-1); (void)(0 >>>>>>>> >>>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>>> >>>>>>>> thanks, >>>>>>>> Calvin >>>>>>>>> Mandy >>>>>> > From calvin.cheung at oracle.com Tue Oct 10 20:39:17 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 10 Oct 2017 13:39:17 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <43ca20cd-0d38-4c5d-dfbb-baee9507e0d9@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> <43ca20cd-0d38-4c5d-dfbb-baee9507e0d9@oracle.com> Message-ID: <59DD2FF5.4010406@oracle.com> Thanks Coleen. Calvin On 10/10/17, 1:21 PM, coleen.phillimore at oracle.com wrote: > > This looks good to me. > > Coleen > > On 10/10/17 2:53 PM, Calvin Cheung wrote: >> I ran into some runtime issue when creating the _java_platform_loader >> before initPhase2. >> I've filed the following to track the above issue: >> https://bugs.openjdk.java.net/browse/JDK-8189120 >> >> I'm going with the fix similar to version.02 - creating the system >> and platform loaders after initPhase3. >> updated webrev: >> http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ >> >> thanks, >> Calvin >> >> On 10/5/17, 10:38 PM, David Holmes wrote: >>> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>>> Hi Coleen, Calvin, >>>>> >>>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>>>> So if you use -Djava.system.loader=myLoader on the command line, >>>>>> setting _java_system_loader, then does that mean that the classes >>>>>> loaded by >>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>>> are not in the system loader? ie. they can be unloaded? What is >>>>>> the result of the call to >>>>>> SystemDictionary::is_system_class_loader() used for? I guess >>>>>> same question applies to the platform class loader. >>>>> >>>>> The classloading delegation hierarchy (as of JDK 9) is: >>>>> - boot loader (native) >>>>> - platform loader (built-in) >>>>> - system (aka application) loader (built-in) >>>>> >>>>> If the user specifies a custom class for the system loader then it >>>>> is loaded by an instance of the default system loader and becomes >>>>> a fourth level in the hierarchy, and it is then technically the >>>>> "system loader". None of these loaders, or the classes they load >>>>> can be unloaded. >>>>> >>>>> Which is presumably why the code checks both: >>>>> >>>>> 180 bool SystemDictionary::is_system_class_loader(oop >>>>> class_loader) { >>>>> 181 if (class_loader == NULL) { >>>>> 182 return false; >>>>> 183 } >>>>> 184 return (class_loader->klass() == >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>> || >>>>> 185 class_loader == _java_system_loader); >>>>> 186 } >>>>> >>>>> because we need to treat both of these instances as the "system >>>>> loader" from a VM perspective? The same does not apply to the >>>>> platform loader. >>>> We're obtaining the _java_system_loader after initPhase3 even >>>> before this change. Roughly, the calling sequence of initPhase3 is >>>> as follows: >>>> >>>> call_initPhase3() >>>> -> ClassLoader.initPhase3() >>>> -> ClassLoader.initSystemClassLoader() which contains the >>>> following code: >>>> >>>> String cn = System.getProperty("java.system.class.loader"); >>>> if (cn != null) { >>>> try { >>>> // custom class loader is only supported to be >>>> loaded from unnamed module >>>> Constructor ctor = Class.forName(cn, false, >>>> builtinLoader) >>>> .getDeclaredConstructor(ClassLoader.class); >>>> scl = (ClassLoader) ctor.newInstance(builtinLoader); >>>> } catch (Exception e) { >>>> throw new Error(e); >>>> } >>>> } else { >>>> scl = builtinLoader; >>>> } >>>> return scl; >>>> >>>> So initSystemClassLoader() will either return the built-in >>>> system loader or a custom loader if it exists. >>>> >>>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>>> >>>> public static ClassLoader getSystemClassLoader() { >>>> switch (VM.initLevel()) { >>>> case 0: >>>> case 1: >>>> case 2: >>>> // the system class loader is the built-in app >>>> class loader during startup >>>> return getBuiltinAppClassLoader(); >>>> case 3: >>>> String msg = "getSystemClassLoader should only be >>>> called after VM booted"; >>>> throw new InternalError(msg); >>>> case 4: >>>> // system fully initialized >>>> assert VM.isBooted() && scl != null; >>>> SecurityManager sm = System.getSecurityManager(); >>>> if (sm != null) { >>>> checkClassLoaderPermission(scl, >>>> Reflection.getCallerClass()); >>>> } >>>> return scl; >>>> default: >>>> throw new InternalError("should not reach here"); >>>> } >>>> } >>>> >>>> So the _java_system_loader will either be the built-in system >>>> loader or a custom loader. (case 4 in the above) >>>> >>>> I don't quite understand why the check in line 184 is required? >>>> Is it for checking if a given class_loader is the same type >>>> (like an instanceof) as the built-in system loader? >>> >>> I believe it is checking if the loader is the built-in default >>> system loader, both to account for the case where/if >>> SystemDictionary::is_system_class_loader is called prior to >>> initPhase3 completing; and also to account for encountering the >>> default-built-in loader when the custom system loader delegates to it. >>> >>> I'd have to examine every call path to >>> SystemDictionary::is_system_class_loader to check all the details. >>> >>> David >>> ----- >>> >>>> thanks, >>>> Calvin >>>>> >>>>> David >>>>> ----- >>>>> >>>>>> thanks, >>>>>> Coleen >>>>>> >>>>>>> >>>>>>>> The implementation is in closed source. >>>>>>> Please clean up the closed code to remove those. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Jiangli >>>>>>> >>>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>>> Yes, it is being used in closed source. >>>>>>>>> Currently >>>>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>>>>> is preloaded. Shouldn't this be removed? What about >>>>>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp >>>>>>>> and also in closed source. >>>>>>>>> thread.cpp >>>>>>>>> >>>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>>> >>>>>>>>> What is the difference between CHECK_(JNI_ERR) vs >>>>>>>>> CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other >>>>>>>>> places? >>>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>>>>> >>>>>>>> and it expands to the following: >>>>>>>> __the_thread__); if >>>>>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) >>>>>>>> return (-1); (void)(0 >>>>>>>> >>>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>>> >>>>>>>> thanks, >>>>>>>> Calvin >>>>>>>>> Mandy >>>>>> > From gerard.ziemski at oracle.com Tue Oct 10 20:40:15 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Tue, 10 Oct 2017 15:40:15 -0500 Subject: RFR: 8184765: Dynamically resize SystemDictionary Message-ID: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> hi all, Please review this change that adds dynamic resizing of system dictionaries. The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock A few notes: - dynamic resizing is off when either dumping or using shared spaces - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) bug: https://bugs.openjdk.java.net/browse/JDK-8184765 webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 cheers From peter.levart at gmail.com Tue Oct 10 21:28:51 2017 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 10 Oct 2017 23:28:51 +0200 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: <96cf434b-aa72-158f-515f-089a63c25c4d@oracle.com> References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> <57e0b49a-bed6-25c2-7d01-b87f08d82a75@oracle.com> <376a0876-d94b-8775-873c-bfd741614273@gmail.com> <96cf434b-aa72-158f-515f-089a63c25c4d@oracle.com> Message-ID: Hi Mandy, On 10/10/17 17:00, mandy chung wrote: >> Now that (system)nativeLibraries is a Map, is it still necessary to >> iterate it and call lib.findEntry(name) on each NativeLibrary until >> the one that returns a non-zero entry or would it be semantically >> equivalent to 1st look-up the Map by the 'name' key to get the >> correct NativeLibrary? >> > The name parameter is the symbol name but not the library name. It's > confusing and therefore I renamed NativeLibrary::find to > NativeLibrary::findEntry.?? I could rename the name parameter to > entryName to make it clearer. > > Mandy Oh, I see. I haven't checked the findNative VM call-back method purpose before. It's reasonable to try all the native libraries of a ClassLoader when looking for a symbol that links to native method implementation. Constructing and keeping an index of symbols for each native library on the Java side is probably not worth since there might be lots of them in libraries and only some of them link to native method implementations and there's typically not many native libraries loaded. Regards, Peter From mandy.chung at oracle.com Tue Oct 10 21:31:55 2017 From: mandy.chung at oracle.com (mandy chung) Date: Tue, 10 Oct 2017 14:31:55 -0700 Subject: Review Request JDK-8164512: Replace ClassLoader use of finalizer with phantom reference to unload native library In-Reply-To: References: <1643128c-8714-4d6d-253a-b7413a5eb8ef@oracle.com> <859f3acc-2330-9c9c-39c8-be894f007585@oracle.com> <8d22132f-b8f4-a65e-88ad-f9af3a6d49a4@oracle.com> <2cadf6bf-6fd5-5c7a-f9f3-53031ee50a9b@oracle.com> <4eeb8628-28af-3c19-7ed6-feb179ed5620@oracle.com> <33edcd76-d057-45cf-2edd-b461d27214b5@oracle.com> <57e0b49a-bed6-25c2-7d01-b87f08d82a75@oracle.com> <376a0876-d94b-8775-873c-bfd741614273@gmail.com> <96cf434b-aa72-158f-515f-089a63c25c4d@oracle.com> Message-ID: On 10/10/17 2:28 PM, Peter Levart wrote: > Hi Mandy, > > On 10/10/17 17:00, mandy chung wrote: >>> Now that (system)nativeLibraries is a Map, is it still necessary to >>> iterate it and call lib.findEntry(name) on each NativeLibrary until >>> the one that returns a non-zero entry or would it be semantically >>> equivalent to 1st look-up the Map by the 'name' key to get the >>> correct NativeLibrary? >>> >> The name parameter is the symbol name but not the library name. It's >> confusing and therefore I renamed NativeLibrary::find to >> NativeLibrary::findEntry.?? I could rename the name parameter to >> entryName to make it clearer. >> >> Mandy > > Oh, I see. I haven't checked the findNative VM call-back method > purpose before. It's reasonable to try all the native libraries of a > ClassLoader when looking for a symbol that links to native method > implementation. Constructing and keeping an index of symbols for each > native library on the Java side is probably not worth since there > might be lots of them in libraries and only some of them link to > native method implementations and there's typically not many native > libraries loaded. Exactly. Mandy From jiangli.zhou at oracle.com Tue Oct 10 21:38:55 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Tue, 10 Oct 2017 14:38:55 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59DD1731.9090006@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> Message-ID: <0DA5ECC0-92CF-4E83-A99C-0A01E256D79A@oracle.com> Hi Calvin, > On Oct 10, 2017, at 11:53 AM, Calvin Cheung wrote: > > I ran into some runtime issue when creating the _java_platform_loader before initPhase2. > I've filed the following to track the above issue: > https://bugs.openjdk.java.net/browse/JDK-8189120 > > I'm going with the fix similar to version.02 - creating the system and platform loaders after initPhase3. > updated webrev: > http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ This looks good to me. SystemDictionary::compyte_java_loader() calls CDS_ONLY(SystemDictionaryShared::initialize()). The function name initialize() is misleading. Since you are touching the related code, could you please file a bug so we can clean up SystemDictionaryShared::initialize() API in the near future? Thanks, Jiangli > > thanks, > Calvin > > On 10/5/17, 10:38 PM, David Holmes wrote: >> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>> Hi Coleen, Calvin, >>>> >>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>>> So if you use -Djava.system.loader=myLoader on the command line, setting _java_system_loader, then does that mean that the classes loaded by SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() are not in the system loader? ie. they can be unloaded? What is the result of the call to SystemDictionary::is_system_class_loader() used for? I guess same question applies to the platform class loader. >>>> >>>> The classloading delegation hierarchy (as of JDK 9) is: >>>> - boot loader (native) >>>> - platform loader (built-in) >>>> - system (aka application) loader (built-in) >>>> >>>> If the user specifies a custom class for the system loader then it is loaded by an instance of the default system loader and becomes a fourth level in the hierarchy, and it is then technically the "system loader". None of these loaders, or the classes they load can be unloaded. >>>> >>>> Which is presumably why the code checks both: >>>> >>>> 180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>>> 181 if (class_loader == NULL) { >>>> 182 return false; >>>> 183 } >>>> 184 return (class_loader->klass() == SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() || >>>> 185 class_loader == _java_system_loader); >>>> 186 } >>>> >>>> because we need to treat both of these instances as the "system loader" from a VM perspective? The same does not apply to the platform loader. >>> We're obtaining the _java_system_loader after initPhase3 even before this change. Roughly, the calling sequence of initPhase3 is as follows: >>> >>> call_initPhase3() >>> -> ClassLoader.initPhase3() >>> -> ClassLoader.initSystemClassLoader() which contains the following code: >>> >>> String cn = System.getProperty("java.system.class.loader"); >>> if (cn != null) { >>> try { >>> // custom class loader is only supported to be loaded from unnamed module >>> Constructor ctor = Class.forName(cn, false, builtinLoader) >>> .getDeclaredConstructor(ClassLoader.class); >>> scl = (ClassLoader) ctor.newInstance(builtinLoader); >>> } catch (Exception e) { >>> throw new Error(e); >>> } >>> } else { >>> scl = builtinLoader; >>> } >>> return scl; >>> >>> So initSystemClassLoader() will either return the built-in system loader or a custom loader if it exists. >>> >>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>> >>> public static ClassLoader getSystemClassLoader() { >>> switch (VM.initLevel()) { >>> case 0: >>> case 1: >>> case 2: >>> // the system class loader is the built-in app class loader during startup >>> return getBuiltinAppClassLoader(); >>> case 3: >>> String msg = "getSystemClassLoader should only be called after VM booted"; >>> throw new InternalError(msg); >>> case 4: >>> // system fully initialized >>> assert VM.isBooted() && scl != null; >>> SecurityManager sm = System.getSecurityManager(); >>> if (sm != null) { >>> checkClassLoaderPermission(scl, Reflection.getCallerClass()); >>> } >>> return scl; >>> default: >>> throw new InternalError("should not reach here"); >>> } >>> } >>> >>> So the _java_system_loader will either be the built-in system loader or a custom loader. (case 4 in the above) >>> >>> I don't quite understand why the check in line 184 is required? >>> Is it for checking if a given class_loader is the same type (like an instanceof) as the built-in system loader? >> >> I believe it is checking if the loader is the built-in default system loader, both to account for the case where/if SystemDictionary::is_system_class_loader is called prior to initPhase3 completing; and also to account for encountering the default-built-in loader when the custom system loader delegates to it. >> >> I'd have to examine every call path to SystemDictionary::is_system_class_loader to check all the details. >> >> David >> ----- >> >>> thanks, >>> Calvin >>>> >>>> David >>>> ----- >>>> >>>>> thanks, >>>>> Coleen >>>>> >>>>>> >>>>>>> The implementation is in closed source. >>>>>> Please clean up the closed code to remove those. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Jiangli >>>>>> >>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>> Yes, it is being used in closed source. >>>>>>>> Currently SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass is preloaded. Shouldn't this be removed? What about jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp and also in closed source. >>>>>>>> thread.cpp >>>>>>>> >>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>> >>>>>>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other places? >>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>>>> >>>>>>> and it expands to the following: >>>>>>> __the_thread__); if ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return (-1); (void)(0 >>>>>>> >>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>> >>>>>>> thanks, >>>>>>> Calvin >>>>>>>> Mandy >>>>> From calvin.cheung at oracle.com Tue Oct 10 21:49:58 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 10 Oct 2017 14:49:58 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <0DA5ECC0-92CF-4E83-A99C-0A01E256D79A@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> <0DA5ECC0-92CF-4E83-A99C-0A01E256D79A@oracle.com> Message-ID: <59DD4086.2080704@oracle.com> On 10/10/17, 2:38 PM, Jiangli Zhou wrote: > Hi Calvin, > >> On Oct 10, 2017, at 11:53 AM, Calvin Cheung > > wrote: >> >> I ran into some runtime issue when creating the _java_platform_loader >> before initPhase2. >> I've filed the following to track the above issue: >> https://bugs.openjdk.java.net/browse/JDK-8189120 >> >> I'm going with the fix similar to version.02 - creating the system >> and platform loaders after initPhase3. >> updated webrev: >> http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ >> > > This looks good to me. Thanks for taking another look. > > SystemDictionary::compyte_java_loader() calls > CDS_ONLY(SystemDictionaryShared::initialize()). The function name > initialize() is misleading. The initialize() function is still initializing some classes. So I don't think the name is misleading. > Since you are touching the related code, could you please file a bug > so we can clean up SystemDictionaryShared::initialize() API in the > near future? If you don't mind, could you file one? thanks, Calvin > > Thanks, > Jiangli > >> >> thanks, >> Calvin >> >> On 10/5/17, 10:38 PM, David Holmes wrote: >>> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>>> Hi Coleen, Calvin, >>>>> >>>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com >>>>> wrote: >>>>>> So if you use -Djava.system.loader=myLoader on the command line, >>>>>> setting _java_system_loader, then does that mean that the classes >>>>>> loaded by >>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>>> are not in the system loader? ie. they can be unloaded? What is >>>>>> the result of the call to >>>>>> SystemDictionary::is_system_class_loader() used for? I guess >>>>>> same question applies to the platform class loader. >>>>> >>>>> The classloading delegation hierarchy (as of JDK 9) is: >>>>> - boot loader (native) >>>>> - platform loader (built-in) >>>>> - system (aka application) loader (built-in) >>>>> >>>>> If the user specifies a custom class for the system loader then it >>>>> is loaded by an instance of the default system loader and becomes >>>>> a fourth level in the hierarchy, and it is then technically the >>>>> "system loader". None of these loaders, or the classes they load >>>>> can be unloaded. >>>>> >>>>> Which is presumably why the code checks both: >>>>> >>>>> 180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>>>> 181 if (class_loader == NULL) { >>>>> 182 return false; >>>>> 183 } >>>>> 184 return (class_loader->klass() == >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>> || >>>>> 185 class_loader == _java_system_loader); >>>>> 186 } >>>>> >>>>> because we need to treat both of these instances as the "system >>>>> loader" from a VM perspective? The same does not apply to the >>>>> platform loader. >>>> We're obtaining the _java_system_loader after initPhase3 even >>>> before this change. Roughly, the calling sequence of initPhase3 is >>>> as follows: >>>> >>>> call_initPhase3() >>>> -> ClassLoader.initPhase3() >>>> -> ClassLoader.initSystemClassLoader() which contains the >>>> following code: >>>> >>>> String cn = System.getProperty("java.system.class.loader"); >>>> if (cn != null) { >>>> try { >>>> // custom class loader is only supported to be >>>> loaded from unnamed module >>>> Constructor ctor = Class.forName(cn, false, >>>> builtinLoader) >>>> .getDeclaredConstructor(ClassLoader.class); >>>> scl = (ClassLoader) ctor.newInstance(builtinLoader); >>>> } catch (Exception e) { >>>> throw new Error(e); >>>> } >>>> } else { >>>> scl = builtinLoader; >>>> } >>>> return scl; >>>> >>>> So initSystemClassLoader() will either return the built-in >>>> system loader or a custom loader if it exists. >>>> >>>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>>> >>>> public static ClassLoader getSystemClassLoader() { >>>> switch (VM.initLevel()) { >>>> case 0: >>>> case 1: >>>> case 2: >>>> // the system class loader is the built-in app >>>> class loader during startup >>>> return getBuiltinAppClassLoader(); >>>> case 3: >>>> String msg = "getSystemClassLoader should only be >>>> called after VM booted"; >>>> throw new InternalError(msg); >>>> case 4: >>>> // system fully initialized >>>> assert VM.isBooted() && scl != null; >>>> SecurityManager sm = System.getSecurityManager(); >>>> if (sm != null) { >>>> checkClassLoaderPermission(scl, >>>> Reflection.getCallerClass()); >>>> } >>>> return scl; >>>> default: >>>> throw new InternalError("should not reach here"); >>>> } >>>> } >>>> >>>> So the _java_system_loader will either be the built-in system >>>> loader or a custom loader. (case 4 in the above) >>>> >>>> I don't quite understand why the check in line 184 is required? >>>> Is it for checking if a given class_loader is the same type >>>> (like an instanceof) as the built-in system loader? >>> >>> I believe it is checking if the loader is the built-in default >>> system loader, both to account for the case where/if >>> SystemDictionary::is_system_class_loader is called prior to >>> initPhase3 completing; and also to account for encountering the >>> default-built-in loader when the custom system loader delegates to it. >>> >>> I'd have to examine every call path to >>> SystemDictionary::is_system_class_loader to check all the details. >>> >>> David >>> ----- >>> >>>> thanks, >>>> Calvin >>>>> >>>>> David >>>>> ----- >>>>> >>>>>> thanks, >>>>>> Coleen >>>>>> >>>>>>> >>>>>>>> The implementation is in closed source. >>>>>>> Please clean up the closed code to remove those. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Jiangli >>>>>>> >>>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>>> Yes, it is being used in closed source. >>>>>>>>> Currently >>>>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>>>>> is preloaded. Shouldn't this be removed? What about >>>>>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp >>>>>>>> and also in closed source. >>>>>>>>> thread.cpp >>>>>>>>> >>>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>>> >>>>>>>>> What is the difference between CHECK_(JNI_ERR) vs >>>>>>>>> CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other >>>>>>>>> places? >>>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>>>>> >>>>>>>> and it expands to the following: >>>>>>>> __the_thread__); if >>>>>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) >>>>>>>> return (-1); (void)(0 >>>>>>>> >>>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>>> >>>>>>>> thanks, >>>>>>>> Calvin >>>>>>>>> Mandy >>>>>> > From jiangli.zhou at oracle.com Tue Oct 10 23:40:06 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Tue, 10 Oct 2017 16:40:06 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59DD4086.2080704@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> <0DA5ECC0-92CF-4E83-A99C-0A01E256D79A@oracle.com> <59DD4086.2080704@oracle.com> Message-ID: <373AABB4-42D6-458C-8F50-D2CD110ACF2C@oracle.com> I filed JDK-8189140 . Thanks, Jiangli > On Oct 10, 2017, at 2:49 PM, Calvin Cheung wrote: > > > > On 10/10/17, 2:38 PM, Jiangli Zhou wrote: >> Hi Calvin, >> >>> On Oct 10, 2017, at 11:53 AM, Calvin Cheung > wrote: >>> >>> I ran into some runtime issue when creating the _java_platform_loader before initPhase2. >>> I've filed the following to track the above issue: >>> https://bugs.openjdk.java.net/browse/JDK-8189120 >>> >>> I'm going with the fix similar to version.02 - creating the system and platform loaders after initPhase3. >>> updated webrev: >>> http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ >> >> This looks good to me. > Thanks for taking another look. >> >> SystemDictionary::compyte_java_loader() calls CDS_ONLY(SystemDictionaryShared::initialize()). The function name initialize() is misleading. > The initialize() function is still initializing some classes. So I don't think the name is misleading. >> Since you are touching the related code, could you please file a bug so we can clean up SystemDictionaryShared::initialize() API in the near future? > If you don't mind, could you file one? > > thanks, > Calvin >> >> Thanks, >> Jiangli >> >>> >>> thanks, >>> Calvin >>> >>> On 10/5/17, 10:38 PM, David Holmes wrote: >>>> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>>>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>>>> Hi Coleen, Calvin, >>>>>> >>>>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>>>>> So if you use -Djava.system.loader=myLoader on the command line, setting _java_system_loader, then does that mean that the classes loaded by SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() are not in the system loader? ie. they can be unloaded? What is the result of the call to SystemDictionary::is_system_class_loader() used for? I guess same question applies to the platform class loader. >>>>>> >>>>>> The classloading delegation hierarchy (as of JDK 9) is: >>>>>> - boot loader (native) >>>>>> - platform loader (built-in) >>>>>> - system (aka application) loader (built-in) >>>>>> >>>>>> If the user specifies a custom class for the system loader then it is loaded by an instance of the default system loader and becomes a fourth level in the hierarchy, and it is then technically the "system loader". None of these loaders, or the classes they load can be unloaded. >>>>>> >>>>>> Which is presumably why the code checks both: >>>>>> >>>>>> 180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>>>>> 181 if (class_loader == NULL) { >>>>>> 182 return false; >>>>>> 183 } >>>>>> 184 return (class_loader->klass() == SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() || >>>>>> 185 class_loader == _java_system_loader); >>>>>> 186 } >>>>>> >>>>>> because we need to treat both of these instances as the "system loader" from a VM perspective? The same does not apply to the platform loader. >>>>> We're obtaining the _java_system_loader after initPhase3 even before this change. Roughly, the calling sequence of initPhase3 is as follows: >>>>> >>>>> call_initPhase3() >>>>> -> ClassLoader.initPhase3() >>>>> -> ClassLoader.initSystemClassLoader() which contains the following code: >>>>> >>>>> String cn = System.getProperty("java.system.class.loader"); >>>>> if (cn != null) { >>>>> try { >>>>> // custom class loader is only supported to be loaded from unnamed module >>>>> Constructor ctor = Class.forName(cn, false, builtinLoader) >>>>> .getDeclaredConstructor(ClassLoader.class); >>>>> scl = (ClassLoader) ctor.newInstance(builtinLoader); >>>>> } catch (Exception e) { >>>>> throw new Error(e); >>>>> } >>>>> } else { >>>>> scl = builtinLoader; >>>>> } >>>>> return scl; >>>>> >>>>> So initSystemClassLoader() will either return the built-in system loader or a custom loader if it exists. >>>>> >>>>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>>>> >>>>> public static ClassLoader getSystemClassLoader() { >>>>> switch (VM.initLevel()) { >>>>> case 0: >>>>> case 1: >>>>> case 2: >>>>> // the system class loader is the built-in app class loader during startup >>>>> return getBuiltinAppClassLoader(); >>>>> case 3: >>>>> String msg = "getSystemClassLoader should only be called after VM booted"; >>>>> throw new InternalError(msg); >>>>> case 4: >>>>> // system fully initialized >>>>> assert VM.isBooted() && scl != null; >>>>> SecurityManager sm = System.getSecurityManager(); >>>>> if (sm != null) { >>>>> checkClassLoaderPermission(scl, Reflection.getCallerClass()); >>>>> } >>>>> return scl; >>>>> default: >>>>> throw new InternalError("should not reach here"); >>>>> } >>>>> } >>>>> >>>>> So the _java_system_loader will either be the built-in system loader or a custom loader. (case 4 in the above) >>>>> >>>>> I don't quite understand why the check in line 184 is required? >>>>> Is it for checking if a given class_loader is the same type (like an instanceof) as the built-in system loader? >>>> >>>> I believe it is checking if the loader is the built-in default system loader, both to account for the case where/if SystemDictionary::is_system_class_loader is called prior to initPhase3 completing; and also to account for encountering the default-built-in loader when the custom system loader delegates to it. >>>> >>>> I'd have to examine every call path to SystemDictionary::is_system_class_loader to check all the details. >>>> >>>> David >>>> ----- >>>> >>>>> thanks, >>>>> Calvin >>>>>> >>>>>> David >>>>>> ----- >>>>>> >>>>>>> thanks, >>>>>>> Coleen >>>>>>> >>>>>>>> >>>>>>>>> The implementation is in closed source. >>>>>>>> Please clean up the closed code to remove those. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Jiangli >>>>>>>> >>>>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>>>> Yes, it is being used in closed source. >>>>>>>>>> Currently SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass is preloaded. Shouldn't this be removed? What about jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp and also in closed source. >>>>>>>>>> thread.cpp >>>>>>>>>> >>>>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>>>> >>>>>>>>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other places? >>>>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>>>>>> >>>>>>>>> and it expands to the following: >>>>>>>>> __the_thread__); if ((((ThreadShadow*)__the_thread__)->has_pending_exception())) return (-1); (void)(0 >>>>>>>>> >>>>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>>>> >>>>>>>>> thanks, >>>>>>>>> Calvin >>>>>>>>>> Mandy >>>>>>> >> From david.holmes at oracle.com Wed Oct 11 01:22:10 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 11 Oct 2017 11:22:10 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <59DD1731.9090006@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> Message-ID: <580694df-3f13-5e32-7f90-24b9f4d1d029@oracle.com> Hi Calvin, On 11/10/2017 4:53 AM, Calvin Cheung wrote: > I ran into some runtime issue when creating the _java_platform_loader > before initPhase2. Not really surprising. > I've filed the following to track the above issue: > ??? https://bugs.openjdk.java.net/browse/JDK-8189120 > > I'm going with the fix similar to version.02 - creating the system and > platform loaders after initPhase3. > updated webrev: > ??? http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ I think this work has gone off-track somewhat. You've ended up simply caching the value of the platform loader, but that seems unrelated to the bug synopsis: "Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader()" as you have not made that replacement! I expect that actual change is in the soon-to-be-open AppCDS code - but such a change does not depend on what you are doing here AFAICS. So why do we need the current change? I now have a somewhat better understanding of the issues here. We are caching the system and (now) platform loaders after initPhase3 when basically all of the core classes have been initialized, and the module system etc. But during all of that initial classloading we will hit the normal classloading logic that has to ask "is this the system/platform loader?" and so the is_system/platform_loader() calls cannot use the cached fields alone as they will not yet have been set! In addition we have the complexity that when a custom system loader is used, we initially have the built-in system loader acting as "the system loader" until the real "system loader" has been created and installed - hence the formulation: bool SystemDictionary::is_system_class_loader(oop class_loader) { if (class_loader == NULL) { return false; } return (class_loader->klass() == SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() || class_loader == _java_system_loader); } For the platform loader there is only ever the built-in platform loader, so it is simply: bool SystemDictionary::is_platform_class_loader(oop class_loader) { if (class_loader == NULL) { return false; } return (class_loader->klass() == SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); } adding "|| class_loader == _java_platform_loader" would be pointless as it could never be true if executed. So caching the value in _java_platform_loader seems unnecessary. I don't think we should be looking at any changes to the timing of the system or platform classloader construction. We could look at caching those values earlier in the initialization process (eg immediately after the classloader instances are created) but I don't see any advantage in doing so - it would not lead to any other code changes AFAICS. David ----- > thanks, > Calvin > > On 10/5/17, 10:38 PM, David Holmes wrote: >> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>> Hi Coleen, Calvin, >>>> >>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>>> So if you use -Djava.system.loader=myLoader on the command line, >>>>> setting _java_system_loader, then does that mean that the classes >>>>> loaded by >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>> are not in the system loader?? ie. they can be unloaded?? What is >>>>> the result of the call to >>>>> SystemDictionary::is_system_class_loader() used for??? I guess same >>>>> question applies to the platform class loader. >>>> >>>> The classloading delegation hierarchy (as of JDK 9) is: >>>> - boot loader (native) >>>> ? - platform loader (built-in) >>>> ??? - system (aka application) loader (built-in) >>>> >>>> If the user specifies a custom class for the system loader then it >>>> is loaded by an instance of the default system loader and becomes a >>>> fourth level in the hierarchy, and it is then technically the >>>> "system loader". None of these loaders, or the classes they load can >>>> be unloaded. >>>> >>>> Which is presumably why the code checks both: >>>> >>>> ?180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>>> ?181?? if (class_loader == NULL) { >>>> ?182???? return false; >>>> ?183?? } >>>> ?184?? return (class_loader->klass() == >>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>> || >>>> ?185?????????? class_loader == _java_system_loader); >>>> ?186 } >>>> >>>> because we need to treat both of these instances as the "system >>>> loader" from a VM perspective? The same does not apply to the >>>> platform loader. >>> We're obtaining the _java_system_loader after initPhase3 even before >>> this change. Roughly, the calling sequence of initPhase3 is as follows: >>> >>> call_initPhase3() >>> ???? -> ClassLoader.initPhase3() >>> ???????? -> ClassLoader.initSystemClassLoader()? which contains the >>> following code: >>> >>> ???????? String cn = System.getProperty("java.system.class.loader"); >>> ???????? if (cn != null) { >>> ???????????? try { >>> ???????????????? // custom class loader is only supported to be >>> loaded from unnamed module >>> ???????????????? Constructor ctor = Class.forName(cn, false, >>> builtinLoader) >>> .getDeclaredConstructor(ClassLoader.class); >>> ???????????????? scl = (ClassLoader) ctor.newInstance(builtinLoader); >>> ???????????? } catch (Exception e) { >>> ???????????????? throw new Error(e); >>> ???????????? } >>> ???????? } else { >>> ???????????? scl = builtinLoader; >>> ???????? } >>> ???????? return scl; >>> >>> ???????? So initSystemClassLoader() will either return the built-in >>> system loader or a custom loader if it exists. >>> >>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>> >>> ???? public static ClassLoader getSystemClassLoader() { >>> ???????? switch (VM.initLevel()) { >>> ???????????? case 0: >>> ???????????? case 1: >>> ???????????? case 2: >>> ???????????????? // the system class loader is the built-in app class >>> loader during startup >>> ???????????????? return getBuiltinAppClassLoader(); >>> ???????????? case 3: >>> ???????????????? String msg = "getSystemClassLoader should only be >>> called after VM booted"; >>> ???????????????? throw new InternalError(msg); >>> ???????????? case 4: >>> ???????????????? // system fully initialized >>> ???????????????? assert VM.isBooted() && scl != null; >>> ???????????????? SecurityManager sm = System.getSecurityManager(); >>> ???????????????? if (sm != null) { >>> ???????????????????? checkClassLoaderPermission(scl, >>> Reflection.getCallerClass()); >>> ???????????????? } >>> ???????????????? return scl; >>> ???????????? default: >>> ???????????????? throw new InternalError("should not reach here"); >>> ???????? } >>> ???? } >>> >>> ???? So the _java_system_loader will either be the built-in system >>> loader or a custom loader. (case 4 in the above) >>> >>> ???? I don't quite understand why the check in line 184 is required? >>> ???? Is it for checking if a given class_loader is the same type >>> (like an instanceof) as the built-in system loader? >> >> I believe it is checking if the loader is the built-in default system >> loader, both to account for the case where/if >> SystemDictionary::is_system_class_loader is called prior to initPhase3 >> completing; and also to account for encountering the default-built-in >> loader when the custom system loader delegates to it. >> >> I'd have to examine every call path to >> SystemDictionary::is_system_class_loader to check all the details. >> >> David >> ----- >> >>> thanks, >>> Calvin >>>> >>>> David >>>> ----- >>>> >>>>> thanks, >>>>> Coleen >>>>> >>>>>> >>>>>>> The implementation is in closed source. >>>>>> Please clean up the closed code to remove those. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Jiangli >>>>>> >>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>> Yes, it is being used in closed source. >>>>>>>> Currently >>>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>>>> is preloaded.? Shouldn't this be removed?? What about >>>>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp >>>>>>> and also in closed source. >>>>>>>> thread.cpp >>>>>>>> >>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>> >>>>>>>> What is the difference between CHECK_(JNI_ERR) vs CHECK_JNI_ERR? >>>>>>>> Should it simply use CHECK_JNI_ERR as in other places? >>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>> #define CHECK_JNI_ERR??????????????????????????? CHECK_(JNI_ERR) >>>>>>> >>>>>>> and it expands to the following: >>>>>>> __the_thread__); if >>>>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) >>>>>>> return (-1); (void)(0 >>>>>>> >>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>> >>>>>>> thanks, >>>>>>> Calvin >>>>>>>> Mandy >>>>> From david.holmes at oracle.com Wed Oct 11 01:41:39 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 11 Oct 2017 11:41:39 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <580694df-3f13-5e32-7f90-24b9f4d1d029@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> <580694df-3f13-5e32-7f90-24b9f4d1d029@oracle.com> Message-ID: <0632a877-1a2a-7506-2eca-f650e9035d09@oracle.com> One correction to the below ... On 11/10/2017 11:22 AM, David Holmes wrote: > Hi Calvin, > > On 11/10/2017 4:53 AM, Calvin Cheung wrote: >> I ran into some runtime issue when creating the _java_platform_loader >> before initPhase2. > > Not really surprising. > >> I've filed the following to track the above issue: >> ???? https://bugs.openjdk.java.net/browse/JDK-8189120 >> >> I'm going with the fix similar to version.02 - creating the system and >> platform loaders after initPhase3. >> updated webrev: >> ???? http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ > > I think this work has gone off-track somewhat. You've ended up simply > caching the value of the platform loader, but that seems unrelated to > the bug synopsis: > > "Replace SystemDictionaryShared::_java_platform_loader with > SystemDictionary::is_platform_class_loader()" > > as you have not made that replacement! I expect that actual change is in > the soon-to-be-open AppCDS code - but such a change does not depend on > what you are doing here AFAICS. So why do we need the current change? There is one case where the cached java_platform_loader() is used, but it is not a case where we are asking is_platform_class_loader(). David ----- > I now have a somewhat better understanding of the issues here. We are > caching the system and (now) platform loaders after initPhase3 when > basically all of the core classes have been initialized, and the module > system etc. But during all of that initial classloading we will hit the > normal classloading logic that has to ask "is this the system/platform > loader?" and so the is_system/platform_loader() calls cannot use the > cached fields alone as they will not yet have been set! > > In addition we have the complexity that when a custom system loader is > used, we initially have the built-in system loader acting as "the system > loader" until the real "system loader" has been created and installed - > hence the formulation: > > bool SystemDictionary::is_system_class_loader(oop class_loader) { > ? if (class_loader == NULL) { > ??? return false; > ? } > ? return (class_loader->klass() == > SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() || > > ?????? class_loader == _java_system_loader); > } > > For the platform loader there is only ever the built-in platform loader, > so it is simply: > > bool SystemDictionary::is_platform_class_loader(oop class_loader) { > ? if (class_loader == NULL) { > ??? return false; > ? } > ? return (class_loader->klass() == > SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); > > } > > adding "|| class_loader == _java_platform_loader" would be pointless as > it could never be true if executed. So caching the value in > _java_platform_loader seems unnecessary. > > I don't think we should be looking at any changes to the timing of the > system or platform classloader construction. We could look at caching > those values earlier in the initialization process (eg immediately after > the classloader instances are created) but I don't see any advantage in > doing so - it would not lead to any other code changes AFAICS. > > David > ----- > > >> thanks, >> Calvin >> >> On 10/5/17, 10:38 PM, David Holmes wrote: >>> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>>> Hi Coleen, Calvin, >>>>> >>>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>>>> So if you use -Djava.system.loader=myLoader on the command line, >>>>>> setting _java_system_loader, then does that mean that the classes >>>>>> loaded by >>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>>> are not in the system loader?? ie. they can be unloaded?? What is >>>>>> the result of the call to >>>>>> SystemDictionary::is_system_class_loader() used for??? I guess >>>>>> same question applies to the platform class loader. >>>>> >>>>> The classloading delegation hierarchy (as of JDK 9) is: >>>>> - boot loader (native) >>>>> ? - platform loader (built-in) >>>>> ??? - system (aka application) loader (built-in) >>>>> >>>>> If the user specifies a custom class for the system loader then it >>>>> is loaded by an instance of the default system loader and becomes a >>>>> fourth level in the hierarchy, and it is then technically the >>>>> "system loader". None of these loaders, or the classes they load >>>>> can be unloaded. >>>>> >>>>> Which is presumably why the code checks both: >>>>> >>>>> ?180 bool SystemDictionary::is_system_class_loader(oop class_loader) { >>>>> ?181?? if (class_loader == NULL) { >>>>> ?182???? return false; >>>>> ?183?? } >>>>> ?184?? return (class_loader->klass() == >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>> || >>>>> ?185?????????? class_loader == _java_system_loader); >>>>> ?186 } >>>>> >>>>> because we need to treat both of these instances as the "system >>>>> loader" from a VM perspective? The same does not apply to the >>>>> platform loader. >>>> We're obtaining the _java_system_loader after initPhase3 even before >>>> this change. Roughly, the calling sequence of initPhase3 is as follows: >>>> >>>> call_initPhase3() >>>> ???? -> ClassLoader.initPhase3() >>>> ???????? -> ClassLoader.initSystemClassLoader()? which contains the >>>> following code: >>>> >>>> ???????? String cn = System.getProperty("java.system.class.loader"); >>>> ???????? if (cn != null) { >>>> ???????????? try { >>>> ???????????????? // custom class loader is only supported to be >>>> loaded from unnamed module >>>> ???????????????? Constructor ctor = Class.forName(cn, false, >>>> builtinLoader) >>>> .getDeclaredConstructor(ClassLoader.class); >>>> ???????????????? scl = (ClassLoader) ctor.newInstance(builtinLoader); >>>> ???????????? } catch (Exception e) { >>>> ???????????????? throw new Error(e); >>>> ???????????? } >>>> ???????? } else { >>>> ???????????? scl = builtinLoader; >>>> ???????? } >>>> ???????? return scl; >>>> >>>> ???????? So initSystemClassLoader() will either return the built-in >>>> system loader or a custom loader if it exists. >>>> >>>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>>> >>>> ???? public static ClassLoader getSystemClassLoader() { >>>> ???????? switch (VM.initLevel()) { >>>> ???????????? case 0: >>>> ???????????? case 1: >>>> ???????????? case 2: >>>> ???????????????? // the system class loader is the built-in app >>>> class loader during startup >>>> ???????????????? return getBuiltinAppClassLoader(); >>>> ???????????? case 3: >>>> ???????????????? String msg = "getSystemClassLoader should only be >>>> called after VM booted"; >>>> ???????????????? throw new InternalError(msg); >>>> ???????????? case 4: >>>> ???????????????? // system fully initialized >>>> ???????????????? assert VM.isBooted() && scl != null; >>>> ???????????????? SecurityManager sm = System.getSecurityManager(); >>>> ???????????????? if (sm != null) { >>>> ???????????????????? checkClassLoaderPermission(scl, >>>> Reflection.getCallerClass()); >>>> ???????????????? } >>>> ???????????????? return scl; >>>> ???????????? default: >>>> ???????????????? throw new InternalError("should not reach here"); >>>> ???????? } >>>> ???? } >>>> >>>> ???? So the _java_system_loader will either be the built-in system >>>> loader or a custom loader. (case 4 in the above) >>>> >>>> ???? I don't quite understand why the check in line 184 is required? >>>> ???? Is it for checking if a given class_loader is the same type >>>> (like an instanceof) as the built-in system loader? >>> >>> I believe it is checking if the loader is the built-in default system >>> loader, both to account for the case where/if >>> SystemDictionary::is_system_class_loader is called prior to >>> initPhase3 completing; and also to account for encountering the >>> default-built-in loader when the custom system loader delegates to it. >>> >>> I'd have to examine every call path to >>> SystemDictionary::is_system_class_loader to check all the details. >>> >>> David >>> ----- >>> >>>> thanks, >>>> Calvin >>>>> >>>>> David >>>>> ----- >>>>> >>>>>> thanks, >>>>>> Coleen >>>>>> >>>>>>> >>>>>>>> The implementation is in closed source. >>>>>>> Please clean up the closed code to remove those. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Jiangli >>>>>>> >>>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>>> Yes, it is being used in closed source. >>>>>>>>> Currently >>>>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>>>>> is preloaded.? Shouldn't this be removed?? What about >>>>>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp >>>>>>>> and also in closed source. >>>>>>>>> thread.cpp >>>>>>>>> >>>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>>> >>>>>>>>> What is the difference between CHECK_(JNI_ERR) vs >>>>>>>>> CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other >>>>>>>>> places? >>>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>>> #define CHECK_JNI_ERR??????????????????????????? CHECK_(JNI_ERR) >>>>>>>> >>>>>>>> and it expands to the following: >>>>>>>> __the_thread__); if >>>>>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) >>>>>>>> return (-1); (void)(0 >>>>>>>> >>>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>>> >>>>>>>> thanks, >>>>>>>> Calvin >>>>>>>>> Mandy >>>>>> From ioi.lam at oracle.com Wed Oct 11 04:45:54 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 10 Oct 2017 21:45:54 -0700 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: <580694df-3f13-5e32-7f90-24b9f4d1d029@oracle.com> References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> <580694df-3f13-5e32-7f90-24b9f4d1d029@oracle.com> Message-ID: On 10/10/17 6:22 PM, David Holmes wrote: > Hi Calvin, > > On 11/10/2017 4:53 AM, Calvin Cheung wrote: >> I ran into some runtime issue when creating the _java_platform_loader >> before initPhase2. > > Not really surprising. > >> I've filed the following to track the above issue: >> ???? https://bugs.openjdk.java.net/browse/JDK-8189120 >> >> I'm going with the fix similar to version.02 - creating the system >> and platform loaders after initPhase3. >> updated webrev: >> ???? http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ > > I think this work has gone off-track somewhat. You've ended up simply > caching the value of the platform loader, but that seems unrelated to > the bug synopsis: > > "Replace SystemDictionaryShared::_java_platform_loader with > SystemDictionary::is_platform_class_loader()" > > as you have not made that replacement! I expect that actual change is > in the soon-to-be-open AppCDS code - but such a change does not depend > on what you are doing here AFAICS. So why do we need the current change? > David, I filed JDK-8185694 as a clean-up task, so we can avoid using the field _java_platform_loader directly, and replace that with a function call. The replacement (from the field to the function call) is currently in the closed code. This code will be moved to open soon (see the JEP: JDK-8185996). As I mentioned in JDK-8188791 (the actual task for moving the code), we will separate the "moving the code" step vs the "cleaning up the code" steps. Otherwise it will be very hard to manage concurrent modifications to the related source files. JDK-8185694 (_java_platform_loader) is just one of those "cleaning up" steps, which happens to be done before the code is actually moved. Thanks - Ioi > I now have a somewhat better understanding of the issues here. We are > caching the system and (now) platform loaders after initPhase3 when > basically all of the core classes have been initialized, and the > module system etc. But during all of that initial classloading we will > hit the normal classloading logic that has to ask "is this the > system/platform loader?" and so the is_system/platform_loader() calls > cannot use the cached fields alone as they will not yet have been set! > > In addition we have the complexity that when a custom system loader is > used, we initially have the built-in system loader acting as "the > system loader" until the real "system loader" has been created and > installed - hence the formulation: > > bool SystemDictionary::is_system_class_loader(oop class_loader) { > ? if (class_loader == NULL) { > ??? return false; > ? } > ? return (class_loader->klass() == > SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() > || > ?????? class_loader == _java_system_loader); > } > > For the platform loader there is only ever the built-in platform > loader, so it is simply: > > bool SystemDictionary::is_platform_class_loader(oop class_loader) { > ? if (class_loader == NULL) { > ??? return false; > ? } > ? return (class_loader->klass() == > SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); > } > > adding "|| class_loader == _java_platform_loader" would be pointless > as it could never be true if executed. So caching the value in > _java_platform_loader seems unnecessary. > > I don't think we should be looking at any changes to the timing of the > system or platform classloader construction. We could look at caching > those values earlier in the initialization process (eg immediately > after the classloader instances are created) but I don't see any > advantage in doing so - it would not lead to any other code changes > AFAICS. > > David > ----- > > >> thanks, >> Calvin >> >> On 10/5/17, 10:38 PM, David Holmes wrote: >>> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>>> Hi Coleen, Calvin, >>>>> >>>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>>>> So if you use -Djava.system.loader=myLoader on the command line, >>>>>> setting _java_system_loader, then does that mean that the classes >>>>>> loaded by >>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>>> are not in the system loader?? ie. they can be unloaded?? What is >>>>>> the result of the call to >>>>>> SystemDictionary::is_system_class_loader() used for??? I guess >>>>>> same question applies to the platform class loader. >>>>> >>>>> The classloading delegation hierarchy (as of JDK 9) is: >>>>> - boot loader (native) >>>>> ? - platform loader (built-in) >>>>> ??? - system (aka application) loader (built-in) >>>>> >>>>> If the user specifies a custom class for the system loader then it >>>>> is loaded by an instance of the default system loader and becomes >>>>> a fourth level in the hierarchy, and it is then technically the >>>>> "system loader". None of these loaders, or the classes they load >>>>> can be unloaded. >>>>> >>>>> Which is presumably why the code checks both: >>>>> >>>>> ?180 bool SystemDictionary::is_system_class_loader(oop >>>>> class_loader) { >>>>> ?181?? if (class_loader == NULL) { >>>>> ?182???? return false; >>>>> ?183?? } >>>>> ?184?? return (class_loader->klass() == >>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>> || >>>>> ?185?????????? class_loader == _java_system_loader); >>>>> ?186 } >>>>> >>>>> because we need to treat both of these instances as the "system >>>>> loader" from a VM perspective? The same does not apply to the >>>>> platform loader. >>>> We're obtaining the _java_system_loader after initPhase3 even >>>> before this change. Roughly, the calling sequence of initPhase3 is >>>> as follows: >>>> >>>> call_initPhase3() >>>> ???? -> ClassLoader.initPhase3() >>>> ???????? -> ClassLoader.initSystemClassLoader()? which contains the >>>> following code: >>>> >>>> ???????? String cn = System.getProperty("java.system.class.loader"); >>>> ???????? if (cn != null) { >>>> ???????????? try { >>>> ???????????????? // custom class loader is only supported to be >>>> loaded from unnamed module >>>> ???????????????? Constructor ctor = Class.forName(cn, false, >>>> builtinLoader) >>>> .getDeclaredConstructor(ClassLoader.class); >>>> ???????????????? scl = (ClassLoader) ctor.newInstance(builtinLoader); >>>> ???????????? } catch (Exception e) { >>>> ???????????????? throw new Error(e); >>>> ???????????? } >>>> ???????? } else { >>>> ???????????? scl = builtinLoader; >>>> ???????? } >>>> ???????? return scl; >>>> >>>> ???????? So initSystemClassLoader() will either return the built-in >>>> system loader or a custom loader if it exists. >>>> >>>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>>> >>>> ???? public static ClassLoader getSystemClassLoader() { >>>> ???????? switch (VM.initLevel()) { >>>> ???????????? case 0: >>>> ???????????? case 1: >>>> ???????????? case 2: >>>> ???????????????? // the system class loader is the built-in app >>>> class loader during startup >>>> ???????????????? return getBuiltinAppClassLoader(); >>>> ???????????? case 3: >>>> ???????????????? String msg = "getSystemClassLoader should only be >>>> called after VM booted"; >>>> ???????????????? throw new InternalError(msg); >>>> ???????????? case 4: >>>> ???????????????? // system fully initialized >>>> ???????????????? assert VM.isBooted() && scl != null; >>>> ???????????????? SecurityManager sm = System.getSecurityManager(); >>>> ???????????????? if (sm != null) { >>>> ???????????????????? checkClassLoaderPermission(scl, >>>> Reflection.getCallerClass()); >>>> ???????????????? } >>>> ???????????????? return scl; >>>> ???????????? default: >>>> ???????????????? throw new InternalError("should not reach here"); >>>> ???????? } >>>> ???? } >>>> >>>> ???? So the _java_system_loader will either be the built-in system >>>> loader or a custom loader. (case 4 in the above) >>>> >>>> ???? I don't quite understand why the check in line 184 is required? >>>> ???? Is it for checking if a given class_loader is the same type >>>> (like an instanceof) as the built-in system loader? >>> >>> I believe it is checking if the loader is the built-in default >>> system loader, both to account for the case where/if >>> SystemDictionary::is_system_class_loader is called prior to >>> initPhase3 completing; and also to account for encountering the >>> default-built-in loader when the custom system loader delegates to it. >>> >>> I'd have to examine every call path to >>> SystemDictionary::is_system_class_loader to check all the details. >>> >>> David >>> ----- >>> >>>> thanks, >>>> Calvin >>>>> >>>>> David >>>>> ----- >>>>> >>>>>> thanks, >>>>>> Coleen >>>>>> >>>>>>> >>>>>>>> The implementation is in closed source. >>>>>>> Please clean up the closed code to remove those. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Jiangli >>>>>>> >>>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>>> Yes, it is being used in closed source. >>>>>>>>> Currently >>>>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>>>>> is preloaded.? Shouldn't this be removed?? What about >>>>>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp >>>>>>>> and also in closed source. >>>>>>>>> thread.cpp >>>>>>>>> >>>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>>> >>>>>>>>> What is the difference between CHECK_(JNI_ERR) vs >>>>>>>>> CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other >>>>>>>>> places? >>>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>>>>> >>>>>>>> and it expands to the following: >>>>>>>> __the_thread__); if >>>>>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) >>>>>>>> return (-1); (void)(0 >>>>>>>> >>>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>>> >>>>>>>> thanks, >>>>>>>> Calvin >>>>>>>>> Mandy >>>>>> From david.holmes at oracle.com Wed Oct 11 05:07:28 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 11 Oct 2017 15:07:28 +1000 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> Message-ID: <58edfce7-4fe5-edbc-2a2d-56a1dcca6ced@oracle.com> Hi Gerard, On 11/10/2017 6:40 AM, Gerard Ziemski wrote: > hi all, > > Please review this change that adds dynamic resizing of system dictionaries. Sounds good - but there is nothing in the bug report to justify any of the resizing choices - eg resize factor - where are the performance numbers? :) In the JDK libraries the policy of always doubling collections when they needed to grow was removed because as the collection gets really big, doubling its size becomes a major problem. What kind of sizes are we likely to be dealing with? Given we don't currently resize, when will we now see resizing happening? ie what apps will be affected by this and what performance hit might they take when the resizing occurs? IIUC resizing is done lazily if we hit the right thresholds when a safepoint happens. I'm unclear what kind of margins we have to ensure we are not likely to run out of space before the next safepoint might occur? > The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock So basically anyone calling hash_to_index must now hold the SD lock when doing that call. In most places it is a simple matter of moving the call to a locked region later in the method; but in SD::resolve_instance_class_or_null you added a new locked region - I am concerned this may introduce a synchronization bottleneck. > A few notes: > > - dynamic resizing is off when either dumping or using shared spaces > - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) I don't think it makes sense to make a flag that restores the old behaviour, experimental. This should be a product flag IMHO. This flag is different to PredictedLoadedClassCount which provided an unsupported (aka experimental) means to influence the initial table size. > - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) > > bug: https://bugs.openjdk.java.net/browse/JDK-8184765 > webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 src/hotspot/share/classfile/dictionary.cpp 109 for (int i=n; i* buckets_new = NEW_C_HEAP_ARRAY2(HashtableBucket, new_size, F, CURRENT_PC); 272 if (buckets_new == NULL) { You're using the NEW_ macro that aborts on OOM not the one that returns NULL. BTW what happens when the SD does fill up? Do we abort or just stop loading classes? --- src/hotspot/share/utilities/hashtable.hpp May be worth adding: assert_locked_or_safepoint(SystemDictionary_lock); to hash_to_index (or just a lock ownwership test if you don't expect this to be done at a safepoint) --- test/runtime/LoadClass/TestResize.java test/runtime/LoadClass/TriggerResize.java Shouldn't these be test/hotspot/jtreg/runtime/LoadClass/... ? --- TestResize.java 24 /* 25 * @test 26 * @bug 8184765 27 * @summary Test dynamic resizing of SystemDictionary 28 * @modules java.base/jdk.internal.misc:open 29 * @modules java.base/java.lang:open 30 * @library /test/lib 31 */ This isn't actually a test - you never run it directly - so I don't expect to see any jtreg tags here ?? I would expect them all to be on the actual test class. Actually I'm not sure why this needs to be a separate public class in its own file ? Please add comments in load() explaining what all the byte/index stuff is doing - presumably making the class representation unique? --- Thanks, David ----- > > cheers > From david.holmes at oracle.com Wed Oct 11 05:26:31 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 11 Oct 2017 15:26:31 +1000 Subject: RFR(S): 8185694: Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader() In-Reply-To: References: <59D4006C.3010000@oracle.com> <015f044d-e735-ec31-07e4-16b87d83420b@oracle.com> <59D51682.1060301@oracle.com> <42ba4527-1062-d524-1ee4-fa367cdc3298@oracle.com> <59D5617D.4050509@oracle.com> <1ACE23D1-BFA9-4041-A9DB-176240A3DF53@oracle.com> <8244c2ee-71cb-4c32-0e80-a0887b9d05ee@oracle.com> <6266de32-fc48-6afb-3f51-24bb0e56a879@oracle.com> <59D7146C.3010206@oracle.com> <59DD1731.9090006@oracle.com> <580694df-3f13-5e32-7f90-24b9f4d1d029@oracle.com> Message-ID: <280d6994-7912-d4c6-f116-87b5db05a27f@oracle.com> Hi Ioi, On 11/10/2017 2:45 PM, Ioi Lam wrote: > > > On 10/10/17 6:22 PM, David Holmes wrote: >> Hi Calvin, >> >> On 11/10/2017 4:53 AM, Calvin Cheung wrote: >>> I ran into some runtime issue when creating the _java_platform_loader >>> before initPhase2. >> >> Not really surprising. >> >>> I've filed the following to track the above issue: >>> ???? https://bugs.openjdk.java.net/browse/JDK-8189120 >>> >>> I'm going with the fix similar to version.02 - creating the system >>> and platform loaders after initPhase3. >>> updated webrev: >>> ???? http://cr.openjdk.java.net/~ccheung/8185694/webrev.04/ >> >> I think this work has gone off-track somewhat. You've ended up simply >> caching the value of the platform loader, but that seems unrelated to >> the bug synopsis: >> >> "Replace SystemDictionaryShared::_java_platform_loader with >> SystemDictionary::is_platform_class_loader()" >> >> as you have not made that replacement! I expect that actual change is >> in the soon-to-be-open AppCDS code - but such a change does not depend >> on what you are doing here AFAICS. So why do we need the current change? >> > > David, I filed JDK-8185694 as a clean-up task, so we can avoid using the > field _java_platform_loader directly, and replace that with a function > call. > > The replacement (from the field to the function call) is currently in > the closed code. This code will be moved to open soon (see the JEP: > JDK-8185996). > > As I mentioned in JDK-8188791 (the actual task for moving the code), we > will separate the "moving the code" step vs the "cleaning up the code" > steps. Otherwise it will be very hard to manage concurrent modifications > to the related source files. JDK-8185694 (_java_platform_loader) is just > one of those "cleaning up" steps, which happens to be done before the > code is actually moved. Yes you are right - I needed to look back at Calvin's original RFR which simply states: "The open code change for this fix involves adding the _java_platform_loader to the SystemDictionary class. " which is all this change actually does. Things got muddled with all the suggestions to change the definitions of is_platform/system_class_loader() - combined with the observation that although the bug is titled "Replace SystemDictionaryShared::_java_platform_loader with SystemDictionary::is_platform_class_loader()" that doesn't actually require the change being implemented here. I'll also note that there may be subtle differences in behaviour in using SystemDictionary::is_platform_class_loader() if you currently call it before the platform loader would have been cached. Thanks, David > Thanks > - Ioi > > > >> I now have a somewhat better understanding of the issues here. We are >> caching the system and (now) platform loaders after initPhase3 when >> basically all of the core classes have been initialized, and the >> module system etc. But during all of that initial classloading we will >> hit the normal classloading logic that has to ask "is this the >> system/platform loader?" and so the is_system/platform_loader() calls >> cannot use the cached fields alone as they will not yet have been set! >> >> In addition we have the complexity that when a custom system loader is >> used, we initially have the built-in system loader acting as "the >> system loader" until the real "system loader" has been created and >> installed - hence the formulation: >> >> bool SystemDictionary::is_system_class_loader(oop class_loader) { >> ? if (class_loader == NULL) { >> ??? return false; >> ? } >> ? return (class_loader->klass() == >> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >> || >> ?????? class_loader == _java_system_loader); >> } >> >> For the platform loader there is only ever the built-in platform >> loader, so it is simply: >> >> bool SystemDictionary::is_platform_class_loader(oop class_loader) { >> ? if (class_loader == NULL) { >> ??? return false; >> ? } >> ? return (class_loader->klass() == >> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass()); >> >> } >> >> adding "|| class_loader == _java_platform_loader" would be pointless >> as it could never be true if executed. So caching the value in >> _java_platform_loader seems unnecessary. >> >> I don't think we should be looking at any changes to the timing of the >> system or platform classloader construction. We could look at caching >> those values earlier in the initialization process (eg immediately >> after the classloader instances are created) but I don't see any >> advantage in doing so - it would not lead to any other code changes >> AFAICS. >> >> David >> ----- >> >> >>> thanks, >>> Calvin >>> >>> On 10/5/17, 10:38 PM, David Holmes wrote: >>>> On 6/10/2017 3:28 PM, Calvin Cheung wrote: >>>>> On 10/5/17, 6:33 PM, David Holmes wrote: >>>>>> Hi Coleen, Calvin, >>>>>> >>>>>> On 6/10/2017 6:54 AM, coleen.phillimore at oracle.com wrote: >>>>>>> So if you use -Djava.system.loader=myLoader on the command line, >>>>>>> setting _java_system_loader, then does that mean that the classes >>>>>>> loaded by >>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>>>> are not in the system loader?? ie. they can be unloaded?? What is >>>>>>> the result of the call to >>>>>>> SystemDictionary::is_system_class_loader() used for??? I guess >>>>>>> same question applies to the platform class loader. >>>>>> >>>>>> The classloading delegation hierarchy (as of JDK 9) is: >>>>>> - boot loader (native) >>>>>> ? - platform loader (built-in) >>>>>> ??? - system (aka application) loader (built-in) >>>>>> >>>>>> If the user specifies a custom class for the system loader then it >>>>>> is loaded by an instance of the default system loader and becomes >>>>>> a fourth level in the hierarchy, and it is then technically the >>>>>> "system loader". None of these loaders, or the classes they load >>>>>> can be unloaded. >>>>>> >>>>>> Which is presumably why the code checks both: >>>>>> >>>>>> ?180 bool SystemDictionary::is_system_class_loader(oop >>>>>> class_loader) { >>>>>> ?181?? if (class_loader == NULL) { >>>>>> ?182???? return false; >>>>>> ?183?? } >>>>>> ?184?? return (class_loader->klass() == >>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_AppClassLoader_klass() >>>>>> || >>>>>> ?185?????????? class_loader == _java_system_loader); >>>>>> ?186 } >>>>>> >>>>>> because we need to treat both of these instances as the "system >>>>>> loader" from a VM perspective? The same does not apply to the >>>>>> platform loader. >>>>> We're obtaining the _java_system_loader after initPhase3 even >>>>> before this change. Roughly, the calling sequence of initPhase3 is >>>>> as follows: >>>>> >>>>> call_initPhase3() >>>>> ???? -> ClassLoader.initPhase3() >>>>> ???????? -> ClassLoader.initSystemClassLoader()? which contains the >>>>> following code: >>>>> >>>>> ???????? String cn = System.getProperty("java.system.class.loader"); >>>>> ???????? if (cn != null) { >>>>> ???????????? try { >>>>> ???????????????? // custom class loader is only supported to be >>>>> loaded from unnamed module >>>>> ???????????????? Constructor ctor = Class.forName(cn, false, >>>>> builtinLoader) >>>>> .getDeclaredConstructor(ClassLoader.class); >>>>> ???????????????? scl = (ClassLoader) ctor.newInstance(builtinLoader); >>>>> ???????????? } catch (Exception e) { >>>>> ???????????????? throw new Error(e); >>>>> ???????????? } >>>>> ???????? } else { >>>>> ???????????? scl = builtinLoader; >>>>> ???????? } >>>>> ???????? return scl; >>>>> >>>>> ???????? So initSystemClassLoader() will either return the built-in >>>>> system loader or a custom loader if it exists. >>>>> >>>>> We use the getSystemClassLoader API to obtain the _java_system_loader: >>>>> >>>>> ???? public static ClassLoader getSystemClassLoader() { >>>>> ???????? switch (VM.initLevel()) { >>>>> ???????????? case 0: >>>>> ???????????? case 1: >>>>> ???????????? case 2: >>>>> ???????????????? // the system class loader is the built-in app >>>>> class loader during startup >>>>> ???????????????? return getBuiltinAppClassLoader(); >>>>> ???????????? case 3: >>>>> ???????????????? String msg = "getSystemClassLoader should only be >>>>> called after VM booted"; >>>>> ???????????????? throw new InternalError(msg); >>>>> ???????????? case 4: >>>>> ???????????????? // system fully initialized >>>>> ???????????????? assert VM.isBooted() && scl != null; >>>>> ???????????????? SecurityManager sm = System.getSecurityManager(); >>>>> ???????????????? if (sm != null) { >>>>> ???????????????????? checkClassLoaderPermission(scl, >>>>> Reflection.getCallerClass()); >>>>> ???????????????? } >>>>> ???????????????? return scl; >>>>> ???????????? default: >>>>> ???????????????? throw new InternalError("should not reach here"); >>>>> ???????? } >>>>> ???? } >>>>> >>>>> ???? So the _java_system_loader will either be the built-in system >>>>> loader or a custom loader. (case 4 in the above) >>>>> >>>>> ???? I don't quite understand why the check in line 184 is required? >>>>> ???? Is it for checking if a given class_loader is the same type >>>>> (like an instanceof) as the built-in system loader? >>>> >>>> I believe it is checking if the loader is the built-in default >>>> system loader, both to account for the case where/if >>>> SystemDictionary::is_system_class_loader is called prior to >>>> initPhase3 completing; and also to account for encountering the >>>> default-built-in loader when the custom system loader delegates to it. >>>> >>>> I'd have to examine every call path to >>>> SystemDictionary::is_system_class_loader to check all the details. >>>> >>>> David >>>> ----- >>>> >>>>> thanks, >>>>> Calvin >>>>>> >>>>>> David >>>>>> ----- >>>>>> >>>>>>> thanks, >>>>>>> Coleen >>>>>>> >>>>>>>> >>>>>>>>> The implementation is in closed source. >>>>>>>> Please clean up the closed code to remove those. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Jiangli >>>>>>>> >>>>>>>>>> Is this new java_platform_loader function used anywhere? >>>>>>>>> Yes, it is being used in closed source. >>>>>>>>>> Currently >>>>>>>>>> SystemDictionary::jdk_internal_loader_ClassLoaders_PlatformClassLoader_klass >>>>>>>>>> is preloaded.? Shouldn't this be removed?? What about >>>>>>>>>> jdk_internal_loader_ClassLoaders_AppClassLoader? >>>>>>>>> They're being used in lines 184 and 193 in systemDictionary.cpp >>>>>>>>> and also in closed source. >>>>>>>>>> thread.cpp >>>>>>>>>> >>>>>>>>>> 3752 SystemDictionary::compute_java_loaders(CHECK_(JNI_ERR)); >>>>>>>>>> >>>>>>>>>> What is the difference between CHECK_(JNI_ERR) vs >>>>>>>>>> CHECK_JNI_ERR? Should it simply use CHECK_JNI_ERR as in other >>>>>>>>>> places? >>>>>>>>> They are the same, in utilities/exceptions.hpp: >>>>>>>>> #define CHECK_JNI_ERR CHECK_(JNI_ERR) >>>>>>>>> >>>>>>>>> and it expands to the following: >>>>>>>>> __the_thread__); if >>>>>>>>> ((((ThreadShadow*)__the_thread__)->has_pending_exception())) >>>>>>>>> return (-1); (void)(0 >>>>>>>>> >>>>>>>>> I can change it to CHECK_JNI_ERR. >>>>>>>>> >>>>>>>>> thanks, >>>>>>>>> Calvin >>>>>>>>>> Mandy >>>>>>> > From robbin.ehn at oracle.com Wed Oct 11 10:06:13 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Wed, 11 Oct 2017 12:06:13 +0200 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> Message-ID: Hi, jtreg now spits out: dockerSupport() threw exception: java.io.IOException: Cannot run program "docker": error=2, No such file or directory Doesn't seem right. /Robbin On 09/22/2017 02:58 AM, mikhailo wrote: > Please review this initial drop of Docker test utils and a sanity test. This change lays ground > for further test development and test utils improvement in this area. > > ??? JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 > ??? Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ > ??? Testing: > ?????? - run this test on machine with Docker enabled - works > ?????? - run this test on Linux-x64 with no Docker engine or Docker disabled - test skipped (as expected) > ?????? - run this test on automated system - in progress > > > Thank you, > Misha > From harold.seigel at oracle.com Wed Oct 11 12:45:27 2017 From: harold.seigel at oracle.com (harold seigel) Date: Wed, 11 Oct 2017 08:45:27 -0400 Subject: RFR 8188922: runtime/CommandLine/VMDeprecatedOptions.java fails with JDK10 release bits Message-ID: <85bd1c02-6a0a-9abd-61da-a375b38d339c@oracle.com> Hi, Please review this small change to fix JDK-8188922.? The fix adds -XX:+UnlockDiagnosticVMOptions to the command line when testing a deprecated diagnostic option. Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8188922/webrev/index.html JBD Bug: https://bugs.openjdk.java.net/browse/JDK-8188922 The fix was tested by running the modified VMDeprecatedOptions.java test with both a product JDK-10 build and a fastdebug JDK-10 build. Thanks, Harold From jamsheed.c.m at oracle.com Wed Oct 11 12:48:05 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Wed, 11 Oct 2017 18:18:05 +0530 Subject: [10] RFR: [AOT] assert(false) failed: DEBUG MESSAGE: InterpreterMacroAssembler::call_VM_base: last_sp != NULL In-Reply-To: References: <8eb949bd-fa14-2722-5eaa-21a0a0c95b26@oracle.com> <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> Message-ID: <34517155-bd8d-56ee-86a1-7f9ffe73e2de@oracle.com> Hi Vladimir, Thank you for pointing this. revised webrev: http://cr.openjdk.java.net/~jcm/8168712/webrev.02/ Best Regards, Jamsheed On Tuesday 10 October 2017 08:41 PM, Vladimir Kozlov wrote: > Why you added !defined(AARCH64) in templateTable_arm.cpp? Is only > 32-bit affected? > > Thanks, > Vladimir > > On 9/13/17 11:54 PM, Dean Long wrote: >> It looks like you accidentally dropped >> hotspot-compiler-dev at openjdk.java.net when you added runtime. >> >> dl >> >> >> On 9/13/2017 11:21 PM, jamsheed wrote: >>> (adding runtime list for inputs) >>> >>> On Monday 11 September 2017 11:43 PM, jamsheed wrote: >>>> brief desc: special handling of Object. in >>>> TemplateInterpreter::deopt_reexecute_entry >>>> >>>> required last_sp to be reset explicitly in normal return path >>>> >>>> address TemplateInterpreter::deopt_reexecute_entry(Method* method, >>>> address bcp) { >>>> ? assert(method->contains(bcp), "just checkin'"); >>>> ? Bytecodes::Code code?? = Bytecodes::java_code_at(method, bcp); >>>> ? if (code == Bytecodes::_return) { >>>> ??? // This is used for deopt during registration of finalizers >>>> ??? // during Object..? We simply need to resume execution at >>>> ??? // the standard return vtos bytecode to pop the frame normally. >>>> ??? // reexecuting the real bytecode would cause double registration >>>> ??? // of the finalizable object. >>>> ??? return _normal_table.entry(Bytecodes::_return).entry(vtos); >>> >>> last_sp ! = null not an issue for this case, so i skip the assert in >>> debug build >>> >>> http://cr.openjdk.java.net/~jcm/8168712/webrev.01/ >>> >>> Please review. >>> >>> Best Regards, >>> Jamsheed >>> >>> >>> >>> >>> >> From coleen.phillimore at oracle.com Wed Oct 11 13:33:44 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 11 Oct 2017 09:33:44 -0400 Subject: RFR 8188922: runtime/CommandLine/VMDeprecatedOptions.java fails with JDK10 release bits In-Reply-To: <85bd1c02-6a0a-9abd-61da-a375b38d339c@oracle.com> References: <85bd1c02-6a0a-9abd-61da-a375b38d339c@oracle.com> Message-ID: This looks really good.? I think you should check this in under trivial rules so we don't have to look at this test failure in our own testing anymore. thanks! Coleen On 10/11/17 8:45 AM, harold seigel wrote: > Hi, > > Please review this small change to fix JDK-8188922.? The fix adds > -XX:+UnlockDiagnosticVMOptions to the command line when testing a > deprecated diagnostic option. > > Open webrev: > http://cr.openjdk.java.net/~hseigel/bug_8188922/webrev/index.html > > JBD Bug: https://bugs.openjdk.java.net/browse/JDK-8188922 > > The fix was tested by running the modified VMDeprecatedOptions.java > test with both a product JDK-10 build and a fastdebug JDK-10 build. > > Thanks, Harold > From lois.foltan at oracle.com Wed Oct 11 13:58:28 2017 From: lois.foltan at oracle.com (Lois Foltan) Date: Wed, 11 Oct 2017 09:58:28 -0400 Subject: RFR 8188922: runtime/CommandLine/VMDeprecatedOptions.java fails with JDK10 release bits In-Reply-To: <85bd1c02-6a0a-9abd-61da-a375b38d339c@oracle.com> References: <85bd1c02-6a0a-9abd-61da-a375b38d339c@oracle.com> Message-ID: <603cb187-9608-c250-4202-989d97861299@oracle.com> Looks good! Lois On 10/11/2017 8:45 AM, harold seigel wrote: > Hi, > > Please review this small change to fix JDK-8188922.? The fix adds > -XX:+UnlockDiagnosticVMOptions to the command line when testing a > deprecated diagnostic option. > > Open webrev: > http://cr.openjdk.java.net/~hseigel/bug_8188922/webrev/index.html > > JBD Bug: https://bugs.openjdk.java.net/browse/JDK-8188922 > > The fix was tested by running the modified VMDeprecatedOptions.java > test with both a product JDK-10 build and a fastdebug JDK-10 build. > > Thanks, Harold > From harold.seigel at oracle.com Wed Oct 11 13:58:55 2017 From: harold.seigel at oracle.com (harold seigel) Date: Wed, 11 Oct 2017 09:58:55 -0400 Subject: RFR 8188922: runtime/CommandLine/VMDeprecatedOptions.java fails with JDK10 release bits In-Reply-To: References: <85bd1c02-6a0a-9abd-61da-a375b38d339c@oracle.com> Message-ID: Thanks Coleen! I'll go ahead and push it. Harold On 10/11/2017 9:33 AM, coleen.phillimore at oracle.com wrote: > > This looks really good.? I think you should check this in under > trivial rules so we don't have to look at this test failure in our own > testing anymore. > thanks! > Coleen > > On 10/11/17 8:45 AM, harold seigel wrote: >> Hi, >> >> Please review this small change to fix JDK-8188922.? The fix adds >> -XX:+UnlockDiagnosticVMOptions to the command line when testing a >> deprecated diagnostic option. >> >> Open webrev: >> http://cr.openjdk.java.net/~hseigel/bug_8188922/webrev/index.html >> >> JBD Bug: https://bugs.openjdk.java.net/browse/JDK-8188922 >> >> The fix was tested by running the modified VMDeprecatedOptions.java >> test with both a product JDK-10 build and a fastdebug JDK-10 build. >> >> Thanks, Harold >> > From harold.seigel at oracle.com Wed Oct 11 13:59:05 2017 From: harold.seigel at oracle.com (harold seigel) Date: Wed, 11 Oct 2017 09:59:05 -0400 Subject: RFR 8188922: runtime/CommandLine/VMDeprecatedOptions.java fails with JDK10 release bits In-Reply-To: <603cb187-9608-c250-4202-989d97861299@oracle.com> References: <85bd1c02-6a0a-9abd-61da-a375b38d339c@oracle.com> <603cb187-9608-c250-4202-989d97861299@oracle.com> Message-ID: <0b10128e-81dc-5f42-f2bb-c9e35ea0b1f8@oracle.com> Thanks Lois! Harold On 10/11/2017 9:58 AM, Lois Foltan wrote: > Looks good! > Lois > > On 10/11/2017 8:45 AM, harold seigel wrote: >> Hi, >> >> Please review this small change to fix JDK-8188922.? The fix adds >> -XX:+UnlockDiagnosticVMOptions to the command line when testing a >> deprecated diagnostic option. >> >> Open webrev: >> http://cr.openjdk.java.net/~hseigel/bug_8188922/webrev/index.html >> >> JBD Bug: https://bugs.openjdk.java.net/browse/JDK-8188922 >> >> The fix was tested by running the modified VMDeprecatedOptions.java >> test with both a product JDK-10 build and a fastdebug JDK-10 build. >> >> Thanks, Harold >> > From coleen.phillimore at oracle.com Wed Oct 11 14:10:34 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 11 Oct 2017 10:10:34 -0400 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> Message-ID: Gerard, some preliminary comments. http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html *+{* *!_MutexLocker mu(SystemDictionary_lock, THREAD_);* *!_Klass* probe = dictionary->find(_d_hash, name, protection_domain);* *if (probe != NULL) return probe;* ** *+ }* I don't think you need this because dictionary->find() should have the NoSafepointVerifier, so the index will not change. http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/classLoaderData.cpp.udiff.html I think we want a global _some_dictionary_needs_resizing to avoid walking through the CLDG. And have Dictionary have a field _resizing_needed to avoid the calculation during the safepoint, which is set when adding an entry. _resizing_needed = *(number_of_entries() > (_resize_load_trigger*table_size()); *CLDG::_any_resizing_needed |= _resizing_needed; ? // or something like that. I can write more about the rationale of this change in the bug report, if needed. Thank you for doing this change. Coleen On 10/10/17 4:40 PM, Gerard Ziemski wrote: > hi all, > > Please review this change that adds dynamic resizing of system dictionaries. > > The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock > > A few notes: > > - dynamic resizing is off when either dumping or using shared spaces > - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) > - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) > > bug: https://bugs.openjdk.java.net/browse/JDK-8184765 > webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 > > > cheers > From coleen.phillimore at oracle.com Wed Oct 11 15:32:10 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 11 Oct 2017 11:32:10 -0400 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> Message-ID: <2b180329-9cf3-d83b-133b-ddd5d33b4f1f@oracle.com> Hi,? From my initial look at this, I really like new interface for walking the threads list, except every instance but 2 uses of JavaThreadIterator has a preceding ThreadsListHandle declaration. + ThreadsListHandle tlh; + JavaThreadIterator jti(tlh.list()); Could there be a wrapper that embeds the ThreadsListHandle, like: class JavaThreadsListIterator { ??? ThreadsListHandle _tlh; ... } Thanks, Coleen On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: > Greetings, > > We have a (eXtra Large) fix for the following bug: > > 8167108 inconsistent handling of SR_lock can lead to crashes > https://bugs.openjdk.java.net/browse/JDK-8167108 > > This fix adds a Safe Memory Reclamation (SMR) mechanism based on > Hazard Pointers to manage JavaThread lifecycle. > > Here's a PDF for the internal wiki that we've been using to describe > and track the work on this project: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf > > > Dan has noticed that the indenting is wrong in some of the code quotes > in the PDF that are not present in the internal wiki. We don't have a > solution for that problem yet. > > Here's the webrev for current JDK10 version of this fix: > > http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full > > This fix has been run through many rounds of JPRT and Mach5 tier[2-5] > testing, additional stress testing on Dan's Solaris X64 server, and > additional testing on Erik and Robbin's machines. > > We welcome comments, suggestions and feedback. > > Daniel Daugherty > Erik Osterlund > Robbin Ehn From rkennke at redhat.com Wed Oct 11 19:30:38 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 11 Oct 2017 21:30:38 +0200 Subject: RFR (S): 8185580: Refactor Threads::possibly_parallel_oops_do() to use Threads::parallel_java_threads_do() Message-ID: This is a follow-up to my earlier safepoint parallel cleanup work. Currently we have 2 places where we apply the parallel claiming protocol when iterating threads, that is Threads::parallel_java_threads_do() and Threads::possibly_parallel_oops_do(). In order to avoid code duplication, the latter should call the former, using a private ThreadClosure. We already had one bug (JDK-8185273) that was caused by an inconsistency between the two. The only other user of parallel_java_threads_do() already filters out any non-Java-threads, which means that doing the extra processing of the VMThread should be ok. I renamed the method to parallel_threads_do() to reflect that it not only does the Java threads. Webrev: http://cr.openjdk.java.net/~rkennke/8185580/webrev.00/ Bug: https://bugs.openjdk.java.net/browse/JDK-8185580 Tested successfully by running hotspot_gc tests, which should cover most if not all uses of those methods. Roman From david.holmes at oracle.com Wed Oct 11 21:30:12 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 12 Oct 2017 07:30:12 +1000 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <2b180329-9cf3-d83b-133b-ddd5d33b4f1f@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <2b180329-9cf3-d83b-133b-ddd5d33b4f1f@oracle.com> Message-ID: <928a46f0-4c87-7ff9-71c3-2734ddc1cbd8@oracle.com> On 12/10/2017 1:32 AM, coleen.phillimore at oracle.com wrote: > > Hi,? From my initial look at this, I really like new interface for > walking the threads list, except every instance but 2 uses of > JavaThreadIterator has a preceding ThreadsListHandle declaration. > > +? ThreadsListHandle tlh; > +? JavaThreadIterator jti(tlh.list()); > > > Could there be a wrapper that embeds the ThreadsListHandle, like: > > class JavaThreadsListIterator { > ??? ThreadsListHandle _tlh; > ... > } The ThreadsListHandle could not itself be a stack-object in that case. David > Thanks, > Coleen > > On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >> Greetings, >> >> We have a (eXtra Large) fix for the following bug: >> >> 8167108 inconsistent handling of SR_lock can lead to crashes >> https://bugs.openjdk.java.net/browse/JDK-8167108 >> >> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >> Hazard Pointers to manage JavaThread lifecycle. >> >> Here's a PDF for the internal wiki that we've been using to describe >> and track the work on this project: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >> >> >> Dan has noticed that the indenting is wrong in some of the code quotes >> in the PDF that are not present in the internal wiki. We don't have a >> solution for that problem yet. >> >> Here's the webrev for current JDK10 version of this fix: >> >> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >> >> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >> testing, additional stress testing on Dan's Solaris X64 server, and >> additional testing on Erik and Robbin's machines. >> >> We welcome comments, suggestions and feedback. >> >> Daniel Daugherty >> Erik Osterlund >> Robbin Ehn > From dean.long at oracle.com Wed Oct 11 21:58:00 2017 From: dean.long at oracle.com (dean.long at oracle.com) Date: Wed, 11 Oct 2017 14:58:00 -0700 Subject: [10] RFR: [AOT] assert(false) failed: DEBUG MESSAGE: InterpreterMacroAssembler::call_VM_base: last_sp != NULL In-Reply-To: <34517155-bd8d-56ee-86a1-7f9ffe73e2de@oracle.com> References: <8eb949bd-fa14-2722-5eaa-21a0a0c95b26@oracle.com> <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> <34517155-bd8d-56ee-86a1-7f9ffe73e2de@oracle.com> Message-ID: <01b62784-332e-8b6b-ac15-5fe4c21f9cd5@oracle.com> For AARCH64 in templateTable_arm.cpp, how about using the same code as generate_deopt_entry_for? ? __ restore_sp_after_call(Rtemp);? // Restore SP to extended SP ? __ restore_stack_top(); dl On 10/11/17 5:48 AM, jamsheed wrote: > Hi Vladimir, > > Thank you for pointing this. > > revised webrev: http://cr.openjdk.java.net/~jcm/8168712/webrev.02/ > > Best Regards, > > Jamsheed > > > On Tuesday 10 October 2017 08:41 PM, Vladimir Kozlov wrote: >> Why you added !defined(AARCH64) in templateTable_arm.cpp? Is only >> 32-bit affected? >> >> Thanks, >> Vladimir >> >> On 9/13/17 11:54 PM, Dean Long wrote: >>> It looks like you accidentally dropped >>> hotspot-compiler-dev at openjdk.java.net when you added runtime. >>> >>> dl >>> >>> >>> On 9/13/2017 11:21 PM, jamsheed wrote: >>>> (adding runtime list for inputs) >>>> >>>> On Monday 11 September 2017 11:43 PM, jamsheed wrote: >>>>> brief desc: special handling of Object. in >>>>> TemplateInterpreter::deopt_reexecute_entry >>>>> >>>>> required last_sp to be reset explicitly in normal return path >>>>> >>>>> address TemplateInterpreter::deopt_reexecute_entry(Method* method, >>>>> address bcp) { >>>>> ? assert(method->contains(bcp), "just checkin'"); >>>>> ? Bytecodes::Code code?? = Bytecodes::java_code_at(method, bcp); >>>>> ? if (code == Bytecodes::_return) { >>>>> ??? // This is used for deopt during registration of finalizers >>>>> ??? // during Object..? We simply need to resume execution at >>>>> ??? // the standard return vtos bytecode to pop the frame normally. >>>>> ??? // reexecuting the real bytecode would cause double registration >>>>> ??? // of the finalizable object. >>>>> ??? return _normal_table.entry(Bytecodes::_return).entry(vtos); >>>> >>>> last_sp ! = null not an issue for this case, so i skip the assert >>>> in debug build >>>> >>>> http://cr.openjdk.java.net/~jcm/8168712/webrev.01/ >>>> >>>> Please review. >>>> >>>> Best Regards, >>>> Jamsheed >>>> >>>> >>>> >>>> >>>> >>> > From david.holmes at oracle.com Thu Oct 12 01:20:27 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 12 Oct 2017 11:20:27 +1000 Subject: RFR(XL): 8167108 - SMR and JavaThread Lifecycle In-Reply-To: <928a46f0-4c87-7ff9-71c3-2734ddc1cbd8@oracle.com> References: <1e50bb73-840c-fc3a-81ad-31f83037093f@oracle.com> <2b180329-9cf3-d83b-133b-ddd5d33b4f1f@oracle.com> <928a46f0-4c87-7ff9-71c3-2734ddc1cbd8@oracle.com> Message-ID: <099bb2f1-486e-d8c0-4ade-0cbf2b31a13d@oracle.com> Please ignore this gaseous brain explosion. :) Thanks, David On 12/10/2017 7:30 AM, David Holmes wrote: > On 12/10/2017 1:32 AM, coleen.phillimore at oracle.com wrote: >> >> Hi,? From my initial look at this, I really like new interface for >> walking the threads list, except every instance but 2 uses of >> JavaThreadIterator has a preceding ThreadsListHandle declaration. >> >> +? ThreadsListHandle tlh; >> +? JavaThreadIterator jti(tlh.list()); >> >> >> Could there be a wrapper that embeds the ThreadsListHandle, like: >> >> class JavaThreadsListIterator { >> ???? ThreadsListHandle _tlh; >> ... >> } > > The ThreadsListHandle could not itself be a stack-object in that case. > > David > >> Thanks, >> Coleen >> >> On 10/9/17 3:41 PM, Daniel D. Daugherty wrote: >>> Greetings, >>> >>> We have a (eXtra Large) fix for the following bug: >>> >>> 8167108 inconsistent handling of SR_lock can lead to crashes >>> https://bugs.openjdk.java.net/browse/JDK-8167108 >>> >>> This fix adds a Safe Memory Reclamation (SMR) mechanism based on >>> Hazard Pointers to manage JavaThread lifecycle. >>> >>> Here's a PDF for the internal wiki that we've been using to describe >>> and track the work on this project: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/SMR_and_JavaThread_Lifecycle-JDK10-04.pdf >>> >>> >>> Dan has noticed that the indenting is wrong in some of the code quotes >>> in the PDF that are not present in the internal wiki. We don't have a >>> solution for that problem yet. >>> >>> Here's the webrev for current JDK10 version of this fix: >>> >>> http://cr.openjdk.java.net/~dcubed/8167108-webrev/jdk10-04-full >>> >>> This fix has been run through many rounds of JPRT and Mach5 tier[2-5] >>> testing, additional stress testing on Dan's Solaris X64 server, and >>> additional testing on Erik and Robbin's machines. >>> >>> We welcome comments, suggestions and feedback. >>> >>> Daniel Daugherty >>> Erik Osterlund >>> Robbin Ehn >> From mikhailo.seledtsov at oracle.com Thu Oct 12 03:37:17 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Wed, 11 Oct 2017 20:37:17 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> Message-ID: <59DEE36D.6020108@oracle.com> Hi Robbin, On 10/11/17, 3:06 AM, Robbin Ehn wrote: > Hi, > > jtreg now spits out: > dockerSupport() threw exception: java.io.IOException: Cannot run > program "docker": error=2, No such file or directory > > Doesn't seem right. > When I saw your message at first I was afraid that this stops/blocks test execution, despite me running 5 rounds on internal automated test system. I just re-tested again manually on a machine w/o docker engine, and see that the test run continues and completes normally. However, I can see the message spitted out. Thank you for pointing this out. I will file a bug for this. By looking at the code of test/jtreg-ext/requires/VMProps.java, the dockerSupport() method has a try/catch clause around the checkDockerSupport(), the method which actually does the check. So I assumed that the exception is handled properly. We must have TraceException enabled at that point, or some other mechanism that makes correctly caught exceptions verbose. Thank you, Misha > /Robbin > > On 09/22/2017 02:58 AM, mikhailo wrote: >> Please review this initial drop of Docker test utils and a sanity >> test. This change lays ground >> for further test development and test utils improvement in this area. >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >> Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ >> Testing: >> - run this test on machine with Docker enabled - works >> - run this test on Linux-x64 with no Docker engine or Docker >> disabled - test skipped (as expected) >> - run this test on automated system - in progress >> >> >> Thank you, >> Misha >> From mikhailo.seledtsov at oracle.com Thu Oct 12 03:50:17 2017 From: mikhailo.seledtsov at oracle.com (Mikhailo Seledtsov) Date: Wed, 11 Oct 2017 20:50:17 -0700 Subject: RFR(S): 8181592: [TESTBUG] Docker test utils and docker jdk basic test In-Reply-To: <59DEE36D.6020108@oracle.com> References: <6b95b720-a2cc-39e1-c0c1-6885b106ac16@oracle.com> <59DEE36D.6020108@oracle.com> Message-ID: <59DEE679.2080203@oracle.com> Filed: JDK-8189213 - [TESTBUG] Running jtreg tests on machine without docker shows extra message On 10/11/17, 8:37 PM, Mikhailo Seledtsov wrote: > Hi Robbin, > > On 10/11/17, 3:06 AM, Robbin Ehn wrote: >> Hi, >> >> jtreg now spits out: >> dockerSupport() threw exception: java.io.IOException: Cannot run >> program "docker": error=2, No such file or directory >> >> Doesn't seem right. >> > When I saw your message at first I was afraid that this stops/blocks > test execution, despite me running 5 rounds on internal automated test > system. I just re-tested again manually on a machine w/o docker > engine, and see that the test run continues and completes normally. > However, I can see the message spitted out. > > Thank you for pointing this out. I will file a bug for this. > > By looking at the code of test/jtreg-ext/requires/VMProps.java, the > dockerSupport() method has a try/catch clause around the > checkDockerSupport(), the method which actually does the check. So I > assumed that the exception is handled properly. We must have > TraceException enabled at that point, or some other mechanism that > makes correctly caught exceptions verbose. > > Thank you, > Misha >> /Robbin >> >> On 09/22/2017 02:58 AM, mikhailo wrote: >>> Please review this initial drop of Docker test utils and a sanity >>> test. This change lays ground >>> for further test development and test utils improvement in this area. >>> >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8181592 >>> Webrev: http://cr.openjdk.java.net/~mseledtsov/8181592.00/ >>> Testing: >>> - run this test on machine with Docker enabled - works >>> - run this test on Linux-x64 with no Docker engine or Docker >>> disabled - test skipped (as expected) >>> - run this test on automated system - in progress >>> >>> >>> Thank you, >>> Misha >>> From yasuenag at gmail.com Thu Oct 12 04:10:58 2017 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 12 Oct 2017 13:10:58 +0900 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: Message-ID: Hi all, This change was tried to push via JPRT, but it failed because test/hotspot/jtreg/runtime/SharedArchiveFile/MaxMetaspaceSizeTest.java was failed. Also I heard the comment that the changes in arguments.cpp which set ergonomic value to CompressedClassSpaceSize should be moved to Metaspace::ergo_initialize(). I uploaded new webrev. Could you review again? This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and test/hotspot/jtreg/runtime/SharedArchiveFile. http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ Thanks, Yasumasa 2017-10-10 21:02 GMT+09:00 : > Hi Yasumasa, I like this better. I will sponsor it. I think it only needs > one reviewer, unless there were more? Please send me the result of hg > export. > thanks, > Coleen > > > On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: >> >> Hi Coleen, >> >>>> I will sponsor this for you. >> >> Thanks! >> >> >>>> Hi, this looks reasonable but I would prefer that the code in >>>> arguments.cpp call something in metaspace.cpp, like >>>> check_metaspace_sizes() >>>> so that you don't have to tell arguments what the MediumChunk size is. >> >> I added new function calculate_min_metaspace_size() in metaspace.cpp >> to get minimum metaspace size, and uses return value from this >> function in metaspace initialization and metaspace size check. >> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ >> >> Could you review again? >> >> >> Thanks, >> >> Yasumasa >> >> >> 2017-10-10 6:07 GMT+09:00 Man Cao : >>> >>> Thanks for you both updating and sponsoring the webrev! >>> We plan to back-port it to our JDK9 once it is submitted. >>> >>> -Man >>> >>> On Mon, Oct 9, 2017 at 6:15 AM, wrote: >>>> >>>> Hi, this looks reasonable but I would prefer that the code in >>>> arguments.cpp call something in metaspace.cpp, like >>>> check_metaspace_sizes() >>>> so that you don't have to tell arguments what the MediumChunk size is. >>>> >>>> I will sponsor this for you. >>>> >>>> thanks, >>>> Coleen >>>> >>>> >>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I added a testcase for this issue in new webrev: >>>>> >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> >>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I uploaded webrev for this issue against jdk10/hs. >>>>>> Could you review it? >>>>>> >>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>>>> >>>>>> >>>>>> I cannot access JPRT. So I need a sponsor. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Yasumasa >>>>>> >>>>>> >>>>>> >>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>>> >>>>>>> Thank Yasumasa and Stefan for the responses. >>>>>>> >>>>>>> Good to know that the patch is not blocked due to breaking some >>>>>>> internal >>>>>>> invariants/assumptions, but just due to its P5 status. >>>>>>> Is it possible to push it to P4? >>>>>>> >>>>>>> -Man >>>>>>> >>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>>> >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>>> >>>>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>>>> space >>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>> this >>>>>>>>> on the >>>>>>>>> hotspot-runtime-dev list. >>>>>>>> >>>>>>>> >>>>>>>> I will send review request against jdk10/master or jdk10/hs after >>>>>>>> repos >>>>>>>> are opened. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Yasumasa >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>>> >>>>>>>>> Hi Man, >>>>>>>>> >>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>>> >>>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>>> >>>>>>>>>> Do you have any thoughts on why this patch has been pending for 2+ >>>>>>>>>> years? This patch could really save us from some annoying issues >>>>>>>>>> since we >>>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>>> >>>>>>>>> >>>>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>>>> space >>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>> this >>>>>>>>> on the >>>>>>>>> hotspot-runtime-dev list. >>>>>>>>> >>>>>>>>> StefanK >>>>>>>>> >>>>>>>>>> -Man >>>>>>>>>> >>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I wonder if there is any recent update on the patch for >>>>>>>>>> JDK-8087291. >>>>>>>>>> Is it possible to push this patch into JDK9? Except for its >>>>>>>>>> low >>>>>>>>>> priority (P5), >>>>>>>>>> is there any complication that prevents this patch getting >>>>>>>>>> approved >>>>>>>>>> (for example, some JVM logic requires >>>>>>>>>> CompressedClassSpaceSize >>>>>>>>>> to be >>>>>>>>>> 1GB by default)? >>>>>>>>>> >>>>>>>>>> I work in the Java Platform Team at Google. We have >>>>>>>>>> encountered >>>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>>>>> confusing 1GB >>>>>>>>>> memory reserved by metaspace, >>>>>>>>>> regardless of MaxMetaspaceSize value. The root cause for >>>>>>>>>> these >>>>>>>>>> issues is that CompressedClassSpaceSize is not automatically >>>>>>>>>> capped >>>>>>>>>> by MaxMetaspaceSize >>>>>>>>>> during VM initialization, and this patch seems fix the root >>>>>>>>>> cause. >>>>>>>>>> (I'm aware that even after this patch, the reserved size >>>>>>>>>> could >>>>>>>>>> still >>>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>>> but it is better than the current situation.) >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Man >>>>>>>>>> >>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>>> >>>>>>>>>> Thank you for your comment! >>>>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>>>> command >>>>>>>>>> line. >>>>>>>>>> > >>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It works on fastdebug VM. >>>>>>>>>> Please review again. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Yasumasa >>>>>>>>>> >>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>>> > Yasumasa, >>>>>>>>>> > >>>>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>>>> command >>>>>>>>>> line. >>>>>>>>>> > >>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>> > >>>>>>>>>> > On a linux system I get this when I build with your >>>>>>>>>> patch. >>>>>>>>>> > >>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>> >> # To suppress the following error report, specify >>>>>>>>>> this >>>>>>>>>> argument >>>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>>> >> # >>>>>>>>>> >> # A fatal error has been detected by the Java >>>>>>>>>> Runtime >>>>>>>>>> Environment: >>>>>>>>>> >> # >>>>>>>>>> >> # Internal Error >>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>>> ClassMediumChunk) >>>>>>>>>> failed: Not a >>>>>>>>>> >> humongous chunk >>>>>>>>>> >> # >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > Jon >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>>>>> >> I want to continue to discuss about >>>>>>>>>> CompressedClassSpace and >>>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>> Should I resize CompressedClassSpaceSize than to >>>>>>>>>> show >>>>>>>>>> error message? >>>>>>>>>> >>> If you add slightly better heuristics for the setup >>>>>>>>>> of >>>>>>>>>> the >>>>>>>>>> CompressedClassSpaceSize flag, for example lowering the >>>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is set, >>>>>>>>>> then >>>>>>>>>> it >>>>>>>>>> might be less likely that you'll hit the >>>>>>>>>> OutOfMemoryError >>>>>>>>>> when >>>>>>>>>> the system is set up with strict overcommit settings. >>>>>>>>>> >> >>>>>>>>>> >> I've uploaded new webrev: >>>>>>>>>> >> >>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >> >>>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>>> >> >>>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is greater >>>>>>>>>> than >>>>>>>>>> MaxMetaspaceSize, >>>>>>>>>> >> VM will fail with error message. >>>>>>>>>> >> >>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will be set to >>>>>>>>>> MaxMetaspaceSize >>>>>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> Thanks, >>>>>>>>>> >> >>>>>>>>>> >> Yasumasa >>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>> > From david.holmes at oracle.com Thu Oct 12 04:22:29 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 12 Oct 2017 14:22:29 +1000 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: Message-ID: Hi Yasumasa, On 12/10/2017 2:10 PM, Yasumasa Suenaga wrote: > Hi all, > > This change was tried to push via JPRT, but it failed because > test/hotspot/jtreg/runtime/SharedArchiveFile/MaxMetaspaceSizeTest.java > was failed. > Also I heard the comment that the changes in arguments.cpp which set > ergonomic value to CompressedClassSpaceSize should be moved to > Metaspace::ergo_initialize(). > > I uploaded new webrev. Could you review again? > This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and > test/hotspot/jtreg/runtime/SharedArchiveFile. > > http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ I'll let Coleen cover the technical details, I'll just pick up a few style nits: 3328 /* Initial virtual space size will be calculated at global_initialize() */ Use // comment 3329 size_t min_metaspace_sz = 3330 VIRTUALSPACEMULTIPLIER * InitialBootClassLoaderMetaspaceSize; ^ should indent only to here 3336 FLAG_SET_ERGO(size_t, CompressedClassSpaceSize, 3337 MaxMetaspaceSize - min_metaspace_sz); ^ should indent only to here 3341 FLAG_SET_ERGO(size_t, InitialBootClassLoaderMetaspaceSize, 3342 min_metaspace_sz); ^ should indent only to here You may have time to fix these in place before anyone else sees the webrev :) Thanks, David ----- > > Thanks, > > Yasumasa > > > > 2017-10-10 21:02 GMT+09:00 : >> Hi Yasumasa, I like this better. I will sponsor it. I think it only needs >> one reviewer, unless there were more? Please send me the result of hg >> export. >> thanks, >> Coleen >> >> >> On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: >>> >>> Hi Coleen, >>> >>>>> I will sponsor this for you. >>> >>> Thanks! >>> >>> >>>>> Hi, this looks reasonable but I would prefer that the code in >>>>> arguments.cpp call something in metaspace.cpp, like >>>>> check_metaspace_sizes() >>>>> so that you don't have to tell arguments what the MediumChunk size is. >>> >>> I added new function calculate_min_metaspace_size() in metaspace.cpp >>> to get minimum metaspace size, and uses return value from this >>> function in metaspace initialization and metaspace size check. >>> >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ >>> >>> Could you review again? >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> 2017-10-10 6:07 GMT+09:00 Man Cao : >>>> >>>> Thanks for you both updating and sponsoring the webrev! >>>> We plan to back-port it to our JDK9 once it is submitted. >>>> >>>> -Man >>>> >>>> On Mon, Oct 9, 2017 at 6:15 AM, wrote: >>>>> >>>>> Hi, this looks reasonable but I would prefer that the code in >>>>> arguments.cpp call something in metaspace.cpp, like >>>>> check_metaspace_sizes() >>>>> so that you don't have to tell arguments what the MediumChunk size is. >>>>> >>>>> I will sponsor this for you. >>>>> >>>>> thanks, >>>>> Coleen >>>>> >>>>> >>>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I added a testcase for this issue in new webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Yasumasa >>>>>> >>>>>> >>>>>> >>>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I uploaded webrev for this issue against jdk10/hs. >>>>>>> Could you review it? >>>>>>> >>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>>>>> >>>>>>> >>>>>>> I cannot access JPRT. So I need a sponsor. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Yasumasa >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>>>> >>>>>>>> Thank Yasumasa and Stefan for the responses. >>>>>>>> >>>>>>>> Good to know that the patch is not blocked due to breaking some >>>>>>>> internal >>>>>>>> invariants/assumptions, but just due to its P5 status. >>>>>>>> Is it possible to push it to P4? >>>>>>>> >>>>>>>> -Man >>>>>>>> >>>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>>>> >>>>>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>>>>> space >>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>> this >>>>>>>>>> on the >>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>> >>>>>>>>> >>>>>>>>> I will send review request against jdk10/master or jdk10/hs after >>>>>>>>> repos >>>>>>>>> are opened. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Yasumasa >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>>>> >>>>>>>>>> Hi Man, >>>>>>>>>> >>>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>>>> >>>>>>>>>>> Do you have any thoughts on why this patch has been pending for 2+ >>>>>>>>>>> years? This patch could really save us from some annoying issues >>>>>>>>>>> since we >>>>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>>>>> space >>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>> this >>>>>>>>>> on the >>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>> >>>>>>>>>> StefanK >>>>>>>>>> >>>>>>>>>>> -Man >>>>>>>>>>> >>>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I wonder if there is any recent update on the patch for >>>>>>>>>>> JDK-8087291. >>>>>>>>>>> Is it possible to push this patch into JDK9? Except for its >>>>>>>>>>> low >>>>>>>>>>> priority (P5), >>>>>>>>>>> is there any complication that prevents this patch getting >>>>>>>>>>> approved >>>>>>>>>>> (for example, some JVM logic requires >>>>>>>>>>> CompressedClassSpaceSize >>>>>>>>>>> to be >>>>>>>>>>> 1GB by default)? >>>>>>>>>>> >>>>>>>>>>> I work in the Java Platform Team at Google. We have >>>>>>>>>>> encountered >>>>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>>>>>> confusing 1GB >>>>>>>>>>> memory reserved by metaspace, >>>>>>>>>>> regardless of MaxMetaspaceSize value. The root cause for >>>>>>>>>>> these >>>>>>>>>>> issues is that CompressedClassSpaceSize is not automatically >>>>>>>>>>> capped >>>>>>>>>>> by MaxMetaspaceSize >>>>>>>>>>> during VM initialization, and this patch seems fix the root >>>>>>>>>>> cause. >>>>>>>>>>> (I'm aware that even after this patch, the reserved size >>>>>>>>>>> could >>>>>>>>>>> still >>>>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>>>> but it is better than the current situation.) >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Man >>>>>>>>>>> >>>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>>>> >>>>>>>>>>> Thank you for your comment! >>>>>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>>>>> command >>>>>>>>>>> line. >>>>>>>>>>> > >>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It works on fastdebug VM. >>>>>>>>>>> Please review again. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Yasumasa >>>>>>>>>>> >>>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>>>> > Yasumasa, >>>>>>>>>>> > >>>>>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>>>>> command >>>>>>>>>>> line. >>>>>>>>>>> > >>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>> > >>>>>>>>>>> > On a linux system I get this when I build with your >>>>>>>>>>> patch. >>>>>>>>>>> > >>>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>> >> # To suppress the following error report, specify >>>>>>>>>>> this >>>>>>>>>>> argument >>>>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>>>> >> # >>>>>>>>>>> >> # A fatal error has been detected by the Java >>>>>>>>>>> Runtime >>>>>>>>>>> Environment: >>>>>>>>>>> >> # >>>>>>>>>>> >> # Internal Error >>>>>>>>>>> >> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>>>> ClassMediumChunk) >>>>>>>>>>> failed: Not a >>>>>>>>>>> >> humongous chunk >>>>>>>>>>> >> # >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > Jon >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>>>>>> >> I want to continue to discuss about >>>>>>>>>>> CompressedClassSpace and >>>>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>> Should I resize CompressedClassSpaceSize than to >>>>>>>>>>> show >>>>>>>>>>> error message? >>>>>>>>>>> >>> If you add slightly better heuristics for the setup >>>>>>>>>>> of >>>>>>>>>>> the >>>>>>>>>>> CompressedClassSpaceSize flag, for example lowering the >>>>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is set, >>>>>>>>>>> then >>>>>>>>>>> it >>>>>>>>>>> might be less likely that you'll hit the >>>>>>>>>>> OutOfMemoryError >>>>>>>>>>> when >>>>>>>>>>> the system is set up with strict overcommit settings. >>>>>>>>>>> >> >>>>>>>>>>> >> I've uploaded new webrev: >>>>>>>>>>> >> >>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >> >>>>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>>>> >> >>>>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is greater >>>>>>>>>>> than >>>>>>>>>>> MaxMetaspaceSize, >>>>>>>>>>> >> VM will fail with error message. >>>>>>>>>>> >> >>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will be set to >>>>>>>>>>> MaxMetaspaceSize >>>>>>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> Thanks, >>>>>>>>>>> >> >>>>>>>>>>> >> Yasumasa >>>>>>>>>>> >> >>>>>>>>>>> >>>>>>>>>>> >> From yasuenag at gmail.com Thu Oct 12 04:47:20 2017 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 12 Oct 2017 13:47:20 +0900 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: Message-ID: Hi David, > You may have time to fix these in place before anyone else sees the webrev I've fixed that: http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.07/ I checked raw text of your email, so I believe the indent is correct. If I have mistake(s), please tell me. Thanks, Yasumasa 2017-10-12 13:22 GMT+09:00 David Holmes : > Hi Yasumasa, > > On 12/10/2017 2:10 PM, Yasumasa Suenaga wrote: >> >> Hi all, >> >> This change was tried to push via JPRT, but it failed because >> test/hotspot/jtreg/runtime/SharedArchiveFile/MaxMetaspaceSizeTest.java >> was failed. >> Also I heard the comment that the changes in arguments.cpp which set >> ergonomic value to CompressedClassSpaceSize should be moved to >> Metaspace::ergo_initialize(). >> >> I uploaded new webrev. Could you review again? >> This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and >> test/hotspot/jtreg/runtime/SharedArchiveFile. >> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ > > > I'll let Coleen cover the technical details, I'll just pick up a few style > nits: > > 3328 /* Initial virtual space size will be calculated at > global_initialize() */ > > Use // comment > > 3329 size_t min_metaspace_sz = > 3330 VIRTUALSPACEMULTIPLIER * > InitialBootClassLoaderMetaspaceSize; > > ^ should indent only to here > > 3336 FLAG_SET_ERGO(size_t, CompressedClassSpaceSize, > 3337 MaxMetaspaceSize - > min_metaspace_sz); > ^ should indent only to here > > 3341 FLAG_SET_ERGO(size_t, InitialBootClassLoaderMetaspaceSize, > 3342 min_metaspace_sz); > > ^ should indent only to here > > You may have time to fix these in place before anyone else sees the webrev > :) > > Thanks, > David > ----- > > >> >> Thanks, >> >> Yasumasa >> >> >> >> 2017-10-10 21:02 GMT+09:00 : >>> >>> Hi Yasumasa, I like this better. I will sponsor it. I think it only >>> needs >>> one reviewer, unless there were more? Please send me the result of hg >>> export. >>> thanks, >>> Coleen >>> >>> >>> On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: >>>> >>>> >>>> Hi Coleen, >>>> >>>>>> I will sponsor this for you. >>>> >>>> >>>> Thanks! >>>> >>>> >>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>> check_metaspace_sizes() >>>>>> so that you don't have to tell arguments what the MediumChunk size is. >>>> >>>> >>>> I added new function calculate_min_metaspace_size() in metaspace.cpp >>>> to get minimum metaspace size, and uses return value from this >>>> function in metaspace initialization and metaspace size check. >>>> >>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ >>>> >>>> Could you review again? >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> 2017-10-10 6:07 GMT+09:00 Man Cao : >>>>> >>>>> >>>>> Thanks for you both updating and sponsoring the webrev! >>>>> We plan to back-port it to our JDK9 once it is submitted. >>>>> >>>>> -Man >>>>> >>>>> On Mon, Oct 9, 2017 at 6:15 AM, wrote: >>>>>> >>>>>> >>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>> check_metaspace_sizes() >>>>>> so that you don't have to tell arguments what the MediumChunk size is. >>>>>> >>>>>> I will sponsor this for you. >>>>>> >>>>>> thanks, >>>>>> Coleen >>>>>> >>>>>> >>>>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>>>>> >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I added a testcase for this issue in new webrev: >>>>>>> >>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Yasumasa >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >>>>>>>> >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I uploaded webrev for this issue against jdk10/hs. >>>>>>>> Could you review it? >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>>>>>> >>>>>>>> >>>>>>>> I cannot access JPRT. So I need a sponsor. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Yasumasa >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>>>>> >>>>>>>>> >>>>>>>>> Thank Yasumasa and Stefan for the responses. >>>>>>>>> >>>>>>>>> Good to know that the patch is not blocked due to breaking some >>>>>>>>> internal >>>>>>>>> invariants/assumptions, but just due to its P5 status. >>>>>>>>> Is it possible to push it to P4? >>>>>>>>> >>>>>>>>> -Man >>>>>>>>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>>>>> >>>>>>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>>>>>> space >>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>> this >>>>>>>>>>> on the >>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I will send review request against jdk10/master or jdk10/hs after >>>>>>>>>> repos >>>>>>>>>> are opened. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Yasumasa >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi Man, >>>>>>>>>>> >>>>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>>>>> >>>>>>>>>>>> Do you have any thoughts on why this patch has been pending for >>>>>>>>>>>> 2+ >>>>>>>>>>>> years? This patch could really save us from some annoying issues >>>>>>>>>>>> since we >>>>>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>>>>>> space >>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>> this >>>>>>>>>>> on the >>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>> >>>>>>>>>>> StefanK >>>>>>>>>>> >>>>>>>>>>>> -Man >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>>>>> > wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi all, >>>>>>>>>>>> >>>>>>>>>>>> I wonder if there is any recent update on the patch for >>>>>>>>>>>> JDK-8087291. >>>>>>>>>>>> Is it possible to push this patch into JDK9? Except for >>>>>>>>>>>> its >>>>>>>>>>>> low >>>>>>>>>>>> priority (P5), >>>>>>>>>>>> is there any complication that prevents this patch >>>>>>>>>>>> getting >>>>>>>>>>>> approved >>>>>>>>>>>> (for example, some JVM logic requires >>>>>>>>>>>> CompressedClassSpaceSize >>>>>>>>>>>> to be >>>>>>>>>>>> 1GB by default)? >>>>>>>>>>>> >>>>>>>>>>>> I work in the Java Platform Team at Google. We have >>>>>>>>>>>> encountered >>>>>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>>>>>>> confusing 1GB >>>>>>>>>>>> memory reserved by metaspace, >>>>>>>>>>>> regardless of MaxMetaspaceSize value. The root cause for >>>>>>>>>>>> these >>>>>>>>>>>> issues is that CompressedClassSpaceSize is not >>>>>>>>>>>> automatically >>>>>>>>>>>> capped >>>>>>>>>>>> by MaxMetaspaceSize >>>>>>>>>>>> during VM initialization, and this patch seems fix the >>>>>>>>>>>> root >>>>>>>>>>>> cause. >>>>>>>>>>>> (I'm aware that even after this patch, the reserved size >>>>>>>>>>>> could >>>>>>>>>>>> still >>>>>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>>>>> but it is better than the current situation.) >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Man >>>>>>>>>>>> >>>>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>>>>> >>>>>>>>>>>> Thank you for your comment! >>>>>>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>>>>>> command >>>>>>>>>>>> line. >>>>>>>>>>>> > >>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>>>>> >>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> It works on fastdebug VM. >>>>>>>>>>>> Please review again. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Yasumasa >>>>>>>>>>>> >>>>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>>>>> > Yasumasa, >>>>>>>>>>>> > >>>>>>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>>>>>> command >>>>>>>>>>>> line. >>>>>>>>>>>> > >>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>> > >>>>>>>>>>>> > On a linux system I get this when I build with >>>>>>>>>>>> your >>>>>>>>>>>> patch. >>>>>>>>>>>> > >>>>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>> >> # To suppress the following error report, specify >>>>>>>>>>>> this >>>>>>>>>>>> argument >>>>>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>>>>> >> # >>>>>>>>>>>> >> # A fatal error has been detected by the Java >>>>>>>>>>>> Runtime >>>>>>>>>>>> Environment: >>>>>>>>>>>> >> # >>>>>>>>>>>> >> # Internal Error >>>>>>>>>>>> >> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>>>>> ClassMediumChunk) >>>>>>>>>>>> failed: Not a >>>>>>>>>>>> >> humongous chunk >>>>>>>>>>>> >> # >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > Jon >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>>>>>>> >> I want to continue to discuss about >>>>>>>>>>>> CompressedClassSpace and >>>>>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>> Should I resize CompressedClassSpaceSize than >>>>>>>>>>>> to >>>>>>>>>>>> show >>>>>>>>>>>> error message? >>>>>>>>>>>> >>> If you add slightly better heuristics for the >>>>>>>>>>>> setup >>>>>>>>>>>> of >>>>>>>>>>>> the >>>>>>>>>>>> CompressedClassSpaceSize flag, for example lowering >>>>>>>>>>>> the >>>>>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is >>>>>>>>>>>> set, >>>>>>>>>>>> then >>>>>>>>>>>> it >>>>>>>>>>>> might be less likely that you'll hit the >>>>>>>>>>>> OutOfMemoryError >>>>>>>>>>>> when >>>>>>>>>>>> the system is set up with strict overcommit settings. >>>>>>>>>>>> >> >>>>>>>>>>>> >> I've uploaded new webrev: >>>>>>>>>>>> >> >>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >> >>>>>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>>>>> >> >>>>>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is greater >>>>>>>>>>>> than >>>>>>>>>>>> MaxMetaspaceSize, >>>>>>>>>>>> >> VM will fail with error message. >>>>>>>>>>>> >> >>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will be set >>>>>>>>>>>> to >>>>>>>>>>>> MaxMetaspaceSize >>>>>>>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> Thanks, >>>>>>>>>>>> >> >>>>>>>>>>>> >> Yasumasa >>>>>>>>>>>> >> >>>>>>>>>>>> >>>>>>>>>>>> >>> > From david.holmes at oracle.com Thu Oct 12 04:55:52 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 12 Oct 2017 14:55:52 +1000 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: Message-ID: <69f62651-dac3-3426-bb4f-6d4dcb03abb8@oracle.com> On 12/10/2017 2:47 PM, Yasumasa Suenaga wrote: > Hi David, > >> You may have time to fix these in place before anyone else sees the webrev > > I've fixed that: > > http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.07/ > > > I checked raw text of your email, so I believe the indent is correct. > If I have mistake(s), please tell me. Sorry they are off. Basic rules: var = some long value; indent by 4. foo(arg1, arg2, arg3, arg4); align the args on the two lines as shown. Hope that is clearer. Thanks, David > > Thanks, > > Yasumasa > > > 2017-10-12 13:22 GMT+09:00 David Holmes : >> Hi Yasumasa, >> >> On 12/10/2017 2:10 PM, Yasumasa Suenaga wrote: >>> >>> Hi all, >>> >>> This change was tried to push via JPRT, but it failed because >>> test/hotspot/jtreg/runtime/SharedArchiveFile/MaxMetaspaceSizeTest.java >>> was failed. >>> Also I heard the comment that the changes in arguments.cpp which set >>> ergonomic value to CompressedClassSpaceSize should be moved to >>> Metaspace::ergo_initialize(). >>> >>> I uploaded new webrev. Could you review again? >>> This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and >>> test/hotspot/jtreg/runtime/SharedArchiveFile. >>> >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ >> >> >> I'll let Coleen cover the technical details, I'll just pick up a few style >> nits: >> >> 3328 /* Initial virtual space size will be calculated at >> global_initialize() */ >> >> Use // comment >> >> 3329 size_t min_metaspace_sz = >> 3330 VIRTUALSPACEMULTIPLIER * >> InitialBootClassLoaderMetaspaceSize; >> >> ^ should indent only to here >> >> 3336 FLAG_SET_ERGO(size_t, CompressedClassSpaceSize, >> 3337 MaxMetaspaceSize - >> min_metaspace_sz); >> ^ should indent only to here >> >> 3341 FLAG_SET_ERGO(size_t, InitialBootClassLoaderMetaspaceSize, >> 3342 min_metaspace_sz); >> >> ^ should indent only to here >> >> You may have time to fix these in place before anyone else sees the webrev >> :) >> >> Thanks, >> David >> ----- >> >> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> >>> 2017-10-10 21:02 GMT+09:00 : >>>> >>>> Hi Yasumasa, I like this better. I will sponsor it. I think it only >>>> needs >>>> one reviewer, unless there were more? Please send me the result of hg >>>> export. >>>> thanks, >>>> Coleen >>>> >>>> >>>> On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: >>>>> >>>>> >>>>> Hi Coleen, >>>>> >>>>>>> I will sponsor this for you. >>>>> >>>>> >>>>> Thanks! >>>>> >>>>> >>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>> check_metaspace_sizes() >>>>>>> so that you don't have to tell arguments what the MediumChunk size is. >>>>> >>>>> >>>>> I added new function calculate_min_metaspace_size() in metaspace.cpp >>>>> to get minimum metaspace size, and uses return value from this >>>>> function in metaspace initialization and metaspace size check. >>>>> >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ >>>>> >>>>> Could you review again? >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> 2017-10-10 6:07 GMT+09:00 Man Cao : >>>>>> >>>>>> >>>>>> Thanks for you both updating and sponsoring the webrev! >>>>>> We plan to back-port it to our JDK9 once it is submitted. >>>>>> >>>>>> -Man >>>>>> >>>>>> On Mon, Oct 9, 2017 at 6:15 AM, wrote: >>>>>>> >>>>>>> >>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>> check_metaspace_sizes() >>>>>>> so that you don't have to tell arguments what the MediumChunk size is. >>>>>>> >>>>>>> I will sponsor this for you. >>>>>>> >>>>>>> thanks, >>>>>>> Coleen >>>>>>> >>>>>>> >>>>>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>>>>>> >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I added a testcase for this issue in new webrev: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Yasumasa >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I uploaded webrev for this issue against jdk10/hs. >>>>>>>>> Could you review it? >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>>>>>>> >>>>>>>>> >>>>>>>>> I cannot access JPRT. So I need a sponsor. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Yasumasa >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thank Yasumasa and Stefan for the responses. >>>>>>>>>> >>>>>>>>>> Good to know that the patch is not blocked due to breaking some >>>>>>>>>> internal >>>>>>>>>> invariants/assumptions, but just due to its P5 status. >>>>>>>>>> Is it possible to push it to P4? >>>>>>>>>> >>>>>>>>>> -Man >>>>>>>>>> >>>>>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>>>>>> >>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>>>>>>> space >>>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>>> this >>>>>>>>>>>> on the >>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I will send review request against jdk10/master or jdk10/hs after >>>>>>>>>>> repos >>>>>>>>>>> are opened. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Yasumasa >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Man, >>>>>>>>>>>> >>>>>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>>>>>> >>>>>>>>>>>>> Do you have any thoughts on why this patch has been pending for >>>>>>>>>>>>> 2+ >>>>>>>>>>>>> years? This patch could really save us from some annoying issues >>>>>>>>>>>>> since we >>>>>>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed class >>>>>>>>>>>> space >>>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>>> this >>>>>>>>>>>> on the >>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>> >>>>>>>>>>>> StefanK >>>>>>>>>>>> >>>>>>>>>>>>> -Man >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>>>>>> > wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi all, >>>>>>>>>>>>> >>>>>>>>>>>>> I wonder if there is any recent update on the patch for >>>>>>>>>>>>> JDK-8087291. >>>>>>>>>>>>> Is it possible to push this patch into JDK9? Except for >>>>>>>>>>>>> its >>>>>>>>>>>>> low >>>>>>>>>>>>> priority (P5), >>>>>>>>>>>>> is there any complication that prevents this patch >>>>>>>>>>>>> getting >>>>>>>>>>>>> approved >>>>>>>>>>>>> (for example, some JVM logic requires >>>>>>>>>>>>> CompressedClassSpaceSize >>>>>>>>>>>>> to be >>>>>>>>>>>>> 1GB by default)? >>>>>>>>>>>>> >>>>>>>>>>>>> I work in the Java Platform Team at Google. We have >>>>>>>>>>>>> encountered >>>>>>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>>>>>>>> confusing 1GB >>>>>>>>>>>>> memory reserved by metaspace, >>>>>>>>>>>>> regardless of MaxMetaspaceSize value. The root cause for >>>>>>>>>>>>> these >>>>>>>>>>>>> issues is that CompressedClassSpaceSize is not >>>>>>>>>>>>> automatically >>>>>>>>>>>>> capped >>>>>>>>>>>>> by MaxMetaspaceSize >>>>>>>>>>>>> during VM initialization, and this patch seems fix the >>>>>>>>>>>>> root >>>>>>>>>>>>> cause. >>>>>>>>>>>>> (I'm aware that even after this patch, the reserved size >>>>>>>>>>>>> could >>>>>>>>>>>>> still >>>>>>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>>>>>> but it is better than the current situation.) >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Man >>>>>>>>>>>>> >>>>>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you for your comment! >>>>>>>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>>>>>>> command >>>>>>>>>>>>> line. >>>>>>>>>>>>> > >>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>>>>>> >>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> It works on fastdebug VM. >>>>>>>>>>>>> Please review again. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>> >>>>>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>>>>>> > Yasumasa, >>>>>>>>>>>>> > >>>>>>>>>>>>> > Try running a debug JVM with your patch with this >>>>>>>>>>>>> command >>>>>>>>>>>>> line. >>>>>>>>>>>>> > >>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>> > >>>>>>>>>>>>> > On a linux system I get this when I build with >>>>>>>>>>>>> your >>>>>>>>>>>>> patch. >>>>>>>>>>>>> > >>>>>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>> >> # To suppress the following error report, specify >>>>>>>>>>>>> this >>>>>>>>>>>>> argument >>>>>>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>>>>>> >> # >>>>>>>>>>>>> >> # A fatal error has been detected by the Java >>>>>>>>>>>>> Runtime >>>>>>>>>>>>> Environment: >>>>>>>>>>>>> >> # >>>>>>>>>>>>> >> # Internal Error >>>>>>>>>>>>> >> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>>>>>> ClassMediumChunk) >>>>>>>>>>>>> failed: Not a >>>>>>>>>>>>> >> humongous chunk >>>>>>>>>>>>> >> # >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > Jon >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>>>>>>>> >> I want to continue to discuss about >>>>>>>>>>>>> CompressedClassSpace and >>>>>>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>> Should I resize CompressedClassSpaceSize than >>>>>>>>>>>>> to >>>>>>>>>>>>> show >>>>>>>>>>>>> error message? >>>>>>>>>>>>> >>> If you add slightly better heuristics for the >>>>>>>>>>>>> setup >>>>>>>>>>>>> of >>>>>>>>>>>>> the >>>>>>>>>>>>> CompressedClassSpaceSize flag, for example lowering >>>>>>>>>>>>> the >>>>>>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is >>>>>>>>>>>>> set, >>>>>>>>>>>>> then >>>>>>>>>>>>> it >>>>>>>>>>>>> might be less likely that you'll hit the >>>>>>>>>>>>> OutOfMemoryError >>>>>>>>>>>>> when >>>>>>>>>>>>> the system is set up with strict overcommit settings. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> I've uploaded new webrev: >>>>>>>>>>>>> >> >>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is greater >>>>>>>>>>>>> than >>>>>>>>>>>>> MaxMetaspaceSize, >>>>>>>>>>>>> >> VM will fail with error message. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will be set >>>>>>>>>>>>> to >>>>>>>>>>>>> MaxMetaspaceSize >>>>>>>>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Yasumasa >>>>>>>>>>>>> >> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>> >> From yasuenag at gmail.com Thu Oct 12 05:16:01 2017 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 12 Oct 2017 14:16:01 +0900 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: <69f62651-dac3-3426-bb4f-6d4dcb03abb8@oracle.com> References: <69f62651-dac3-3426-bb4f-6d4dcb03abb8@oracle.com> Message-ID: Thanks David! I've fixed: http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.08/ Yasumasa 2017-10-12 13:55 GMT+09:00 David Holmes : > On 12/10/2017 2:47 PM, Yasumasa Suenaga wrote: >> >> Hi David, >> >>> You may have time to fix these in place before anyone else sees the >>> webrev >> >> >> I've fixed that: >> >> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.07/ >> >> >> I checked raw text of your email, so I believe the indent is correct. >> If I have mistake(s), please tell me. > > > Sorry they are off. Basic rules: > > var = > some long value; > > indent by 4. > > foo(arg1, arg2, > arg3, arg4); > > align the args on the two lines as shown. > > Hope that is clearer. > > Thanks, > David > > >> >> Thanks, >> >> Yasumasa >> >> >> 2017-10-12 13:22 GMT+09:00 David Holmes : >>> >>> Hi Yasumasa, >>> >>> On 12/10/2017 2:10 PM, Yasumasa Suenaga wrote: >>>> >>>> >>>> Hi all, >>>> >>>> This change was tried to push via JPRT, but it failed because >>>> test/hotspot/jtreg/runtime/SharedArchiveFile/MaxMetaspaceSizeTest.java >>>> was failed. >>>> Also I heard the comment that the changes in arguments.cpp which set >>>> ergonomic value to CompressedClassSpaceSize should be moved to >>>> Metaspace::ergo_initialize(). >>>> >>>> I uploaded new webrev. Could you review again? >>>> This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and >>>> test/hotspot/jtreg/runtime/SharedArchiveFile. >>>> >>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ >>> >>> >>> >>> I'll let Coleen cover the technical details, I'll just pick up a few >>> style >>> nits: >>> >>> 3328 /* Initial virtual space size will be calculated at >>> global_initialize() */ >>> >>> Use // comment >>> >>> 3329 size_t min_metaspace_sz = >>> 3330 VIRTUALSPACEMULTIPLIER * >>> InitialBootClassLoaderMetaspaceSize; >>> >>> ^ should indent only to here >>> >>> 3336 FLAG_SET_ERGO(size_t, CompressedClassSpaceSize, >>> 3337 MaxMetaspaceSize - >>> min_metaspace_sz); >>> ^ should indent only to here >>> >>> 3341 FLAG_SET_ERGO(size_t, InitialBootClassLoaderMetaspaceSize, >>> 3342 min_metaspace_sz); >>> >>> ^ should indent only to here >>> >>> You may have time to fix these in place before anyone else sees the >>> webrev >>> :) >>> >>> Thanks, >>> David >>> ----- >>> >>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> >>>> 2017-10-10 21:02 GMT+09:00 : >>>>> >>>>> >>>>> Hi Yasumasa, I like this better. I will sponsor it. I think it only >>>>> needs >>>>> one reviewer, unless there were more? Please send me the result of hg >>>>> export. >>>>> thanks, >>>>> Coleen >>>>> >>>>> >>>>> On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: >>>>>> >>>>>> >>>>>> >>>>>> Hi Coleen, >>>>>> >>>>>>>> I will sponsor this for you. >>>>>> >>>>>> >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>> check_metaspace_sizes() >>>>>>>> so that you don't have to tell arguments what the MediumChunk size >>>>>>>> is. >>>>>> >>>>>> >>>>>> >>>>>> I added new function calculate_min_metaspace_size() in metaspace.cpp >>>>>> to get minimum metaspace size, and uses return value from this >>>>>> function in metaspace initialization and metaspace size check. >>>>>> >>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ >>>>>> >>>>>> Could you review again? >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Yasumasa >>>>>> >>>>>> >>>>>> 2017-10-10 6:07 GMT+09:00 Man Cao : >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks for you both updating and sponsoring the webrev! >>>>>>> We plan to back-port it to our JDK9 once it is submitted. >>>>>>> >>>>>>> -Man >>>>>>> >>>>>>> On Mon, Oct 9, 2017 at 6:15 AM, wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>> check_metaspace_sizes() >>>>>>>> so that you don't have to tell arguments what the MediumChunk size >>>>>>>> is. >>>>>>>> >>>>>>>> I will sponsor this for you. >>>>>>>> >>>>>>>> thanks, >>>>>>>> Coleen >>>>>>>> >>>>>>>> >>>>>>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I added a testcase for this issue in new webrev: >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Yasumasa >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I uploaded webrev for this issue against jdk10/hs. >>>>>>>>>> Could you review it? >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I cannot access JPRT. So I need a sponsor. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Yasumasa >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thank Yasumasa and Stefan for the responses. >>>>>>>>>>> >>>>>>>>>>> Good to know that the patch is not blocked due to breaking some >>>>>>>>>>> internal >>>>>>>>>>> invariants/assumptions, but just due to its P5 status. >>>>>>>>>>> Is it possible to push it to P4? >>>>>>>>>>> >>>>>>>>>>> -Man >>>>>>>>>>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>>>>>>> >>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>> class >>>>>>>>>>>>> space >>>>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>>>> this >>>>>>>>>>>>> on the >>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I will send review request against jdk10/master or jdk10/hs >>>>>>>>>>>> after >>>>>>>>>>>> repos >>>>>>>>>>>> are opened. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Yasumasa >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Man, >>>>>>>>>>>>> >>>>>>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Do you have any thoughts on why this patch has been pending >>>>>>>>>>>>>> for >>>>>>>>>>>>>> 2+ >>>>>>>>>>>>>> years? This patch could really save us from some annoying >>>>>>>>>>>>>> issues >>>>>>>>>>>>>> since we >>>>>>>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>> class >>>>>>>>>>>>> space >>>>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>>>> this >>>>>>>>>>>>> on the >>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>> >>>>>>>>>>>>> StefanK >>>>>>>>>>>>> >>>>>>>>>>>>>> -Man >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I wonder if there is any recent update on the patch >>>>>>>>>>>>>> for >>>>>>>>>>>>>> JDK-8087291. >>>>>>>>>>>>>> Is it possible to push this patch into JDK9? Except >>>>>>>>>>>>>> for >>>>>>>>>>>>>> its >>>>>>>>>>>>>> low >>>>>>>>>>>>>> priority (P5), >>>>>>>>>>>>>> is there any complication that prevents this patch >>>>>>>>>>>>>> getting >>>>>>>>>>>>>> approved >>>>>>>>>>>>>> (for example, some JVM logic requires >>>>>>>>>>>>>> CompressedClassSpaceSize >>>>>>>>>>>>>> to be >>>>>>>>>>>>>> 1GB by default)? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I work in the Java Platform Team at Google. We have >>>>>>>>>>>>>> encountered >>>>>>>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>>>>>>>>> confusing 1GB >>>>>>>>>>>>>> memory reserved by metaspace, >>>>>>>>>>>>>> regardless of MaxMetaspaceSize value. The root cause >>>>>>>>>>>>>> for >>>>>>>>>>>>>> these >>>>>>>>>>>>>> issues is that CompressedClassSpaceSize is not >>>>>>>>>>>>>> automatically >>>>>>>>>>>>>> capped >>>>>>>>>>>>>> by MaxMetaspaceSize >>>>>>>>>>>>>> during VM initialization, and this patch seems fix the >>>>>>>>>>>>>> root >>>>>>>>>>>>>> cause. >>>>>>>>>>>>>> (I'm aware that even after this patch, the reserved >>>>>>>>>>>>>> size >>>>>>>>>>>>>> could >>>>>>>>>>>>>> still >>>>>>>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>>>>>>> but it is better than the current situation.) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Man >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you for your comment! >>>>>>>>>>>>>> > Try running a debug JVM with your patch with >>>>>>>>>>>>>> this >>>>>>>>>>>>>> command >>>>>>>>>>>>>> line. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> It works on fastdebug VM. >>>>>>>>>>>>>> Please review again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>>>>>>> > Yasumasa, >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Try running a debug JVM with your patch with >>>>>>>>>>>>>> this >>>>>>>>>>>>>> command >>>>>>>>>>>>>> line. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > On a linux system I get this when I build with >>>>>>>>>>>>>> your >>>>>>>>>>>>>> patch. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>> >> # To suppress the following error report, >>>>>>>>>>>>>> specify >>>>>>>>>>>>>> this >>>>>>>>>>>>>> argument >>>>>>>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>>>>>>> >> # >>>>>>>>>>>>>> >> # A fatal error has been detected by the Java >>>>>>>>>>>>>> Runtime >>>>>>>>>>>>>> Environment: >>>>>>>>>>>>>> >> # >>>>>>>>>>>>>> >> # Internal Error >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>>>>>>> ClassMediumChunk) >>>>>>>>>>>>>> failed: Not a >>>>>>>>>>>>>> >> humongous chunk >>>>>>>>>>>>>> >> # >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Jon >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>>>>>>>>> >> I want to continue to discuss about >>>>>>>>>>>>>> CompressedClassSpace and >>>>>>>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>> Should I resize CompressedClassSpaceSize >>>>>>>>>>>>>> than >>>>>>>>>>>>>> to >>>>>>>>>>>>>> show >>>>>>>>>>>>>> error message? >>>>>>>>>>>>>> >>> If you add slightly better heuristics for the >>>>>>>>>>>>>> setup >>>>>>>>>>>>>> of >>>>>>>>>>>>>> the >>>>>>>>>>>>>> CompressedClassSpaceSize flag, for example >>>>>>>>>>>>>> lowering >>>>>>>>>>>>>> the >>>>>>>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is >>>>>>>>>>>>>> set, >>>>>>>>>>>>>> then >>>>>>>>>>>>>> it >>>>>>>>>>>>>> might be less likely that you'll hit the >>>>>>>>>>>>>> OutOfMemoryError >>>>>>>>>>>>>> when >>>>>>>>>>>>>> the system is set up with strict overcommit >>>>>>>>>>>>>> settings. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> I've uploaded new webrev: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is >>>>>>>>>>>>>> greater >>>>>>>>>>>>>> than >>>>>>>>>>>>>> MaxMetaspaceSize, >>>>>>>>>>>>>> >> VM will fail with error message. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will be >>>>>>>>>>>>>> set >>>>>>>>>>>>>> to >>>>>>>>>>>>>> MaxMetaspaceSize >>>>>>>>>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Yasumasa >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> >>> > From david.holmes at oracle.com Thu Oct 12 05:37:03 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 12 Oct 2017 15:37:03 +1000 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: <69f62651-dac3-3426-bb4f-6d4dcb03abb8@oracle.com> Message-ID: <9933e37d-149b-8ecf-b146-88c0d3fc73fd@oracle.com> Thanks. David On 12/10/2017 3:16 PM, Yasumasa Suenaga wrote: > Thanks David! > > I've fixed: > > http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.08/ > > > Yasumasa > > > 2017-10-12 13:55 GMT+09:00 David Holmes : >> On 12/10/2017 2:47 PM, Yasumasa Suenaga wrote: >>> >>> Hi David, >>> >>>> You may have time to fix these in place before anyone else sees the >>>> webrev >>> >>> >>> I've fixed that: >>> >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.07/ >>> >>> >>> I checked raw text of your email, so I believe the indent is correct. >>> If I have mistake(s), please tell me. >> >> >> Sorry they are off. Basic rules: >> >> var = >> some long value; >> >> indent by 4. >> >> foo(arg1, arg2, >> arg3, arg4); >> >> align the args on the two lines as shown. >> >> Hope that is clearer. >> >> Thanks, >> David >> >> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> 2017-10-12 13:22 GMT+09:00 David Holmes : >>>> >>>> Hi Yasumasa, >>>> >>>> On 12/10/2017 2:10 PM, Yasumasa Suenaga wrote: >>>>> >>>>> >>>>> Hi all, >>>>> >>>>> This change was tried to push via JPRT, but it failed because >>>>> test/hotspot/jtreg/runtime/SharedArchiveFile/MaxMetaspaceSizeTest.java >>>>> was failed. >>>>> Also I heard the comment that the changes in arguments.cpp which set >>>>> ergonomic value to CompressedClassSpaceSize should be moved to >>>>> Metaspace::ergo_initialize(). >>>>> >>>>> I uploaded new webrev. Could you review again? >>>>> This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and >>>>> test/hotspot/jtreg/runtime/SharedArchiveFile. >>>>> >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ >>>> >>>> >>>> >>>> I'll let Coleen cover the technical details, I'll just pick up a few >>>> style >>>> nits: >>>> >>>> 3328 /* Initial virtual space size will be calculated at >>>> global_initialize() */ >>>> >>>> Use // comment >>>> >>>> 3329 size_t min_metaspace_sz = >>>> 3330 VIRTUALSPACEMULTIPLIER * >>>> InitialBootClassLoaderMetaspaceSize; >>>> >>>> ^ should indent only to here >>>> >>>> 3336 FLAG_SET_ERGO(size_t, CompressedClassSpaceSize, >>>> 3337 MaxMetaspaceSize - >>>> min_metaspace_sz); >>>> ^ should indent only to here >>>> >>>> 3341 FLAG_SET_ERGO(size_t, InitialBootClassLoaderMetaspaceSize, >>>> 3342 min_metaspace_sz); >>>> >>>> ^ should indent only to here >>>> >>>> You may have time to fix these in place before anyone else sees the >>>> webrev >>>> :) >>>> >>>> Thanks, >>>> David >>>> ----- >>>> >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> >>>>> 2017-10-10 21:02 GMT+09:00 : >>>>>> >>>>>> >>>>>> Hi Yasumasa, I like this better. I will sponsor it. I think it only >>>>>> needs >>>>>> one reviewer, unless there were more? Please send me the result of hg >>>>>> export. >>>>>> thanks, >>>>>> Coleen >>>>>> >>>>>> >>>>>> On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi Coleen, >>>>>>> >>>>>>>>> I will sponsor this for you. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>>> check_metaspace_sizes() >>>>>>>>> so that you don't have to tell arguments what the MediumChunk size >>>>>>>>> is. >>>>>>> >>>>>>> >>>>>>> >>>>>>> I added new function calculate_min_metaspace_size() in metaspace.cpp >>>>>>> to get minimum metaspace size, and uses return value from this >>>>>>> function in metaspace initialization and metaspace size check. >>>>>>> >>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ >>>>>>> >>>>>>> Could you review again? >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Yasumasa >>>>>>> >>>>>>> >>>>>>> 2017-10-10 6:07 GMT+09:00 Man Cao : >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks for you both updating and sponsoring the webrev! >>>>>>>> We plan to back-port it to our JDK9 once it is submitted. >>>>>>>> >>>>>>>> -Man >>>>>>>> >>>>>>>> On Mon, Oct 9, 2017 at 6:15 AM, wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>>> check_metaspace_sizes() >>>>>>>>> so that you don't have to tell arguments what the MediumChunk size >>>>>>>>> is. >>>>>>>>> >>>>>>>>> I will sponsor this for you. >>>>>>>>> >>>>>>>>> thanks, >>>>>>>>> Coleen >>>>>>>>> >>>>>>>>> >>>>>>>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I added a testcase for this issue in new webrev: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Yasumasa >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I uploaded webrev for this issue against jdk10/hs. >>>>>>>>>>> Could you review it? >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I cannot access JPRT. So I need a sponsor. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Yasumasa >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thank Yasumasa and Stefan for the responses. >>>>>>>>>>>> >>>>>>>>>>>> Good to know that the patch is not blocked due to breaking some >>>>>>>>>>>> internal >>>>>>>>>>>> invariants/assumptions, but just due to its P5 status. >>>>>>>>>>>> Is it possible to push it to P4? >>>>>>>>>>>> >>>>>>>>>>>> -Man >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>>>>>>>> >>>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>>> class >>>>>>>>>>>>>> space >>>>>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>>>>> this >>>>>>>>>>>>>> on the >>>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I will send review request against jdk10/master or jdk10/hs >>>>>>>>>>>>> after >>>>>>>>>>>>> repos >>>>>>>>>>>>> are opened. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Man, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you have any thoughts on why this patch has been pending >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> 2+ >>>>>>>>>>>>>>> years? This patch could really save us from some annoying >>>>>>>>>>>>>>> issues >>>>>>>>>>>>>>> since we >>>>>>>>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>>> class >>>>>>>>>>>>>> space >>>>>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>>>>> this >>>>>>>>>>>>>> on the >>>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>>> >>>>>>>>>>>>>> StefanK >>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Man >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I wonder if there is any recent update on the patch >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> JDK-8087291. >>>>>>>>>>>>>>> Is it possible to push this patch into JDK9? Except >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> its >>>>>>>>>>>>>>> low >>>>>>>>>>>>>>> priority (P5), >>>>>>>>>>>>>>> is there any complication that prevents this patch >>>>>>>>>>>>>>> getting >>>>>>>>>>>>>>> approved >>>>>>>>>>>>>>> (for example, some JVM logic requires >>>>>>>>>>>>>>> CompressedClassSpaceSize >>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>> 1GB by default)? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I work in the Java Platform Team at Google. We have >>>>>>>>>>>>>>> encountered >>>>>>>>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>>>>>>>>>> confusing 1GB >>>>>>>>>>>>>>> memory reserved by metaspace, >>>>>>>>>>>>>>> regardless of MaxMetaspaceSize value. The root cause >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> these >>>>>>>>>>>>>>> issues is that CompressedClassSpaceSize is not >>>>>>>>>>>>>>> automatically >>>>>>>>>>>>>>> capped >>>>>>>>>>>>>>> by MaxMetaspaceSize >>>>>>>>>>>>>>> during VM initialization, and this patch seems fix the >>>>>>>>>>>>>>> root >>>>>>>>>>>>>>> cause. >>>>>>>>>>>>>>> (I'm aware that even after this patch, the reserved >>>>>>>>>>>>>>> size >>>>>>>>>>>>>>> could >>>>>>>>>>>>>>> still >>>>>>>>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>>>>>>>> but it is better than the current situation.) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Man >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you for your comment! >>>>>>>>>>>>>>> > Try running a debug JVM with your patch with >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> command >>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It works on fastdebug VM. >>>>>>>>>>>>>>> Please review again. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>>>>>>>> > Yasumasa, >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Try running a debug JVM with your patch with >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> command >>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > On a linux system I get this when I build with >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>> patch. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>> >> # To suppress the following error report, >>>>>>>>>>>>>>> specify >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> argument >>>>>>>>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>> >> # A fatal error has been detected by the Java >>>>>>>>>>>>>>> Runtime >>>>>>>>>>>>>>> Environment: >>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>> >> # Internal Error >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>>>>>>>> ClassMediumChunk) >>>>>>>>>>>>>>> failed: Not a >>>>>>>>>>>>>>> >> humongous chunk >>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Jon >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>>>>>>>>>> >> I want to continue to discuss about >>>>>>>>>>>>>>> CompressedClassSpace and >>>>>>>>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>> Should I resize CompressedClassSpaceSize >>>>>>>>>>>>>>> than >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> show >>>>>>>>>>>>>>> error message? >>>>>>>>>>>>>>> >>> If you add slightly better heuristics for the >>>>>>>>>>>>>>> setup >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> CompressedClassSpaceSize flag, for example >>>>>>>>>>>>>>> lowering >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is >>>>>>>>>>>>>>> set, >>>>>>>>>>>>>>> then >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>> might be less likely that you'll hit the >>>>>>>>>>>>>>> OutOfMemoryError >>>>>>>>>>>>>>> when >>>>>>>>>>>>>>> the system is set up with strict overcommit >>>>>>>>>>>>>>> settings. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> I've uploaded new webrev: >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is >>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>> than >>>>>>>>>>>>>>> MaxMetaspaceSize, >>>>>>>>>>>>>>> >> VM will fail with error message. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will be >>>>>>>>>>>>>>> set >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> MaxMetaspaceSize >>>>>>>>>>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>>>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Yasumasa >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>> >>>> >> From jamsheed.c.m at oracle.com Thu Oct 12 10:33:34 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Thu, 12 Oct 2017 16:03:34 +0530 Subject: [10] RFR: [AOT] assert(false) failed: DEBUG MESSAGE: InterpreterMacroAssembler::call_VM_base: last_sp != NULL In-Reply-To: <01b62784-332e-8b6b-ac15-5fe4c21f9cd5@oracle.com> References: <8eb949bd-fa14-2722-5eaa-21a0a0c95b26@oracle.com> <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> <34517155-bd8d-56ee-86a1-7f9ffe73e2de@oracle.com> <01b62784-332e-8b6b-ac15-5fe4c21f9cd5@oracle.com> Message-ID: <497a7224-6aab-e5b0-4e72-5475b2ab5579@oracle.com> Dean, Thank you for the review, yes there is check for extended sp equality too. made the change http://cr.openjdk.java.net/~jcm/8168712/webrev.02/ Best regards, Jamsheed On Thursday 12 October 2017 03:28 AM, dean.long at oracle.com wrote: > For AARCH64 in templateTable_arm.cpp, how about using the same code as > generate_deopt_entry_for? > > ? __ restore_sp_after_call(Rtemp);? // Restore SP to extended SP > ? __ restore_stack_top(); > > > dl > > On 10/11/17 5:48 AM, jamsheed wrote: >> Hi Vladimir, >> >> Thank you for pointing this. >> >> revised webrev: http://cr.openjdk.java.net/~jcm/8168712/webrev.02/ >> >> Best Regards, >> >> Jamsheed >> >> >> On Tuesday 10 October 2017 08:41 PM, Vladimir Kozlov wrote: >>> Why you added !defined(AARCH64) in templateTable_arm.cpp? Is only >>> 32-bit affected? >>> >>> Thanks, >>> Vladimir >>> >>> On 9/13/17 11:54 PM, Dean Long wrote: >>>> It looks like you accidentally dropped >>>> hotspot-compiler-dev at openjdk.java.net when you added runtime. >>>> >>>> dl >>>> >>>> >>>> On 9/13/2017 11:21 PM, jamsheed wrote: >>>>> (adding runtime list for inputs) >>>>> >>>>> On Monday 11 September 2017 11:43 PM, jamsheed wrote: >>>>>> brief desc: special handling of Object. in >>>>>> TemplateInterpreter::deopt_reexecute_entry >>>>>> >>>>>> required last_sp to be reset explicitly in normal return path >>>>>> >>>>>> address TemplateInterpreter::deopt_reexecute_entry(Method* >>>>>> method, address bcp) { >>>>>> ? assert(method->contains(bcp), "just checkin'"); >>>>>> ? Bytecodes::Code code?? = Bytecodes::java_code_at(method, bcp); >>>>>> ? if (code == Bytecodes::_return) { >>>>>> ??? // This is used for deopt during registration of finalizers >>>>>> ??? // during Object..? We simply need to resume execution at >>>>>> ??? // the standard return vtos bytecode to pop the frame normally. >>>>>> ??? // reexecuting the real bytecode would cause double registration >>>>>> ??? // of the finalizable object. >>>>>> ??? return _normal_table.entry(Bytecodes::_return).entry(vtos); >>>>> >>>>> last_sp ! = null not an issue for this case, so i skip the assert >>>>> in debug build >>>>> >>>>> http://cr.openjdk.java.net/~jcm/8168712/webrev.01/ >>>>> >>>>> Please review. >>>>> >>>>> Best Regards, >>>>> Jamsheed >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >> > From dean.long at oracle.com Thu Oct 12 18:21:29 2017 From: dean.long at oracle.com (dean.long at oracle.com) Date: Thu, 12 Oct 2017 11:21:29 -0700 Subject: [10] RFR: [AOT] assert(false) failed: DEBUG MESSAGE: InterpreterMacroAssembler::call_VM_base: last_sp != NULL In-Reply-To: <497a7224-6aab-e5b0-4e72-5475b2ab5579@oracle.com> References: <8eb949bd-fa14-2722-5eaa-21a0a0c95b26@oracle.com> <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> <34517155-bd8d-56ee-86a1-7f9ffe73e2de@oracle.com> <01b62784-332e-8b6b-ac15-5fe4c21f9cd5@oracle.com> <497a7224-6aab-e5b0-4e72-5475b2ab5579@oracle.com> Message-ID: Looks good. dl On 10/12/17 3:33 AM, jamsheed wrote: > Dean, > > Thank you for the review, yes there is check for extended sp equality > too. made the change > > http://cr.openjdk.java.net/~jcm/8168712/webrev.02/ > > Best regards, > > Jamsheed > > > On Thursday 12 October 2017 03:28 AM, dean.long at oracle.com wrote: >> For AARCH64 in templateTable_arm.cpp, how about using the same code >> as generate_deopt_entry_for? >> >> ? __ restore_sp_after_call(Rtemp);? // Restore SP to extended SP >> ? __ restore_stack_top(); >> >> >> dl >> >> On 10/11/17 5:48 AM, jamsheed wrote: >>> Hi Vladimir, >>> >>> Thank you for pointing this. >>> >>> revised webrev: http://cr.openjdk.java.net/~jcm/8168712/webrev.02/ >>> >>> Best Regards, >>> >>> Jamsheed >>> >>> >>> On Tuesday 10 October 2017 08:41 PM, Vladimir Kozlov wrote: >>>> Why you added !defined(AARCH64) in templateTable_arm.cpp? Is only >>>> 32-bit affected? >>>> >>>> Thanks, >>>> Vladimir >>>> >>>> On 9/13/17 11:54 PM, Dean Long wrote: >>>>> It looks like you accidentally dropped >>>>> hotspot-compiler-dev at openjdk.java.net when you added runtime. >>>>> >>>>> dl >>>>> >>>>> >>>>> On 9/13/2017 11:21 PM, jamsheed wrote: >>>>>> (adding runtime list for inputs) >>>>>> >>>>>> On Monday 11 September 2017 11:43 PM, jamsheed wrote: >>>>>>> brief desc: special handling of Object. in >>>>>>> TemplateInterpreter::deopt_reexecute_entry >>>>>>> >>>>>>> required last_sp to be reset explicitly in normal return path >>>>>>> >>>>>>> address TemplateInterpreter::deopt_reexecute_entry(Method* >>>>>>> method, address bcp) { >>>>>>> ? assert(method->contains(bcp), "just checkin'"); >>>>>>> ? Bytecodes::Code code?? = Bytecodes::java_code_at(method, bcp); >>>>>>> ? if (code == Bytecodes::_return) { >>>>>>> ??? // This is used for deopt during registration of finalizers >>>>>>> ??? // during Object..? We simply need to resume execution at >>>>>>> ??? // the standard return vtos bytecode to pop the frame normally. >>>>>>> ??? // reexecuting the real bytecode would cause double >>>>>>> registration >>>>>>> ??? // of the finalizable object. >>>>>>> ??? return _normal_table.entry(Bytecodes::_return).entry(vtos); >>>>>> >>>>>> last_sp ! = null not an issue for this case, so i skip the assert >>>>>> in debug build >>>>>> >>>>>> http://cr.openjdk.java.net/~jcm/8168712/webrev.01/ >>>>>> >>>>>> Please review. >>>>>> >>>>>> Best Regards, >>>>>> Jamsheed >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>> >> > From rkennke at redhat.com Thu Oct 12 18:34:53 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 12 Oct 2017 20:34:53 +0200 Subject: RFR: 8179387: Factor out CMS specific code from GenCollectedHeap into its own subclass In-Reply-To: <7a9cdfff-bcd5-b616-f512-78adf42c70f9@redhat.com> References: <4d5e6af8-d975-7803-64c5-7295e0d56154@redhat.com> <13358626-e399-e352-1711-587416621aac@redhat.com> <27af0ad2-fe78-3536-2143-996dd42583ab@oracle.com> <4bc53aaa-b98a-8a61-73bf-d30ac3f402b8@redhat.com> <666af7f2-27e9-48c6-91e4-eaefa5289e18@redhat.com> <3ec8a6a3-5a4b-a910-f6ec-ed1c0dad4cad@oracle.com> <5417889c-5289-37cd-eb31-a2b55f70e85e@redhat.com> <088d467c-8038-60bc-1eab-b34061ad20d9@redhat.com> <7abbeec1-b353-c6e9-9827-e70f49269d50@oracle.com> <6e3b0ca0-68c9-f5b6-1cb7-ddca71df2c0e@redhat.com> <51d7f145-27e3-5bfb-f1da-decd6e19858c@oracle.com> <51036840-16f4-c8eb-082e-76e0a5b1534f@redhat.com> <2c1bdd7c-005c-4376-cd9f-3db739165bc2@oracle.com> <7a9cdfff-bcd5-b616-f512-78adf42c70f9@redhat.com> Message-ID: <280d54f9-aca9-b916-50bf-5364c2bd6e13@redhat.com> Am 02.10.2017 um 16:54 schrieb Roman Kennke: > Am 02.10.2017 um 14:22 schrieb Erik Helin: >> On 09/29/2017 03:47 AM, Roman Kennke wrote:>>> Differential: >>>>> http://cr.openjdk.java.net/~rkennke/8179387/webrev.08.diff/ >>>>> >>>> >>>> Thanks, this is good.? I don't know enough about the rest of the >>>> change to be a reviewer, but I think you have your reviews. >>>> Coleen >>>> >>>>> Full: >>>>> http://cr.openjdk.java.net/~rkennke/8179387/webrev.08/ >>>>> >>> >>> Ping? I believe this never actually went in. Is there anything missing? >> >> Sigh, sorry, I completely forgot about this patch, thanks for >> pinging. Yes, this looks good, and thank you for taking on this work! >> >>> Do you want me to re-base it onto the single-repo and post another >>> webrev? >> >> If you don't mind, then yes, please rebase it on top of the >> consolidated repo. Then I can sponsor it for you. > Ok, here it comes: > > http://cr.openjdk.java.net/~rkennke/8179387/webrev.11/ > > > I've run hotspot_gc tests again. Looking good. > > Thanks for reviewing and sponsoring! There've been some changes in some of the code that this changeset touches which caused conflicts. Rebased again, with some minor touchups: http://cr.openjdk.java.net/~rkennke/8179387/webrev.12/ Is this good to go in now? Roman From vladimir.kozlov at oracle.com Thu Oct 12 18:50:45 2017 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Thu, 12 Oct 2017 11:50:45 -0700 Subject: [10] RFR: [AOT] assert(false) failed: DEBUG MESSAGE: InterpreterMacroAssembler::call_VM_base: last_sp != NULL In-Reply-To: References: <8eb949bd-fa14-2722-5eaa-21a0a0c95b26@oracle.com> <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> <34517155-bd8d-56ee-86a1-7f9ffe73e2de@oracle.com> <01b62784-332e-8b6b-ac15-5fe4c21f9cd5@oracle.com> <497a7224-6aab-e5b0-4e72-5475b2ab5579@oracle.com> Message-ID: <8f40236b-c8ff-4ae5-2e9a-89de588f610e@oracle.com> +1 Thanks, Vladimir On 10/12/17 11:21 AM, dean.long at oracle.com wrote: > Looks good. > > dl > > > On 10/12/17 3:33 AM, jamsheed wrote: >> Dean, >> >> Thank you for the review, yes there is check for extended sp equality too. made the change >> >> http://cr.openjdk.java.net/~jcm/8168712/webrev.02/ >> >> Best regards, >> >> Jamsheed >> >> >> On Thursday 12 October 2017 03:28 AM, dean.long at oracle.com wrote: >>> For AARCH64 in templateTable_arm.cpp, how about using the same code as generate_deopt_entry_for? >>> >>> ? __ restore_sp_after_call(Rtemp);? // Restore SP to extended SP >>> ? __ restore_stack_top(); >>> >>> >>> dl >>> >>> On 10/11/17 5:48 AM, jamsheed wrote: >>>> Hi Vladimir, >>>> >>>> Thank you for pointing this. >>>> >>>> revised webrev: http://cr.openjdk.java.net/~jcm/8168712/webrev.02/ >>>> >>>> Best Regards, >>>> >>>> Jamsheed >>>> >>>> >>>> On Tuesday 10 October 2017 08:41 PM, Vladimir Kozlov wrote: >>>>> Why you added !defined(AARCH64) in templateTable_arm.cpp? Is only 32-bit affected? >>>>> >>>>> Thanks, >>>>> Vladimir >>>>> >>>>> On 9/13/17 11:54 PM, Dean Long wrote: >>>>>> It looks like you accidentally dropped hotspot-compiler-dev at openjdk.java.net when you added runtime. >>>>>> >>>>>> dl >>>>>> >>>>>> >>>>>> On 9/13/2017 11:21 PM, jamsheed wrote: >>>>>>> (adding runtime list for inputs) >>>>>>> >>>>>>> On Monday 11 September 2017 11:43 PM, jamsheed wrote: >>>>>>>> brief desc: special handling of Object. in TemplateInterpreter::deopt_reexecute_entry >>>>>>>> >>>>>>>> required last_sp to be reset explicitly in normal return path >>>>>>>> >>>>>>>> address TemplateInterpreter::deopt_reexecute_entry(Method* method, address bcp) { >>>>>>>> ? assert(method->contains(bcp), "just checkin'"); >>>>>>>> ? Bytecodes::Code code?? = Bytecodes::java_code_at(method, bcp); >>>>>>>> ? if (code == Bytecodes::_return) { >>>>>>>> ??? // This is used for deopt during registration of finalizers >>>>>>>> ??? // during Object..? We simply need to resume execution at >>>>>>>> ??? // the standard return vtos bytecode to pop the frame normally. >>>>>>>> ??? // reexecuting the real bytecode would cause double registration >>>>>>>> ??? // of the finalizable object. >>>>>>>> ??? return _normal_table.entry(Bytecodes::_return).entry(vtos); >>>>>>> >>>>>>> last_sp ! = null not an issue for this case, so i skip the assert in debug build >>>>>>> >>>>>>> http://cr.openjdk.java.net/~jcm/8168712/webrev.01/ >>>>>>> >>>>>>> Please review. >>>>>>> >>>>>>> Best Regards, >>>>>>> Jamsheed >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>> >> > From rkennke at redhat.com Thu Oct 12 19:59:43 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 12 Oct 2017 21:59:43 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes Message-ID: I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev because it touches both areas (i.e. the GC interface). Currently, all GC related argument processing is done in arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all sorts of GC specific methods etc. In order to have a cleaner GC interface, I propose to move all that code into GC specific classes under their GC specific directories. Since at this point in time we haven't created the XXXHeap subclasses yet (and cannot, before having set up all the flags and arguments), I propose to create a GC interface of which we create an instance as soon as we have selected a GC, and then do argument processing there. This 'GC' interface is, at this point, meant as a sort of GC factory interface. With this patch, it will only do argument processing. Later I plan to add heap creation to it (which is currently done in universe.cpp, with INCLUDE_ALL_GCS stuff and all the if (UseBlahGC) .. else if .. else branch chains. My current prototype in the jdk10 sandbox also provides servicabilty support classes and some more stuff, but we should decide later where to put this. I broke out some code paths into gc_factory.cpp (i.e. not in gc.cpp). Those are all the code paths that select GC, create the right instances, etc, in other words, the code paths that need to know about all GCs. Ideally, this should be the only file that needs to know about all the GCs. (Suggestions for improvement are welcome of course.) I tested this by running hotspot_gc jtreg tests, with normal fastdebug and no-precompiled configurations. Everything passes fine. http://cr.openjdk.java.net/~rkennke/8189171/webrev.00/ Opinions? Roman From ioi.lam at oracle.com Thu Oct 12 23:48:49 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 12 Oct 2017 16:48:49 -0700 Subject: RFR(M) JDK-8188791 Move AppCDS implementation from closed repo to open repo Message-ID: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> Hi, Please review this change set. http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ ??? https://bugs.openjdk.java.net/browse/JDK-8188791 This is the first step of implementing the following JEP, which moves AppCDS from closed repos into the openjdk repo: ??? https://bugs.openjdk.java.net/browse/JDK-8185996 In JDK 9, significant portion of AppCDS code resided in the closed repo. As part of the open-sourcing effort of JDK 18.3, we will move the source code into the open repo. In this changeset, the code is moved verbatim as much as possible. The intention is only to relocate the sources, not to changing existing behaviors, and not to do any sort of refactoring. Most of the "diffs" shown in this webrev are the result of copying the closed source files on top of files of the same name in the open repo. So in reviewing, instead of focusing on what's "changed", it's better to focus on the entire content of the new version of each file. The only functional change in this task is that the UseAppCDS flag is changed from a "commercial" flag to a regular "product" flag. This is because "commercial" flags are not supported by the OpenJDK build. Source code refactoring may be desirable, because the old open/closed source code structure had introduced some intermediary APIs to connect code between the two repos. Such API should be removed in a separate RFE. Also, some AppCDS tests are currently in the closed repo. These tests will be moved in a separate task. See JDK-8188792 for details. All the AppCDS tests (currently still in closed sources) passed with both Oracle JDK and OpenJDK. Thanks - Ioi From david.holmes at oracle.com Fri Oct 13 01:20:48 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 13 Oct 2017 11:20:48 +1000 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: References: Message-ID: Hi Roman, Not a review as GC folk need to do that. On 13/10/2017 5:59 AM, Roman Kennke wrote: > I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev because it > touches both areas (i.e. the GC interface). > > Currently, all GC related argument processing is done in arguments.cpp, > littering it with #ifdef INCLUDE_ALL_GCS and all sorts of GC specific > methods etc. From a runtime perspective I like all the GC specific ifdefs and settings moved out-of-line of the main argument processing. From a refactoring perspective I noticed that set_object_alignment() no longer calls set_cms_values(). I presume setting it elsewhere is okay? Thanks, David > In order to have a cleaner GC interface, I propose to move all that code > into GC specific classes under their GC specific directories. > > Since at this point in time we haven't created the XXXHeap subclasses > yet (and cannot, before having set up all the flags and arguments), I > propose to create a GC interface of which we create an instance as soon > as we have selected a GC, and then do argument processing there. > > This 'GC' interface is, at this point, meant as a sort of GC factory > interface. With this patch, it will only do argument processing. Later I > plan to add heap creation to it (which is currently done in > universe.cpp, with INCLUDE_ALL_GCS stuff and all the if (UseBlahGC) .. > else if .. else branch chains. My current prototype in the jdk10 sandbox > also provides servicabilty support classes and some more stuff, but we > should decide later where to put this. > > I broke out some code paths into gc_factory.cpp (i.e. not in gc.cpp). > Those are all the code paths that select GC, create the right instances, > etc, in other words, the code paths that need to know about all GCs. > Ideally, this should be the only file that needs to know about all the > GCs. (Suggestions for improvement are welcome of course.) > > I tested this by running hotspot_gc jtreg tests, with normal fastdebug > and no-precompiled configurations. Everything passes fine. > > http://cr.openjdk.java.net/~rkennke/8189171/webrev.00/ > > > Opinions? > > Roman > From jamsheed.c.m at oracle.com Fri Oct 13 05:38:46 2017 From: jamsheed.c.m at oracle.com (jamsheed) Date: Fri, 13 Oct 2017 11:08:46 +0530 Subject: [10] RFR: [AOT] assert(false) failed: DEBUG MESSAGE: InterpreterMacroAssembler::call_VM_base: last_sp != NULL In-Reply-To: <8f40236b-c8ff-4ae5-2e9a-89de588f610e@oracle.com> References: <8eb949bd-fa14-2722-5eaa-21a0a0c95b26@oracle.com> <069663cb-4b48-483e-7d1b-8619dafe616d@oracle.com> <34517155-bd8d-56ee-86a1-7f9ffe73e2de@oracle.com> <01b62784-332e-8b6b-ac15-5fe4c21f9cd5@oracle.com> <497a7224-6aab-e5b0-4e72-5475b2ab5579@oracle.com> <8f40236b-c8ff-4ae5-2e9a-89de588f610e@oracle.com> Message-ID: <37326ba7-f520-ca96-bdc2-31c8fe99a52d@oracle.com> Thanks for the review, Dean, Vladimir Best regards, Jamsheed On Friday 13 October 2017 12:20 AM, Vladimir Kozlov wrote: > +1 > > Thanks, > Vladimir > > On 10/12/17 11:21 AM, dean.long at oracle.com wrote: >> Looks good. >> >> dl >> >> >> On 10/12/17 3:33 AM, jamsheed wrote: >>> Dean, >>> >>> Thank you for the review, yes there is check for extended sp >>> equality too. made the change >>> >>> http://cr.openjdk.java.net/~jcm/8168712/webrev.02/ >>> >>> Best regards, >>> >>> Jamsheed >>> >>> >>> On Thursday 12 October 2017 03:28 AM, dean.long at oracle.com wrote: >>>> For AARCH64 in templateTable_arm.cpp, how about using the same code >>>> as generate_deopt_entry_for? >>>> >>>> ? __ restore_sp_after_call(Rtemp);? // Restore SP to extended SP >>>> ? __ restore_stack_top(); >>>> >>>> >>>> dl >>>> >>>> On 10/11/17 5:48 AM, jamsheed wrote: >>>>> Hi Vladimir, >>>>> >>>>> Thank you for pointing this. >>>>> >>>>> revised webrev: http://cr.openjdk.java.net/~jcm/8168712/webrev.02/ >>>>> >>>>> Best Regards, >>>>> >>>>> Jamsheed >>>>> >>>>> >>>>> On Tuesday 10 October 2017 08:41 PM, Vladimir Kozlov wrote: >>>>>> Why you added !defined(AARCH64) in templateTable_arm.cpp? Is only >>>>>> 32-bit affected? >>>>>> >>>>>> Thanks, >>>>>> Vladimir >>>>>> >>>>>> On 9/13/17 11:54 PM, Dean Long wrote: >>>>>>> It looks like you accidentally dropped >>>>>>> hotspot-compiler-dev at openjdk.java.net when you added runtime. >>>>>>> >>>>>>> dl >>>>>>> >>>>>>> >>>>>>> On 9/13/2017 11:21 PM, jamsheed wrote: >>>>>>>> (adding runtime list for inputs) >>>>>>>> >>>>>>>> On Monday 11 September 2017 11:43 PM, jamsheed wrote: >>>>>>>>> brief desc: special handling of Object. in >>>>>>>>> TemplateInterpreter::deopt_reexecute_entry >>>>>>>>> >>>>>>>>> required last_sp to be reset explicitly in normal return path >>>>>>>>> >>>>>>>>> address TemplateInterpreter::deopt_reexecute_entry(Method* >>>>>>>>> method, address bcp) { >>>>>>>>> ? assert(method->contains(bcp), "just checkin'"); >>>>>>>>> ? Bytecodes::Code code?? = Bytecodes::java_code_at(method, bcp); >>>>>>>>> ? if (code == Bytecodes::_return) { >>>>>>>>> ??? // This is used for deopt during registration of finalizers >>>>>>>>> ??? // during Object..? We simply need to resume >>>>>>>>> execution at >>>>>>>>> ??? // the standard return vtos bytecode to pop the frame >>>>>>>>> normally. >>>>>>>>> ??? // reexecuting the real bytecode would cause double >>>>>>>>> registration >>>>>>>>> ??? // of the finalizable object. >>>>>>>>> ??? return _normal_table.entry(Bytecodes::_return).entry(vtos); >>>>>>>> >>>>>>>> last_sp ! = null not an issue for this case, so i skip the >>>>>>>> assert in debug build >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~jcm/8168712/webrev.01/ >>>>>>>> >>>>>>>> Please review. >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Jamsheed >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>> >> From karen.kinnear at oracle.com Fri Oct 13 10:42:50 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Fri, 13 Oct 2017 12:42:50 +0200 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> Message-ID: <5C7CF716-E381-4274-8CE6-1A3C1B23346A@oracle.com> A quick note - we are resizing the hash table, i.e. number of linked lists, so as to speed up lookup if it is too small, or waste less space if it is too large. Each dictionary may have multiple entries off of it - one per loaded class - so there is no such thing as running out of space based on the sizing (just based on total OOM). I have not yet looked at the change - but just wanted to clarify the goal and concept for folks first. thanks Karen > On Oct 11, 2017, at 4:10 PM, coleen.phillimore at oracle.com wrote: > > > Gerard, some preliminary comments. > > http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html > > *+{* > *!_MutexLocker mu(SystemDictionary_lock, THREAD_);* > *!_Klass* probe = dictionary->find(_d_hash, name, protection_domain);* > *if (probe != NULL) return probe;* > ** > *+ }* > > I don't think you need this because dictionary->find() should have the NoSafepointVerifier, so the index will not change. > > http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/classLoaderData.cpp.udiff.html > > I think we want a global _some_dictionary_needs_resizing to avoid walking through the CLDG. > > And have Dictionary have a field _resizing_needed to avoid the calculation during the safepoint, which is set when adding an entry. > > _resizing_needed = *(number_of_entries() > (_resize_load_trigger*table_size()); > *CLDG::_any_resizing_needed |= _resizing_needed; // or something like that. > > I can write more about the rationale of this change in the bug report, if needed. > > Thank you for doing this change. > Coleen > > > On 10/10/17 4:40 PM, Gerard Ziemski wrote: >> hi all, >> >> Please review this change that adds dynamic resizing of system dictionaries. >> >> The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock >> >> A few notes: >> >> - dynamic resizing is off when either dumping or using shared spaces >> - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) >> - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8184765 >> webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 >> >> >> cheers >> > From adam.farley at uk.ibm.com Fri Oct 13 12:16:41 2017 From: adam.farley at uk.ibm.com (Adam Farley8) Date: Fri, 13 Oct 2017 12:16:41 +0000 Subject: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be exited by java Message-ID: Hi All, Here's a summary of the email below (which is intended, partly, as a summary of the emails before it). Let me know if you agree/disagree with any of these points. 1) Exit(#) during vm startup is a bug because it should return a code regardless of the state of the VM. 2) Exit(0) is an *especially* big bug because it imitates successful completion of external cpp code accessing JNI methods. 3) One solution is to specify a new return code for JNI. 4) The supplied code (diff) generates, facilitates, and handles that return code for the exit(0) scenario: -agentlib:jdwp=help 5) The supplied test confirms that the supplied code works (run via unzip, and then bash TestStart.sh ). 6) To implement this new return code, plus the code that handles it, we would need to follow the CSR process. 7) To implement the fix for the scenario used as an example of the new return code's use, we would need to modify the JVM TI spec. 8) To address all of the worst instances of exit(#), we would need to search for exit(#) and raise a bug for each significant one (or group). 9) To solve (8) in one bug would be a lot of work, arguably too much work for a single bug. 10) If the new return code is chosen as the appropriate solution to this problem, we may need to choose a better name for the return code. Is this a fair assessment of the current state of the debate? I'm on IRC every weekday from 9am-5pm (4pm fridays) BST (GMT+1) if anyone wants to discuss this in real-time on the openjdk channel. Best Regards Adam Farley -- Previous Email -- Hi David, Alan, You are right in that the changes to HotSpot would be nontrivial. I see a number of places in (e.g.) arguments.cpp that seem to exit in the same manner as Xlog (such as -Xinternalversion). I would advise ploughing through the CSR process to alter the JNI spec, and simultaneously identify some key paths that can be raised as bugs. That way, when people have time to address these issues, the mechanism to handle a silent exit is already in place. The JDWP fix can be raised separately as one of these bugs, if it would make things simpler. As for the name, JNI_SILENT_EXIT is a placeholder, and can be readily changed. Do you have any suggestions? Lastly, in an ideal world, the VM initialisation should never exit(#). It should return a return code that tells the caller something, pass or fail, messy or tidy. That way, if someone is using the JNI as part of something bigger (like a database or a web server), one of these scenarios is just a bug, rather than a world-ender like exit(#). And now for the individual messages. :) David: Having help data returned by the launcher seems like a good way to avoid exit(0) calls, but I'm not sure how we'd prevent a JNI-caller using those options. Ultimately, to be sure, we'd have to remove the logic for those options, centralise the data to better enable launcher access, and add some logic in there so it can find any other help data (e.g. from the jdwp agent library). I feel this would be a bigger task than adding the new return code and changing the vm, plus it wouldn't provide for any non-help scenarios where the vm wants to shut down without error during initialisation. Alan: I should mention that the silent exit solution is already in use in the OpenJ9 VM. Not all of the exit paths have been resolved, but many have. The code is open and can be found here: https://github.com/eclipse/openj9 And though the silent exit code is disabled for the time being, it can be re-enabled by entering this class: runtime/vm/jvminit.c and altering line 2343 ( ctrl-f for exit(1) if it's not there). I won't paste the full code here in case people are concerned about contamination, but I would assert that this code (and the associated vm files) prove that the concept is possible. Note that that code should not be enabled until after we've integrated the code that can handle a silent exit. Best Regards Adam Farley P.S. Thank you both for your efforts on this. :) From: David Holmes To: Alan Bateman , Adam Farley8 , core-libs-dev at openjdk.java.net, hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com Date: 15/09/2017 12:03 Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be exited by java On 15/09/2017 8:17 PM, Alan Bateman wrote: > On 15/09/2017 02:47, David Holmes wrote: >> Hi Adam, >> >> I am still very much torn over this one. I think the idea of >> print-and-exit flags for a potentially hosted library like the JVM is >> just wrong - we should never have done that, but we did. Fixing that >> by moving the flags to the launcher is far from trivial**. Endorsing >> and encouraging these sorts of flag by adding JNI support seems to be >> sending the wrong message. >> >> ** I can envisage a "help xxx" Dcmd that can read back the info from >> the VM. The launcher can send the Dcmd, print the output and exit. The >> launcher would not need to know what the xxx values mean, but would >> have to intercept the existing ones. >> >> Another option is just to be aware of these flags (are there more than >> jdwp and Xlog?) and deal with them specially in your custom launcher - >> either filter them out and ignore them, or else launch the VM in its >> own process to respond to them. >> >> Any changes to the JNI specification need to go through the CSR process. > Yes, it would require an update to the JNI spec, also a change to the > JVM TI spec where Agent_OnLoad returning a non-0 value is specified to > terminates the VM. The name and value needs discussion too, esp. as the > JNI spec uses negative values for failure. > > In any case, I'm also torn over this one as it's a corner case that is > only interesting for custom launchers that load agents with options that > print usage messages. It wouldn't be hard to have the Agent_OnLoad > specify a printf hook that the agent could use for output although there > are complications with agents such as JDWP that also announce their > transport end point. Beyond that there is still the issue of the custom > launcher that would need to know to destroy the VM without reporting an > error. > > So what happened to the more meaty part to this which is fixing the > various cases in HotSpot that terminate the process during > initialization? I would expect some progress could be made on those > cases while trying to decide whether to rev the JNI and JVM TI specs to > cover the help case. Trying to eliminate the vm_exit_during_initialization paths in hotspot is a huge undertaking IMHO. David > > -Alan Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From Alan.Bateman at oracle.com Fri Oct 13 12:58:17 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 13 Oct 2017 13:58:17 +0100 Subject: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be exited by java In-Reply-To: References: Message-ID: On 13/10/2017 13:16, Adam Farley8 wrote: > Hi All, > > Here's a summary of the email below (which is intended, partly, as a > summary of the emails before it). > > Let me know if you agree/disagree with any of these points. > > : > 3) One solution is to specify a new return code for JNI. Yes, hence the need to update the JNI spec. A discussion point to add to your point #10 is the return code value as the JNI spec uses negative values for errors. > > 6) To implement this new return code, plus the code that handles it, > we would need to follow the CSR process. Yes, a CSR will be needed if this goes ahead as it will need changes to both the JNI and JVM TI specs. > > 7) To implement the fix for the scenario used as an example of the new > return code's use, we would need to modify the JVM TI spec. Yes, because the JVM TI spec is very clear that the Agent_OnLoad returning a non-0 value is an error that terminates the VM. > > 8) To address all of the worst instances of exit(#), we would need to > search for exit(#) and raise a bug for each significant one (or group). In the discussion to date then I think there is an acknowledgement that there are issues in hotspot for several error or resource exhaustion cases that would need a lot of work to recover from. I don't think there was any suggestion that they would need to be addressed or even identified as part of deciding if the agent "-help" scenario is worth trying to support. -Alan From david.holmes at oracle.com Fri Oct 13 13:11:41 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 13 Oct 2017 23:11:41 +1000 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <5C7CF716-E381-4274-8CE6-1A3C1B23346A@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <5C7CF716-E381-4274-8CE6-1A3C1B23346A@oracle.com> Message-ID: <21eef3ba-9fa2-8a76-fe3b-f9669c94f7bb@oracle.com> Hi Karen, On 13/10/2017 8:42 PM, Karen Kinnear wrote: > A quick note - we are resizing the hash table, i.e. number of linked lists, so as to speed up lookup if it is too small, or waste less space if it is > too large. Each dictionary may have multiple entries off of it - one per loaded class - so there is no such thing as running out > of space based on the sizing (just based on total OOM). I have not yet looked at the change - but just wanted to clarify the > goal and concept for folks first. Thanks for clarifying - though AFAICS this proposal only grows the dictionary. Is the "waste less space" to be achieved by starting with a smaller dictionary initially? Do we have a way to control the initial size? Thanks, David > thanks > Karen > >> On Oct 11, 2017, at 4:10 PM, coleen.phillimore at oracle.com wrote: >> >> >> Gerard, some preliminary comments. >> >> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html >> >> *+{* >> *!_MutexLocker mu(SystemDictionary_lock, THREAD_);* >> *!_Klass* probe = dictionary->find(_d_hash, name, protection_domain);* >> *if (probe != NULL) return probe;* >> ** >> *+ }* >> >> I don't think you need this because dictionary->find() should have the NoSafepointVerifier, so the index will not change. >> >> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/classLoaderData.cpp.udiff.html >> >> I think we want a global _some_dictionary_needs_resizing to avoid walking through the CLDG. >> >> And have Dictionary have a field _resizing_needed to avoid the calculation during the safepoint, which is set when adding an entry. >> >> _resizing_needed = *(number_of_entries() > (_resize_load_trigger*table_size()); >> *CLDG::_any_resizing_needed |= _resizing_needed; // or something like that. >> >> I can write more about the rationale of this change in the bug report, if needed. >> >> Thank you for doing this change. >> Coleen >> >> >> On 10/10/17 4:40 PM, Gerard Ziemski wrote: >>> hi all, >>> >>> Please review this change that adds dynamic resizing of system dictionaries. >>> >>> The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock >>> >>> A few notes: >>> >>> - dynamic resizing is off when either dumping or using shared spaces >>> - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) >>> - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) >>> >>> bug: https://bugs.openjdk.java.net/browse/JDK-8184765 >>> webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 >>> >>> >>> cheers >>> >> > From david.holmes at oracle.com Fri Oct 13 13:29:52 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 13 Oct 2017 23:29:52 +1000 Subject: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be exited by java In-Reply-To: References: Message-ID: <1c824799-e58c-1ff8-bc0a-d13be651a3ec@oracle.com> Hi Adam, On 13/10/2017 10:16 PM, Adam Farley8 wrote: > Hi All, > > Here's a summary of the email below (which is intended, partly, as a > summary of the emails before it). > > Let me know if you agree/disagree with any of these points. > > 1) Exit(#) during vm startup is a bug because it should return a code > regardless of the state of the VM. Yes it's a bug but not one that is likely to be addressed in any foreseeable timeframe. There are simply too many "exit on error" paths. If we were to start using C++ exceptions within the VM that might provide a way to quickly get back to the CreateJavaVM routine where we could return an error code - but that is itself a major project that has barely even been discussed AFAIK. (Compiler folk have talked about it because compiler paths are fairly self-contained - though that was before Graal and AOT.) > 2) Exit(0) is an *especially* big bug because it imitates successful > completion of external cpp code accessing JNI methods. > 3) One solution is to specify a new return code for JNI. A solution for 2) yes. > 4) The supplied code (diff) generates, facilitates, and handles that > return code for the exit(0) scenario: -agentlib:jdwp=help > 5) The supplied test confirms that the supplied code works (run via > unzip, and then bash TestStart.sh bin dir>). > 6) To implement this new return code, plus the code that handles it, we > would need to follow the CSR process. > 7) To implement the fix for the scenario used as an example of the new > return code's use, we would need to modify the JVM TI spec. Yes you have demonstrated a potential solution for the agent case. The question is, is it the right solution? Is it a worthwhile solution? (As I've said I'd prefer not to have any "do something then exit" VM arguments.) And can we make it fit into the existing specs without contorting things too much. I still think it easier/preferable for whatever loads the VM to filter out the VM args that trigger this behaviour. I mean if you pass -Xshare:dump you don't have any right to expect a functioning VM after JNI_CreateJavaVM returns - at least I don't think so. Just don't do it. Yes it is an imperfection in the invocation API, but life isn't perfect. > 8) To address all of the worst instances of exit(#), we would need to > search for exit(#) and raise a bug for each significant one (or group). > 9) To solve (8) in one bug would be a lot of work, arguably too much > work for a single bug. This is simply impractical. You may be able to pick off a few low-hanging cases, but that won't really make any practical difference. > 10) If the new return code is chosen as the appropriate solution to this > problem, we may need to choose a better name for the return code. > > Is this a fair assessment of the current state of the debate? It's a fair summary of your position and proposal. Cheers, David > I'm on IRC every weekday from 9am-5pm (4pm fridays) BST (GMT+1) if > anyone wants to discuss this in real-time on the openjdk channel. > > Best Regards > > Adam Farley > > > > -- Previous Email -- > > Hi David, Alan, > > You are right in that the changes to HotSpot would be nontrivial. > I see a number of places in (e.g.) arguments.cpp that seem to > exit in the same manner as Xlog (such as -Xinternalversion). > > I would advise ploughing through the CSR process to alter the > JNI spec, and simultaneously identify some key paths that can > be raised as bugs. That way, when people have time to address > these issues, the mechanism to handle a silent exit is already > in place. > > The JDWP fix can be raised separately as one of these bugs, if > it would make things simpler. > > As for the name, JNI_SILENT_EXIT is a placeholder, and can be > readily changed. Do you have any suggestions? > > Lastly, in an ideal world, the VM initialisation should never exit(#). It > should return a return code that tells the caller something, pass or > fail, messy or tidy. That way, if someone is using the JNI as part of > something bigger (like a database or a web server), one of these > scenarios is just a bug, rather than a world-ender like exit(#). > > And now for the individual messages. :) > > David: Having help data returned by the launcher seems like a > good way to avoid exit(0) calls, but I'm not sure how we'd prevent > a JNI-caller using those options. Ultimately, to be sure, we'd have > to remove the logic for those options, centralise the data to better > enable launcher access, and add some logic in there so it can find > any other help data (e.g. from the jdwp agent library). I feel this would > be a bigger task than adding the new return code and changing the > vm, plus it wouldn't provide for any non-help scenarios where the > vm wants to shut down without error during initialisation. > > Alan: I should mention that the silent exit solution is already in > use in the OpenJ9 VM. Not all of the exit paths have been > resolved, but many have. > > The code is open and can be found here: > https://github.com/eclipse/openj9 > > And though the silent exit code is disabled for the time being, it > can be re-enabled by entering this class: > > runtime/vm/jvminit.c > > and altering line 2343 ( ctrl-f for exit(1) if it's not there). > > I won't paste the full code here in case people are concerned > about contamination, but I would assert that this code (and the > associated vm files) prove that the concept is possible. > > Note that that code should not be enabled until after we've > integrated the code that can handle a silent exit. > > Best Regards > > Adam Farley > > P.S. Thank you both for your efforts on this. :) > > > > From: David Holmes > To: Alan Bateman , Adam Farley8 > , core-libs-dev at openjdk.java.net, > hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com > Date: 15/09/2017 12:03 > Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be > exited by java > ------------------------------------------------------------------------ > > > > > > On 15/09/2017 8:17 PM, Alan Bateman wrote: > > On 15/09/2017 02:47, David Holmes wrote: > >> Hi Adam, > >> > >> I am still very much torn over this one. I think the idea of > >> print-and-exit flags for a potentially hosted library like the JVM is > >> just wrong - we should never have done that, but we did. Fixing that > >> by moving the flags to the launcher is far from trivial**. Endorsing > >> and encouraging these sorts of flag by adding JNI support seems to be > >> sending the wrong message. > >> > >> ** I can envisage a "help xxx" Dcmd that can read back the info from > >> the VM. The launcher can send the Dcmd, print the output and exit. The > >> launcher would not need to know what the xxx values mean, but would > >> have to intercept the existing ones. > >> > >> Another option is just to be aware of these flags (are there more than > >> jdwp and Xlog?) and deal with them specially in your custom launcher - > >> either filter them out and ignore them, or else launch the VM in its > >> own process to respond to them. > >> > >> Any changes to the JNI specification need to go through the CSR process. > > Yes, it would require an update to the JNI spec, also a change to the > > JVM TI spec where Agent_OnLoad returning a non-0 value is specified to > > terminates the VM. The name and value needs discussion too, esp. as the > > JNI spec uses negative values for failure. > > > > In any case, I'm also torn over this one as it's a corner case that is > > only interesting for custom launchers that load agents with options that > > print usage messages. It wouldn't be hard to have the Agent_OnLoad > > specify a printf hook that the agent could use for output although there > > are complications with agents such as JDWP that also announce their > > transport end point. Beyond that there is still the issue of the custom > > launcher that would need to know to destroy the VM without reporting an > > error. > > > > So what happened to the more meaty part to this which is fixing the > > various cases in HotSpot that terminate the process during > > initialization? I would expect some progress could be made on those > > cases while trying to decide whether to rev the JNI and JVM TI specs to > > cover the help case. > > Trying to eliminate the vm_exit_during_initialization paths in hotspot > is a huge undertaking IMHO. > > David > > > > > -Alan > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From coleen.phillimore at oracle.com Fri Oct 13 14:21:42 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 13 Oct 2017 10:21:42 -0400 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <21eef3ba-9fa2-8a76-fe3b-f9669c94f7bb@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <5C7CF716-E381-4274-8CE6-1A3C1B23346A@oracle.com> <21eef3ba-9fa2-8a76-fe3b-f9669c94f7bb@oracle.com> Message-ID: <80640482-0c4f-71b0-1ef8-3dd4792a4b6d@oracle.com> On 10/13/17 9:11 AM, David Holmes wrote: > Hi Karen, > > On 13/10/2017 8:42 PM, Karen Kinnear wrote: >> A quick note - we are resizing the hash table, i.e. number of linked >> lists, so as to speed up lookup if it is too small, or waste less >> space if it is >> too large. Each dictionary may have multiple entries off of it - one >> per loaded class - so there is no such thing as running out >> of space based on the sizing (just based on total OOM). I have not >> yet looked at the change - but just wanted to clarify the >> goal and concept for folks first. > > Thanks for clarifying - though AFAICS this proposal only grows the > dictionary. Is the "waste less space" to be achieved by starting with > a smaller dictionary initially? Do we have a way to control the > initial size? I made the change so that there is one dictionary per class loader data.? The dictionary starts at 1009 and from what I've seen in testing has about 900 entries in it on average. So seems appropriately sized.? The application class loader dictionary (as default) or dictionary defined with -Djava.system.class.loader and the other dictionaries start at 107.?? The latter can be started with a larger size with -XX:+PredictedLoadedClassCount.?? We had trouble finding any users who set this option. The application class loader dictionary is hard to find an initial size for, and rather than have the user guess which size they want, we will do automatic resizing to make it larger.? I think the initial size of application class loader dictionary should be more like 1009.? It was a bug that I initialized it with 107 actually. We do not have a way to specify initial size for class loader dictionaries.?? And we don't resize down because that doesn't make any sense.?? Entries are almost never deleted(*), rather the whole dictionary goes away when the class is unloaded. So this change is mostly for a class loader that loads a huge number of classes, that might experience slower than average lookups.? We do have a user who does this internally. Coleen > Thanks, > David > >> thanks >> Karen >> >>> On Oct 11, 2017, at 4:10 PM, coleen.phillimore at oracle.com wrote: >>> >>> >>> Gerard, some preliminary comments. >>> >>> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html >>> >>> >>> *+{* >>> *!_MutexLocker mu(SystemDictionary_lock, THREAD_);* >>> *!_Klass* probe = dictionary->find(_d_hash, name, protection_domain);* >>> *if (probe != NULL) return probe;* >>> ** >>> *+ }* >>> >>> I don't think you need this because dictionary->find() should have >>> the NoSafepointVerifier, so the index will not change. >>> >>> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/classLoaderData.cpp.udiff.html >>> >>> >>> I think we want a global _some_dictionary_needs_resizing to avoid >>> walking through the CLDG. >>> >>> And have Dictionary have a field _resizing_needed to avoid the >>> calculation during the safepoint, which is set when adding an entry. >>> >>> _resizing_needed = *(number_of_entries() > >>> (_resize_load_trigger*table_size()); >>> *CLDG::_any_resizing_needed |= _resizing_needed;?? // or something >>> like that. >>> >>> I can write more about the rationale of this change in the bug >>> report, if needed. >>> >>> Thank you for doing this change. >>> Coleen >>> >>> >>> On 10/10/17 4:40 PM, Gerard Ziemski wrote: >>>> hi all, >>>> >>>> Please review this change that adds dynamic resizing of system >>>> dictionaries. >>>> >>>> The biggest change is refactoring of the code that protects >>>> calculating of the table entry?s index using SystemDictionary_lock >>>> >>>> A few notes: >>>> >>>> - dynamic resizing is off when either dumping or using shared spaces >>>> - we remove the experimental runtime option >>>> ?PredictedLoadedClassCount? and add >>>> ?DynamicallyResizeSystemDictionaries? to turn the resizing off >>>> (it?s on by default) >>>> - the jtreg test uses stream of bytes to dynamically load numbered >>>> classes from memory instead of disk (thank you Ioi) >>>> >>>> bug:??? https://bugs.openjdk.java.net/browse/JDK-8184765 >>>> webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 >>>> >>>> >>>> cheers >>>> >>> >> From david.holmes at oracle.com Sat Oct 14 01:11:36 2017 From: david.holmes at oracle.com (David Holmes) Date: Sat, 14 Oct 2017 11:11:36 +1000 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <80640482-0c4f-71b0-1ef8-3dd4792a4b6d@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <5C7CF716-E381-4274-8CE6-1A3C1B23346A@oracle.com> <21eef3ba-9fa2-8a76-fe3b-f9669c94f7bb@oracle.com> <80640482-0c4f-71b0-1ef8-3dd4792a4b6d@oracle.com> Message-ID: Thanks for the additional background Coleen! David On 14/10/2017 12:21 AM, coleen.phillimore at oracle.com wrote: > > > On 10/13/17 9:11 AM, David Holmes wrote: >> Hi Karen, >> >> On 13/10/2017 8:42 PM, Karen Kinnear wrote: >>> A quick note - we are resizing the hash table, i.e. number of linked >>> lists, so as to speed up lookup if it is too small, or waste less >>> space if it is >>> too large. Each dictionary may have multiple entries off of it - one >>> per loaded class - so there is no such thing as running out >>> of space based on the sizing (just based on total OOM). I have not >>> yet looked at the change - but just wanted to clarify the >>> goal and concept for folks first. >> >> Thanks for clarifying - though AFAICS this proposal only grows the >> dictionary. Is the "waste less space" to be achieved by starting with >> a smaller dictionary initially? Do we have a way to control the >> initial size? > > I made the change so that there is one dictionary per class loader > data.? The dictionary starts at 1009 and from what I've > seen in testing has about 900 entries in it on average. So seems > appropriately sized.? The application class loader dictionary (as > default) or dictionary defined with -Djava.system.class.loader and the > other dictionaries start at 107.?? The latter can be started with a > larger size with -XX:+PredictedLoadedClassCount.?? We had trouble > finding any users who set this option. > > The application class loader dictionary is hard to find an initial size > for, and rather than have the user guess which size they want, we will > do automatic resizing to make it larger.? I think the initial size of > application class loader dictionary should be more like 1009.? It was a > bug that I initialized it with 107 actually. > > We do not have a way to specify initial size for class loader > dictionaries.?? And we don't resize down because that doesn't make any > sense.?? Entries are almost never deleted(*), rather the whole > dictionary goes away when the class is unloaded. > > So this change is mostly for a class loader that loads a huge number of > classes, that might experience slower than average lookups.? We do have > a user who does this internally. > > Coleen > >> Thanks, >> David >> >>> thanks >>> Karen >>> >>>> On Oct 11, 2017, at 4:10 PM, coleen.phillimore at oracle.com wrote: >>>> >>>> >>>> Gerard, some preliminary comments. >>>> >>>> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html >>>> >>>> >>>> *+{* >>>> *!_MutexLocker mu(SystemDictionary_lock, THREAD_);* >>>> *!_Klass* probe = dictionary->find(_d_hash, name, protection_domain);* >>>> *if (probe != NULL) return probe;* >>>> ** >>>> *+ }* >>>> >>>> I don't think you need this because dictionary->find() should have >>>> the NoSafepointVerifier, so the index will not change. >>>> >>>> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/classLoaderData.cpp.udiff.html >>>> >>>> >>>> I think we want a global _some_dictionary_needs_resizing to avoid >>>> walking through the CLDG. >>>> >>>> And have Dictionary have a field _resizing_needed to avoid the >>>> calculation during the safepoint, which is set when adding an entry. >>>> >>>> _resizing_needed = *(number_of_entries() > >>>> (_resize_load_trigger*table_size()); >>>> *CLDG::_any_resizing_needed |= _resizing_needed;?? // or something >>>> like that. >>>> >>>> I can write more about the rationale of this change in the bug >>>> report, if needed. >>>> >>>> Thank you for doing this change. >>>> Coleen >>>> >>>> >>>> On 10/10/17 4:40 PM, Gerard Ziemski wrote: >>>>> hi all, >>>>> >>>>> Please review this change that adds dynamic resizing of system >>>>> dictionaries. >>>>> >>>>> The biggest change is refactoring of the code that protects >>>>> calculating of the table entry?s index using SystemDictionary_lock >>>>> >>>>> A few notes: >>>>> >>>>> - dynamic resizing is off when either dumping or using shared spaces >>>>> - we remove the experimental runtime option >>>>> ?PredictedLoadedClassCount? and add >>>>> ?DynamicallyResizeSystemDictionaries? to turn the resizing off >>>>> (it?s on by default) >>>>> - the jtreg test uses stream of bytes to dynamically load numbered >>>>> classes from memory instead of disk (thank you Ioi) >>>>> >>>>> bug:??? https://bugs.openjdk.java.net/browse/JDK-8184765 >>>>> webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 >>>>> >>>>> >>>>> cheers >>>>> >>>> >>> > From harold.seigel at oracle.com Mon Oct 16 14:20:40 2017 From: harold.seigel at oracle.com (harold seigel) Date: Mon, 16 Oct 2017 10:20:40 -0400 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution Message-ID: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> Hi, Please review this new JDK-10 fix for bug JDK-8174954.? If the initial resolution attempt fails with a LinkageError exception then the fix saves the exception in the resolution_errors table and sets a flag in the constant pool Cache, indicating the failure.? Subsequent attempts to do the same resolution will see that the flag is set, retrieve the exception from the resolution_errors table, and throw it, instead of re-trying the resolution. The fix also prevents a race condition if two or more threads try to do the same resolution concurrently.? When a thread tries to record the result of its resolution attempt, it will check to see if another thread recorded its result first.? If so, then it will use the result recorded by the other thread.? Otherwise, it will record and use its result.? Access to the resolution result is protected by the 'resolved_references' lock. Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 The fix was tested with the JCK Lang and VM tests, the JTreg hotspot, java/io, java/lang, java/util, and other tests, the co-located NSK tests, JPRT, with Mach5 tier2 - tier5 tests, and by hand to check race condition handling. Thanks, Harold From rkennke at redhat.com Mon Oct 16 16:19:44 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 16 Oct 2017 18:19:44 +0200 Subject: RFR (S): 8185580: Refactor Threads::possibly_parallel_oops_do() to use Threads::parallel_java_threads_do() In-Reply-To: References: Message-ID: <60d5ee80-c3b9-273b-39fe-6279faae1827@redhat.com> Am 11.10.2017 um 21:30 schrieb Roman Kennke: > This is a follow-up to my earlier safepoint parallel cleanup work. > > Currently we have 2 places where we apply the parallel claiming > protocol when iterating threads, that is > Threads::parallel_java_threads_do() and > Threads::possibly_parallel_oops_do(). In order to avoid code > duplication, the latter should call the former, using a private > ThreadClosure. We already had one bug (JDK-8185273) that was caused by > an inconsistency between the two. > > The only other user of parallel_java_threads_do() already filters out > any non-Java-threads, which means that doing the extra processing of > the VMThread should be ok. I renamed the method to > parallel_threads_do() to reflect that it not only does the Java threads. > > Webrev: > http://cr.openjdk.java.net/~rkennke/8185580/webrev.00/ > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8185580 > > Tested successfully by running hotspot_gc tests, which should cover > most if not all uses of those methods. > > Roman Ping? Roman From jiangli.zhou at oracle.com Mon Oct 16 17:51:26 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 16 Oct 2017 10:51:26 -0700 Subject: RFR(M) JDK-8188791 Move AppCDS implementation from closed repo to open repo In-Reply-To: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> Message-ID: Hi Ioi, The move looks good to me. Thanks, Jiangli > On Oct 12, 2017, at 4:48 PM, Ioi Lam wrote: > > Hi, > > Please review this change set. > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ > https://bugs.openjdk.java.net/browse/JDK-8188791 > > This is the first step of implementing the following JEP, which moves AppCDS from > closed repos into the openjdk repo: > > https://bugs.openjdk.java.net/browse/JDK-8185996 > > In JDK 9, significant portion of AppCDS code resided in the closed repo. As part > of the open-sourcing effort of JDK 18.3, we will move the source code into the > open repo. > > In this changeset, the code is moved verbatim as much as possible. The intention is > only to relocate the sources, not to changing existing behaviors, and not > to do any sort of refactoring. > > Most of the "diffs" shown in this webrev are the result of copying the closed source > files on top of files of the same name in the open repo. So in reviewing, instead of > focusing on what's "changed", it's better to focus on the entire content of the new > version of each file. > > The only functional change in this task is that the UseAppCDS flag is changed from > a "commercial" flag to a regular "product" flag. This is because "commercial" > flags are not supported by the OpenJDK build. > > Source code refactoring may be desirable, because the old open/closed source > code structure had introduced some intermediary APIs to connect code between > the two repos. Such API should be removed in a separate RFE. > > Also, some AppCDS tests are currently in the closed repo. These tests will be > moved in a separate task. See JDK-8188792 for details. > > All the AppCDS tests (currently still in closed sources) passed with both Oracle JDK > and OpenJDK. > > Thanks > - Ioi From coleen.phillimore at oracle.com Mon Oct 16 20:23:07 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 16 Oct 2017 16:23:07 -0400 Subject: RFR (S): 8185580: Refactor Threads::possibly_parallel_oops_do() to use Threads::parallel_java_threads_do() In-Reply-To: <60d5ee80-c3b9-273b-39fe-6279faae1827@redhat.com> References: <60d5ee80-c3b9-273b-39fe-6279faae1827@redhat.com> Message-ID: This looks good to me.? I will sponsor it for you.?? Maybe Dan can provide a second review since he tracked down the bug caused by the former inconsistency of these functions. thanks, Coleen On 10/16/17 12:19 PM, Roman Kennke wrote: > Am 11.10.2017 um 21:30 schrieb Roman Kennke: >> This is a follow-up to my earlier safepoint parallel cleanup work. >> >> Currently we have 2 places where we apply the parallel claiming >> protocol when iterating threads, that is >> Threads::parallel_java_threads_do() and >> Threads::possibly_parallel_oops_do(). In order to avoid code >> duplication, the latter should call the former, using a private >> ThreadClosure. We already had one bug (JDK-8185273) that was caused >> by an inconsistency between the two. >> >> The only other user of parallel_java_threads_do() already filters out >> any non-Java-threads, which means that doing the extra processing of >> the VMThread should be ok. I renamed the method to >> parallel_threads_do() to reflect that it not only does the Java threads. >> >> Webrev: >> http://cr.openjdk.java.net/~rkennke/8185580/webrev.00/ >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8185580 >> >> Tested successfully by running hotspot_gc tests, which should cover >> most if not all uses of those methods. >> >> Roman > > Ping? > > Roman > From rkennke at redhat.com Mon Oct 16 20:26:58 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 16 Oct 2017 22:26:58 +0200 Subject: RFR (S): 8185580: Refactor Threads::possibly_parallel_oops_do() to use Threads::parallel_java_threads_do() In-Reply-To: References: <60d5ee80-c3b9-273b-39fe-6279faae1827@redhat.com> Message-ID: <0fd8dd7d-d67b-18f0-0ac4-891d4c62b62e@redhat.com> Thank you! Roman > > This looks good to me.? I will sponsor it for you.?? Maybe Dan can > provide a second review since he tracked down the bug caused by the > former inconsistency of these functions. > thanks, > Coleen > > On 10/16/17 12:19 PM, Roman Kennke wrote: >> Am 11.10.2017 um 21:30 schrieb Roman Kennke: >>> This is a follow-up to my earlier safepoint parallel cleanup work. >>> >>> Currently we have 2 places where we apply the parallel claiming >>> protocol when iterating threads, that is >>> Threads::parallel_java_threads_do() and >>> Threads::possibly_parallel_oops_do(). In order to avoid code >>> duplication, the latter should call the former, using a private >>> ThreadClosure. We already had one bug (JDK-8185273) that was caused >>> by an inconsistency between the two. >>> >>> The only other user of parallel_java_threads_do() already filters >>> out any non-Java-threads, which means that doing the extra >>> processing of the VMThread should be ok. I renamed the method to >>> parallel_threads_do() to reflect that it not only does the Java >>> threads. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~rkennke/8185580/webrev.00/ >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8185580 >>> >>> Tested successfully by running hotspot_gc tests, which should cover >>> most if not all uses of those methods. >>> >>> Roman >> >> Ping? >> >> Roman >> > From calvin.cheung at oracle.com Mon Oct 16 22:15:53 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Mon, 16 Oct 2017 15:15:53 -0700 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures Message-ID: <59E52F99.8090903@oracle.com> JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 Adding a warning message if the full path or the directory length exceeds MAX_PATH on windows. Example warning messages. 1) The full path exceeds MAX_PATH: Java HotSpot(TM) 64-Bit Server VM warning: construct full path name failed: total length 270 exceeds max length of 260 dir T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx length 259 name Hello.class length 11 2) The directory path exceeds MAX_PATH: Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: path length 265 exceeds max length 260 path T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx webrev: http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ Testing: JPRT (including the new test) thanks, Calvin From ioi.lam at oracle.com Mon Oct 16 23:02:37 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 16 Oct 2017 16:02:37 -0700 Subject: RFR (XS) - 8176827 Test can't find libXext.so Message-ID: Hi, JDK-8176827 is a closed issue, but basically we have some tests that have unnecessary dependencies on desktop features, and would sporadically fail on hosts that are? configured to run only headless tests. https://bugs.openjdk.java.net/browse/JDK-8176827 http://cr.openjdk.java.net/~iklam/jdk10/8176827-cant-find-libxext.v01/ The fix is simple -- replace desktop classes used by the tests with headless classes. E.g., java.awt.Button -> java.util.logging.Level Thanks - Ioi From david.holmes at oracle.com Mon Oct 16 23:06:47 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 09:06:47 +1000 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <59E52F99.8090903@oracle.com> References: <59E52F99.8090903@oracle.com> Message-ID: <736614c0-ed94-8adc-0e39-760f86b3b085@oracle.com> Hi Calvin, Isn't there a Unicode API we can use for this? "The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. " [1] Thanks, David [1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath On 17/10/2017 8:15 AM, Calvin Cheung wrote: > JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 > > Adding a warning message if the full path or the directory length > exceeds MAX_PATH on windows. > > Example warning messages. > > 1) The full path exceeds MAX_PATH: > > Java HotSpot(TM) 64-Bit Server VM warning: construct full path name > failed: total length 270 exceeds max length of 260 > ??? dir > T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > length 259 > ??? name Hello.class length 11 > > 2) The directory path exceeds MAX_PATH: > > Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: path length > 265 exceeds max length 260 > ??? path > T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx > > > webrev: > ??? http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ > > Testing: > ??? JPRT (including the new test) > > thanks, > Calvin From david.holmes at oracle.com Mon Oct 16 23:07:49 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 09:07:49 +1000 Subject: RFR (XS) - 8176827 Test can't find libXext.so In-Reply-To: References: Message-ID: Looks good. Thanks, David On 17/10/2017 9:02 AM, Ioi Lam wrote: > Hi, > > JDK-8176827 is a closed issue, but basically we have some tests that > have unnecessary dependencies on desktop features, and would > sporadically fail on hosts that are? configured to run only headless tests. > > https://bugs.openjdk.java.net/browse/JDK-8176827 > http://cr.openjdk.java.net/~iklam/jdk10/8176827-cant-find-libxext.v01/ > > The fix is simple -- replace desktop classes used by the tests with > headless classes. E.g., java.awt.Button -> java.util.logging.Level > > Thanks > - Ioi From ioi.lam at oracle.com Mon Oct 16 23:15:38 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 16 Oct 2017 16:15:38 -0700 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <59E52F99.8090903@oracle.com> References: <59E52F99.8090903@oracle.com> Message-ID: <1fd5de54-6d7a-d847-b60b-0480b3d981fb@oracle.com> Hi Calvin, From your test case: ? 44???? private static final int MAX_PATH = 260; ? ... ? 53???????? int subDirLen = MAX_PATH - classDir.toString().length() - 2; ? 54???????? if (subDirLen > 0) { ? 55???????????? char[] chars = new char[subDirLen]; ? 56???????????? Arrays.fill(chars, 'x'); ? 57???????????? String subPath = new String(chars); ? 58???????????? destDir = Paths.get(System.getProperty("test.classes"), subPath); ? 59???????? } ? 60 ? 61???????? CompilerUtils.compile(sourceDir, destDir); It looks like at least part of the JDK is able to read and write files that are in paths longer than 260 characters. For JDK-8188122, it seems the problem exists only with -Xbootclasspath/a is used. Have you tested if you can use -cp to load classes from the long directory? If that's the case, maybe we should fix -Xbootclasspath/a so that it can handle over 260 chars on Windows? Thanks - Ioi On 10/16/17 3:15 PM, Calvin Cheung wrote: > JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 > > Adding a warning message if the full path or the directory length > exceeds MAX_PATH on windows. > > Example warning messages. > > 1) The full path exceeds MAX_PATH: > > Java HotSpot(TM) 64-Bit Server VM warning: construct full path name > failed: total length 270 exceeds max length of 260 > ??? dir > T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > length 259 > ??? name Hello.class length 11 > > 2) The directory path exceeds MAX_PATH: > > Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: path > length 265 exceeds max length 260 > ??? path > T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx > > webrev: > ??? http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ > > Testing: > ??? JPRT (including the new test) > > thanks, > Calvin From calvin.cheung at oracle.com Mon Oct 16 23:23:37 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Mon, 16 Oct 2017 16:23:37 -0700 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <736614c0-ed94-8adc-0e39-760f86b3b085@oracle.com> References: <59E52F99.8090903@oracle.com> <736614c0-ed94-8adc-0e39-760f86b3b085@oracle.com> Message-ID: <59E53F79.1080707@oracle.com> Hi David, In order to use the Unicode API, the hotspot windows version needs to be compiled with UNICODE defined. I can file an RFE for that if we want to pursue that enhancement. For now, I think adding more informative message other than just a CNFE should be sufficient. What do you think? thanks, Calvin On 10/16/17, 4:06 PM, David Holmes wrote: > Hi Calvin, > > Isn't there a Unicode API we can use for this? > > "The Windows API has many functions that also have Unicode versions to > permit an extended-length path for a maximum total path length of > 32,767 characters. " [1] > > Thanks, > David > > [1] > https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath > > On 17/10/2017 8:15 AM, Calvin Cheung wrote: >> JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 >> >> Adding a warning message if the full path or the directory length >> exceeds MAX_PATH on windows. >> >> Example warning messages. >> >> 1) The full path exceeds MAX_PATH: >> >> Java HotSpot(TM) 64-Bit Server VM warning: construct full path name >> failed: total length 270 exceeds max length of 260 >> dir >> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> length 259 >> name Hello.class length 11 >> >> 2) The directory path exceeds MAX_PATH: >> >> Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: path >> length 265 exceeds max length 260 >> path >> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >> >> >> webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >> >> Testing: >> JPRT (including the new test) >> >> thanks, >> Calvin From calvin.cheung at oracle.com Tue Oct 17 00:47:24 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Mon, 16 Oct 2017 17:47:24 -0700 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <1fd5de54-6d7a-d847-b60b-0480b3d981fb@oracle.com> References: <59E52F99.8090903@oracle.com> <1fd5de54-6d7a-d847-b60b-0480b3d981fb@oracle.com> Message-ID: <59E5531C.70300@oracle.com> Hi Ioi, On 10/16/17, 4:15 PM, Ioi Lam wrote: > Hi Calvin, > > From your test case: > > 44 private static final int MAX_PATH = 260; > ... > > 53 int subDirLen = MAX_PATH - classDir.toString().length() - 2; > 54 if (subDirLen > 0) { > 55 char[] chars = new char[subDirLen]; > 56 Arrays.fill(chars, 'x'); > 57 String subPath = new String(chars); > 58 destDir = > Paths.get(System.getProperty("test.classes"), subPath); > 59 } > 60 > 61 CompilerUtils.compile(sourceDir, destDir); > > It looks like at least part of the JDK is able to read and write files > that are in paths longer than 260 characters. > > For JDK-8188122, it seems the problem exists only with > -Xbootclasspath/a is used. Have you tested if you can use -cp to load > classes from the long directory? Yes, the problem exists only with -Xbootclasspath/a because the call path is different comparing with -cp. Call path with -Xbootclasspath/a: ClassPathDirEntry::open_stream() at classLoader.cpp:274 0x7ffff5ca48d0 ClassLoader::load_class() at classLoader.cpp:1,412 0x7ffff5ca94a8 SystemDictionary::load_instance_class() at systemDictionary.cpp:1,524 0x7ffff671e141 SystemDictionary::resolve_instance_class_or_null() at systemDictionary.cpp:847 0x7ffff671bb3f SystemDictionary::resolve_or_null() at systemDictionary.cpp:263 0x7ffff671a1a1 SystemDictionary::resolve_or_null() at systemDictionary.cpp:268 0x7ffff671a215 JVM_FindClassFromBootLoader() at jvm.cpp:777 0x7ffff6204544 Java_java_lang_ClassLoader_findBootstrapClass() at 0x7ffff4a8a0e6 Call path with -cp: ClassFileParser::fill_instance_klass() at classFileParser.cpp:5,309 0x7ffff5c93a6d ClassFileParser::create_instance_klass() at classFileParser.cpp:5,289 0x7ffff5c938bd KlassFactory::create_from_stream() at klassFactory.cpp:214 0x7ffff63462ef SystemDictionary::resolve_from_stream() at systemDictionary.cpp:1,144 0x7ffff671cadd jvm_define_class_common() at jvm.cpp:943 0x7ffff6205376 JVM_DefineClassWithSource() at jvm.cpp:963 0x7ffff620588d Java_java_lang_ClassLoader_defineClass1() at 0x7ffff4a89c68 Note that ClassPathDirEntry::open_stream() isn't called when -cp is used. > > If that's the case, maybe we should fix -Xbootclasspath/a so that it > can handle over 260 chars on Windows? No easy way of fixing it other than using the unicode version of windows API. thanks, Calvin > > Thanks > > - Ioi > > > > On 10/16/17 3:15 PM, Calvin Cheung wrote: >> JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 >> >> Adding a warning message if the full path or the directory length >> exceeds MAX_PATH on windows. >> >> Example warning messages. >> >> 1) The full path exceeds MAX_PATH: >> >> Java HotSpot(TM) 64-Bit Server VM warning: construct full path name >> failed: total length 270 exceeds max length of 260 >> dir >> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> length 259 >> name Hello.class length 11 >> >> 2) The directory path exceeds MAX_PATH: >> >> Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: path >> length 265 exceeds max length 260 >> path >> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >> >> webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >> >> Testing: >> JPRT (including the new test) >> >> thanks, >> Calvin > From david.holmes at oracle.com Tue Oct 17 01:26:55 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 11:26:55 +1000 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <59E5531C.70300@oracle.com> References: <59E52F99.8090903@oracle.com> <1fd5de54-6d7a-d847-b60b-0480b3d981fb@oracle.com> <59E5531C.70300@oracle.com> Message-ID: On 17/10/2017 10:47 AM, Calvin Cheung wrote: > Hi Ioi, > > On 10/16/17, 4:15 PM, Ioi Lam wrote: >> Hi Calvin, >> >> From your test case: >> >> ? 44???? private static final int MAX_PATH = 260; >> ? ... >> >> ? 53???????? int subDirLen = MAX_PATH - classDir.toString().length() - 2; >> ? 54???????? if (subDirLen > 0) { >> ? 55???????????? char[] chars = new char[subDirLen]; >> ? 56???????????? Arrays.fill(chars, 'x'); >> ? 57???????????? String subPath = new String(chars); >> ? 58???????????? destDir = >> Paths.get(System.getProperty("test.classes"), subPath); >> ? 59???????? } >> ? 60 >> ? 61???????? CompilerUtils.compile(sourceDir, destDir); >> >> It looks like at least part of the JDK is able to read and write files >> that are in paths longer than 260 characters. >> >> For JDK-8188122, it seems the problem exists only with >> -Xbootclasspath/a is used. Have you tested if you can use -cp to load >> classes from the long directory? > Yes, the problem exists only with -Xbootclasspath/a because the call > path is different comparing with -cp. The point is if -cp can handle it then -Xbootclasspath must be able to handle it too! They both have to read directories! Thanks, David > Call path with -Xbootclasspath/a: > ??? ClassPathDirEntry::open_stream() at classLoader.cpp:274 0x7ffff5ca48d0 > ??? ClassLoader::load_class() at classLoader.cpp:1,412 0x7ffff5ca94a8 > ??? SystemDictionary::load_instance_class() at > systemDictionary.cpp:1,524 0x7ffff671e141 > ??? SystemDictionary::resolve_instance_class_or_null() at > systemDictionary.cpp:847 0x7ffff671bb3f > ??? SystemDictionary::resolve_or_null() at systemDictionary.cpp:263 > 0x7ffff671a1a1 > ??? SystemDictionary::resolve_or_null() at systemDictionary.cpp:268 > 0x7ffff671a215 > ??? JVM_FindClassFromBootLoader() at jvm.cpp:777 0x7ffff6204544 > ??? Java_java_lang_ClassLoader_findBootstrapClass() at 0x7ffff4a8a0e6 > > Call path with -cp: > > ??? ClassFileParser::fill_instance_klass() at classFileParser.cpp:5,309 > 0x7ffff5c93a6d > ??? ClassFileParser::create_instance_klass() at > classFileParser.cpp:5,289 0x7ffff5c938bd > ??? KlassFactory::create_from_stream() at klassFactory.cpp:214 > 0x7ffff63462ef > ??? SystemDictionary::resolve_from_stream() at > systemDictionary.cpp:1,144 0x7ffff671cadd > ??? jvm_define_class_common() at jvm.cpp:943 0x7ffff6205376 > ??? JVM_DefineClassWithSource() at jvm.cpp:963 0x7ffff620588d > ??? Java_java_lang_ClassLoader_defineClass1() at 0x7ffff4a89c68 > > Note that ClassPathDirEntry::open_stream() isn't called when -cp is used. >> >> If that's the case, maybe we should fix -Xbootclasspath/a so that it >> can handle over 260 chars on Windows? > No easy way of fixing it other than using the unicode version of windows > API. > > thanks, > Calvin >> >> Thanks >> >> - Ioi >> >> >> >> On 10/16/17 3:15 PM, Calvin Cheung wrote: >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 >>> >>> Adding a warning message if the full path or the directory length >>> exceeds MAX_PATH on windows. >>> >>> Example warning messages. >>> >>> 1) The full path exceeds MAX_PATH: >>> >>> Java HotSpot(TM) 64-Bit Server VM warning: construct full path name >>> failed: total length 270 exceeds max length of 260 >>> ??? dir >>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> length 259 >>> ??? name Hello.class length 11 >>> >>> 2) The directory path exceeds MAX_PATH: >>> >>> Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: path >>> length 265 exceeds max length 260 >>> ??? path >>> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >>> >>> >>> webrev: >>> ??? http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >>> >>> Testing: >>> ??? JPRT (including the new test) >>> >>> thanks, >>> Calvin >> From david.holmes at oracle.com Tue Oct 17 03:33:10 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 13:33:10 +1000 Subject: RFR (S): 8185580: Refactor Threads::possibly_parallel_oops_do() to use Threads::parallel_java_threads_do() In-Reply-To: References: Message-ID: <04615da8-68f1-cbda-a7a6-628dabd82e97@oracle.com> Hi Roman, On 12/10/2017 5:30 AM, Roman Kennke wrote: > This is a follow-up to my earlier safepoint parallel cleanup work. > > Currently we have 2 places where we apply the parallel claiming protocol > when iterating threads, that is Threads::parallel_java_threads_do() and > Threads::possibly_parallel_oops_do(). In order to avoid code > duplication, the latter should call the former, using a private > ThreadClosure. We already had one bug (JDK-8185273) that was caused by > an inconsistency between the two. > > The only other user of parallel_java_threads_do() already filters out > any non-Java-threads, which means that doing the extra processing of the > VMThread should be ok. "should" be okay? It's not at all clear to me what processing the VMThread where previously it was not processed, will actually do ?? > I renamed the method to parallel_threads_do() to > reflect that it not only does the Java threads. Given it takes an is_par argument it really should be possibly_parallel_threads_do() Thanks, David > Webrev: > http://cr.openjdk.java.net/~rkennke/8185580/webrev.00/ > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8185580 > > Tested successfully by running hotspot_gc tests, which should cover most > if not all uses of those methods. > > Roman From rkennke at redhat.com Tue Oct 17 04:55:31 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 17 Oct 2017 06:55:31 +0200 Subject: RFR (S): 8185580: Refactor Threads::possibly_parallel_oops_do() to use Threads::parallel_java_threads_do() In-Reply-To: <04615da8-68f1-cbda-a7a6-628dabd82e97@oracle.com> References: <04615da8-68f1-cbda-a7a6-628dabd82e97@oracle.com> Message-ID: <1e638965-2df0-f495-95da-0e13a0984359@redhat.com> Am 17.10.2017 um 05:33 schrieb David Holmes: > Hi Roman, > > On 12/10/2017 5:30 AM, Roman Kennke wrote: >> This is a follow-up to my earlier safepoint parallel cleanup work. >> >> Currently we have 2 places where we apply the parallel claiming >> protocol when iterating threads, that is >> Threads::parallel_java_threads_do() and >> Threads::possibly_parallel_oops_do(). In order to avoid code >> duplication, the latter should call the former, using a private >> ThreadClosure. We already had one bug (JDK-8185273) that was caused >> by an inconsistency between the two. >> >> The only other user of parallel_java_threads_do() already filters out >> any non-Java-threads, which means that doing the extra processing of >> the VMThread should be ok. > > "should" be okay? It's not at all clear to me what processing the > VMThread where previously it was not processed, will actually do ?? It ensures that both parallel thread iteration routines are doing the same, and thus claim the same threads, which is later assert_all_threads_are_claimed(). If we don't do this consistently, we throw off the thread claiming mechanics. > >> I renamed the method to parallel_threads_do() to reflect that it not >> only does the Java threads. > > Given it takes an is_par argument it really should be > possibly_parallel_threads_do() Ok. Differential diff: http://cr.openjdk.java.net/~rkennke/8185580/webrev.01.diff/ Full webrev: http://cr.openjdk.java.net/~rkennke/8185580/webrev.01/ Good? Roman From david.holmes at oracle.com Tue Oct 17 05:11:17 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 15:11:17 +1000 Subject: RFR (S): 8185580: Refactor Threads::possibly_parallel_oops_do() to use Threads::parallel_java_threads_do() In-Reply-To: <1e638965-2df0-f495-95da-0e13a0984359@redhat.com> References: <04615da8-68f1-cbda-a7a6-628dabd82e97@oracle.com> <1e638965-2df0-f495-95da-0e13a0984359@redhat.com> Message-ID: On 17/10/2017 2:55 PM, Roman Kennke wrote: > Am 17.10.2017 um 05:33 schrieb David Holmes: >> Hi Roman, >> >> On 12/10/2017 5:30 AM, Roman Kennke wrote: >>> This is a follow-up to my earlier safepoint parallel cleanup work. >>> >>> Currently we have 2 places where we apply the parallel claiming >>> protocol when iterating threads, that is >>> Threads::parallel_java_threads_do() and >>> Threads::possibly_parallel_oops_do(). In order to avoid code >>> duplication, the latter should call the former, using a private >>> ThreadClosure. We already had one bug (JDK-8185273) that was caused >>> by an inconsistency between the two. >>> >>> The only other user of parallel_java_threads_do() already filters out >>> any non-Java-threads, which means that doing the extra processing of >>> the VMThread should be ok. >> >> "should" be okay? It's not at all clear to me what processing the >> VMThread where previously it was not processed, will actually do ?? > It ensures that both parallel thread iteration routines are doing the > same, and thus claim the same threads, which is later > assert_all_threads_are_claimed(). If we don't do this consistently, we > throw off the thread claiming mechanics. Sorry that's not answering my question. Before we did not call tc->do_thread(vmt); for the VMThread, but now we do. So what is the affect of this change in behaviour? AFAICS the "claiming" behaviour of both versions is the same. Thanks, David ----- >> >>> I renamed the method to parallel_threads_do() to reflect that it not >>> only does the Java threads. >> >> Given it takes an is_par argument it really should be >> possibly_parallel_threads_do() > Ok. Differential diff: > http://cr.openjdk.java.net/~rkennke/8185580/webrev.01.diff/ > > Full webrev: > http://cr.openjdk.java.net/~rkennke/8185580/webrev.01/ > > > Good? > > Roman > From rkennke at redhat.com Tue Oct 17 05:27:29 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 17 Oct 2017 07:27:29 +0200 Subject: RFR (S): 8185580: Refactor Threads::possibly_parallel_oops_do() to use Threads::parallel_java_threads_do() In-Reply-To: References: <04615da8-68f1-cbda-a7a6-628dabd82e97@oracle.com> <1e638965-2df0-f495-95da-0e13a0984359@redhat.com> Message-ID: <74b27f36-eca1-f81d-d970-55d32ed84c21@redhat.com> Am 17.10.2017 um 07:11 schrieb David Holmes: > On 17/10/2017 2:55 PM, Roman Kennke wrote: >> Am 17.10.2017 um 05:33 schrieb David Holmes: >>> Hi Roman, >>> >>> On 12/10/2017 5:30 AM, Roman Kennke wrote: >>>> This is a follow-up to my earlier safepoint parallel cleanup work. >>>> >>>> Currently we have 2 places where we apply the parallel claiming >>>> protocol when iterating threads, that is >>>> Threads::parallel_java_threads_do() and >>>> Threads::possibly_parallel_oops_do(). In order to avoid code >>>> duplication, the latter should call the former, using a private >>>> ThreadClosure. We already had one bug (JDK-8185273) that was caused >>>> by an inconsistency between the two. >>>> >>>> The only other user of parallel_java_threads_do() already filters >>>> out any non-Java-threads, which means that doing the extra >>>> processing of the VMThread should be ok. >>> >>> "should" be okay? It's not at all clear to me what processing the >>> VMThread where previously it was not processed, will actually do ?? >> It ensures that both parallel thread iteration routines are doing the >> same, and thus claim the same threads, which is later >> assert_all_threads_are_claimed(). If we don't do this consistently, >> we throw off the thread claiming mechanics. > > Sorry that's not answering my question. Before we did not call > > tc->do_thread(vmt); > > for the VMThread, but now we do. So what is the affect of this change > in behaviour? The effect is that for the safepointing code we now also call do_thread(vmt), but since we're filtering out non-Java threads in the ThreadClosure, the net effect is zero. There is no actual bug here to fix. The only point of this refactoring is to make both code path use the same iterator and thus ensure consistency and *not change* any effects. (Note: there will be future uses of possibly_parallel_threads_do() that need to see the VMThread, e.g. we have one such case in Shenandoah.) Does that answer your question? Roman From david.holmes at oracle.com Tue Oct 17 06:04:50 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 16:04:50 +1000 Subject: RFR (S): 8185580: Refactor Threads::possibly_parallel_oops_do() to use Threads::parallel_java_threads_do() In-Reply-To: <74b27f36-eca1-f81d-d970-55d32ed84c21@redhat.com> References: <04615da8-68f1-cbda-a7a6-628dabd82e97@oracle.com> <1e638965-2df0-f495-95da-0e13a0984359@redhat.com> <74b27f36-eca1-f81d-d970-55d32ed84c21@redhat.com> Message-ID: <31ca7fff-5bbb-0b76-b6e8-6a3e57f84290@oracle.com> Hi Roman, On 17/10/2017 3:27 PM, Roman Kennke wrote: > Am 17.10.2017 um 07:11 schrieb David Holmes: >> On 17/10/2017 2:55 PM, Roman Kennke wrote: >>> Am 17.10.2017 um 05:33 schrieb David Holmes: >>>> Hi Roman, >>>> >>>> On 12/10/2017 5:30 AM, Roman Kennke wrote: >>>>> This is a follow-up to my earlier safepoint parallel cleanup work. >>>>> >>>>> Currently we have 2 places where we apply the parallel claiming >>>>> protocol when iterating threads, that is >>>>> Threads::parallel_java_threads_do() and >>>>> Threads::possibly_parallel_oops_do(). In order to avoid code >>>>> duplication, the latter should call the former, using a private >>>>> ThreadClosure. We already had one bug (JDK-8185273) that was caused >>>>> by an inconsistency between the two. >>>>> >>>>> The only other user of parallel_java_threads_do() already filters >>>>> out any non-Java-threads, which means that doing the extra >>>>> processing of the VMThread should be ok. >>>> >>>> "should" be okay? It's not at all clear to me what processing the >>>> VMThread where previously it was not processed, will actually do ?? >>> It ensures that both parallel thread iteration routines are doing the >>> same, and thus claim the same threads, which is later >>> assert_all_threads_are_claimed(). If we don't do this consistently, >>> we throw off the thread claiming mechanics. >> >> Sorry that's not answering my question. Before we did not call >> >> tc->do_thread(vmt); >> >> for the VMThread, but now we do. So what is the affect of this change >> in behaviour? > > The effect is that for the safepointing code we now also call > do_thread(vmt), but since we're filtering out non-Java threads in the > ThreadClosure, the net effect is zero. There is no actual bug here to > fix. The only point of this refactoring is to make both code path use > the same iterator and thus ensure consistency and *not change* any > effects. (Note: there will be future uses of > possibly_parallel_threads_do() that need to see the VMThread, e.g. we > have one such case in Shenandoah.) > > Does that answer your question? Yes - thank you. I find it odd that a closure that was passed to parallel_java_threads_do additionally filtered on is_Java_thread, but with the new entry point it's perfectly fine. Thanks, David > Roman > From david.holmes at oracle.com Tue Oct 17 07:22:10 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 17:22:10 +1000 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM Message-ID: CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 Could I please have a reviewer for this CSR request so I can fast-track it. Comments on the proposal are of course welcome. Thanks, David From thomas.stuefe at gmail.com Tue Oct 17 08:04:28 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 17 Oct 2017 10:04:28 +0200 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: Message-ID: Hi David, Just some notes: - I am confused about the "primordial thread" naming; I always understood the primordial thread to be the thread running main(), not the thread invoking JNI_CreateJavaVM. I always used this term in this way for AIX related problems. How about "VM-Initializing thread"? - Would this not also be a useful feature for other threads attached to the VM (AttachCurrentThread), as those threads may use a different SO-handling scheme too ? Other than that, I think this is a useful addition. Would have helped me before on AIX (see https://bugs.openjdk.java.net/browse/JDK-8179327) Kind Regards, Thomas On Tue, Oct 17, 2017 at 9:22 AM, David Holmes wrote: > CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 > > Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 > > Could I please have a reviewer for this CSR request so I can fast-track it. > > Comments on the proposal are of course welcome. > > Thanks, > David > From dmitry.samersoff at bell-sw.com Tue Oct 17 08:31:59 2017 From: dmitry.samersoff at bell-sw.com (dmitry.samersov) Date: Tue, 17 Oct 2017 11:31:59 +0300 Subject: RFR(S): AArch64: NMT detail stack trace cleanup Message-ID: <81369691-46f1-a0f0-c3f8-8bc17ae94e7d@bell-sw.com> Everybody, Please, review a minimal, aarch64 only patch that makes aarch64 behavior similar to x86 one. http://cr.openjdk.java.net/~dsamersoff/JDK-8163011/aarch64_only/webrev.01/ This patch is not a replacement for a larger refactoring I proposed early, but the refactoring requires significant effort to test it on different platforms, so I plan to file a separate CR and address it when possible. -Dmitry From david.holmes at oracle.com Tue Oct 17 09:06:44 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 19:06:44 +1000 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: Message-ID: <674c235d-b79c-695a-abae-9b354c335927@oracle.com> Hi Thomas, On 17/10/2017 6:04 PM, Thomas St?fe wrote: > Hi David, > > Just some notes: > > - I am confused about the "primordial thread" naming; I always > understood the primordial thread to be the thread running main(), not > the thread invoking JNI_CreateJavaVM. I always used this term in this > way for AIX related problems. How about "VM-Initializing thread"? This is about the primordial thread only, not about the VM-initializing thread in general. The primordial thread is the initial thread of a process that executes main as you note. It is only when that thread is used to create the VM that we want to allow disabling of the stack guard pages. > - Would this not also be a useful feature for other threads attached to > the VM (AttachCurrentThread), as those threads may use a different > SO-handling scheme too ? Perhaps but I'm not going there :) This is just a concession to a particular execution scenario. > Other than that, I think this is a useful addition. Would have helped me > before on AIX (see https://bugs.openjdk.java.net/browse/JDK-8179327) Thanks, David > Kind Regards, Thomas > > > > On Tue, Oct 17, 2017 at 9:22 AM, David Holmes > wrote: > > CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 > > > Could I please have a reviewer for this CSR request so I can > fast-track it. > > Comments on the proposal are of course welcome. > > Thanks, > David > > From robbin.ehn at oracle.com Tue Oct 17 09:15:02 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Tue, 17 Oct 2017 11:15:02 +0200 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: Message-ID: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> Hi, On 10/17/2017 10:04 AM, Thomas St?fe wrote: > Hi David, > > Just some notes: > > - I am confused about the "primordial thread" naming; I always understood > the primordial thread to be the thread running main(), not the thread > invoking JNI_CreateJavaVM. I always used this term in this way for AIX > related problems. How about "VM-Initializing thread"? That is how I also have interpret it. > > - Would this not also be a useful feature for other threads attached to the > VM (AttachCurrentThread), as those threads may use a different SO-handling > scheme too ? This is much bigger problem imho. In a product I work with we had several different stack-sizes on attached threads. E.g. a 3rd party library might have created a thread which you want to attach for a jni call. > > Other than that, I think this is a useful addition. Would have helped me > before on AIX (see https://bugs.openjdk.java.net/browse/JDK-8179327) Since we had similar problems, we used the advice do not call CreateJavaVM with a thread you care about. Just spawn a new thread and call CreateJavaVM, not sure if that is worth having an option for? I much rather have an option that say do not try to figure out stacksize, because you can't. Alternative you should have the option of attaching individual threads with different stacksizes. It's also very confusing when you are using a 3rd party library which require large stacks. A normal unix user would just set ulimit -s 32768, but you would also need to set Xss to 32M. A not set Xss should be the same value as pthread create uses. So my opinion is that R can fix this with a new thread for CreateJavaVM (temporarily) and we should spend some time figuring all this out before adding a new option? Thanks Robbin > > Kind Regards, Thomas > > > > On Tue, Oct 17, 2017 at 9:22 AM, David Holmes > wrote: > >> CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >> >> Could I please have a reviewer for this CSR request so I can fast-track it. >> >> Comments on the proposal are of course welcome. >> >> Thanks, >> David >> From david.holmes at oracle.com Tue Oct 17 09:22:16 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 19:22:16 +1000 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> Message-ID: <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> Hi Robbin, On 17/10/2017 7:15 PM, Robbin Ehn wrote: > Hi, > > On 10/17/2017 10:04 AM, Thomas St?fe wrote: >> Hi David, >> >> Just some notes: >> >> - I am confused about the "primordial thread" naming; I always understood >> the primordial thread to be the thread running main(), not the thread >> invoking JNI_CreateJavaVM. I always used this term in this way for AIX >> related problems. How about "VM-Initializing thread"? > > That is how I also have interpret it. See my response to Thomas. >> >> - Would this not also be a useful feature for other threads attached >> to the >> VM (AttachCurrentThread), as those threads may use a different >> SO-handling >> scheme too ? > > This is much bigger problem imho. In a product I work with we had > several different stack-sizes on attached threads. > E.g. a 3rd party library might have created a thread which you want to > attach for a jni call. > >> >> Other than that, I think this is a useful addition. Would have helped me >> before on AIX (see https://bugs.openjdk.java.net/browse/JDK-8179327) > > Since we had similar problems, we used the advice do not call > CreateJavaVM with a thread you care about. > Just spawn a new thread and call CreateJavaVM, not sure if that is worth > having an option for? > > I much rather have an option that say do not try to figure out > stacksize, because you can't. > Alternative you should have the option of attaching individual threads > with different stacksizes. > > It's also very confusing when you are using a 3rd party library which > require large stacks. > A normal unix user would just set ulimit -s 32768, but you would also > need to set Xss to 32M. > A not set Xss should be the same value as pthread create uses. > > So my opinion is that R can fix this with a new thread for CreateJavaVM > (temporarily) and we should spend some time figuring all this out before > adding a new option? My understanding from prior discussion (ie when we fixed the 2MB stack limit) is that it doesn't work for them to use a new thread for the JVM. The change to support anything but unlimited stack size also helped (and the 8M limit when 'unlimited' is tolerable) but really they just want a simple way to say "please don't bother with stack guards, I can deal with that myself". This is intended to be a very simple, very specific proposal. I suppose it could be flagged as "experimental" if we wanted the flexibility to do something different later. But I don't see that being on the cards. Cheers, David > > Thanks Robbin > >> >> Kind Regards, Thomas >> >> >> >> On Tue, Oct 17, 2017 at 9:22 AM, David Holmes >> wrote: >> >>> CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >>> >>> Could I please have a reviewer for this CSR request so I can >>> fast-track it. >>> >>> Comments on the proposal are of course welcome. >>> >>> Thanks, >>> David >>> From robbin.ehn at oracle.com Tue Oct 17 09:28:49 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Tue, 17 Oct 2017 11:28:49 +0200 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> Message-ID: <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> > My understanding from prior discussion (ie when we fixed the 2MB stack limit) is that it doesn't work for them to use a new thread for the JVM. The change to support > anything but unlimited stack size also helped (and the 8M limit when 'unlimited' is tolerable) but really they just want a simple way to say "please don't bother with stack > guards, I can deal with that myself". Is there a problem I'm not seeing to extends this to all attaching threads? E.g. -XX:+DisableAttachingThreadGuardPages (more useful option I would say) Thanks, Robbin > > This is intended to be a very simple, very specific proposal. I suppose it could be flagged as "experimental" if we wanted the flexibility to do something different later. > But I don't see that being on the cards. > > Cheers, > David > >> >> Thanks Robbin >> >>> >>> Kind Regards, Thomas >>> >>> >>> >>> On Tue, Oct 17, 2017 at 9:22 AM, David Holmes >>> wrote: >>> >>>> CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >>>> >>>> Could I please have a reviewer for this CSR request so I can fast-track it. >>>> >>>> Comments on the proposal are of course welcome. >>>> >>>> Thanks, >>>> David >>>> From thomas.stuefe at gmail.com Tue Oct 17 09:46:35 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 17 Oct 2017 11:46:35 +0200 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> Message-ID: On Tue, Oct 17, 2017 at 11:28 AM, Robbin Ehn wrote: > My understanding from prior discussion (ie when we fixed the 2MB stack >> limit) is that it doesn't work for them to use a new thread for the JVM. >> The change to support anything but unlimited stack size also helped (and >> the 8M limit when 'unlimited' is tolerable) but really they just want a >> simple way to say "please don't bother with stack guards, I can deal with >> that myself". >> > > Is there a problem I'm not seeing to extends this to all attaching threads? > E.g. -XX:+DisableAttachingThreadGuardPages (more useful option I would > say) > > +1 for that. This would be easy to understand and more useful. I might want to skip java stack guard creation for reasons beyond issues with the primordial thread. I also like the idea to specify this on a per-thread base when calling AttachCurrentThread(), though I guess this is more complicated due to API changes. Thanks, Thomas > Thanks, Robbin > > > >> This is intended to be a very simple, very specific proposal. I suppose >> it could be flagged as "experimental" if we wanted the flexibility to do >> something different later. But I don't see that being on the cards. >> >> Cheers, >> David >> >> >>> Thanks Robbin >>> >>> >>>> Kind Regards, Thomas >>>> >>>> >>>> >>>> On Tue, Oct 17, 2017 at 9:22 AM, David Holmes >>>> wrote: >>>> >>>> CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 >>>>> >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >>>>> >>>>> Could I please have a reviewer for this CSR request so I can >>>>> fast-track it. >>>>> >>>>> Comments on the proposal are of course welcome. >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> From david.holmes at oracle.com Tue Oct 17 10:29:52 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 20:29:52 +1000 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> Message-ID: <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> On 17/10/2017 7:28 PM, Robbin Ehn wrote: >> My understanding from prior discussion (ie when we fixed the 2MB stack >> limit) is that it doesn't work for them to use a new thread for the >> JVM. The change to support anything but unlimited stack size also >> helped (and the 8M limit when 'unlimited' is tolerable) but really >> they just want a simple way to say "please don't bother with stack >> guards, I can deal with that myself". > > Is there a problem I'm not seeing to extends this to all attaching threads? > E.g. -XX:+DisableAttachingThreadGuardPages (more useful option I would say) Yes - that allows it to impact all of our launchers as well - not something I want to allow for. It opens the door for a myriad of bug reports if people use this flag without understanding it's consequences. The aim here is to solve a specific problem, not introduce new ways to break things. David > Thanks, Robbin > >> >> This is intended to be a very simple, very specific proposal. I >> suppose it could be flagged as "experimental" if we wanted the >> flexibility to do something different later. But I don't see that >> being on the cards. >> >> Cheers, >> David >> >>> >>> Thanks Robbin >>> >>>> >>>> Kind Regards, Thomas >>>> >>>> >>>> >>>> On Tue, Oct 17, 2017 at 9:22 AM, David Holmes >>>> wrote: >>>> >>>>> CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 >>>>> >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >>>>> >>>>> Could I please have a reviewer for this CSR request so I can >>>>> fast-track it. >>>>> >>>>> Comments on the proposal are of course welcome. >>>>> >>>>> Thanks, >>>>> David >>>>> From aph at redhat.com Tue Oct 17 12:14:55 2017 From: aph at redhat.com (Andrew Haley) Date: Tue, 17 Oct 2017 13:14:55 +0100 Subject: RFR(S): AArch64: NMT detail stack trace cleanup In-Reply-To: <81369691-46f1-a0f0-c3f8-8bc17ae94e7d@bell-sw.com> References: <81369691-46f1-a0f0-c3f8-8bc17ae94e7d@bell-sw.com> Message-ID: <87196637-2b40-8c33-bd1f-f0990e1a6884@redhat.com> On 17/10/17 09:31, dmitry.samersov wrote: > Please, review a minimal, aarch64 only patch that makes aarch64 behavior > similar to x86 one. It'll want a proper bug id; and please rename the local variable to fp. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From robbin.ehn at oracle.com Tue Oct 17 12:41:41 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Tue, 17 Oct 2017 14:41:41 +0200 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> Message-ID: <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> On 10/17/2017 12:29 PM, David Holmes wrote: > On 17/10/2017 7:28 PM, Robbin Ehn wrote: >>> My understanding from prior discussion (ie when we fixed the 2MB >>> stack limit) is that it doesn't work for them to use a new thread for >>> the JVM. The change to support anything but unlimited stack size also >>> helped (and the 8M limit when 'unlimited' is tolerable) but really >>> they just want a simple way to say "please don't bother with stack >>> guards, I can deal with that myself". >> >> Is there a problem I'm not seeing to extends this to all attaching >> threads? >> E.g. -XX:+DisableAttachingThreadGuardPages (more useful option I would >> say) > > Yes - that allows it to impact all of our launchers as well - not > something I want to allow for. It opens the door for a myriad of bug > reports if people use this flag without understanding it's consequences. > > The aim here is to solve a specific problem, not introduce new ways to > break things. First, since this is broken, you would fix the issue with different stacksizes. And secondly, I think the solution to that is to move all 'text' options to the launcher and have the CreateVM taking a data structure with pre-parsed options. And let the launcher choose which options to expose to the end-user. Meaning our own java launcher would never expose this option. (but this is much bigger discussion) Would the option would skip adding guard to any other thread calling CreateVM? If not, have the option really the correct name? E.g. -XX:+DisableCreateVMThreadGuardPages ? Thanks, Robbin > > David > >> Thanks, Robbin >> >>> >>> This is intended to be a very simple, very specific proposal. I >>> suppose it could be flagged as "experimental" if we wanted the >>> flexibility to do something different later. But I don't see that >>> being on the cards. >>> >>> Cheers, >>> David >>> >>>> >>>> Thanks Robbin >>>> >>>>> >>>>> Kind Regards, Thomas >>>>> >>>>> >>>>> >>>>> On Tue, Oct 17, 2017 at 9:22 AM, David Holmes >>>>> >>>>> wrote: >>>>> >>>>>> CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 >>>>>> >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >>>>>> >>>>>> Could I please have a reviewer for this CSR request so I can >>>>>> fast-track it. >>>>>> >>>>>> Comments on the proposal are of course welcome. >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> From david.holmes at oracle.com Tue Oct 17 13:00:17 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 17 Oct 2017 23:00:17 +1000 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> Message-ID: On 17/10/2017 10:41 PM, Robbin Ehn wrote: > On 10/17/2017 12:29 PM, David Holmes wrote: >> On 17/10/2017 7:28 PM, Robbin Ehn wrote: >>>> My understanding from prior discussion (ie when we fixed the 2MB >>>> stack limit) is that it doesn't work for them to use a new thread >>>> for the JVM. The change to support anything but unlimited stack size >>>> also helped (and the 8M limit when 'unlimited' is tolerable) but >>>> really they just want a simple way to say "please don't bother with >>>> stack guards, I can deal with that myself". >>> >>> Is there a problem I'm not seeing to extends this to all attaching >>> threads? >>> E.g. -XX:+DisableAttachingThreadGuardPages (more useful option I >>> would say) >> >> Yes - that allows it to impact all of our launchers as well - not >> something I want to allow for. It opens the door for a myriad of bug >> reports if people use this flag without understanding it's consequences. >> >> The aim here is to solve a specific problem, not introduce new ways to >> break things. > > First, since this is broken, you would fix the issue with different > stacksizes. Not sure what you consider "broken". The current approach causes a problem for runtimes that want to do their own stack guards. The proposal is to allow them by giving a flag to turn ours off - but only for the primordial thread, as that is where the problem lies. > And secondly, I think the solution to that is to move all 'text' options > to the launcher and have the CreateVM taking a data structure with > pre-parsed options. And let the launcher choose which options to expose > to the end-user. Meaning our own java launcher would never expose this > option. (but this is much bigger discussion) Let's not start to envisage a complete re-design of how you might communicate arguments between a host process and the VM. Please. > Would the option would skip adding guard to any other thread calling > CreateVM? > If not, have the option really the correct name? > > E.g. -XX:+DisableCreateVMThreadGuardPages ? The option would ONLY skip for the PRIMORDIAL thread of the process when it loads the VM - hence it has exactly the right name. I specifically do not want to allow this for an arbitrary thread that happens to do the loading of the VM. Thanks, David > Thanks, Robbin > >> >> David >> >>> Thanks, Robbin >>> >>>> >>>> This is intended to be a very simple, very specific proposal. I >>>> suppose it could be flagged as "experimental" if we wanted the >>>> flexibility to do something different later. But I don't see that >>>> being on the cards. >>>> >>>> Cheers, >>>> David >>>> >>>>> >>>>> Thanks Robbin >>>>> >>>>>> >>>>>> Kind Regards, Thomas >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Oct 17, 2017 at 9:22 AM, David Holmes >>>>>> >>>>>> wrote: >>>>>> >>>>>>> CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 >>>>>>> >>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >>>>>>> >>>>>>> Could I please have a reviewer for this CSR request so I can >>>>>>> fast-track it. >>>>>>> >>>>>>> Comments on the proposal are of course welcome. >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> From harold.seigel at oracle.com Tue Oct 17 13:19:16 2017 From: harold.seigel at oracle.com (harold seigel) Date: Tue, 17 Oct 2017 09:19:16 -0400 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: <69f62651-dac3-3426-bb4f-6d4dcb03abb8@oracle.com> Message-ID: <81a66147-6a69-f33b-859e-f1626fa2e0ba@oracle.com> This looks good. Thanks, Harold On 10/12/2017 1:16 AM, Yasumasa Suenaga wrote: > Thanks David! > > I've fixed: > > http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.08/ > > > Yasumasa > > > 2017-10-12 13:55 GMT+09:00 David Holmes : >> On 12/10/2017 2:47 PM, Yasumasa Suenaga wrote: >>> Hi David, >>> >>>> You may have time to fix these in place before anyone else sees the >>>> webrev >>> >>> I've fixed that: >>> >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.07/ >>> >>> >>> I checked raw text of your email, so I believe the indent is correct. >>> If I have mistake(s), please tell me. >> >> Sorry they are off. Basic rules: >> >> var = >> some long value; >> >> indent by 4. >> >> foo(arg1, arg2, >> arg3, arg4); >> >> align the args on the two lines as shown. >> >> Hope that is clearer. >> >> Thanks, >> David >> >> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> 2017-10-12 13:22 GMT+09:00 David Holmes : >>>> Hi Yasumasa, >>>> >>>> On 12/10/2017 2:10 PM, Yasumasa Suenaga wrote: >>>>> >>>>> Hi all, >>>>> >>>>> This change was tried to push via JPRT, but it failed because >>>>> test/hotspot/jtreg/runtime/SharedArchiveFile/MaxMetaspaceSizeTest.java >>>>> was failed. >>>>> Also I heard the comment that the changes in arguments.cpp which set >>>>> ergonomic value to CompressedClassSpaceSize should be moved to >>>>> Metaspace::ergo_initialize(). >>>>> >>>>> I uploaded new webrev. Could you review again? >>>>> This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and >>>>> test/hotspot/jtreg/runtime/SharedArchiveFile. >>>>> >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ >>>> >>>> >>>> I'll let Coleen cover the technical details, I'll just pick up a few >>>> style >>>> nits: >>>> >>>> 3328 /* Initial virtual space size will be calculated at >>>> global_initialize() */ >>>> >>>> Use // comment >>>> >>>> 3329 size_t min_metaspace_sz = >>>> 3330 VIRTUALSPACEMULTIPLIER * >>>> InitialBootClassLoaderMetaspaceSize; >>>> >>>> ^ should indent only to here >>>> >>>> 3336 FLAG_SET_ERGO(size_t, CompressedClassSpaceSize, >>>> 3337 MaxMetaspaceSize - >>>> min_metaspace_sz); >>>> ^ should indent only to here >>>> >>>> 3341 FLAG_SET_ERGO(size_t, InitialBootClassLoaderMetaspaceSize, >>>> 3342 min_metaspace_sz); >>>> >>>> ^ should indent only to here >>>> >>>> You may have time to fix these in place before anyone else sees the >>>> webrev >>>> :) >>>> >>>> Thanks, >>>> David >>>> ----- >>>> >>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> >>>>> 2017-10-10 21:02 GMT+09:00 : >>>>>> >>>>>> Hi Yasumasa, I like this better. I will sponsor it. I think it only >>>>>> needs >>>>>> one reviewer, unless there were more? Please send me the result of hg >>>>>> export. >>>>>> thanks, >>>>>> Coleen >>>>>> >>>>>> >>>>>> On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: >>>>>>> >>>>>>> >>>>>>> Hi Coleen, >>>>>>> >>>>>>>>> I will sponsor this for you. >>>>>>> >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>>> check_metaspace_sizes() >>>>>>>>> so that you don't have to tell arguments what the MediumChunk size >>>>>>>>> is. >>>>>>> >>>>>>> >>>>>>> I added new function calculate_min_metaspace_size() in metaspace.cpp >>>>>>> to get minimum metaspace size, and uses return value from this >>>>>>> function in metaspace initialization and metaspace size check. >>>>>>> >>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ >>>>>>> >>>>>>> Could you review again? >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Yasumasa >>>>>>> >>>>>>> >>>>>>> 2017-10-10 6:07 GMT+09:00 Man Cao : >>>>>>>> >>>>>>>> >>>>>>>> Thanks for you both updating and sponsoring the webrev! >>>>>>>> We plan to back-port it to our JDK9 once it is submitted. >>>>>>>> >>>>>>>> -Man >>>>>>>> >>>>>>>> On Mon, Oct 9, 2017 at 6:15 AM, wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>>> check_metaspace_sizes() >>>>>>>>> so that you don't have to tell arguments what the MediumChunk size >>>>>>>>> is. >>>>>>>>> >>>>>>>>> I will sponsor this for you. >>>>>>>>> >>>>>>>>> thanks, >>>>>>>>> Coleen >>>>>>>>> >>>>>>>>> >>>>>>>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I added a testcase for this issue in new webrev: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Yasumasa >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I uploaded webrev for this issue against jdk10/hs. >>>>>>>>>>> Could you review it? >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I cannot access JPRT. So I need a sponsor. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Yasumasa >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thank Yasumasa and Stefan for the responses. >>>>>>>>>>>> >>>>>>>>>>>> Good to know that the patch is not blocked due to breaking some >>>>>>>>>>>> internal >>>>>>>>>>>> invariants/assumptions, but just due to its P5 status. >>>>>>>>>>>> Is it possible to push it to P4? >>>>>>>>>>>> >>>>>>>>>>>> -Man >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>>>>>>>> >>>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>>> class >>>>>>>>>>>>>> space >>>>>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>>>>> this >>>>>>>>>>>>>> on the >>>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I will send review request against jdk10/master or jdk10/hs >>>>>>>>>>>>> after >>>>>>>>>>>>> repos >>>>>>>>>>>>> are opened. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Man, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you have any thoughts on why this patch has been pending >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> 2+ >>>>>>>>>>>>>>> years? This patch could really save us from some annoying >>>>>>>>>>>>>>> issues >>>>>>>>>>>>>>> since we >>>>>>>>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>>> class >>>>>>>>>>>>>> space >>>>>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>>>>> this >>>>>>>>>>>>>> on the >>>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>>> >>>>>>>>>>>>>> StefanK >>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Man >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I wonder if there is any recent update on the patch >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> JDK-8087291. >>>>>>>>>>>>>>> Is it possible to push this patch into JDK9? Except >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> its >>>>>>>>>>>>>>> low >>>>>>>>>>>>>>> priority (P5), >>>>>>>>>>>>>>> is there any complication that prevents this patch >>>>>>>>>>>>>>> getting >>>>>>>>>>>>>>> approved >>>>>>>>>>>>>>> (for example, some JVM logic requires >>>>>>>>>>>>>>> CompressedClassSpaceSize >>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>> 1GB by default)? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I work in the Java Platform Team at Google. We have >>>>>>>>>>>>>>> encountered >>>>>>>>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>>>>>>>>>> confusing 1GB >>>>>>>>>>>>>>> memory reserved by metaspace, >>>>>>>>>>>>>>> regardless of MaxMetaspaceSize value. The root cause >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> these >>>>>>>>>>>>>>> issues is that CompressedClassSpaceSize is not >>>>>>>>>>>>>>> automatically >>>>>>>>>>>>>>> capped >>>>>>>>>>>>>>> by MaxMetaspaceSize >>>>>>>>>>>>>>> during VM initialization, and this patch seems fix the >>>>>>>>>>>>>>> root >>>>>>>>>>>>>>> cause. >>>>>>>>>>>>>>> (I'm aware that even after this patch, the reserved >>>>>>>>>>>>>>> size >>>>>>>>>>>>>>> could >>>>>>>>>>>>>>> still >>>>>>>>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>>>>>>>> but it is better than the current situation.) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Man >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you for your comment! >>>>>>>>>>>>>>> > Try running a debug JVM with your patch with >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> command >>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It works on fastdebug VM. >>>>>>>>>>>>>>> Please review again. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>>>>>>>> > Yasumasa, >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Try running a debug JVM with your patch with >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> command >>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > On a linux system I get this when I build with >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>> patch. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>> >> # To suppress the following error report, >>>>>>>>>>>>>>> specify >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> argument >>>>>>>>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>> >> # A fatal error has been detected by the Java >>>>>>>>>>>>>>> Runtime >>>>>>>>>>>>>>> Environment: >>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>> >> # Internal Error >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>>>>>>>> ClassMediumChunk) >>>>>>>>>>>>>>> failed: Not a >>>>>>>>>>>>>>> >> humongous chunk >>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Jon >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>>>>>>>>>> >> I want to continue to discuss about >>>>>>>>>>>>>>> CompressedClassSpace and >>>>>>>>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>> Should I resize CompressedClassSpaceSize >>>>>>>>>>>>>>> than >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> show >>>>>>>>>>>>>>> error message? >>>>>>>>>>>>>>> >>> If you add slightly better heuristics for the >>>>>>>>>>>>>>> setup >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> CompressedClassSpaceSize flag, for example >>>>>>>>>>>>>>> lowering >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is >>>>>>>>>>>>>>> set, >>>>>>>>>>>>>>> then >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>> might be less likely that you'll hit the >>>>>>>>>>>>>>> OutOfMemoryError >>>>>>>>>>>>>>> when >>>>>>>>>>>>>>> the system is set up with strict overcommit >>>>>>>>>>>>>>> settings. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> I've uploaded new webrev: >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is >>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>> than >>>>>>>>>>>>>>> MaxMetaspaceSize, >>>>>>>>>>>>>>> >> VM will fail with error message. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will be >>>>>>>>>>>>>>> set >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> MaxMetaspaceSize >>>>>>>>>>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>>>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Yasumasa >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From rkennke at redhat.com Tue Oct 17 13:24:04 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 17 Oct 2017 15:24:04 +0200 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs Message-ID: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> The SuspendibleThreadSet API for synchronizing any non-Java thread with safepoints currently resides under gc/g1. It is very useful for other GCs too (in particular, Shenandoah does use it too), so I wanted to move it to a common location like gc/common. Then Kim Barrett commented that it might actually be useful for other threads outside GC land and to put it under runtime/. So I did: http://cr.openjdk.java.net/~rkennke/8189276/webrev.00/ I also added a generic hook to call the STS from safepoint sync/desync, which is consequently used by G1 now. In other words, the CollectedHeap API that Erik ? introduced is no longer used by G1. Only CMS still uses that API because it has its own way to sync with safepoints. I filed another bug for this. Although I have my doubt it will ever be fixed. This seems to have been very carefully evolved (to put it positive), and the risk of breaking it is relatively high, and thus doesn't seem worth the struggle to make it fit the STS. Issue: https://bugs.openjdk.java.net/browse/JDK-8189276 What do you think? Ok to go in? Roman From thomas.stuefe at gmail.com Tue Oct 17 15:11:16 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 17 Oct 2017 17:11:16 +0200 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> Message-ID: Hi David, I understand that you want to have a quick solution to a specific problem. I do not like adding a switch with such a narrow usage and would prefer, like Robin, a more well rounded and more general solution. That said, I do not see any specific problems with the proposed switch. Not even sure if that matters, as I am not even sure I can review CSRs :) Kind Regards, Thomas On Tue, Oct 17, 2017 at 3:00 PM, David Holmes wrote: > On 17/10/2017 10:41 PM, Robbin Ehn wrote: > >> On 10/17/2017 12:29 PM, David Holmes wrote: >> >>> On 17/10/2017 7:28 PM, Robbin Ehn wrote: >>> >>>> My understanding from prior discussion (ie when we fixed the 2MB stack >>>>> limit) is that it doesn't work for them to use a new thread for the JVM. >>>>> The change to support anything but unlimited stack size also helped (and >>>>> the 8M limit when 'unlimited' is tolerable) but really they just want a >>>>> simple way to say "please don't bother with stack guards, I can deal with >>>>> that myself". >>>>> >>>> >>>> Is there a problem I'm not seeing to extends this to all attaching >>>> threads? >>>> E.g. -XX:+DisableAttachingThreadGuardPages (more useful option I would >>>> say) >>>> >>> >>> Yes - that allows it to impact all of our launchers as well - not >>> something I want to allow for. It opens the door for a myriad of bug >>> reports if people use this flag without understanding it's consequences. >>> >>> The aim here is to solve a specific problem, not introduce new ways to >>> break things. >>> >> >> First, since this is broken, you would fix the issue with different >> stacksizes. >> > > Not sure what you consider "broken". The current approach causes a problem > for runtimes that want to do their own stack guards. The proposal is to > allow them by giving a flag to turn ours off - but only for the primordial > thread, as that is where the problem lies. > > And secondly, I think the solution to that is to move all 'text' options >> to the launcher and have the CreateVM taking a data structure with >> pre-parsed options. And let the launcher choose which options to expose to >> the end-user. Meaning our own java launcher would never expose this option. >> (but this is much bigger discussion) >> > > Let's not start to envisage a complete re-design of how you might > communicate arguments between a host process and the VM. Please. > > Would the option would skip adding guard to any other thread calling >> CreateVM? >> If not, have the option really the correct name? >> >> E.g. -XX:+DisableCreateVMThreadGuardPages ? >> > > The option would ONLY skip for the PRIMORDIAL thread of the process when > it loads the VM - hence it has exactly the right name. I specifically do > not want to allow this for an arbitrary thread that happens to do the > loading of the VM. > > Thanks, > David > > > Thanks, Robbin >> >> >>> David >>> >>> Thanks, Robbin >>>> >>>> >>>>> This is intended to be a very simple, very specific proposal. I >>>>> suppose it could be flagged as "experimental" if we wanted the flexibility >>>>> to do something different later. But I don't see that being on the cards. >>>>> >>>>> Cheers, >>>>> David >>>>> >>>>> >>>>>> Thanks Robbin >>>>>> >>>>>> >>>>>>> Kind Regards, Thomas >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 17, 2017 at 9:22 AM, David Holmes < >>>>>>> david.holmes at oracle.com> >>>>>>> wrote: >>>>>>> >>>>>>> CSR: https://bugs.openjdk.java.net/browse/JDK-8189423 >>>>>>>> >>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8189170 >>>>>>>> >>>>>>>> Could I please have a reviewer for this CSR request so I can >>>>>>>> fast-track it. >>>>>>>> >>>>>>>> Comments on the proposal are of course welcome. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> David >>>>>>>> >>>>>>>> From calvin.cheung at oracle.com Tue Oct 17 16:42:39 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 17 Oct 2017 09:42:39 -0700 Subject: RFR(M) JDK-8188791 Move AppCDS implementation from closed repo to open repo In-Reply-To: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> Message-ID: <59E632FF.7040001@oracle.com> Hi Ioi, The AppCDS source code relocation looks good to me. thanks, Calvin On 10/12/17, 4:48 PM, Ioi Lam wrote: > Hi, > > Please review this change set. > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ > https://bugs.openjdk.java.net/browse/JDK-8188791 > > This is the first step of implementing the following JEP, which moves > AppCDS from > closed repos into the openjdk repo: > > https://bugs.openjdk.java.net/browse/JDK-8185996 > > In JDK 9, significant portion of AppCDS code resided in the closed > repo. As part > of the open-sourcing effort of JDK 18.3, we will move the source code > into the > open repo. > > In this changeset, the code is moved verbatim as much as possible. The > intention is > only to relocate the sources, not to changing existing behaviors, and not > to do any sort of refactoring. > > Most of the "diffs" shown in this webrev are the result of copying the > closed source > files on top of files of the same name in the open repo. So in > reviewing, instead of > focusing on what's "changed", it's better to focus on the entire > content of the new > version of each file. > > The only functional change in this task is that the UseAppCDS flag is > changed from > a "commercial" flag to a regular "product" flag. This is because > "commercial" > flags are not supported by the OpenJDK build. > > Source code refactoring may be desirable, because the old open/closed > source > code structure had introduced some intermediary APIs to connect code > between > the two repos. Such API should be removed in a separate RFE. > > Also, some AppCDS tests are currently in the closed repo. These tests > will be > moved in a separate task. See JDK-8188792 for details. > > All the AppCDS tests (currently still in closed sources) passed with > both Oracle JDK > and OpenJDK. > > Thanks > - Ioi From robbin.ehn at oracle.com Tue Oct 17 17:22:50 2017 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Tue, 17 Oct 2017 19:22:50 +0200 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> Message-ID: Hi David, On 2017-10-17 17:11, Thomas St?fe wrote: > Hi David, > > I understand that you want to have a quick solution to a specific problem. I do not like adding a switch with such a narrow usage and would prefer, like Robin, a more well rounded and more general solution. > > That said, I do not see any specific problems with the proposed switch. Not even sure if that matters, as I am not even sure I can review CSRs :) +1 > > Not sure what you consider "broken". The current approach causes a problem for runtimes that want to do their own stack guards. The proposal is to allow them by giving a flag to turn ours off - but only for the primordial thread, as that is where the problem lies. > The issue I had in 1.6 seems to work fine now. I would still prefer the generic one, but I see your point in not wanting to expose that. > > The option would ONLY skip for the PRIMORDIAL thread of the process when it loads the VM - hence it has exactly the right name. I specifically do not want to allow this for an arbitrary thread that happens to do the loading of the VM. > Yes your option have exactly the right name :) I didn't know that we had cross platform reliable ways of knowing if a thread is the primordial one, but it looks like we have something like that. Thanks, Robbin From coleen.phillimore at oracle.com Tue Oct 17 18:29:03 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 17 Oct 2017 14:29:03 -0400 Subject: RFR (S): 8185580: Refactor Threads::possibly_parallel_oops_do() to use Threads::parallel_java_threads_do() In-Reply-To: <1e638965-2df0-f495-95da-0e13a0984359@redhat.com> References: <04615da8-68f1-cbda-a7a6-628dabd82e97@oracle.com> <1e638965-2df0-f495-95da-0e13a0984359@redhat.com> Message-ID: Yes, I prefer the new possibly_parallel_threads_do name. I'll push it. Thanks, Coleen On 10/17/17 12:55 AM, Roman Kennke wrote: > Am 17.10.2017 um 05:33 schrieb David Holmes: >> Hi Roman, >> >> On 12/10/2017 5:30 AM, Roman Kennke wrote: >>> This is a follow-up to my earlier safepoint parallel cleanup work. >>> >>> Currently we have 2 places where we apply the parallel claiming >>> protocol when iterating threads, that is >>> Threads::parallel_java_threads_do() and >>> Threads::possibly_parallel_oops_do(). In order to avoid code >>> duplication, the latter should call the former, using a private >>> ThreadClosure. We already had one bug (JDK-8185273) that was caused >>> by an inconsistency between the two. >>> >>> The only other user of parallel_java_threads_do() already filters >>> out any non-Java-threads, which means that doing the extra >>> processing of the VMThread should be ok. >> >> "should" be okay? It's not at all clear to me what processing the >> VMThread where previously it was not processed, will actually do ?? > It ensures that both parallel thread iteration routines are doing the > same, and thus claim the same threads, which is later > assert_all_threads_are_claimed(). If we don't do this consistently, we > throw off the thread claiming mechanics. > >> >>> I renamed the method to parallel_threads_do() to reflect that it not >>> only does the Java threads. >> >> Given it takes an is_par argument it really should be >> possibly_parallel_threads_do() > Ok. Differential diff: > http://cr.openjdk.java.net/~rkennke/8185580/webrev.01.diff/ > > Full webrev: > http://cr.openjdk.java.net/~rkennke/8185580/webrev.01/ > > > Good? > > Roman > From coleen.phillimore at oracle.com Tue Oct 17 19:46:17 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 17 Oct 2017 15:46:17 -0400 Subject: RFR: JDK-8087291: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: References: <69f62651-dac3-3426-bb4f-6d4dcb03abb8@oracle.com> Message-ID: <79bcec31-20d1-1aef-b9a6-1134676aa289@oracle.com> Hi Yasumasa, This looks good.? I can sponsor this for you. thanks, Coleen On 10/12/17 1:16 AM, Yasumasa Suenaga wrote: > Thanks David! > > I've fixed: > > http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.08/ > > > Yasumasa > > > 2017-10-12 13:55 GMT+09:00 David Holmes : >> On 12/10/2017 2:47 PM, Yasumasa Suenaga wrote: >>> Hi David, >>> >>>> You may have time to fix these in place before anyone else sees the >>>> webrev >>> >>> I've fixed that: >>> >>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.07/ >>> >>> >>> I checked raw text of your email, so I believe the indent is correct. >>> If I have mistake(s), please tell me. >> >> Sorry they are off. Basic rules: >> >> var = >> some long value; >> >> indent by 4. >> >> foo(arg1, arg2, >> arg3, arg4); >> >> align the args on the two lines as shown. >> >> Hope that is clearer. >> >> Thanks, >> David >> >> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> 2017-10-12 13:22 GMT+09:00 David Holmes : >>>> Hi Yasumasa, >>>> >>>> On 12/10/2017 2:10 PM, Yasumasa Suenaga wrote: >>>>> >>>>> Hi all, >>>>> >>>>> This change was tried to push via JPRT, but it failed because >>>>> test/hotspot/jtreg/runtime/SharedArchiveFile/MaxMetaspaceSizeTest.java >>>>> was failed. >>>>> Also I heard the comment that the changes in arguments.cpp which set >>>>> ergonomic value to CompressedClassSpaceSize should be moved to >>>>> Metaspace::ergo_initialize(). >>>>> >>>>> I uploaded new webrev. Could you review again? >>>>> This webrev works fine on test/hotspot/jtreg/runtime/Metaspace and >>>>> test/hotspot/jtreg/runtime/SharedArchiveFile. >>>>> >>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.06/ >>>> >>>> >>>> I'll let Coleen cover the technical details, I'll just pick up a few >>>> style >>>> nits: >>>> >>>> 3328 /* Initial virtual space size will be calculated at >>>> global_initialize() */ >>>> >>>> Use // comment >>>> >>>> 3329 size_t min_metaspace_sz = >>>> 3330 VIRTUALSPACEMULTIPLIER * >>>> InitialBootClassLoaderMetaspaceSize; >>>> >>>> ^ should indent only to here >>>> >>>> 3336 FLAG_SET_ERGO(size_t, CompressedClassSpaceSize, >>>> 3337 MaxMetaspaceSize - >>>> min_metaspace_sz); >>>> ^ should indent only to here >>>> >>>> 3341 FLAG_SET_ERGO(size_t, InitialBootClassLoaderMetaspaceSize, >>>> 3342 min_metaspace_sz); >>>> >>>> ^ should indent only to here >>>> >>>> You may have time to fix these in place before anyone else sees the >>>> webrev >>>> :) >>>> >>>> Thanks, >>>> David >>>> ----- >>>> >>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> >>>>> 2017-10-10 21:02 GMT+09:00 : >>>>>> >>>>>> Hi Yasumasa, I like this better. I will sponsor it. I think it only >>>>>> needs >>>>>> one reviewer, unless there were more? Please send me the result of hg >>>>>> export. >>>>>> thanks, >>>>>> Coleen >>>>>> >>>>>> >>>>>> On 10/9/17 11:09 PM, Yasumasa Suenaga wrote: >>>>>>> >>>>>>> >>>>>>> Hi Coleen, >>>>>>> >>>>>>>>> I will sponsor this for you. >>>>>>> >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>>> check_metaspace_sizes() >>>>>>>>> so that you don't have to tell arguments what the MediumChunk size >>>>>>>>> is. >>>>>>> >>>>>>> >>>>>>> I added new function calculate_min_metaspace_size() in metaspace.cpp >>>>>>> to get minimum metaspace size, and uses return value from this >>>>>>> function in metaspace initialization and metaspace size check. >>>>>>> >>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.05/ >>>>>>> >>>>>>> Could you review again? >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Yasumasa >>>>>>> >>>>>>> >>>>>>> 2017-10-10 6:07 GMT+09:00 Man Cao : >>>>>>>> >>>>>>>> >>>>>>>> Thanks for you both updating and sponsoring the webrev! >>>>>>>> We plan to back-port it to our JDK9 once it is submitted. >>>>>>>> >>>>>>>> -Man >>>>>>>> >>>>>>>> On Mon, Oct 9, 2017 at 6:15 AM, wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi, this looks reasonable but I would prefer that the code in >>>>>>>>> arguments.cpp call something in metaspace.cpp, like >>>>>>>>> check_metaspace_sizes() >>>>>>>>> so that you don't have to tell arguments what the MediumChunk size >>>>>>>>> is. >>>>>>>>> >>>>>>>>> I will sponsor this for you. >>>>>>>>> >>>>>>>>> thanks, >>>>>>>>> Coleen >>>>>>>>> >>>>>>>>> >>>>>>>>> On 9/26/17 11:05 PM, Yasumasa Suenaga wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I added a testcase for this issue in new webrev: >>>>>>>>>> >>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.04/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Yasumasa >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2017-09-26 18:36 GMT+09:00 Yasumasa Suenaga : >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I uploaded webrev for this issue against jdk10/hs. >>>>>>>>>>> Could you review it? >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.03/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I cannot access JPRT. So I need a sponsor. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Yasumasa >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2017-09-21 3:11 GMT+09:00 Man Cao : >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thank Yasumasa and Stefan for the responses. >>>>>>>>>>>> >>>>>>>>>>>> Good to know that the patch is not blocked due to breaking some >>>>>>>>>>>> internal >>>>>>>>>>>> invariants/assumptions, but just due to its P5 status. >>>>>>>>>>>> Is it possible to push it to P4? >>>>>>>>>>>> >>>>>>>>>>>> -Man >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:16 AM, Yasumasa Suenaga >>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> (CC'ed hotspot-runtime-dev) >>>>>>>>>>>>> >>>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>>> class >>>>>>>>>>>>>> space >>>>>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>>>>> this >>>>>>>>>>>>>> on the >>>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I will send review request against jdk10/master or jdk10/hs >>>>>>>>>>>>> after >>>>>>>>>>>>> repos >>>>>>>>>>>>> are opened. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 2017/09/20 20:53, Stefan Karlsson wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Man, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2017-09-13 20:55, Man Cao wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Yasumasa, Stefan, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you have any thoughts on why this patch has been pending >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> 2+ >>>>>>>>>>>>>>> years? This patch could really save us from some annoying >>>>>>>>>>>>>>> issues >>>>>>>>>>>>>>> since we >>>>>>>>>>>>>>> are automatically monitoring hsperfdata counters. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think the reason is that this bug is a P5. The compressed >>>>>>>>>>>>>> class >>>>>>>>>>>>>> space >>>>>>>>>>>>>> belongs to the runtime code, so you might get more traction for >>>>>>>>>>>>>> this >>>>>>>>>>>>>> on the >>>>>>>>>>>>>> hotspot-runtime-dev list. >>>>>>>>>>>>>> >>>>>>>>>>>>>> StefanK >>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Man >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 3:46 PM, Man Cao >>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I wonder if there is any recent update on the patch >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> JDK-8087291. >>>>>>>>>>>>>>> Is it possible to push this patch into JDK9? Except >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> its >>>>>>>>>>>>>>> low >>>>>>>>>>>>>>> priority (P5), >>>>>>>>>>>>>>> is there any complication that prevents this patch >>>>>>>>>>>>>>> getting >>>>>>>>>>>>>>> approved >>>>>>>>>>>>>>> (for example, some JVM logic requires >>>>>>>>>>>>>>> CompressedClassSpaceSize >>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>> 1GB by default)? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I work in the Java Platform Team at Google. We have >>>>>>>>>>>>>>> encountered >>>>>>>>>>>>>>> annoying issues that the hsperfdata counter >>>>>>>>>>>>>>> "sun_gc_metaspace_maxCapacity" reporting >>>>>>>>>>>>>>> a too large value (about 1GB) even if user sets >>>>>>>>>>>>>>> -XX:MaxMetaspaceSize=100m, as well as GC log shows the >>>>>>>>>>>>>>> confusing 1GB >>>>>>>>>>>>>>> memory reserved by metaspace, >>>>>>>>>>>>>>> regardless of MaxMetaspaceSize value. The root cause >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> these >>>>>>>>>>>>>>> issues is that CompressedClassSpaceSize is not >>>>>>>>>>>>>>> automatically >>>>>>>>>>>>>>> capped >>>>>>>>>>>>>>> by MaxMetaspaceSize >>>>>>>>>>>>>>> during VM initialization, and this patch seems fix the >>>>>>>>>>>>>>> root >>>>>>>>>>>>>>> cause. >>>>>>>>>>>>>>> (I'm aware that even after this patch, the reserved >>>>>>>>>>>>>>> size >>>>>>>>>>>>>>> could >>>>>>>>>>>>>>> still >>>>>>>>>>>>>>> be up to 2*MaxMetaspaceSize, >>>>>>>>>>>>>>> but it is better than the current situation.) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Man >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 6/19/2015 00:34, Yasumasa Suenaga wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you for your comment! >>>>>>>>>>>>>>> > Try running a debug JVM with your patch with >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> command >>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>> Sorry, I've fixed it and uploaded webrev: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.02/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It works on fastdebug VM. >>>>>>>>>>>>>>> Please review again. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Yasumasa >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2015/06/18 10:45, Jon Masamitsu wrote: >>>>>>>>>>>>>>> > Yasumasa, >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Try running a debug JVM with your patch with >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> command >>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > On a linux system I get this when I build with >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>> patch. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >> java -XX:MaxMetaspaceSize=4195328 -version >>>>>>>>>>>>>>> >> # To suppress the following error report, >>>>>>>>>>>>>>> specify >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> argument >>>>>>>>>>>>>>> >> # after -XX: or in .hotspotrc: >>>>>>>>>>>>>>> SuppressErrorAt=/metaspace.cpp:2324 >>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>> >> # A fatal error has been detected by the Java >>>>>>>>>>>>>>> Runtime >>>>>>>>>>>>>>> Environment: >>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>> >> # Internal Error >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (/export/jmasa/java/jdk9-gc-code_review/src/share/vm/memory/metaspace.cpp:2324), >>>>>>>>>>>>>>> >> pid=19099, tid=0x00007ff4b9b92700 >>>>>>>>>>>>>>> >> # assert(size > MediumChunk || size > >>>>>>>>>>>>>>> ClassMediumChunk) >>>>>>>>>>>>>>> failed: Not a >>>>>>>>>>>>>>> >> humongous chunk >>>>>>>>>>>>>>> >> # >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Jon >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > On 6/17/2015 7:54 AM, Yasumasa Suenaga wrote: >>>>>>>>>>>>>>> >> I want to continue to discuss about >>>>>>>>>>>>>>> CompressedClassSpace and >>>>>>>>>>>>>>> MaxMetaspace in this (RFR) thread. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-June/013873.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>> Should I resize CompressedClassSpaceSize >>>>>>>>>>>>>>> than >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> show >>>>>>>>>>>>>>> error message? >>>>>>>>>>>>>>> >>> If you add slightly better heuristics for the >>>>>>>>>>>>>>> setup >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> CompressedClassSpaceSize flag, for example >>>>>>>>>>>>>>> lowering >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> CompressedClassSpaceSize when MaxMetaspaceSize is >>>>>>>>>>>>>>> set, >>>>>>>>>>>>>>> then >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>> might be less likely that you'll hit the >>>>>>>>>>>>>>> OutOfMemoryError >>>>>>>>>>>>>>> when >>>>>>>>>>>>>>> the system is set up with strict overcommit >>>>>>>>>>>>>>> settings. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> I've uploaded new webrev: >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~ysuenaga/JDK-8087291/webrev.01/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> This patch checkes MaxMetaspaceSize, >>>>>>>>>>>>>>> CompressedClassSpaceSize, and >>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> I add to check CompressedClassSpaceSize in >>>>>>>>>>>>>>> Arguments::set_use_compressed_klass_ptrs(). >>>>>>>>>>>>>>> >> If InitialBootClassLoaderMetaspaceSize is >>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>> than >>>>>>>>>>>>>>> MaxMetaspaceSize, >>>>>>>>>>>>>>> >> VM will fail with error message. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> InitialBootClassLoaderMetaspaceSize will be >>>>>>>>>>>>>>> set >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> MaxMetaspaceSize >>>>>>>>>>>>>>> >> when UseCompressedClassPointers is not set in >>>>>>>>>>>>>>> Metaspace::ergo_initialize(). >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Yasumasa >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From rkennke at redhat.com Tue Oct 17 20:22:37 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 17 Oct 2017 22:22:37 +0200 Subject: RFR: 8184914: Use MacroAssembler::cmpoop() consistently when comparing heap objects Message-ID: <8d667010-f17e-7d1b-088b-106999e3b005@redhat.com> (Not sure if this is the correct list to ask.. if not, please let me know and/or redirect me) Currently, cmpoop() is only declared for 32-bit x86, and only used in 2 places in C1 to compare oops. In other places, oops are compared using cmpptr(). It would be useful to distinguish normal pointer comparisons from heap object comparisons, and use cmpoop() consistently for heap object comparisons. This would remove clutter in several places where we have #ifdef _LP64 around comparisons, and would also allow to insert necessary barriers for GCs that need them (e.g. Shenandoah) later. http://cr.openjdk.java.net/~rkennke/8184914/webrev.00/ Tested by running hotspot_gc jtreg tests. Can I get a review please? Thanks, Roman From zgu at redhat.com Tue Oct 17 20:23:51 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 17 Oct 2017 16:23:51 -0400 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: References: Message-ID: <3577bda2-7fdd-d458-99b1-79f0ebd72bc2@redhat.com> Hi Roman, Look good overall. 1) Probably make sense to factor out static methods from GC into separate GCFactory class. 2) Stylish: #ifndef SHARE_VM_GC_G1_G1_GC_HPP -> #ifndef SHARE_VM_GC_G1_G1GC_HPP Thanks, -Zhengyu On 10/12/2017 03:59 PM, Roman Kennke wrote: > I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev because it > touches both areas (i.e. the GC interface). > > Currently, all GC related argument processing is done in arguments.cpp, > littering it with #ifdef INCLUDE_ALL_GCS and all sorts of GC specific > methods etc. > > In order to have a cleaner GC interface, I propose to move all that code > into GC specific classes under their GC specific directories. > > Since at this point in time we haven't created the XXXHeap subclasses > yet (and cannot, before having set up all the flags and arguments), I > propose to create a GC interface of which we create an instance as soon > as we have selected a GC, and then do argument processing there. > > This 'GC' interface is, at this point, meant as a sort of GC factory > interface. With this patch, it will only do argument processing. Later I > plan to add heap creation to it (which is currently done in > universe.cpp, with INCLUDE_ALL_GCS stuff and all the if (UseBlahGC) .. > else if .. else branch chains. My current prototype in the jdk10 sandbox > also provides servicabilty support classes and some more stuff, but we > should decide later where to put this. > > I broke out some code paths into gc_factory.cpp (i.e. not in gc.cpp). > Those are all the code paths that select GC, create the right instances, > etc, in other words, the code paths that need to know about all GCs. > Ideally, this should be the only file that needs to know about all the > GCs. (Suggestions for improvement are welcome of course.) > > I tested this by running hotspot_gc jtreg tests, with normal fastdebug > and no-precompiled configurations. Everything passes fine. > > http://cr.openjdk.java.net/~rkennke/8189171/webrev.00/ > > > Opinions? > > Roman > From calvin.cheung at oracle.com Tue Oct 17 20:25:03 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Tue, 17 Oct 2017 13:25:03 -0700 Subject: RFR (XS) - 8176827 Test can't find libXext.so In-Reply-To: References: Message-ID: <59E6671F.2000201@oracle.com> Looks good. thanks, Calvin On 10/16/17, 4:02 PM, Ioi Lam wrote: > Hi, > > JDK-8176827 is a closed issue, but basically we have some tests that > have unnecessary dependencies on desktop features, and would > sporadically fail on hosts that are configured to run only headless > tests. > > https://bugs.openjdk.java.net/browse/JDK-8176827 > http://cr.openjdk.java.net/~iklam/jdk10/8176827-cant-find-libxext.v01/ > > The fix is simple -- replace desktop classes used by the tests with > headless classes. E.g., java.awt.Button -> java.util.logging.Level > > Thanks > - Ioi From david.holmes at oracle.com Tue Oct 17 21:00:27 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 18 Oct 2017 07:00:27 +1000 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> Message-ID: Hi Thomas, On 18/10/2017 1:11 AM, Thomas St?fe wrote: > Hi David, > > I understand that you want to have a quick solution to a specific > problem. I do not like adding a switch with such a narrow usage and > would prefer, like Robin, a more well rounded and more general solution. To have a "more well rounded and more general solution" you first need a more general problem. :) I'm not at all clear what problem the ability to control stack guards for any attaching thread would be solving. > That said, I do not see any specific problems with the proposed switch. Thank you. > Not even sure if that matters, as I am not even sure I can review CSRs :) You certainly can! Q: Who should be a reviewer on a CSR proposal? A: One or more engineers with expertise in the areas impacted by the proposed change should review the CSR request and be listed as a reviewer before the proposal is reviewed by the CSR membership. (These engineers may or may not be Reviewers on the corresponding JDK project.) [1] Thanks, David [1] https://wiki.openjdk.java.net/display/csr/CSR+FAQs > Kind Regards, Thomas > > > On Tue, Oct 17, 2017 at 3:00 PM, David Holmes > wrote: > > On 17/10/2017 10:41 PM, Robbin Ehn wrote: > > On 10/17/2017 12:29 PM, David Holmes wrote: > > On 17/10/2017 7:28 PM, Robbin Ehn wrote: > > My understanding from prior discussion (ie when we > fixed the 2MB stack limit) is that it doesn't work > for them to use a new thread for the JVM. The change > to support anything but unlimited stack size also > helped (and the 8M limit when 'unlimited' is > tolerable) but really they just want a simple way to > say "please don't bother with stack guards, I can > deal with that myself". > > > Is there a problem I'm not seeing to extends this to all > attaching threads? > E.g. -XX:+DisableAttachingThreadGuardPages (more useful > option I would say) > > > Yes - that allows it to impact all of our launchers as well > - not something I want to allow for. It opens the door for a > myriad of bug reports if people use this flag without > understanding it's consequences. > > The aim here is to solve a specific problem, not introduce > new ways to break things. > > > First, since this is broken, you would fix the issue with > different stacksizes. > > > Not sure what you consider "broken". The current approach causes a > problem for runtimes that want to do their own stack guards. The > proposal is to allow them by giving a flag to turn ours off - but > only for the primordial thread, as that is where the problem lies. > > And secondly, I think the solution to that is to move all 'text' > options to the launcher and have the CreateVM taking a data > structure with pre-parsed options. And let the launcher choose > which options to expose to the end-user. Meaning our own java > launcher would never expose this option. (but this is much > bigger discussion) > > > Let's not start to envisage a complete re-design of how you might > communicate arguments between a host process and the VM. Please. > > Would the option would skip adding guard to any other thread > calling CreateVM? > If not, have the option really the correct name? > > E.g. -XX:+DisableCreateVMThreadGuardPages ? > > > The option would ONLY skip for the PRIMORDIAL thread of the process > when it loads the VM - hence it has exactly the right name. I > specifically do not want to allow this for an arbitrary thread that > happens to do the loading of the VM. > > Thanks, > David > > > Thanks, Robbin > > > David > > Thanks, Robbin > > > This is intended to be a very simple, very specific > proposal. I suppose it could be flagged as > "experimental" if we wanted the flexibility to do > something different later. But I don't see that > being on the cards. > > Cheers, > David > > > Thanks Robbin > > > Kind Regards, Thomas > > > > On Tue, Oct 17, 2017 at 9:22 AM, David > Holmes > > wrote: > > CSR: > https://bugs.openjdk.java.net/browse/JDK-8189423 > > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8189170 > > > Could I please have a reviewer for this > CSR request so I can fast-track it. > > Comments on the proposal are of course > welcome. > > Thanks, > David > > From rkennke at redhat.com Tue Oct 17 21:10:15 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 17 Oct 2017 23:10:15 +0200 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> Message-ID: <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> > The SuspendibleThreadSet API for synchronizing any non-Java thread > with safepoints currently resides under gc/g1. It is very useful for > other GCs too (in particular, Shenandoah does use it too), so I wanted > to move it to a common location like gc/common. Then Kim Barrett > commented that it might actually be useful for other threads outside > GC land and to put it under runtime/. So I did: > > http://cr.openjdk.java.net/~rkennke/8189276/webrev.00/ > > > I also added a generic hook to call the STS from safepoint > sync/desync, which is consequently used by G1 now. In other words, the > CollectedHeap API that Erik ? introduced is no longer used by G1. Only > CMS still uses that API because it has its own way to sync with > safepoints. I filed another bug > for this. Although > I have my doubt it will ever be fixed. This seems to have been very > carefully evolved (to put it positive), and the risk of breaking it is > relatively high, and thus doesn't seem worth the struggle to make it > fit the STS. > > Issue: > https://bugs.openjdk.java.net/browse/JDK-8189276 > > What do you think? Ok to go in? > Replying to myself here. I must admit that I am a bit reluctant to expose it to runtime/ unless there's a specific need for it. So maybe go back to the original plan to move it into gc/common and leave all the rest alone for now? What do others think? Roman From rkennke at redhat.com Tue Oct 17 21:21:58 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 17 Oct 2017 23:21:58 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <3577bda2-7fdd-d458-99b1-79f0ebd72bc2@redhat.com> References: <3577bda2-7fdd-d458-99b1-79f0ebd72bc2@redhat.com> Message-ID: Hi Zhengyu, thanks for reviewing! > Hi Roman, > > Look good overall. > > 1) Probably make sense to factor out static methods from GC into > separate GCFactory class. Right, this looks better. > 2) Stylish: > ? #ifndef SHARE_VM_GC_G1_G1_GC_HPP -> #ifndef SHARE_VM_GC_G1_G1GC_HPP Ok, right. I also removed the '_VM' parts to account for the changed directory structure in mono-repo. Differential: http://cr.openjdk.java.net/~rkennke/8189171/webrev.01.diff/ Full: http://cr.openjdk.java.net/~rkennke/8189171/webrev.01/ Better now? Roman > > Thanks, > > -Zhengyu > > On 10/12/2017 03:59 PM, Roman Kennke wrote: >> I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev because >> it touches both areas (i.e. the GC interface). >> >> Currently, all GC related argument processing is done in >> arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all sorts >> of GC specific methods etc. >> >> In order to have a cleaner GC interface, I propose to move all that >> code into GC specific classes under their GC specific directories. >> >> Since at this point in time we haven't created the XXXHeap subclasses >> yet (and cannot, before having set up all the flags and arguments), I >> propose to create a GC interface of which we create an instance as >> soon as we have selected a GC, and then do argument processing there. >> >> This 'GC' interface is, at this point, meant as a sort of GC factory >> interface. With this patch, it will only do argument processing. >> Later I plan to add heap creation to it (which is currently done in >> universe.cpp, with INCLUDE_ALL_GCS stuff and all the if (UseBlahGC) >> .. else if .. else branch chains. My current prototype in the jdk10 >> sandbox also provides servicabilty support classes and some more >> stuff, but we should decide later where to put this. >> >> I broke out some code paths into gc_factory.cpp (i.e. not in gc.cpp). >> Those are all the code paths that select GC, create the right >> instances, etc, in other words, the code paths that need to know >> about all GCs. Ideally, this should be the only file that needs to >> know about all the GCs. (Suggestions for improvement are welcome of >> course.) >> >> I tested this by running hotspot_gc jtreg tests, with normal >> fastdebug and no-precompiled configurations. Everything passes fine. >> >> http://cr.openjdk.java.net/~rkennke/8189171/webrev.00/ >> >> >> Opinions? >> >> Roman >> From david.holmes at oracle.com Tue Oct 17 21:34:37 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 18 Oct 2017 07:34:37 +1000 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> Message-ID: <6c720dba-1462-f14f-ac19-1a26a61266ee@oracle.com> +1 on not exposing this to runtime, but leave it as GC. David On 18/10/2017 7:10 AM, Roman Kennke wrote: > >> The SuspendibleThreadSet API for synchronizing any non-Java thread >> with safepoints currently resides under gc/g1. It is very useful for >> other GCs too (in particular, Shenandoah does use it too), so I wanted >> to move it to a common location like gc/common. Then Kim Barrett >> commented that it might actually be useful for other threads outside >> GC land and to put it under runtime/. So I did: >> >> http://cr.openjdk.java.net/~rkennke/8189276/webrev.00/ >> >> >> I also added a generic hook to call the STS from safepoint >> sync/desync, which is consequently used by G1 now. In other words, the >> CollectedHeap API that Erik ? introduced is no longer used by G1. Only >> CMS still uses that API because it has its own way to sync with >> safepoints. I filed another bug >> for this. Although >> I have my doubt it will ever be fixed. This seems to have been very >> carefully evolved (to put it positive), and the risk of breaking it is >> relatively high, and thus doesn't seem worth the struggle to make it >> fit the STS. >> >> Issue: >> https://bugs.openjdk.java.net/browse/JDK-8189276 >> >> What do you think? Ok to go in? >> > Replying to myself here. > I must admit that I am a bit reluctant to expose it to runtime/ unless > there's a specific need for it. So maybe go back to the original plan to > move it into gc/common and leave all the rest alone for now? What do > others think? > > Roman From rkennke at redhat.com Tue Oct 17 21:34:46 2017 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 17 Oct 2017 23:34:46 +0200 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> Message-ID: <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> Am 17.10.2017 um 23:10 schrieb Roman Kennke: > >> The SuspendibleThreadSet API for synchronizing any non-Java thread >> with safepoints currently resides under gc/g1. It is very useful for >> other GCs too (in particular, Shenandoah does use it too), so I >> wanted to move it to a common location like gc/common. Then Kim >> Barrett commented that it might actually be useful for other threads >> outside GC land and to put it under runtime/. So I did: >> >> http://cr.openjdk.java.net/~rkennke/8189276/webrev.00/ >> >> >> I also added a generic hook to call the STS from safepoint >> sync/desync, which is consequently used by G1 now. In other words, >> the CollectedHeap API that Erik ? introduced is no longer used by G1. >> Only CMS still uses that API because it has its own way to sync with >> safepoints. I filed another bug >> for this. Although >> I have my doubt it will ever be fixed. This seems to have been very >> carefully evolved (to put it positive), and the risk of breaking it >> is relatively high, and thus doesn't seem worth the struggle to make >> it fit the STS. >> >> Issue: >> https://bugs.openjdk.java.net/browse/JDK-8189276 >> >> What do you think? Ok to go in? >> > Replying to myself here. > I must admit that I am a bit reluctant to expose it to runtime/ unless > there's a specific need for it. So maybe go back to the original plan > to move it into gc/common and leave all the rest alone for now? What > do others think? Sorry, forgot to add something. One reason why we originally wanted to move it to gc/common (a new directory) instead of the existing gc/shared was that Erik H objected that gc/shared would be built into the minimal VM, and we probably wouldn't want that unless needed. However, moving it to runtime/ (without actual need), would achieve just that. So unless we can name an actual thread that would benefit from synchronizing using the STS, I suggest to leave the STS code in gc/common for now? That would be: http://cr.openjdk.java.net/~rkennke/8189276/webrev.01/ Roman From coleen.phillimore at oracle.com Tue Oct 17 22:43:20 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 17 Oct 2017 18:43:20 -0400 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> Message-ID: <9623b576-5152-d873-c135-16a8b930fcbb@oracle.com> On 10/17/17 5:34 PM, Roman Kennke wrote: > Am 17.10.2017 um 23:10 schrieb Roman Kennke: >> >>> The SuspendibleThreadSet API for synchronizing any non-Java thread >>> with safepoints currently resides under gc/g1. It is very useful for >>> other GCs too (in particular, Shenandoah does use it too), so I >>> wanted to move it to a common location like gc/common. Then Kim >>> Barrett commented that it might actually be useful for other threads >>> outside GC land and to put it under runtime/. So I did: >>> >>> http://cr.openjdk.java.net/~rkennke/8189276/webrev.00/ >>> >>> >>> I also added a generic hook to call the STS from safepoint >>> sync/desync, which is consequently used by G1 now. In other words, >>> the CollectedHeap API that Erik ? introduced is no longer used by >>> G1. Only CMS still uses that API because it has its own way to sync >>> with safepoints. I filed another bug >>> for this. >>> Although I have my doubt it will ever be fixed. This seems to have >>> been very carefully evolved (to put it positive), and the risk of >>> breaking it is relatively high, and thus doesn't seem worth the >>> struggle to make it fit the STS. >>> >>> Issue: >>> https://bugs.openjdk.java.net/browse/JDK-8189276 >>> >>> What do you think? Ok to go in? >>> >> Replying to myself here. >> I must admit that I am a bit reluctant to expose it to runtime/ >> unless there's a specific need for it. So maybe go back to the >> original plan to move it into gc/common and leave all the rest alone >> for now? What do others think? > > Sorry, forgot to add something. One reason why we originally wanted to > move it to gc/common (a new directory) instead of the existing > gc/shared was that Erik H objected that gc/shared would be built into > the minimal VM, and we probably wouldn't want that unless needed. > However, moving it to runtime/ (without actual need), would achieve > just that. So unless we can name an actual thread that would benefit > from synchronizing using the STS, I suggest to leave the STS code in > gc/common for now? > > That would be: > http://cr.openjdk.java.net/~rkennke/8189276/webrev.01/ > > > Roman Oh, please don't invent a directory for this.? It doesn't really matter that it's built into the minimal vm.? The minimal vm wasn't designed to be the strictly minimal set of things needed for small platforms, just smaller.? ie. that doesn't matter. It seems like it should be gc/shared with things like taskqueue. Thanks, Coleen From david.holmes at oracle.com Wed Oct 18 02:22:10 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 18 Oct 2017 12:22:10 +1000 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> Message-ID: <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> Hi Harold, On 17/10/2017 12:20 AM, harold seigel wrote: > Hi, > > Please review this new JDK-10 fix for bug JDK-8174954.? If the initial > resolution attempt fails with a LinkageError exception then the fix > saves the exception in the resolution_errors table and sets a flag in > the constant pool Cache, indicating the failure.? Subsequent attempts to > do the same resolution will see that the flag is set, retrieve the > exception from the resolution_errors table, and throw it, instead of > re-trying the resolution. I'm a little confused. The fix seems all about indy, which seems to be the topic of: JDK-8174942 Bootstrap Method Called Multiple Times Despite Initial Resolution Failure whereas this bug seems to be about a more general problem. Is this fix addressing both issues?? BTW there's no reason for JDK-8174954 to remain confidential. Thanks, David > The fix also prevents a race condition if two or more threads try to do > the same resolution concurrently.? When a thread tries to record the > result of its resolution attempt, it will check to see if another thread > recorded its result first.? If so, then it will use the result recorded > by the other thread.? Otherwise, it will record and use its result. > Access to the resolution result is protected by the > 'resolved_references' lock. > > Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ > > JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 > > The fix was tested with the JCK Lang and VM tests, the JTreg hotspot, > java/io, java/lang, java/util, and other tests, the co-located NSK > tests, JPRT, with Mach5 tier2 - tier5 tests, and by hand to check race > condition handling. > > Thanks, Harold > From ioi.lam at oracle.com Wed Oct 18 03:52:15 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 17 Oct 2017 20:52:15 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes Message-ID: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ https://bugs.openjdk.java.net/browse/JDK-8185160 Summary: DumpLoadedClassList skips classes that can't be stored into the CDS archive. For boot and platform loaders, only classes from the JRT image are listed. The test for "is this class from JRT image" was insufficient. It checked only classes whose source ends with "...../modules". For the platform loader, which loads the graal classes, the source is in the form "jrt:/jdk.internal.vm.compiler". The fix is to also check for a "jrt:" prefix. I also restructured the code to make it easier to read -- the nesting of || and && is hard to follow. The original code had a subtle bug with the "!" operator: 5938?? if (((class_loader == NULL && !ClassLoader::contains_append_entry(stream->source())) || 5939 SystemDictionary::is_platform_class_loader(class_loader)) && 5940?????? !ClassLoader::is_jrt(stream->source())) { So if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source()) was true, the class would NOT be skipped. The new version removes the "!". Thanks - Ioi From david.holmes at oracle.com Wed Oct 18 04:04:25 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 18 Oct 2017 14:04:25 +1000 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> Message-ID: <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> Hi Ioi, On 18/10/2017 1:52 PM, Ioi Lam wrote: > http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ > > https://bugs.openjdk.java.net/browse/JDK-8185160 > > Summary: > > DumpLoadedClassList skips classes that can't be stored into the CDS > archive. For boot and platform loaders, only classes from the JRT image > are listed. > > The test for "is this class from JRT image" was insufficient. It > checked only classes whose source ends with "...../modules". For the > platform loader, which loads the graal classes, the source is in the > form "jrt:/jdk.internal.vm.compiler". > > The fix is to also check for a "jrt:" prefix. So ./share/classfile/classLoader.hpp: static bool is_jrt(const char* name) { return string_ends_with(name, MODULES_IMAGE_NAME); } is a badly named function or just plain broken? > I also restructured the code to make it easier to read -- the nesting of > || and && is hard to follow. > > The original code had a subtle bug with the "!" operator: > > 5938?? if (((class_loader == NULL && > !ClassLoader::contains_append_entry(stream->source())) || > 5939 SystemDictionary::is_platform_class_loader(class_loader)) && > 5940?????? !ClassLoader::is_jrt(stream->source())) { > > So if (class_loader == NULL && > ClassLoader::contains_append_entry(stream->source()) was true, > the class would NOT be skipped. > > The new version removes the "!". So there should be an updated test to trigger the original bug. Thanks, David > Thanks > - Ioi From thomas.stuefe at gmail.com Wed Oct 18 06:14:49 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 18 Oct 2017 08:14:49 +0200 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> Message-ID: Hi David, On Tue, Oct 17, 2017 at 11:00 PM, David Holmes wrote: > Hi Thomas, > > On 18/10/2017 1:11 AM, Thomas St?fe wrote: > >> Hi David, >> >> I understand that you want to have a quick solution to a specific >> problem. I do not like adding a switch with such a narrow usage and would >> prefer, like Robin, a more well rounded and more general solution. >> > > To have a "more well rounded and more general solution" you first need a > more general problem. :) Sure :) The main example I was thinking of was the desire to attach native threads which have a different stack size than Xss - however, after inspecting the code, I see that this point is moot (maybe always was) as the VM does already the correct thing by placing the VM guards at the end of the real stack size, not at Xss. The other situations I could think of are cases where the thread stack lives in memory which is not protectable, or at least not protectable the way hotspot does it, using mprotect. On AIX, that would be the primordial thread stack as well as any memory allocated with SystemV shm. Beside that, I could imagine certain embedders just not wanting the VM to place guards, for whatever reason. I could imagine scenarios where one wants to pool thread stack memory and does not want that memory polluted with guard pages. But I think now that these cases would best served with an addition to the invocation API, by giving JNI_CreateAttachedThread an option to skip vm guard creation. That would prevent normal users to fall over a misunderstood option. Embedders typically know better what they do. > I'm not at all clear what problem the ability to control stack guards for > any attaching thread would be solving. > > That said, I do not see any specific problems with the proposed switch. >> > > Thank you. > > Not even sure if that matters, as I am not even sure I can review CSRs :) >> > > You certainly can! > > Q: Who should be a reviewer on a CSR proposal? > A: One or more engineers with expertise in the areas impacted by the > proposed change should review the CSR request and be listed as a reviewer > before the proposal is reviewed by the CSR membership. (These engineers may > or may not be Reviewers on the corresponding JDK project.) [1] > > Ah :) Okay then, consider it reviewed :) ..Thomas > Thanks, > David > > [1] https://wiki.openjdk.java.net/display/csr/CSR+FAQs > > Kind Regards, Thomas >> >> >> >> On Tue, Oct 17, 2017 at 3:00 PM, David Holmes > > wrote: >> >> On 17/10/2017 10:41 PM, Robbin Ehn wrote: >> >> On 10/17/2017 12:29 PM, David Holmes wrote: >> >> On 17/10/2017 7:28 PM, Robbin Ehn wrote: >> >> My understanding from prior discussion (ie when we >> fixed the 2MB stack limit) is that it doesn't work >> for them to use a new thread for the JVM. The change >> to support anything but unlimited stack size also >> helped (and the 8M limit when 'unlimited' is >> tolerable) but really they just want a simple way to >> say "please don't bother with stack guards, I can >> deal with that myself". >> >> >> Is there a problem I'm not seeing to extends this to all >> attaching threads? >> E.g. -XX:+DisableAttachingThreadGuardPages (more useful >> option I would say) >> >> >> Yes - that allows it to impact all of our launchers as well >> - not something I want to allow for. It opens the door for a >> myriad of bug reports if people use this flag without >> understanding it's consequences. >> >> The aim here is to solve a specific problem, not introduce >> new ways to break things. >> >> >> First, since this is broken, you would fix the issue with >> different stacksizes. >> >> >> Not sure what you consider "broken". The current approach causes a >> problem for runtimes that want to do their own stack guards. The >> proposal is to allow them by giving a flag to turn ours off - but >> only for the primordial thread, as that is where the problem lies. >> >> And secondly, I think the solution to that is to move all 'text' >> options to the launcher and have the CreateVM taking a data >> structure with pre-parsed options. And let the launcher choose >> which options to expose to the end-user. Meaning our own java >> launcher would never expose this option. (but this is much >> bigger discussion) >> >> >> Let's not start to envisage a complete re-design of how you might >> communicate arguments between a host process and the VM. Please. >> >> Would the option would skip adding guard to any other thread >> calling CreateVM? >> If not, have the option really the correct name? >> >> E.g. -XX:+DisableCreateVMThreadGuardPages ? >> >> >> The option would ONLY skip for the PRIMORDIAL thread of the process >> when it loads the VM - hence it has exactly the right name. I >> specifically do not want to allow this for an arbitrary thread that >> happens to do the loading of the VM. >> >> Thanks, >> David >> >> >> Thanks, Robbin >> >> >> David >> >> Thanks, Robbin >> >> >> This is intended to be a very simple, very specific >> proposal. I suppose it could be flagged as >> "experimental" if we wanted the flexibility to do >> something different later. But I don't see that >> being on the cards. >> >> Cheers, >> David >> >> >> Thanks Robbin >> >> >> Kind Regards, Thomas >> >> >> >> On Tue, Oct 17, 2017 at 9:22 AM, David >> Holmes > > >> wrote: >> >> CSR: >> https://bugs.openjdk.java.net/ >> browse/JDK-8189423 >> > /browse/JDK-8189423> >> >> Bug: >> https://bugs.openjdk.java.net/ >> browse/JDK-8189170 >> > /browse/JDK-8189170> >> >> Could I please have a reviewer for this >> CSR request so I can fast-track it. >> >> Comments on the proposal are of course >> welcome. >> >> Thanks, >> David >> >> >> From david.holmes at oracle.com Wed Oct 18 06:58:08 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 18 Oct 2017 16:58:08 +1000 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> Message-ID: <86b5ffc2-b859-1fa2-5d83-cea892d5037e@oracle.com> On 18/10/2017 4:14 PM, Thomas St?fe wrote: > Hi David, > > On Tue, Oct 17, 2017 at 11:00 PM, David Holmes > wrote: > > Hi Thomas, > > On 18/10/2017 1:11 AM, Thomas St?fe wrote: > > Hi David, > > I understand that you want to have a quick solution to a > specific problem. I do not like adding a switch with such a > narrow usage and would prefer, like Robin, a more well rounded > and more general solution. > > > To have a "more well rounded and more general solution" you first > need a more general problem. :) > > > Sure :) > > The main example I was thinking of was the desire to attach native > threads which have a different stack size than Xss - however, after > inspecting the code, I see that this point is moot (maybe always was) as > the VM does already the correct thing by placing the VM guards at the > end of the real stack size, not at Xss. Right. Xss is for threads created by the VM - and the launcher also uses it for the thread that launches the VM and runs main. The relationship between Xss and the use of the primordial thread to launch the VM has been where things got messy.** ** IIRC the argument was to artificially make it appear as-if the primordial thread had the same stack size as all other created threads so that you wouldn't get in a situation where a computation would pass on the "main thread" and fail with StackOverflowError on any other thread. I think it was after this that we just bit the bullet and had the launcher always kick-off a new thread to become the "main thread". > The other situations I could think of are cases where the thread stack > lives in memory which is not protectable, or at least not protectable > the way hotspot does it, using mprotect. On AIX, that would be the > primordial thread stack as well as any memory allocated with SystemV shm. Ok. Is it possible to augment os::uses_stack_guard_pages() to report false if current thread can't support them ? > Beside that, I could imagine certain embedders just not wanting the VM > to place guards, for whatever reason. I could imagine scenarios where > one wants to pool thread stack memory and does not want that memory > polluted with guard pages. I can imagine lots of things too, but we haven't had many (any?) real requests in 20 years so ... these kinds problems tend to need immediate solutions so people just work around them. > But I think now that these cases would best served with an addition to > the invocation API, by giving JNI_CreateAttachedThread an option to skip > vm guard creation. That would prevent normal users to fall over a > misunderstood option. Embedders typically know better what they do. I agree. If this were to be pursued then a modified attach API would be the way to do it. Feel free to file an RFE if you'd like to do this. ;-) > I'm not at all clear what problem the ability to control stack > guards for any attaching thread would be solving. > > That said, I do not see any specific problems with the proposed > switch. > > > Thank you. > > Not even sure if that matters, as I am not even sure I can > review CSRs :) > > > You certainly can! > > Q: Who should be a reviewer on a CSR proposal? > A: One or more engineers with expertise in the areas impacted by the > proposed change should review the CSR request and be listed as a > reviewer before the proposal is reviewed by the CSR membership. > (These engineers may or may not be Reviewers on the corresponding > JDK project.) [1] > > > Ah :) Okay then, consider it reviewed :) If you really want to be a reviewer (totally up to you) please "edit" the CSR and scroll down to the "reviewed by" box and add your OpenJDK user name. I've updated the CSR to make the flag "experimental". Though I need to check with the R folk to see if that is okay. And added some additional text about a more general attach API. Thanks, David > ..Thomas > > Thanks, > David > > [1] https://wiki.openjdk.java.net/display/csr/CSR+FAQs > > > Kind Regards, Thomas > > > > On Tue, Oct 17, 2017 at 3:00 PM, David Holmes > > >> wrote: > > ? ? On 17/10/2017 10:41 PM, Robbin Ehn wrote: > > ? ? ? ? On 10/17/2017 12:29 PM, David Holmes wrote: > > ? ? ? ? ? ? On 17/10/2017 7:28 PM, Robbin Ehn wrote: > > ? ? ? ? ? ? ? ? ? ? My understanding from prior discussion (ie > when we > ? ? ? ? ? ? ? ? ? ? fixed the 2MB stack limit) is that it > doesn't work > ? ? ? ? ? ? ? ? ? ? for them to use a new thread for the JVM. > The change > ? ? ? ? ? ? ? ? ? ? to support anything but unlimited stack > size also > ? ? ? ? ? ? ? ? ? ? helped (and the 8M limit when 'unlimited' is > ? ? ? ? ? ? ? ? ? ? tolerable) but really they just want a > simple way to > ? ? ? ? ? ? ? ? ? ? say "please don't bother with stack guards, > I can > ? ? ? ? ? ? ? ? ? ? deal with that myself". > > > ? ? ? ? ? ? ? ? Is there a problem I'm not seeing to extends > this to all > ? ? ? ? ? ? ? ? attaching threads? > ? ? ? ? ? ? ? ? E.g. -XX:+DisableAttachingThreadGuardPages > (more useful > ? ? ? ? ? ? ? ? option I would say) > > > ? ? ? ? ? ? Yes - that allows it to impact all of our launchers > as well > ? ? ? ? ? ? - not something I want to allow for. It opens the > door for a > ? ? ? ? ? ? myriad of bug reports if people use this flag without > ? ? ? ? ? ? understanding it's consequences. > > ? ? ? ? ? ? The aim here is to solve a specific problem, not > introduce > ? ? ? ? ? ? new ways to break things. > > > ? ? ? ? First, since this is broken, you would fix the issue with > ? ? ? ? different stacksizes. > > > ? ? Not sure what you consider "broken". The current approach > causes a > ? ? problem for runtimes that want to do their own stack > guards. The > ? ? proposal is to allow them by giving a flag to turn ours off > - but > ? ? only for the primordial thread, as that is where the > problem lies. > > ? ? ? ? And secondly, I think the solution to that is to move > all 'text' > ? ? ? ? options to the launcher and have the CreateVM taking a data > ? ? ? ? structure with pre-parsed options. And let the launcher > choose > ? ? ? ? which options to expose to the end-user. Meaning our > own java > ? ? ? ? launcher would never expose this option. (but this is much > ? ? ? ? bigger discussion) > > > ? ? Let's not start to envisage a complete re-design of how you > might > ? ? communicate arguments between a host process and the VM. > Please. > > ? ? ? ? Would the option would skip adding guard to any other > thread > ? ? ? ? calling CreateVM? > ? ? ? ? If not, have the option really the correct name? > > ? ? ? ? E.g. -XX:+DisableCreateVMThreadGuardPages ? > > > ? ? The option would ONLY skip for the PRIMORDIAL thread of the > process > ? ? when it loads the VM - hence it has exactly the right name. I > ? ? specifically do not want to allow this for an arbitrary > thread that > ? ? happens to do the loading of the VM. > > ? ? Thanks, > ? ? David > > > ? ? ? ? Thanks, Robbin > > > ? ? ? ? ? ? David > > ? ? ? ? ? ? ? ? Thanks, Robbin > > > ? ? ? ? ? ? ? ? ? ? This is intended to be a very simple, very > specific > ? ? ? ? ? ? ? ? ? ? proposal. I suppose it could be flagged as > ? ? ? ? ? ? ? ? ? ? "experimental" if we wanted the flexibility > to do > ? ? ? ? ? ? ? ? ? ? something different later. But I don't see that > ? ? ? ? ? ? ? ? ? ? being on the cards. > > ? ? ? ? ? ? ? ? ? ? Cheers, > ? ? ? ? ? ? ? ? ? ? David > > > ? ? ? ? ? ? ? ? ? ? ? ? Thanks Robbin > > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? Kind Regards, Thomas > > > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? On Tue, Oct 17, 2017 at 9:22 AM, David > ? ? ? ? ? ? ? ? ? ? ? ? ? ? Holmes > ? ? ? ? ? ? ? ? ? ? ? ? ? ? >> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? wrote: > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? CSR: > https://bugs.openjdk.java.net/browse/JDK-8189423 > > > > > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Bug: > https://bugs.openjdk.java.net/browse/JDK-8189170 > > > > > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Could I please have a reviewer > for this > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? CSR request so I can fast-track it. > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Comments on the proposal are of > course > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? welcome. > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Thanks, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? David > > > From thomas.stuefe at gmail.com Wed Oct 18 08:05:30 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 18 Oct 2017 10:05:30 +0200 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <86b5ffc2-b859-1fa2-5d83-cea892d5037e@oracle.com> References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> <86b5ffc2-b859-1fa2-5d83-cea892d5037e@oracle.com> Message-ID: Hi David, On Wed, Oct 18, 2017 at 8:58 AM, David Holmes wrote: > On 18/10/2017 4:14 PM, Thomas St?fe wrote: > >> Hi David, >> >> On Tue, Oct 17, 2017 at 11:00 PM, David Holmes > > wrote: >> >> Hi Thomas, >> >> On 18/10/2017 1:11 AM, Thomas St?fe wrote: >> >> Hi David, >> >> I understand that you want to have a quick solution to a >> specific problem. I do not like adding a switch with such a >> narrow usage and would prefer, like Robin, a more well rounded >> and more general solution. >> >> >> To have a "more well rounded and more general solution" you first >> need a more general problem. :) >> >> Sure :) >> >> The main example I was thinking of was the desire to attach native >> threads which have a different stack size than Xss - however, after >> inspecting the code, I see that this point is moot (maybe always was) as >> the VM does already the correct thing by placing the VM guards at the end >> of the real stack size, not at Xss. >> > > Right. Xss is for threads created by the VM - and the launcher also uses > it for the thread that launches the VM and runs main. The relationship > between Xss and the use of the primordial thread to launch the VM has been > where things got messy.** > > ** IIRC the argument was to artificially make it appear as-if the > primordial thread had the same stack size as all other created threads so > that you wouldn't get in a situation where a computation would pass on the > "main thread" and fail with StackOverflowError on any other thread. I think > it was after this that we just bit the bullet and had the launcher always > kick-off a new thread to become the "main thread". > > I see. > The other situations I could think of are cases where the thread stack >> lives in memory which is not protectable, or at least not protectable the >> way hotspot does it, using mprotect. On AIX, that would be the primordial >> thread stack as well as any memory allocated with SystemV shm. >> > > Ok. Is it possible to augment os::uses_stack_guard_pages() to report > false if current thread can't support them ? > > os::uses_stack_guard_pages() is not necessarily used from the same thread (e.g. cpu/x86/frame_x86.cpp, frame::safe_for_sender()). btw, if we do not augment it, this could be a candidate for cleaning up, no? All platforms just return "true". > Beside that, I could imagine certain embedders just not wanting the VM to >> place guards, for whatever reason. I could imagine scenarios where one >> wants to pool thread stack memory and does not want that memory polluted >> with guard pages. >> > > I can imagine lots of things too, but we haven't had many (any?) real > requests in 20 years so ... these kinds problems tend to need immediate > solutions so people just work around them. > > But I think now that these cases would best served with an addition to the >> invocation API, by giving JNI_CreateAttachedThread an option to skip vm >> guard creation. That would prevent normal users to fall over a >> misunderstood option. Embedders typically know better what they do. >> > > I agree. If this were to be pursued then a modified attach API would be > the way to do it. Feel free to file an RFE if you'd like to do this. ;-) > > I'll do if I have spare time :) > I'm not at all clear what problem the ability to control stack >> guards for any attaching thread would be solving. >> >> That said, I do not see any specific problems with the proposed >> switch. >> >> >> Thank you. >> >> Not even sure if that matters, as I am not even sure I can >> review CSRs :) >> >> >> You certainly can! >> >> Q: Who should be a reviewer on a CSR proposal? >> A: One or more engineers with expertise in the areas impacted by the >> proposed change should review the CSR request and be listed as a >> reviewer before the proposal is reviewed by the CSR membership. >> (These engineers may or may not be Reviewers on the corresponding >> JDK project.) [1] >> >> >> Ah :) Okay then, consider it reviewed :) >> > > If you really want to be a reviewer (totally up to you) please "edit" the > CSR and scroll down to the "reviewed by" box and add your OpenJDK user name. > > I've updated the CSR to make the flag "experimental". Though I need to > check with the R folk to see if that is okay. And added some additional > text about a more general attach API. > > Okay, I did add myself as reviewer. ..Thomas > Thanks, > David > > ..Thomas >> >> Thanks, >> David >> >> [1] https://wiki.openjdk.java.net/display/csr/CSR+FAQs >> >> >> Kind Regards, Thomas >> >> >> >> On Tue, Oct 17, 2017 at 3:00 PM, David Holmes >> >> > >> >> wrote: >> >> On 17/10/2017 10:41 PM, Robbin Ehn wrote: >> >> On 10/17/2017 12:29 PM, David Holmes wrote: >> >> On 17/10/2017 7:28 PM, Robbin Ehn wrote: >> >> My understanding from prior discussion (ie >> when we >> fixed the 2MB stack limit) is that it >> doesn't work >> for them to use a new thread for the JVM. >> The change >> to support anything but unlimited stack >> size also >> helped (and the 8M limit when 'unlimited' is >> tolerable) but really they just want a >> simple way to >> say "please don't bother with stack guards, >> I can >> deal with that myself". >> >> >> Is there a problem I'm not seeing to extends >> this to all >> attaching threads? >> E.g. -XX:+DisableAttachingThreadGuardPages >> (more useful >> option I would say) >> >> >> Yes - that allows it to impact all of our launchers >> as well >> - not something I want to allow for. It opens the >> door for a >> myriad of bug reports if people use this flag without >> understanding it's consequences. >> >> The aim here is to solve a specific problem, not >> introduce >> new ways to break things. >> >> >> First, since this is broken, you would fix the issue with >> different stacksizes. >> >> >> Not sure what you consider "broken". The current approach >> causes a >> problem for runtimes that want to do their own stack >> guards. The >> proposal is to allow them by giving a flag to turn ours off >> - but >> only for the primordial thread, as that is where the >> problem lies. >> >> And secondly, I think the solution to that is to move >> all 'text' >> options to the launcher and have the CreateVM taking a >> data >> structure with pre-parsed options. And let the launcher >> choose >> which options to expose to the end-user. Meaning our >> own java >> launcher would never expose this option. (but this is >> much >> bigger discussion) >> >> >> Let's not start to envisage a complete re-design of how you >> might >> communicate arguments between a host process and the VM. >> Please. >> >> Would the option would skip adding guard to any other >> thread >> calling CreateVM? >> If not, have the option really the correct name? >> >> E.g. -XX:+DisableCreateVMThreadGuardPages ? >> >> >> The option would ONLY skip for the PRIMORDIAL thread of the >> process >> when it loads the VM - hence it has exactly the right name. I >> specifically do not want to allow this for an arbitrary >> thread that >> happens to do the loading of the VM. >> >> Thanks, >> David >> >> >> Thanks, Robbin >> >> >> David >> >> Thanks, Robbin >> >> >> This is intended to be a very simple, very >> specific >> proposal. I suppose it could be flagged as >> "experimental" if we wanted the flexibility >> to do >> something different later. But I don't see >> that >> being on the cards. >> >> Cheers, >> David >> >> >> Thanks Robbin >> >> >> Kind Regards, Thomas >> >> >> >> On Tue, Oct 17, 2017 at 9:22 AM, >> David >> Holmes > >> > >> >> wrote: >> >> CSR: >> https://bugs.openjdk.java.net/browse/JDK-8189423 >> >> < >> https://bugs.openjdk.java.net/browse/JDK-8189423 >> > >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8189170 >> >> < >> https://bugs.openjdk.java.net/browse/JDK-8189170 >> > >> >> Could I please have a reviewer >> for this >> CSR request so I can fast-track >> it. >> >> Comments on the proposal are of >> course >> welcome. >> >> Thanks, >> David >> >> >> >> From erik.osterlund at oracle.com Wed Oct 18 08:18:17 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 18 Oct 2017 10:18:17 +0200 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> Message-ID: <59E70E49.4040207@oracle.com> Hi Roman, On 2017-10-17 23:34, Roman Kennke wrote: > Am 17.10.2017 um 23:10 schrieb Roman Kennke: >> >>> The SuspendibleThreadSet API for synchronizing any non-Java thread >>> with safepoints currently resides under gc/g1. It is very useful for >>> other GCs too (in particular, Shenandoah does use it too), so I >>> wanted to move it to a common location like gc/common. Then Kim >>> Barrett commented that it might actually be useful for other threads >>> outside GC land and to put it under runtime/. So I did: >>> >>> http://cr.openjdk.java.net/~rkennke/8189276/webrev.00/ >>> >>> >>> I also added a generic hook to call the STS from safepoint >>> sync/desync, which is consequently used by G1 now. In other words, >>> the CollectedHeap API that Erik ? introduced is no longer used by >>> G1. Only CMS still uses that API because it has its own way to sync >>> with safepoints. I filed another bug >>> for this. >>> Although I have my doubt it will ever be fixed. This seems to have >>> been very carefully evolved (to put it positive), and the risk of >>> breaking it is relatively high, and thus doesn't seem worth the >>> struggle to make it fit the STS. >>> >>> Issue: >>> https://bugs.openjdk.java.net/browse/JDK-8189276 >>> >>> What do you think? Ok to go in? >>> >> Replying to myself here. >> I must admit that I am a bit reluctant to expose it to runtime/ >> unless there's a specific need for it. So maybe go back to the >> original plan to move it into gc/common and leave all the rest alone >> for now? What do others think? > > Sorry, forgot to add something. One reason why we originally wanted to > move it to gc/common (a new directory) instead of the existing > gc/shared was that Erik H objected that gc/shared would be built into > the minimal VM, and we probably wouldn't want that unless needed. > However, moving it to runtime/ (without actual need), would achieve > just that. So unless we can name an actual thread that would benefit > from synchronizing using the STS, I suggest to leave the STS code in > gc/common for now? > > That would be: > http://cr.openjdk.java.net/~rkennke/8189276/webrev.01/ > We had some internal discussions, and the conclusion seems to be that we would like to not have a new gc/common directory for build-only purposes, but rather put it in gc/shared and explicitly mark files we do not want to have compiled in minimal VM in the make/hotspot/lib/JvmFeatures.gmk build file. You can remove files from the minimal VM build by explicitly adding files to an exclude list in that file. 133: JVM_EXCLUDE_FILES += \ 134: concurrentGCThread.cpp \ +++ add your files here 135: plab.cpp I tend to agree that avoiding the new gc/common directory is probably a win right now. Hope you agree with this. Thanks, /Erik > Roman From adam.farley at uk.ibm.com Wed Oct 18 11:17:53 2017 From: adam.farley at uk.ibm.com (Adam Farley8) Date: Wed, 18 Oct 2017 12:17:53 +0100 Subject: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be exited by java In-Reply-To: References: Message-ID: Hi All, Updated summary, based on the responses: 1) Exit(0) (during VM startup) is a big bug because it imitates successful completion of external cpp code accessing JNI methods. 2) One solution for (1) is to specify a new return code for JNI. 3) The supplied code (diff) generates, facilitates, and handles that return code for the exit(0) scenario: -agentlib:jdwp=help 4) The exit(0) problem (in general) is worth fixing, however we may choose not to support the use of this new code in the jdwp example case. 5) The supplied test confirms that the supplied code works (run via unzip, and then bash TestStart.sh ). 6) To implement this new return code, plus the code that handles it, we would need to follow the CSR process to modify the JNI spec. 7) To implement the jdwp scenario fix, if we choose to support it at all, we would also need to use the CSR process the JVM TI spec. 8) To address all of the worst instances of exit(0), we would need to search for exit(0) and raise a bug for each significant one (or group). 9) To solve (8) in one bug would be a lot of work, arguably too much work for a single bug. 10) If the new return code is identified as the appropriate solution to this problem, we will need to agree on the right name and return code value. Also, here's a shortlist of the main questions I recall being raised here, plus answers people have given. A) What are the potential solutions for the exit(0) problem? i) New JNI Return Code. ii) Remove the info-only options, at least via the JNI, and return a standard error code if they are used. iii) Remove the info-only options, at least via the JNI, and filter them out if they are used. iv) Retain existing behaviour, and document a need for the user to filter out help options before starting the VM via the JNI. B) What are the criteria for the "Best" solution? i) It must prevent "exit(0)" calls. ii) It must be proven to work. iii) It should require minimal (none, ideally) behaviour change from the java.exe user. iv) It should allow the external cpp code accessing the JNI to complete normally, without being prematurely terminated. C) Which solutions meet the (B) criteria? i) Both the "new return code" and the "remove info-only options" solutions meet the (B) criteria. D) Is it right to have any "Do X and then exit." arguments at all, and (if no) what would be the alternatives? i) ? Best Regards Adam Farley P.S. As per Alan and David's emails, the exit(#) references have been removed entirely, as discussing them alongside the original exit(0) problem risks scope creep. This bug, if raised, should only cover the exit(0) cases. I believe we have consensus here. From: David Holmes To: Adam Farley8 , Alan Bateman , core-libs-dev at openjdk.java.net, hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com Date: 13/10/2017 14:31 Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be exited by java Hi Adam, On 13/10/2017 10:16 PM, Adam Farley8 wrote: > Hi All, > > Here's a summary of the email below (which is intended, partly, as a > summary of the emails before it). > > Let me know if you agree/disagree with any of these points. > > 1) Exit(#) during vm startup is a bug because it should return a code > regardless of the state of the VM. Yes it's a bug but not one that is likely to be addressed in any foreseeable timeframe. There are simply too many "exit on error" paths. If we were to start using C++ exceptions within the VM that might provide a way to quickly get back to the CreateJavaVM routine where we could return an error code - but that is itself a major project that has barely even been discussed AFAIK. (Compiler folk have talked about it because compiler paths are fairly self-contained - though that was before Graal and AOT.) > 2) Exit(0) is an *especially* big bug because it imitates successful > completion of external cpp code accessing JNI methods. > 3) One solution is to specify a new return code for JNI. A solution for 2) yes. > 4) The supplied code (diff) generates, facilitates, and handles that > return code for the exit(0) scenario: -agentlib:jdwp=help > 5) The supplied test confirms that the supplied code works (run via > unzip, and then bash TestStart.sh bin dir>). > 6) To implement this new return code, plus the code that handles it, we > would need to follow the CSR process. > 7) To implement the fix for the scenario used as an example of the new > return code's use, we would need to modify the JVM TI spec. Yes you have demonstrated a potential solution for the agent case. The question is, is it the right solution? Is it a worthwhile solution? (As I've said I'd prefer not to have any "do something then exit" VM arguments.) And can we make it fit into the existing specs without contorting things too much. I still think it easier/preferable for whatever loads the VM to filter out the VM args that trigger this behaviour. I mean if you pass -Xshare:dump you don't have any right to expect a functioning VM after JNI_CreateJavaVM returns - at least I don't think so. Just don't do it. Yes it is an imperfection in the invocation API, but life isn't perfect. > 8) To address all of the worst instances of exit(#), we would need to > search for exit(#) and raise a bug for each significant one (or group). > 9) To solve (8) in one bug would be a lot of work, arguably too much > work for a single bug. This is simply impractical. You may be able to pick off a few low-hanging cases, but that won't really make any practical difference. > 10) If the new return code is chosen as the appropriate solution to this > problem, we may need to choose a better name for the return code. > > Is this a fair assessment of the current state of the debate? It's a fair summary of your position and proposal. Cheers, David > I'm on IRC every weekday from 9am-5pm (4pm fridays) BST (GMT+1) if > anyone wants to discuss this in real-time on the openjdk channel. > > Best Regards > > Adam Farley > > > > -- Previous Email -- > > Hi David, Alan, > > You are right in that the changes to HotSpot would be nontrivial. > I see a number of places in (e.g.) arguments.cpp that seem to > exit in the same manner as Xlog (such as -Xinternalversion). > > I would advise ploughing through the CSR process to alter the > JNI spec, and simultaneously identify some key paths that can > be raised as bugs. That way, when people have time to address > these issues, the mechanism to handle a silent exit is already > in place. > > The JDWP fix can be raised separately as one of these bugs, if > it would make things simpler. > > As for the name, JNI_SILENT_EXIT is a placeholder, and can be > readily changed. Do you have any suggestions? > > Lastly, in an ideal world, the VM initialisation should never exit(#). It > should return a return code that tells the caller something, pass or > fail, messy or tidy. That way, if someone is using the JNI as part of > something bigger (like a database or a web server), one of these > scenarios is just a bug, rather than a world-ender like exit(#). > > And now for the individual messages. :) > > David: Having help data returned by the launcher seems like a > good way to avoid exit(0) calls, but I'm not sure how we'd prevent > a JNI-caller using those options. Ultimately, to be sure, we'd have > to remove the logic for those options, centralise the data to better > enable launcher access, and add some logic in there so it can find > any other help data (e.g. from the jdwp agent library). I feel this would > be a bigger task than adding the new return code and changing the > vm, plus it wouldn't provide for any non-help scenarios where the > vm wants to shut down without error during initialisation. > > Alan: I should mention that the silent exit solution is already in > use in the OpenJ9 VM. Not all of the exit paths have been > resolved, but many have. > > The code is open and can be found here: < https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_eclipse_openj9&d=DwIC-g&c=jf_iaSHvJObTbx-siA1ZOg&r=P5m8KWUXJf-CeVJc0hDGD9AQ2LkcXDC0PMV9ntVw5Ho&m=c_McLiDSKtlzF_gNlWcAvBTNbDqyGHyW325GY3_3QkU&s=0QVowxRejNRAXI0Vv5whKQFWGTO36XVICmPYCG8EqIU&e= > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_eclipse_openj9&d=DwIC-g&c=jf_iaSHvJObTbx-siA1ZOg&r=P5m8KWUXJf-CeVJc0hDGD9AQ2LkcXDC0PMV9ntVw5Ho&m=c_McLiDSKtlzF_gNlWcAvBTNbDqyGHyW325GY3_3QkU&s=0QVowxRejNRAXI0Vv5whKQFWGTO36XVICmPYCG8EqIU&e= > > And though the silent exit code is disabled for the time being, it > can be re-enabled by entering this class: > > runtime/vm/jvminit.c > > and altering line 2343 ( ctrl-f for exit(1) if it's not there). > > I won't paste the full code here in case people are concerned > about contamination, but I would assert that this code (and the > associated vm files) prove that the concept is possible. > > Note that that code should not be enabled until after we've > integrated the code that can handle a silent exit. > > Best Regards > > Adam Farley > > P.S. Thank you both for your efforts on this. :) > > > > From: David Holmes > To: Alan Bateman , Adam Farley8 > , core-libs-dev at openjdk.java.net, > hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com > Date: 15/09/2017 12:03 > Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be > exited by java > ------------------------------------------------------------------------ > > > > > > On 15/09/2017 8:17 PM, Alan Bateman wrote: > > On 15/09/2017 02:47, David Holmes wrote: > >> Hi Adam, > >> > >> I am still very much torn over this one. I think the idea of > >> print-and-exit flags for a potentially hosted library like the JVM is > >> just wrong - we should never have done that, but we did. Fixing that > >> by moving the flags to the launcher is far from trivial**. Endorsing > >> and encouraging these sorts of flag by adding JNI support seems to be > >> sending the wrong message. > >> > >> ** I can envisage a "help xxx" Dcmd that can read back the info from > >> the VM. The launcher can send the Dcmd, print the output and exit. The > >> launcher would not need to know what the xxx values mean, but would > >> have to intercept the existing ones. > >> > >> Another option is just to be aware of these flags (are there more than > >> jdwp and Xlog?) and deal with them specially in your custom launcher - > >> either filter them out and ignore them, or else launch the VM in its > >> own process to respond to them. > >> > >> Any changes to the JNI specification need to go through the CSR process. > > Yes, it would require an update to the JNI spec, also a change to the > > JVM TI spec where Agent_OnLoad returning a non-0 value is specified to > > terminates the VM. The name and value needs discussion too, esp. as the > > JNI spec uses negative values for failure. > > > > In any case, I'm also torn over this one as it's a corner case that is > > only interesting for custom launchers that load agents with options that > > print usage messages. It wouldn't be hard to have the Agent_OnLoad > > specify a printf hook that the agent could use for output although there > > are complications with agents such as JDWP that also announce their > > transport end point. Beyond that there is still the issue of the custom > > launcher that would need to know to destroy the VM without reporting an > > error. > > > > So what happened to the more meaty part to this which is fixing the > > various cases in HotSpot that terminate the process during > > initialization? I would expect some progress could be made on those > > cases while trying to decide whether to rev the JNI and JVM TI specs to > > cover the help case. > > Trying to eliminate the vm_exit_during_initialization paths in hotspot > is a huge undertaking IMHO. > > David > > > > > -Alan > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From magnus.ihse.bursie at oracle.com Wed Oct 18 13:10:33 2017 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Wed, 18 Oct 2017 15:10:33 +0200 Subject: Unification of jni_.h Message-ID: In the context of JDK-8167078 (Duplicate header files in hotspot and jdk), I was looking at unifying the platform specific extensions to jni.h, the jni_md.h files between hotspot and java.base. The main difference here is that the hotspot jni_* files are divided into individual files based on cpu, while the java.base versions are divided according to operating system (which, in turn, implicates compiler). On a second look, however, it turns out to not be so problematic -- a hotspot file like jni_x86.h contains basically #ifdef WIN32 ... // do windows stuff #else ... // do posix/macos stuff #endif and the two blocks match very well what the java.base versions are doing for the different operating systems. In fact, the OS (and compiler) division seems more natural, since there is much redundancy in the hotspot files, and the java.base OS-based versions are much simpler. I'm proposing to remove the hotspot CPU-based files and just replace them with the java.base versions. However, a few differences crept out. I'd like to discuss them before proceeding. For jni_aarch64.h: There's a windows (or rather, "not unix") part of the jni_aarch64.h that I do not believe have ever been used. Nevertheless, it contains "typedef int jint" rather than "typedef long jint", which the java.base windows version uses. Since I've never heard of aarch64 building on Windows, I presume this is irrelevant. For jni_aarch64.h and jni_sparc.h: The unix version will match with the java.base version as long as _LP64 is defined. This should be just fine, since we define that when building hotspot/the JDK, and the java.base version that is exported by the JDK on linux/aarch64 and solaris/sparc has always required the _LP64 define. (In fact, the macos and unix versions in java.base only contain trivial differences. The macos version should probably be removed in favor of a single unix version.) For jni_x86.h: There windows part suffers the same problem as for aarch64, but here it's potentially problematic. The hotspot version uses "typedef int jint" while the java.base version uses "typedef long jint". If this *really* were a problem, we would probably have gotten reports from JNI code unable to work on Windows, so I assume this turns out to be the same datatype in the end. Don't know if it's due to luck, compiler flags or by definition. For jni_s390.h: Here's the most problematic version. The exported java.base version uses: #ifdef _LP64 typedef long jlong; #else typedef long long jlong; #endif but s390 uses: typedef long int jlong; My best assumption here is that we're only really using s390x (the 64-bit version) and that "long int" is functionally equivalent to "long". Or that jni is broken on s390 and nobody really noticed. Also, s390 uses a simplified version of the gcc attribute dance used for JNIEXPORT/JNIIMPORT. I think the s390 version is just not really updated, and that using the newer in java.base is harmless. For jni_arm.h: Finally, the jni_arm.h (the 32-bit formerly closed Oracle port), the JNIEXPORT/JNIIMPORT is different, by defining __attribute__((externally_visible)). This might have been relevant due to compile/link time symbol processing for that platform, but technically it should probably not have been connected to the platform per se, but rather to the compilation procedure. Since the arm-32 platform is kind of abandoned right now, I propose to modify the compilation to be more standard, if this is required, rather than keeping these attributes. /Magnus From tomas.kalibera at gmail.com Wed Oct 18 13:12:19 2017 From: tomas.kalibera at gmail.com (Tomas Kalibera) Date: Wed, 18 Oct 2017 15:12:19 +0200 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: <86b5ffc2-b859-1fa2-5d83-cea892d5037e@oracle.com> References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> <86b5ffc2-b859-1fa2-5d83-cea892d5037e@oracle.com> Message-ID: Hi David, On 10/18/2017 08:58 AM, david.holmes at oracle.com (David Holmes) wrote: > On 18/10/2017 4:14 PM, Thomas St?fe wrote: >> Hi David, >> >> On Tue, Oct 17, 2017 at 11:00 PM, David Holmes > > wrote: >> >> Hi Thomas, >> >> On 18/10/2017 1:11 AM, Thomas St?fe wrote: >> >> Hi David, >> >> I understand that you want to have a quick solution to a >> specific problem. I do not like adding a switch with such a >> narrow usage and would prefer, like Robin, a more well rounded >> and more general solution. >> >> >> To have a "more well rounded and more general solution" you first >> need a more general problem. :) >> >> >> Sure :) >> >> The main example I was thinking of was the desire to attach native >> threads which have a different stack size than Xss - however, after >> inspecting the code, I see that this point is moot (maybe always was) as >> the VM does already the correct thing by placing the VM guards at the >> end of the real stack size, not at Xss. > Right. Xss is for threads created by the VM - and the launcher also uses > it for the thread that launches the VM and runs main. The relationship > between Xss and the use of the primordial thread to launch the VM has > been where things got messy.** > > ** IIRC the argument was to artificially make it appear as-if the > primordial thread had the same stack size as all other created threads > so that you wouldn't get in a situation where a computation would pass > on the "main thread" and fail with StackOverflowError on any other > thread. I think it was after this that we just bit the bullet and had > the launcher always kick-off a new thread to become the "main thread". > >> The other situations I could think of are cases where the thread stack >> lives in memory which is not protectable, or at least not protectable >> the way hotspot does it, using mprotect. On AIX, that would be the >> primordial thread stack as well as any memory allocated with SystemV shm. > Ok. Is it possible to augment os::uses_stack_guard_pages() to report > false if current thread can't support them ? > >> Beside that, I could imagine certain embedders just not wanting the VM >> to place guards, for whatever reason. I could imagine scenarios where >> one wants to pool thread stack memory and does not want that memory >> polluted with guard pages. > I can imagine lots of things too, but we haven't had many (any?) real > requests in 20 years so ... these kinds problems tend to need immediate > solutions so people just work around them. > >> But I think now that these cases would best served with an addition to >> the invocation API, by giving JNI_CreateAttachedThread an option to skip >> vm guard creation. That would prevent normal users to fall over a >> misunderstood option. Embedders typically know better what they do. > I agree. If this were to be pursued then a modified attach API would be > the way to do it. Feel free to file an RFE if you'd like to do this. ;-) > >> I'm not at all clear what problem the ability to control stack >> guards for any attaching thread would be solving. >> >> That said, I do not see any specific problems with the proposed >> switch. >> >> >> Thank you. >> >> Not even sure if that matters, as I am not even sure I can >> review CSRs :) >> >> >> You certainly can! >> >> Q: Who should be a reviewer on a CSR proposal? >> A: One or more engineers with expertise in the areas impacted by the >> proposed change should review the CSR request and be listed as a >> reviewer before the proposal is reviewed by the CSR membership. >> (These engineers may or may not be Reviewers on the corresponding >> JDK project.) [1] >> >> >> Ah :) Okay then, consider it reviewed :) > If you really want to be a reviewer (totally up to you) please "edit" > the CSR and scroll down to the "reviewed by" box and add your OpenJDK > user name. > > I've updated the CSR to make the flag "experimental". Though I need to > check with the R folk to see if that is okay. And added some additional > text about a more general attach API. The proposal looks good to me and I think is a clean and simple solution to the problem that Java is reducing our stack on the R thread (R is single threaded). With this new option, we would have to worry neither about -Xss/ThreadStackSize reducing our stack nor about various caps like the 2M/Linux in the past or the 8M for unlimited stack now. To me the option seems potentially useful also to any other language runtime that is embedding Java. Thanks Tomas > > Thanks, > David > >> ..Thomas >> >> Thanks, >> David >> >> [1] https://wiki.openjdk.java.net/display/csr/CSR+FAQs >> >> >> Kind Regards, Thomas >> >> >> >> On Tue, Oct 17, 2017 at 3:00 PM, David Holmes >> >> > >> wrote: >> >> ? ? On 17/10/2017 10:41 PM, Robbin Ehn wrote: >> >> ? ? ? ? On 10/17/2017 12:29 PM, David Holmes wrote: >> >> ? ? ? ? ? ? On 17/10/2017 7:28 PM, Robbin Ehn wrote: >> >> ? ? ? ? ? ? ? ? ? ? My understanding from prior discussion (ie >> when we >> ? ? ? ? ? ? ? ? ? ? fixed the 2MB stack limit) is that it >> doesn't work >> ? ? ? ? ? ? ? ? ? ? for them to use a new thread for the JVM. >> The change >> ? ? ? ? ? ? ? ? ? ? to support anything but unlimited stack >> size also >> ? ? ? ? ? ? ? ? ? ? helped (and the 8M limit when 'unlimited' is >> ? ? ? ? ? ? ? ? ? ? tolerable) but really they just want a >> simple way to >> ? ? ? ? ? ? ? ? ? ? say "please don't bother with stack guards, >> I can >> ? ? ? ? ? ? ? ? ? ? deal with that myself". >> >> >> ? ? ? ? ? ? ? ? Is there a problem I'm not seeing to extends >> this to all >> ? ? ? ? ? ? ? ? attaching threads? >> ? ? ? ? ? ? ? ? E.g. -XX:+DisableAttachingThreadGuardPages >> (more useful >> ? ? ? ? ? ? ? ? option I would say) >> >> >> ? ? ? ? ? ? Yes - that allows it to impact all of our launchers >> as well >> ? ? ? ? ? ? - not something I want to allow for. It opens the >> door for a >> ? ? ? ? ? ? myriad of bug reports if people use this flag without >> ? ? ? ? ? ? understanding it's consequences. >> >> ? ? ? ? ? ? The aim here is to solve a specific problem, not >> introduce >> ? ? ? ? ? ? new ways to break things. >> >> >> ? ? ? ? First, since this is broken, you would fix the issue with >> ? ? ? ? different stacksizes. >> >> >> ? ? Not sure what you consider "broken". The current approach >> causes a >> ? ? problem for runtimes that want to do their own stack >> guards. The >> ? ? proposal is to allow them by giving a flag to turn ours off >> - but >> ? ? only for the primordial thread, as that is where the >> problem lies. >> >> ? ? ? ? And secondly, I think the solution to that is to move >> all 'text' >> ? ? ? ? options to the launcher and have the CreateVM taking a data >> ? ? ? ? structure with pre-parsed options. And let the launcher >> choose >> ? ? ? ? which options to expose to the end-user. Meaning our >> own java >> ? ? ? ? launcher would never expose this option. (but this is much >> ? ? ? ? bigger discussion) >> >> >> ? ? Let's not start to envisage a complete re-design of how you >> might >> ? ? communicate arguments between a host process and the VM. >> Please. >> >> ? ? ? ? Would the option would skip adding guard to any other >> thread >> ? ? ? ? calling CreateVM? >> ? ? ? ? If not, have the option really the correct name? >> >> ? ? ? ? E.g. -XX:+DisableCreateVMThreadGuardPages ? >> >> >> ? ? The option would ONLY skip for the PRIMORDIAL thread of the >> process >> ? ? when it loads the VM - hence it has exactly the right name. I >> ? ? specifically do not want to allow this for an arbitrary >> thread that >> ? ? happens to do the loading of the VM. >> >> ? ? Thanks, >> ? ? David >> >> >> ? ? ? ? Thanks, Robbin >> >> >> ? ? ? ? ? ? David >> >> ? ? ? ? ? ? ? ? Thanks, Robbin >> >> >> ? ? ? ? ? ? ? ? ? ? This is intended to be a very simple, very >> specific >> ? ? ? ? ? ? ? ? ? ? proposal. I suppose it could be flagged as >> ? ? ? ? ? ? ? ? ? ? "experimental" if we wanted the flexibility >> to do >> ? ? ? ? ? ? ? ? ? ? something different later. But I don't see that >> ? ? ? ? ? ? ? ? ? ? being on the cards. >> >> ? ? ? ? ? ? ? ? ? ? Cheers, >> ? ? ? ? ? ? ? ? ? ? David >> >> >> ? ? ? ? ? ? ? ? ? ? ? ? Thanks Robbin >> >> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? Kind Regards, Thomas >> >> >> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? On Tue, Oct 17, 2017 at 9:22 AM, David >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? Holmes > >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? > >> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? wrote: >> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? CSR: >> https://bugs.openjdk.java.net/browse/JDK-8189423 >> >> >> > > >> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Bug: >> https://bugs.openjdk.java.net/browse/JDK-8189170 >> >> >> > > >> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Could I please have a reviewer >> for this >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? CSR request so I can fast-track it. >> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Comments on the proposal are of >> course >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? welcome. >> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Thanks, >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? David >> >> >> From harold.seigel at oracle.com Wed Oct 18 14:20:58 2017 From: harold.seigel at oracle.com (harold seigel) Date: Wed, 18 Oct 2017 10:20:58 -0400 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> Message-ID: Hi David, Thanks for looking at this. Please see inline comments. Thanks, Harold On 10/17/2017 10:22 PM, David Holmes wrote: > Hi Harold, > > On 17/10/2017 12:20 AM, harold seigel wrote: >> Hi, >> >> Please review this new JDK-10 fix for bug JDK-8174954.? If the >> initial resolution attempt fails with a LinkageError exception then >> the fix saves the exception in the resolution_errors table and sets a >> flag in the constant pool Cache, indicating the failure.? Subsequent >> attempts to do the same resolution will see that the flag is set, >> retrieve the exception from the resolution_errors table, and throw >> it, instead of re-trying the resolution. > > I'm a little confused. The fix seems all about indy, which seems to be > the topic of: > > ?JDK-8174942 Bootstrap Method Called Multiple Times Despite Initial > Resolution Failure > > whereas this bug seems to be about a more general problem. Is this fix > addressing both issues?? Thanks for noticing this.? This fix addresses both bugs.? The sample program for 8174942 is one of the test cases for this fix.? I'll close 8174942 as a duplicate. > > BTW there's no reason for JDK-8174954 to remain confidential. Agreed.? I opened the bug. > > Thanks, > David > >> The fix also prevents a race condition if two or more threads try to >> do the same resolution concurrently. When a thread tries to record >> the result of its resolution attempt, it will check to see if another >> thread recorded its result first.? If so, then it will use the result >> recorded by the other thread.? Otherwise, it will record and use its >> result. Access to the resolution result is protected by the >> 'resolved_references' lock. >> >> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ >> >> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 >> >> The fix was tested with the JCK Lang and VM tests, the JTreg hotspot, >> java/io, java/lang, java/util, and other tests, the co-located NSK >> tests, JPRT, with Mach5 tier2 - tier5 tests, and by hand to check >> race condition handling. >> >> Thanks, Harold >> From rkennke at redhat.com Wed Oct 18 14:48:37 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 18 Oct 2017 16:48:37 +0200 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: <59E70E49.4040207@oracle.com> References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> <59E70E49.4040207@oracle.com> Message-ID: Am 18.10.2017 um 10:18 schrieb Erik ?sterlund: > Hi Roman, > > On 2017-10-17 23:34, Roman Kennke wrote: >> Am 17.10.2017 um 23:10 schrieb Roman Kennke: >>> >>>> The SuspendibleThreadSet API for synchronizing any non-Java thread >>>> with safepoints currently resides under gc/g1. It is very useful >>>> for other GCs too (in particular, Shenandoah does use it too), so I >>>> wanted to move it to a common location like gc/common. Then Kim >>>> Barrett commented that it might actually be useful for other >>>> threads outside GC land and to put it under runtime/. So I did: >>>> >>>> http://cr.openjdk.java.net/~rkennke/8189276/webrev.00/ >>>> >>>> >>>> I also added a generic hook to call the STS from safepoint >>>> sync/desync, which is consequently used by G1 now. In other words, >>>> the CollectedHeap API that Erik ? introduced is no longer used by >>>> G1. Only CMS still uses that API because it has its own way to sync >>>> with safepoints. I filed another bug >>>> for this. >>>> Although I have my doubt it will ever be fixed. This seems to have >>>> been very carefully evolved (to put it positive), and the risk of >>>> breaking it is relatively high, and thus doesn't seem worth the >>>> struggle to make it fit the STS. >>>> >>>> Issue: >>>> https://bugs.openjdk.java.net/browse/JDK-8189276 >>>> >>>> What do you think? Ok to go in? >>>> >>> Replying to myself here. >>> I must admit that I am a bit reluctant to expose it to runtime/ >>> unless there's a specific need for it. So maybe go back to the >>> original plan to move it into gc/common and leave all the rest alone >>> for now? What do others think? >> >> Sorry, forgot to add something. One reason why we originally wanted >> to move it to gc/common (a new directory) instead of the existing >> gc/shared was that Erik H objected that gc/shared would be built into >> the minimal VM, and we probably wouldn't want that unless needed. >> However, moving it to runtime/ (without actual need), would achieve >> just that. So unless we can name an actual thread that would benefit >> from synchronizing using the STS, I suggest to leave the STS code in >> gc/common for now? >> >> That would be: >> http://cr.openjdk.java.net/~rkennke/8189276/webrev.01/ >> > > We had some internal discussions, and the conclusion seems to be that > we would like to not have a new gc/common directory for build-only > purposes, but rather put it in gc/shared and explicitly mark files we > do not want to have compiled in minimal VM in the > make/hotspot/lib/JvmFeatures.gmk build file. You can remove files from > the minimal VM build by explicitly adding files to an exclude list in > that file. > > 133:? JVM_EXCLUDE_FILES += \ > 134: ???? concurrentGCThread.cpp \ > +++ add your files here > 135: ???? plab.cpp > > I tend to agree that avoiding the new gc/common directory is probably > a win right now. Hope you agree with this. Sure. This sounds like the best solution so far. Basically back to my original plan :-) http://cr.openjdk.java.net/~rkennke/8189276/webrev.02/ Haven't tested minimal build though. Roman From zgu at redhat.com Wed Oct 18 14:54:07 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 18 Oct 2017 10:54:07 -0400 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> <59E70E49.4040207@oracle.com> Message-ID: <57477642-590a-da89-901b-2513b325bf49@redhat.com> Hi Roman, This looks much better. Please fix this: src/hotspot/share/gc/shared/suspendibleThreadSet.hpp 25 #ifndef SHARE_GC_COMMON_SUSPENDIBLETHREADSET_HPP 26 #define SHARE_GC_COMMON_SUSPENDIBLETHREADSET_HPP Thanks, -Zhengyu >> I tend to agree that avoiding the new gc/common directory is probably >> a win right now. Hope you agree with this. > Sure. This sounds like the best solution so far. Basically back to my > original plan :-) > > http://cr.openjdk.java.net/~rkennke/8189276/webrev.02/ > > > Haven't tested minimal build though. > > Roman > From rkennke at redhat.com Wed Oct 18 14:54:28 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 18 Oct 2017 16:54:28 +0200 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> <59E70E49.4040207@oracle.com> Message-ID: Am 18.10.2017 um 16:48 schrieb Roman Kennke: > Am 18.10.2017 um 10:18 schrieb Erik ?sterlund: >> Hi Roman, >> >> On 2017-10-17 23:34, Roman Kennke wrote: >>> Am 17.10.2017 um 23:10 schrieb Roman Kennke: >>>> >>>>> The SuspendibleThreadSet API for synchronizing any non-Java thread >>>>> with safepoints currently resides under gc/g1. It is very useful >>>>> for other GCs too (in particular, Shenandoah does use it too), so >>>>> I wanted to move it to a common location like gc/common. Then Kim >>>>> Barrett commented that it might actually be useful for other >>>>> threads outside GC land and to put it under runtime/. So I did: >>>>> >>>>> http://cr.openjdk.java.net/~rkennke/8189276/webrev.00/ >>>>> >>>>> >>>>> I also added a generic hook to call the STS from safepoint >>>>> sync/desync, which is consequently used by G1 now. In other words, >>>>> the CollectedHeap API that Erik ? introduced is no longer used by >>>>> G1. Only CMS still uses that API because it has its own way to >>>>> sync with safepoints. I filed another bug >>>>> for this. >>>>> Although I have my doubt it will ever be fixed. This seems to have >>>>> been very carefully evolved (to put it positive), and the risk of >>>>> breaking it is relatively high, and thus doesn't seem worth the >>>>> struggle to make it fit the STS. >>>>> >>>>> Issue: >>>>> https://bugs.openjdk.java.net/browse/JDK-8189276 >>>>> >>>>> What do you think? Ok to go in? >>>>> >>>> Replying to myself here. >>>> I must admit that I am a bit reluctant to expose it to runtime/ >>>> unless there's a specific need for it. So maybe go back to the >>>> original plan to move it into gc/common and leave all the rest >>>> alone for now? What do others think? >>> >>> Sorry, forgot to add something. One reason why we originally wanted >>> to move it to gc/common (a new directory) instead of the existing >>> gc/shared was that Erik H objected that gc/shared would be built >>> into the minimal VM, and we probably wouldn't want that unless >>> needed. However, moving it to runtime/ (without actual need), would >>> achieve just that. So unless we can name an actual thread that would >>> benefit from synchronizing using the STS, I suggest to leave the STS >>> code in gc/common for now? >>> >>> That would be: >>> http://cr.openjdk.java.net/~rkennke/8189276/webrev.01/ >>> >> >> We had some internal discussions, and the conclusion seems to be that >> we would like to not have a new gc/common directory for build-only >> purposes, but rather put it in gc/shared and explicitly mark files we >> do not want to have compiled in minimal VM in the >> make/hotspot/lib/JvmFeatures.gmk build file. You can remove files >> from the minimal VM build by explicitly adding files to an exclude >> list in that file. >> >> 133:? JVM_EXCLUDE_FILES += \ >> 134: ???? concurrentGCThread.cpp \ >> +++ add your files here >> 135: ???? plab.cpp >> >> I tend to agree that avoiding the new gc/common directory is probably >> a win right now. Hope you agree with this. > Sure. This sounds like the best solution so far. Basically back to my > original plan :-) > > http://cr.openjdk.java.net/~rkennke/8189276/webrev.02/ > > Sorry, forget that patch. I uploaded the wrong webrev. Here's the correct one: http://cr.openjdk.java.net/~rkennke/8189276/webrev.03/ Roman From rkennke at redhat.com Wed Oct 18 14:57:25 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 18 Oct 2017 16:57:25 +0200 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: <57477642-590a-da89-901b-2513b325bf49@redhat.com> References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> <59E70E49.4040207@oracle.com> <57477642-590a-da89-901b-2513b325bf49@redhat.com> Message-ID: <5d643c1a-ebae-202f-fc38-254bbf15be9a@redhat.com> Hi Zhengyu, thanks for reviewing. I changed to SHARE_GC_SHARED_... : http://cr.openjdk.java.net/~rkennke/8189276/webrev.04/ Good now? Roman > Hi Roman, > > This looks much better. > > Please fix this: > > src/hotspot/share/gc/shared/suspendibleThreadSet.hpp > > 25 #ifndef SHARE_GC_COMMON_SUSPENDIBLETHREADSET_HPP > 26 #define SHARE_GC_COMMON_SUSPENDIBLETHREADSET_HPP > > > Thanks, > > -Zhengyu > >>> I tend to agree that avoiding the new gc/common directory is >>> probably a win right now. Hope you agree with this. >> Sure. This sounds like the best solution so far. Basically back to my >> original plan :-) >> >> http://cr.openjdk.java.net/~rkennke/8189276/webrev.02/ >> >> >> Haven't tested minimal build though. >> >> Roman >> From zgu at redhat.com Wed Oct 18 15:00:38 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 18 Oct 2017 11:00:38 -0400 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: <5d643c1a-ebae-202f-fc38-254bbf15be9a@redhat.com> References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> <59E70E49.4040207@oracle.com> <57477642-590a-da89-901b-2513b325bf49@redhat.com> <5d643c1a-ebae-202f-fc38-254bbf15be9a@redhat.com> Message-ID: Looks good to me. -Zhengyu On 10/18/2017 10:57 AM, Roman Kennke wrote: > Hi Zhengyu, > > thanks for reviewing. I changed to SHARE_GC_SHARED_... : > > http://cr.openjdk.java.net/~rkennke/8189276/webrev.04/ > > > Good now? > > Roman > >> Hi Roman, >> >> This looks much better. >> >> Please fix this: >> >> src/hotspot/share/gc/shared/suspendibleThreadSet.hpp >> >> 25 #ifndef SHARE_GC_COMMON_SUSPENDIBLETHREADSET_HPP >> 26 #define SHARE_GC_COMMON_SUSPENDIBLETHREADSET_HPP >> >> >> Thanks, >> >> -Zhengyu >> >>>> I tend to agree that avoiding the new gc/common directory is >>>> probably a win right now. Hope you agree with this. >>> Sure. This sounds like the best solution so far. Basically back to my >>> original plan :-) >>> >>> http://cr.openjdk.java.net/~rkennke/8189276/webrev.02/ >>> >>> >>> Haven't tested minimal build though. >>> >>> Roman >>> > From coleen.phillimore at oracle.com Wed Oct 18 17:50:00 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 18 Oct 2017 13:50:00 -0400 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> <59E70E49.4040207@oracle.com> <57477642-590a-da89-901b-2513b325bf49@redhat.com> <5d643c1a-ebae-202f-fc38-254bbf15be9a@redhat.com> Message-ID: This looks good.? It builds the minimal vm fine. Need a sponsor? thanks, Coleen On 10/18/17 11:00 AM, Zhengyu Gu wrote: > Looks good to me. > > -Zhengyu > > On 10/18/2017 10:57 AM, Roman Kennke wrote: >> Hi Zhengyu, >> >> thanks for reviewing. I changed to SHARE_GC_SHARED_... : >> >> http://cr.openjdk.java.net/~rkennke/8189276/webrev.04/ >> >> >> Good now? >> >> Roman >> >>> Hi Roman, >>> >>> This looks much better. >>> >>> Please fix this: >>> >>> src/hotspot/share/gc/shared/suspendibleThreadSet.hpp >>> >>> 25 #ifndef SHARE_GC_COMMON_SUSPENDIBLETHREADSET_HPP >>> 26 #define SHARE_GC_COMMON_SUSPENDIBLETHREADSET_HPP >>> >>> >>> Thanks, >>> >>> -Zhengyu >>> >>>>> I tend to agree that avoiding the new gc/common directory is >>>>> probably a win right now. Hope you agree with this. >>>> Sure. This sounds like the best solution so far. Basically back to >>>> my original plan :-) >>>> >>>> http://cr.openjdk.java.net/~rkennke/8189276/webrev.02/ >>>> >>>> >>>> Haven't tested minimal build though. >>>> >>>> Roman >>>> >> From rkennke at redhat.com Wed Oct 18 17:58:49 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 18 Oct 2017 19:58:49 +0200 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> <3a72da30-b74f-a65e-1f06-adefb0e610c8@redhat.com> <59E70E49.4040207@oracle.com> <57477642-590a-da89-901b-2513b325bf49@redhat.com> <5d643c1a-ebae-202f-fc38-254bbf15be9a@redhat.com> Message-ID: <1c22a0cc-cd1b-7224-03ba-851033ae454a@redhat.com> Hi Coleen, yes, please! Thanks! Roman > This looks good.? It builds the minimal vm fine. > Need a sponsor? > thanks, > Coleen > > On 10/18/17 11:00 AM, Zhengyu Gu wrote: >> Looks good to me. >> >> -Zhengyu >> >> On 10/18/2017 10:57 AM, Roman Kennke wrote: >>> Hi Zhengyu, >>> >>> thanks for reviewing. I changed to SHARE_GC_SHARED_... : >>> >>> http://cr.openjdk.java.net/~rkennke/8189276/webrev.04/ >>> >>> >>> Good now? >>> >>> Roman >>> >>>> Hi Roman, >>>> >>>> This looks much better. >>>> >>>> Please fix this: >>>> >>>> src/hotspot/share/gc/shared/suspendibleThreadSet.hpp >>>> >>>> 25 #ifndef SHARE_GC_COMMON_SUSPENDIBLETHREADSET_HPP >>>> 26 #define SHARE_GC_COMMON_SUSPENDIBLETHREADSET_HPP >>>> >>>> >>>> Thanks, >>>> >>>> -Zhengyu >>>> >>>>>> I tend to agree that avoiding the new gc/common directory is >>>>>> probably a win right now. Hope you agree with this. >>>>> Sure. This sounds like the best solution so far. Basically back to >>>>> my original plan :-) >>>>> >>>>> http://cr.openjdk.java.net/~rkennke/8189276/webrev.02/ >>>>> >>>>> >>>>> Haven't tested minimal build though. >>>>> >>>>> Roman >>>>> >>> > From zgu at redhat.com Wed Oct 18 18:31:59 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 18 Oct 2017 14:31:59 -0400 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: References: <3577bda2-7fdd-d458-99b1-79f0ebd72bc2@redhat.com> Message-ID: <11c6536b-2d83-15c5-5205-bdf055e147f2@redhat.com> Good to me. -Zhengyu On 10/17/2017 05:21 PM, Roman Kennke wrote: > Hi Zhengyu, > > thanks for reviewing! > >> Hi Roman, >> >> Look good overall. >> >> 1) Probably make sense to factor out static methods from GC into >> separate GCFactory class. > Right, this looks better. > >> 2) Stylish: >> #ifndef SHARE_VM_GC_G1_G1_GC_HPP -> #ifndef SHARE_VM_GC_G1_G1GC_HPP > Ok, right. I also removed the '_VM' parts to account for the changed > directory structure in mono-repo. > > Differential: > http://cr.openjdk.java.net/~rkennke/8189171/webrev.01.diff/ > > Full: > http://cr.openjdk.java.net/~rkennke/8189171/webrev.01/ > > > Better now? > > Roman > >> >> Thanks, >> >> -Zhengyu >> >> On 10/12/2017 03:59 PM, Roman Kennke wrote: >>> I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev because >>> it touches both areas (i.e. the GC interface). >>> >>> Currently, all GC related argument processing is done in >>> arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all sorts >>> of GC specific methods etc. >>> >>> In order to have a cleaner GC interface, I propose to move all that >>> code into GC specific classes under their GC specific directories. >>> >>> Since at this point in time we haven't created the XXXHeap subclasses >>> yet (and cannot, before having set up all the flags and arguments), I >>> propose to create a GC interface of which we create an instance as >>> soon as we have selected a GC, and then do argument processing there. >>> >>> This 'GC' interface is, at this point, meant as a sort of GC factory >>> interface. With this patch, it will only do argument processing. >>> Later I plan to add heap creation to it (which is currently done in >>> universe.cpp, with INCLUDE_ALL_GCS stuff and all the if (UseBlahGC) >>> .. else if .. else branch chains. My current prototype in the jdk10 >>> sandbox also provides servicabilty support classes and some more >>> stuff, but we should decide later where to put this. >>> >>> I broke out some code paths into gc_factory.cpp (i.e. not in gc.cpp). >>> Those are all the code paths that select GC, create the right >>> instances, etc, in other words, the code paths that need to know >>> about all GCs. Ideally, this should be the only file that needs to >>> know about all the GCs. (Suggestions for improvement are welcome of >>> course.) >>> >>> I tested this by running hotspot_gc jtreg tests, with normal >>> fastdebug and no-precompiled configurations. Everything passes fine. >>> >>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.00/ >>> >>> >>> Opinions? >>> >>> Roman >>> > From kim.barrett at oracle.com Wed Oct 18 20:35:53 2017 From: kim.barrett at oracle.com (Kim Barrett) Date: Wed, 18 Oct 2017 16:35:53 -0400 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> Message-ID: > On Oct 17, 2017, at 5:10 PM, Roman Kennke wrote: > > >> The SuspendibleThreadSet API for synchronizing any non-Java thread with safepoints currently resides under gc/g1. It is very useful for other GCs too (in particular, Shenandoah does use it too), so I wanted to move it to a common location like gc/common. Then Kim Barrett commented that it might actually be useful for other threads outside GC land and to put it under runtime/. So I did: >> >> http://cr.openjdk.java.net/~rkennke/8189276/webrev.00/ >> >> I also added a generic hook to call the STS from safepoint sync/desync, which is consequently used by G1 now. In other words, the CollectedHeap API that Erik ? introduced is no longer used by G1. Only CMS still uses that API because it has its own way to sync with safepoints. I filed another bug for this. Although I have my doubt it will ever be fixed. This seems to have been very carefully evolved (to put it positive), and the risk of breaking it is relatively high, and thus doesn't seem worth the struggle to make it fit the STS. >> Issue: >> https://bugs.openjdk.java.net/browse/JDK-8189276 >> >> What do you think? Ok to go in? >> > Replying to myself here. > I must admit that I am a bit reluctant to expose it to runtime/ unless there's a specific need for it. So maybe go back to the original plan to move it into gc/common and leave all the rest alone for now? What do others think? > > Roman My reasoning for suggesting it belongs in runtime is that I think it is a part of the API for SafepointSynchronize. The fact that it was backed in via GC (and in a really ugly way, initially) doesn't change my opinion about that. But since runtime folks all seem to be saying "nope, never heard of it, don't need it for anything non-GC, don't want any responsibility for it", I guess keeping it a GC thing is what we'll do. But I still think it's strange that SafepointSynchronize is calling into GC code. (With the CMS mechanism being its own additional oddity.) From mandy.chung at oracle.com Wed Oct 18 21:30:56 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 18 Oct 2017 14:30:56 -0700 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader Message-ID: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> This is a regression caused by JDK-8188052. FindClass should only use system class loader when there is no caller class or when it's called from JNI_OnUnload.? When there is a caller class, it should either use its class loader or the one associated with JNI_Onload. Webrev: http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 thanks Mandy From david.holmes at oracle.com Wed Oct 18 21:48:01 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Oct 2017 07:48:01 +1000 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> Message-ID: Hi Mandy, Do you still have the webrev for 8188052? I need to see how this got through given we looked at it so many times. :( Thanks, David On 19/10/2017 7:30 AM, mandy chung wrote: > This is a regression caused by JDK-8188052. FindClass should only use > system class loader when there is no caller class or when it's called > from JNI_OnUnload.? When there is a caller class, it should either use > its class loader or the one associated with JNI_Onload. > > Webrev: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ > > JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 > > thanks > Mandy From mandy.chung at oracle.com Wed Oct 18 21:52:06 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 18 Oct 2017 14:52:06 -0700 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> Message-ID: <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> The bug creeped in when I improved the code to use loader.is_null() to set to system class loader per Coleen's suggestion.? It's my oversight of the null loader case.? The is the version I pushed. http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ The previous version was correct. http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ I didn't go back to webrev.01.? The fix is an improvement to it. Mandy On 10/18/17 2:48 PM, David Holmes wrote: > Hi Mandy, > > Do you still have the webrev for 8188052? I need to see how this got > through given we looked at it so many times. :( > > Thanks, > David > > On 19/10/2017 7:30 AM, mandy chung wrote: >> This is a regression caused by JDK-8188052. FindClass should only use >> system class loader when there is no caller class or when it's called >> from JNI_OnUnload. When there is a caller class, it should either use >> its class loader or the one associated with JNI_Onload. >> >> Webrev: >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >> >> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >> >> thanks >> Mandy From david.holmes at oracle.com Wed Oct 18 21:57:13 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Oct 2017 07:57:13 +1000 Subject: CSR Review: 8189423: Add option to disable stack overflow checking in primordial thread for use with JNI_CreateJavaJVM In-Reply-To: References: <1827e535-2c86-4cf9-5a3b-6d4748f1a022@oracle.com> <922b23c2-7cdf-f940-202d-e00cdd208af7@oracle.com> <70b63d61-f2e2-3fda-053c-d68ba36f8256@oracle.com> <6e19a11f-3b71-ca3a-44db-12cb65ce4f88@oracle.com> <64529d12-7a97-5c3a-d083-10058005d90b@oracle.com> <86b5ffc2-b859-1fa2-5d83-cea892d5037e@oracle.com> Message-ID: Thanks Tomas! David On 18/10/2017 11:12 PM, Tomas Kalibera wrote: > > Hi David, > > On 10/18/2017 08:58 AM, david.holmes at oracle.com (David Holmes) wrote: >> On 18/10/2017 4:14 PM, Thomas St?fe wrote: >>> Hi David, >>> >>> On Tue, Oct 17, 2017 at 11:00 PM, David Holmes >> oracle.com >>> > wrote: >>> >>> ???? Hi Thomas, >>> >>> ???? On 18/10/2017 1:11 AM, Thomas St?fe wrote: >>> >>> ???????? Hi David, >>> >>> ???????? I understand that you want to have a quick solution to a >>> ???????? specific problem. I do not like adding a switch with such a >>> ???????? narrow usage and would prefer, like Robin, a more well rounded >>> ???????? and more general solution. >>> >>> >>> ???? To have a "more well rounded and more general solution" you first >>> ???? need a more general problem. :) >>> >>> >>> Sure :) >>> >>> The main example I was thinking of was the desire to attach native >>> threads which have a different stack size than Xss - however, after >>> inspecting the code, I see that this point is moot (maybe always was) as >>> the VM does already the correct thing by placing the VM guards at the >>> end of the real stack size, not at Xss. >> Right. Xss is for threads created by the VM - and the launcher also uses >> it for the thread that launches the VM and runs main. The relationship >> between Xss and the use of the primordial thread to launch the VM has >> been where things got messy.** >> >> ** IIRC the argument was to artificially make it appear as-if the >> primordial thread had the same stack size as all other created threads >> so that you wouldn't get in a situation where a computation would pass >> on the "main thread" and fail with StackOverflowError on any other >> thread. I think it was after this that we just bit the bullet and had >> the launcher always kick-off a new thread to become the "main thread". >> >>> The other situations I could think of are cases where the thread stack >>> lives in memory which is not protectable, or at least not protectable >>> the way hotspot does it, using mprotect. On AIX, that would be the >>> primordial thread stack as well as any memory allocated with SystemV >>> shm. >> Ok. Is it possible to augment os::uses_stack_guard_pages() to? report >> false if current thread can't support them ? >> >>> Beside that, I could imagine certain embedders just not wanting the VM >>> to place guards, for whatever reason. I could imagine scenarios where >>> one wants to pool thread stack memory and does not want that memory >>> polluted with guard pages. >> I can imagine lots of things too, but we haven't had many (any?) real >> requests in 20 years so ... these kinds problems tend to need immediate >> solutions so people just work around them. >> >>> But I think now that these cases would best served with an addition to >>> the invocation API, by giving JNI_CreateAttachedThread an option to skip >>> vm guard creation. That would prevent normal users to fall over a >>> misunderstood option. Embedders typically know better what they do. >> I agree. If this were to be pursued then a modified attach API would be >> the way to do it. Feel free to file an RFE if you'd like to do this. ;-) >> >>> ???? I'm not at all clear what problem the ability to control stack >>> ???? guards for any attaching thread would be solving. >>> >>> ???????? That said, I do not see any specific problems with the proposed >>> ???????? switch. >>> >>> >>> ???? Thank you. >>> >>> ???????? Not even sure if that matters, as I am not even sure I can >>> ???????? review CSRs :) >>> >>> >>> ???? You certainly can! >>> >>> ???? Q: Who should be a reviewer on a CSR proposal? >>> ???? A: One or more engineers with expertise in the areas impacted by >>> the >>> ???? proposed change should review the CSR request and be listed as a >>> ???? reviewer before the proposal is reviewed by the CSR membership. >>> ???? (These engineers may or may not be Reviewers on the corresponding >>> ???? JDK project.) [1] >>> >>> >>> Ah :) Okay then, consider it reviewed :) >> If you really want to be a reviewer (totally up to you) please "edit" >> the CSR and scroll down to the "reviewed by" box and add your OpenJDK >> user name. >> >> I've updated the CSR to make the flag "experimental". Though I need to >> check with the R folk to see if that is okay. And added some additional >> text about a more general attach API. > The proposal looks good to me and I think is a clean and simple solution > to the problem that Java is reducing our stack on the R thread (R is > single threaded). With this new option, we would have to worry neither > about -Xss/ThreadStackSize reducing our stack nor about various caps > like the 2M/Linux in the past or the 8M for unlimited stack now. To me > the option seems potentially useful also to any other language runtime > that is embedding Java. > > Thanks > Tomas > > > >> >> Thanks, >> David >> >>> ..Thomas >>> >>> ???? Thanks, >>> ???? David >>> >>> ???? [1] https://wiki.openjdk.java.net/display/csr/CSR+FAQs >>> ???? >>> >>> ???????? Kind Regards, Thomas >>> >>> >>> >>> ???????? On Tue, Oct 17, 2017 at 3:00 PM, David Holmes >>> ???????? >>> ???????? >> ???????? >> wrote: >>> >>> ????????? ? ? On 17/10/2017 10:41 PM, Robbin Ehn wrote: >>> >>> ????????? ? ? ? ? On 10/17/2017 12:29 PM, David Holmes wrote: >>> >>> ????????? ? ? ? ? ? ? On 17/10/2017 7:28 PM, Robbin Ehn wrote: >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? My understanding from prior discussion (ie >>> ???????? when we >>> ????????? ? ? ? ? ? ? ? ? ? ? fixed the 2MB stack limit) is that it >>> ???????? doesn't work >>> ????????? ? ? ? ? ? ? ? ? ? ? for them to use a new thread for the JVM. >>> ???????? The change >>> ????????? ? ? ? ? ? ? ? ? ? ? to support anything but unlimited stack >>> ???????? size also >>> ????????? ? ? ? ? ? ? ? ? ? ? helped (and the 8M limit when >>> 'unlimited' is >>> ????????? ? ? ? ? ? ? ? ? ? ? tolerable) but really they just want a >>> ???????? simple way to >>> ????????? ? ? ? ? ? ? ? ? ? ? say "please don't bother with stack >>> guards, >>> ???????? I can >>> ????????? ? ? ? ? ? ? ? ? ? ? deal with that myself". >>> >>> >>> ????????? ? ? ? ? ? ? ? ? Is there a problem I'm not seeing to extends >>> ???????? this to all >>> ????????? ? ? ? ? ? ? ? ? attaching threads? >>> ????????? ? ? ? ? ? ? ? ? E.g. -XX:+DisableAttachingThreadGuardPages >>> ???????? (more useful >>> ????????? ? ? ? ? ? ? ? ? option I would say) >>> >>> >>> ????????? ? ? ? ? ? ? Yes - that allows it to impact all of our >>> launchers >>> ???????? as well >>> ????????? ? ? ? ? ? ? - not something I want to allow for. It opens the >>> ???????? door for a >>> ????????? ? ? ? ? ? ? myriad of bug reports if people use this flag >>> without >>> ????????? ? ? ? ? ? ? understanding it's consequences. >>> >>> ????????? ? ? ? ? ? ? The aim here is to solve a specific problem, not >>> ???????? introduce >>> ????????? ? ? ? ? ? ? new ways to break things. >>> >>> >>> ????????? ? ? ? ? First, since this is broken, you would fix the >>> issue with >>> ????????? ? ? ? ? different stacksizes. >>> >>> >>> ????????? ? ? Not sure what you consider "broken". The current approach >>> ???????? causes a >>> ????????? ? ? problem for runtimes that want to do their own stack >>> ???????? guards. The >>> ????????? ? ? proposal is to allow them by giving a flag to turn ours >>> off >>> ???????? - but >>> ????????? ? ? only for the primordial thread, as that is where the >>> ???????? problem lies. >>> >>> ????????? ? ? ? ? And secondly, I think the solution to that is to move >>> ???????? all 'text' >>> ????????? ? ? ? ? options to the launcher and have the CreateVM >>> taking a data >>> ????????? ? ? ? ? structure with pre-parsed options. And let the >>> launcher >>> ???????? choose >>> ????????? ? ? ? ? which options to expose to the end-user. Meaning our >>> ???????? own java >>> ????????? ? ? ? ? launcher would never expose this option. (but this >>> is much >>> ????????? ? ? ? ? bigger discussion) >>> >>> >>> ????????? ? ? Let's not start to envisage a complete re-design of how >>> you >>> ???????? might >>> ????????? ? ? communicate arguments between a host process and the VM. >>> ???????? Please. >>> >>> ????????? ? ? ? ? Would the option would skip adding guard to any other >>> ???????? thread >>> ????????? ? ? ? ? calling CreateVM? >>> ????????? ? ? ? ? If not, have the option really the correct name? >>> >>> ????????? ? ? ? ? E.g. -XX:+DisableCreateVMThreadGuardPages ? >>> >>> >>> ????????? ? ? The option would ONLY skip for the PRIMORDIAL thread of >>> the >>> ???????? process >>> ????????? ? ? when it loads the VM - hence it has exactly the right >>> name. I >>> ????????? ? ? specifically do not want to allow this for an arbitrary >>> ???????? thread that >>> ????????? ? ? happens to do the loading of the VM. >>> >>> ????????? ? ? Thanks, >>> ????????? ? ? David >>> >>> >>> ????????? ? ? ? ? Thanks, Robbin >>> >>> >>> ????????? ? ? ? ? ? ? David >>> >>> ????????? ? ? ? ? ? ? ? ? Thanks, Robbin >>> >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? This is intended to be a very simple, very >>> ???????? specific >>> ????????? ? ? ? ? ? ? ? ? ? ? proposal. I suppose it could be flagged as >>> ????????? ? ? ? ? ? ? ? ? ? ? "experimental" if we wanted the >>> flexibility >>> ???????? to do >>> ????????? ? ? ? ? ? ? ? ? ? ? something different later. But I don't >>> see that >>> ????????? ? ? ? ? ? ? ? ? ? ? being on the cards. >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? Cheers, >>> ????????? ? ? ? ? ? ? ? ? ? ? David >>> >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? Thanks Robbin >>> >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Kind Regards, Thomas >>> >>> >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? On Tue, Oct 17, 2017 at 9:22 >>> AM, David >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Holmes >> ???????? >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? >> ???????? >> >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? wrote: >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? CSR: >>> ???????? https://bugs.openjdk.java.net/browse/JDK-8189423 >>> ???????? >>> ???????? >> ???????? > >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Bug: >>> ???????? https://bugs.openjdk.java.net/browse/JDK-8189170 >>> ???????? >>> ???????? >> ???????? > >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Could I please have a reviewer >>> ???????? for this >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? CSR request so I can >>> fast-track it. >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Comments on the proposal >>> are of >>> ???????? course >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? welcome. >>> >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Thanks, >>> ????????? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? David >>> >>> >>> > From coleen.phillimore at oracle.com Wed Oct 18 23:15:10 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 18 Oct 2017 19:15:10 -0400 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> Message-ID: Hi Mandy, In the wrong version, for the case where k->class_loader() is NULL at line 419 or 423, would get reset to the system class loader? I'm so sorry for the unhelpful help, I was hoping to make the logic simpler but it backfired. The new versions logic is a lot more straightforward and nice.? It takes some studying to determine that loader isn't a naked oop though so it might be safer to make it a Handle at 403.? Or else: 428 Handle loader_h = Handle(THREAD, loader); could be 428 Handle loader_h(THREAD, loader); Looks good.? Sorry again. thanks for fixing it quickly. Coleen On 10/18/17 5:52 PM, mandy chung wrote: > The bug creeped in when I improved the code to use loader.is_null() to > set to system class loader per Coleen's suggestion.? It's my oversight > of the null loader case.? The is the version I pushed. > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ > > The previous version was correct. > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ > > I didn't go back to webrev.01.? The fix is an improvement to it. > > Mandy > > On 10/18/17 2:48 PM, David Holmes wrote: >> Hi Mandy, >> >> Do you still have the webrev for 8188052? I need to see how this got >> through given we looked at it so many times. :( >> >> Thanks, >> David >> >> On 19/10/2017 7:30 AM, mandy chung wrote: >>> This is a regression caused by JDK-8188052. FindClass should only >>> use system class loader when there is no caller class or when it's >>> called from JNI_OnUnload. When there is a caller class, it should >>> either use its class loader or the one associated with JNI_Onload. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>> >>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>> >>> thanks >>> Mandy > From mandy.chung at oracle.com Wed Oct 18 23:23:22 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 18 Oct 2017 16:23:22 -0700 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> Message-ID: <40fcad8c-0fa0-0877-a538-fa3b39807812@oracle.com> On 10/18/17 4:15 PM, coleen.phillimore at oracle.com wrote: > > Hi Mandy, > > In the wrong version, for the case where k->class_loader() is NULL at > line 419 or 423, would get reset to the system class loader? Yes and that's the bug. > > I'm so sorry for the unhelpful help, I was hoping to make the logic > simpler but it backfired. > No worries.? I should have caught that myself sorry. > The new versions logic is a lot more straightforward and nice.? It > takes some studying to determine that loader isn't a naked oop though > so it might be safer to make it a Handle at 403. Let me take a look. > ? Or else: > > 428 Handle loader_h = Handle(THREAD, loader); > > > could be > > 428 Handle loader_h(THREAD, loader); > Will change to the above. > > Looks good.? Sorry again. > thanks for fixing it quickly. Thanks for your prompt review. Mandy > Coleen > > > On 10/18/17 5:52 PM, mandy chung wrote: >> The bug creeped in when I improved the code to use loader.is_null() >> to set to system class loader per Coleen's suggestion.? It's my >> oversight of the null loader case.? The is the version I pushed. >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >> >> The previous version was correct. >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ >> >> I didn't go back to webrev.01.? The fix is an improvement to it. >> >> Mandy >> >> On 10/18/17 2:48 PM, David Holmes wrote: >>> Hi Mandy, >>> >>> Do you still have the webrev for 8188052? I need to see how this got >>> through given we looked at it so many times. :( >>> >>> Thanks, >>> David >>> >>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>> This is a regression caused by JDK-8188052. FindClass should only >>>> use system class loader when there is no caller class or when it's >>>> called from JNI_OnUnload. When there is a caller class, it should >>>> either use its class loader or the one associated with JNI_Onload. >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>> >>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>> >>>> thanks >>>> Mandy >> > From ioi.lam at oracle.com Wed Oct 18 23:54:37 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 18 Oct 2017 16:54:37 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> Message-ID: On 10/17/17 9:04 PM, David Holmes wrote: > Hi Ioi, > > On 18/10/2017 1:52 PM, Ioi Lam wrote: >> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >> >> https://bugs.openjdk.java.net/browse/JDK-8185160 >> >> Summary: >> >> DumpLoadedClassList skips classes that can't be stored into the CDS >> archive. For boot and platform loaders, only classes from the JRT image >> are listed. >> >> The test for "is this class from JRT image" was insufficient. It >> checked only classes whose source ends with "...../modules". For the >> platform loader, which loads the graal classes, the source is in the >> form "jrt:/jdk.internal.vm.compiler". >> >> The fix is to also check for a "jrt:" prefix. > > So ./share/classfile/classLoader.hpp: > > static bool is_jrt(const char* name) { return string_ends_with(name, > MODULES_IMAGE_NAME); } > > is a badly named function or just plain broken? > I've renamed all the is_jrt() functions to is_modules_image(), so it's clear as to what it does. See http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >> I also restructured the code to make it easier to read -- the nesting >> of || and && is hard to follow. >> >> The original code had a subtle bug with the "!" operator: >> >> 5938?? if (((class_loader == NULL && >> !ClassLoader::contains_append_entry(stream->source())) || >> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >> 5940?????? !ClassLoader::is_jrt(stream->source())) { >> >> So if (class_loader == NULL && >> ClassLoader::contains_append_entry(stream->source()) was true, >> the class would NOT be skipped. >> >> The new version removes the "!". > > So there should be an updated test to trigger the original bug. > I added a test case, but it's in the closed repo for now (since the test case uses utilities in the closed AppCDS tests). This test will be opened as part of JDK-8185996 Thanks - Ioi > Thanks, > David > >> Thanks >> - Ioi From david.holmes at oracle.com Thu Oct 19 00:26:15 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Oct 2017 10:26:15 +1000 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> Message-ID: Hi Mandy, On 19/10/2017 7:52 AM, mandy chung wrote: > The bug creeped in when I improved the code to use loader.is_null() to > set to system class loader per Coleen's suggestion.? It's my oversight > of the null loader case.? The is the version I pushed. > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ > > The previous version was correct. > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ Yes I see what happened - and we all missed it. And there was no existing test it seems :( > I didn't go back to webrev.01.? The fix is an improvement to it. Okay, only concern I have is that this: 403 oop loader = SystemDictionary::java_system_loader(); certainly looks like a "naked oop" - and the Java call may lead to GC occurring. The test seems very lonely, where are the onLoad tests? In Driver: 30 * @run main/othervm/native what does the "native" do? I don't see it documented. Regarding this comment in BootLoaderTest: // The JNI spec says FindClass returns NULL if class not found but // it also says that NCDFE is thrown if no definition can be found. // The current implementation throws NCDFE. The NCDFE is set pending by FindClass. If you return (or call into) java code then it will be thrown. But if you were remaining in native then you'd use the NULL return to determine success or failure (or check pending exception). Thanks, David > Mandy > > On 10/18/17 2:48 PM, David Holmes wrote: >> Hi Mandy, >> >> Do you still have the webrev for 8188052? I need to see how this got >> through given we looked at it so many times. :( >> >> Thanks, >> David >> >> On 19/10/2017 7:30 AM, mandy chung wrote: >>> This is a regression caused by JDK-8188052. FindClass should only use >>> system class loader when there is no caller class or when it's called >>> from JNI_OnUnload. When there is a caller class, it should either use >>> its class loader or the one associated with JNI_Onload. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>> >>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>> >>> thanks >>> Mandy > From david.holmes at oracle.com Thu Oct 19 01:26:58 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Oct 2017 11:26:58 +1000 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> Message-ID: Hi Ioi, On 19/10/2017 9:54 AM, Ioi Lam wrote: > > > On 10/17/17 9:04 PM, David Holmes wrote: >> Hi Ioi, >> >> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>> >>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>> >>> Summary: >>> >>> DumpLoadedClassList skips classes that can't be stored into the CDS >>> archive. For boot and platform loaders, only classes from the JRT image >>> are listed. >>> >>> The test for "is this class from JRT image" was insufficient. It >>> checked only classes whose source ends with "...../modules". For the >>> platform loader, which loads the graal classes, the source is in the >>> form "jrt:/jdk.internal.vm.compiler". >>> >>> The fix is to also check for a "jrt:" prefix. >> >> So ./share/classfile/classLoader.hpp: >> >> static bool is_jrt(const char* name) { return string_ends_with(name, >> MODULES_IMAGE_NAME); } >> >> is a badly named function or just plain broken? >> > > I've renamed all the is_jrt() functions to is_modules_image(), so it's > clear as to what it does. > > See > http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ Okay. But I'm still a bit confused about the expectations of is_modules_image() - where are these names defined? What does jrt: mean if not "in a module image" ?? That aside your fix does what you needed, so looks fine. > > >>> I also restructured the code to make it easier to read -- the nesting >>> of || and && is hard to follow. >>> >>> The original code had a subtle bug with the "!" operator: >>> >>> 5938?? if (((class_loader == NULL && >>> !ClassLoader::contains_append_entry(stream->source())) || >>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>> 5940?????? !ClassLoader::is_jrt(stream->source())) { >>> >>> So if (class_loader == NULL && >>> ClassLoader::contains_append_entry(stream->source()) was true, >>> the class would NOT be skipped. >>> >>> The new version removes the "!". >> >> So there should be an updated test to trigger the original bug. >> > > I added a test case, but it's in the closed repo for now (since the test > case uses utilities in the closed AppCDS tests). This test will be > opened as part of JDK-8185996 Ok. Thanks, David > Thanks > - Ioi > >> Thanks, >> David >> >>> Thanks >>> - Ioi > From david.holmes at oracle.com Thu Oct 19 01:51:13 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Oct 2017 11:51:13 +1000 Subject: RFR: 8189276: Make SuspendibleThreadSet and related code available to other GCs In-Reply-To: References: <0b96508f-2522-5b93-a271-9fd18aaeda28@redhat.com> <2dbc333b-d020-ca62-e5ed-7d3396728e7e@redhat.com> Message-ID: <0be57ff2-84f5-6eaf-60dd-558b63e35192@oracle.com> On 19/10/2017 6:35 AM, Kim Barrett wrote: >> On Oct 17, 2017, at 5:10 PM, Roman Kennke wrote: >> >> >>> The SuspendibleThreadSet API for synchronizing any non-Java thread with safepoints currently resides under gc/g1. It is very useful for other GCs too (in particular, Shenandoah does use it too), so I wanted to move it to a common location like gc/common. Then Kim Barrett commented that it might actually be useful for other threads outside GC land and to put it under runtime/. So I did: >>> >>> http://cr.openjdk.java.net/~rkennke/8189276/webrev.00/ >>> >>> I also added a generic hook to call the STS from safepoint sync/desync, which is consequently used by G1 now. In other words, the CollectedHeap API that Erik ? introduced is no longer used by G1. Only CMS still uses that API because it has its own way to sync with safepoints. I filed another bug for this. Although I have my doubt it will ever be fixed. This seems to have been very carefully evolved (to put it positive), and the risk of breaking it is relatively high, and thus doesn't seem worth the struggle to make it fit the STS. >>> Issue: >>> https://bugs.openjdk.java.net/browse/JDK-8189276 >>> >>> What do you think? Ok to go in? >>> >> Replying to myself here. >> I must admit that I am a bit reluctant to expose it to runtime/ unless there's a specific need for it. So maybe go back to the original plan to move it into gc/common and leave all the rest alone for now? What do others think? >> >> Roman > > My reasoning for suggesting it belongs in runtime is that I think it > is a part of the API for SafepointSynchronize. The fact that it was > backed in via GC (and in a really ugly way, initially) doesn't change > my opinion about that. But since runtime folks all seem to be saying > "nope, never heard of it, don't need it for anything non-GC, don't > want any responsibility for it", I guess keeping it a GC thing is what > we'll do. But I still think it's strange that SafepointSynchronize is > calling into GC code. (With the CMS mechanism being its own > additional oddity.) I also find it strange, but this is GC code so ... if GC is doing something strange that requires the safepoint code to coordinate with it then we can oblige. That doesn't mean we want to own all that crazy GC code! ;-) Cheers, David > > From ioi.lam at oracle.com Thu Oct 19 02:32:49 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 18 Oct 2017 19:32:49 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> Message-ID: <0a9de83e-b424-1f31-9025-9498fddafcf8@oracle.com> On 10/18/17 6:26 PM, David Holmes wrote: > Hi Ioi, > > On 19/10/2017 9:54 AM, Ioi Lam wrote: >> >> >> On 10/17/17 9:04 PM, David Holmes wrote: >>> Hi Ioi, >>> >>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>> >>>> Summary: >>>> >>>> DumpLoadedClassList skips classes that can't be stored into the CDS >>>> archive. For boot and platform loaders, only classes from the JRT >>>> image >>>> are listed. >>>> >>>> The test for "is this class from JRT image" was insufficient. It >>>> checked only classes whose source ends with "...../modules". For the >>>> platform loader, which loads the graal classes, the source is in the >>>> form "jrt:/jdk.internal.vm.compiler". >>>> >>>> The fix is to also check for a "jrt:" prefix. >>> >>> So ./share/classfile/classLoader.hpp: >>> >>> static bool is_jrt(const char* name) { return string_ends_with(name, >>> MODULES_IMAGE_NAME); } >>> >>> is a badly named function or just plain broken? >>> >> >> I've renamed all the is_jrt() functions to is_modules_image(), so >> it's clear as to what it does. >> >> See >> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ > > > Okay. But I'm still a bit confused about the expectations of > is_modules_image() - where are these names defined? What does jrt: > mean if not "in a module image" ?? > > That aside your fix does what you needed, so looks fine. > The problem we have is the ClassFileParser::source()? returns "/lib/modules" for the boot loader (C code), and "jrt:/java.compiler" for the platform loader (Java code). This should probably be fixed, but I don't want to deal with that now :-( >> >> >>>> I also restructured the code to make it easier to read -- the >>>> nesting of || and && is hard to follow. >>>> >>>> The original code had a subtle bug with the "!" operator: >>>> >>>> 5938?? if (((class_loader == NULL && >>>> !ClassLoader::contains_append_entry(stream->source())) || >>>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>>> 5940?????? !ClassLoader::is_jrt(stream->source())) { >>>> >>>> So if (class_loader == NULL && >>>> ClassLoader::contains_append_entry(stream->source()) was true, >>>> the class would NOT be skipped. >>>> >>>> The new version removes the "!". >>> >>> So there should be an updated test to trigger the original bug. >>> >> >> I added a test case, but it's in the closed repo for now (since the >> test case uses utilities in the closed AppCDS tests). This test will >> be opened as part of JDK-8185996 > > Ok. > Thanks - Ioi > Thanks, > David > >> Thanks >> - Ioi >> >>> Thanks, >>> David >>> >>>> Thanks >>>> - Ioi >> From dean.long at oracle.com Thu Oct 19 03:16:41 2017 From: dean.long at oracle.com (dean.long at oracle.com) Date: Wed, 18 Oct 2017 20:16:41 -0700 Subject: Unification of jni_.h In-Reply-To: References: Message-ID: <2e45d810-d9d6-4acf-50fc-e964c2e31272@oracle.com> On 10/18/17 6:10 AM, Magus Ilse Bursie wrote: > For jni_arm.h: Finally, the jni_arm.h (the 32-bit formerly closed > Oracle port), the JNIEXPORT/JNIIMPORT is different, by defining > __attribute__((externally_visible)). This might have been relevant due > to compile/link time symbol processing for that platform, but > technically it should probably not have been connected to the platform > per se, but rather to the compilation procedure. Since the arm-32 > platform is kind of abandoned right now, I propose to modify the > compilation to be more standard, if this is required, rather than > keeping these attributes. I think that was needed for gcc link-time optimization support, so it's really gcc-specific and not arm-specific. dl From mandy.chung at oracle.com Thu Oct 19 04:03:48 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 18 Oct 2017 21:03:48 -0700 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> Message-ID: <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> On 10/18/17 5:26 PM, David Holmes wrote: > Hi Mandy, > > On 19/10/2017 7:52 AM, mandy chung wrote: >> The bug creeped in when I improved the code to use loader.is_null() >> to set to system class loader per Coleen's suggestion.? It's my >> oversight of the null loader case.? The is the version I pushed. >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >> >> The previous version was correct. >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ > > Yes I see what happened - and we all missed it. And there was no > existing test it seems :( > >> I didn't go back to webrev.01.? The fix is an improvement to it. > > Okay, only concern I have is that this: > > 403?? oop loader = SystemDictionary::java_system_loader(); > > certainly looks like a "naked oop" - and the Java call may lead to GC > occurring. > Coleen also studied this and no naked oop (GC could happen in the java method call but loader is set after it). I will look into making it a handle which may feel more secure. > The test seems very lonely, where are the onLoad tests? > The onLoad test is with JDK-8164512 which I plan to push to jdk10/master but pending for this hotspot integration.? In this case, I think it's good to have a separate test case (it's not the same as the onLoad tests). BTW there is another onLoad test in the closed test. > In Driver: > > 30? * @run main/othervm/native > > what does the "native" do? I don't see it documented. > I have to find the documentation.? There are many hotspot tests using native that builds native library. > Regarding this comment in BootLoaderTest: > > // The JNI spec says FindClass returns NULL if class not found but > // it also says that NCDFE is thrown if no definition can be found. > // The current implementation throws NCDFE. > > The NCDFE is set pending by FindClass. If you return (or call into) > java code then it will be thrown. But if you were remaining in native > then you'd use the NULL return to determine success or failure (or > check pending exception). > Hmm.... let me take a closer look at it. thanks Mandy > Thanks, > David > >> Mandy >> >> On 10/18/17 2:48 PM, David Holmes wrote: >>> Hi Mandy, >>> >>> Do you still have the webrev for 8188052? I need to see how this got >>> through given we looked at it so many times. :( >>> >>> Thanks, >>> David >>> >>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>> This is a regression caused by JDK-8188052. FindClass should only >>>> use system class loader when there is no caller class or when it's >>>> called from JNI_OnUnload. When there is a caller class, it should >>>> either use its class loader or the one associated with JNI_Onload. >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>> >>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>> >>>> thanks >>>> Mandy >> From david.holmes at oracle.com Thu Oct 19 04:09:36 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Oct 2017 14:09:36 +1000 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> Message-ID: <58eb212e-2ef7-03ce-f0c3-e02e1205c41a@oracle.com> On 19/10/2017 2:03 PM, mandy chung wrote: > On 10/18/17 5:26 PM, David Holmes wrote: >> Hi Mandy, >> >> On 19/10/2017 7:52 AM, mandy chung wrote: >>> The bug creeped in when I improved the code to use loader.is_null() >>> to set to system class loader per Coleen's suggestion.? It's my >>> oversight of the null loader case.? The is the version I pushed. >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >>> >>> The previous version was correct. >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ >> >> Yes I see what happened - and we all missed it. And there was no >> existing test it seems :( >> >>> I didn't go back to webrev.01.? The fix is an improvement to it. >> >> Okay, only concern I have is that this: >> >> 403?? oop loader = SystemDictionary::java_system_loader(); >> >> certainly looks like a "naked oop" - and the Java call may lead to GC >> occurring. >> > > Coleen also studied this and no naked oop (GC could happen in the java > method call but loader is set after it). loader will be unchanged if the call is from OnUnload. > I will look into making it a handle which may feel more secure. >> The test seems very lonely, where are the onLoad tests? >> > > The onLoad test is with JDK-8164512 which I plan to push to jdk10/master > but pending for this hotspot integration.? In this case, I think it's > good to have a separate test case (it's not the same as the onLoad tests). No problem with separate test, just wondering if it should be somewhere else (with the other tests), or whether those other tests will end up in the FindClass directory? BTW may be a while before hs pushes up to master. > BTW there is another onLoad test in the closed test. >> In Driver: >> >> 30? * @run main/othervm/native >> >> what does the "native" do? I don't see it documented. >> > > I have to find the documentation.? There are many hotspot tests using > native that builds native library. > >> Regarding this comment in BootLoaderTest: >> >> // The JNI spec says FindClass returns NULL if class not found but >> // it also says that NCDFE is thrown if no definition can be found. >> // The current implementation throws NCDFE. >> >> The NCDFE is set pending by FindClass. If you return (or call into) >> java code then it will be thrown. But if you were remaining in native >> then you'd use the NULL return to determine success or failure (or >> check pending exception). >> > > Hmm.... let me take a closer look at it. I already saw and commented on the bug you filed. Thanks, David > thanks > Mandy >> Thanks, >> David >> >>> Mandy >>> >>> On 10/18/17 2:48 PM, David Holmes wrote: >>>> Hi Mandy, >>>> >>>> Do you still have the webrev for 8188052? I need to see how this got >>>> through given we looked at it so many times. :( >>>> >>>> Thanks, >>>> David >>>> >>>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>>> This is a regression caused by JDK-8188052. FindClass should only >>>>> use system class loader when there is no caller class or when it's >>>>> called from JNI_OnUnload. When there is a caller class, it should >>>>> either use its class loader or the one associated with JNI_Onload. >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>>> >>>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>>> >>>>> thanks >>>>> Mandy >>> > From jiangli.zhou at Oracle.COM Thu Oct 19 04:35:01 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Wed, 18 Oct 2017 21:35:01 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> Message-ID: <732B9B1D-6593-4E2F-ACAC-0BD25C55FA43@oracle.com> Hi Ioi, > On Oct 18, 2017, at 4:54 PM, Ioi Lam wrote: > > > > On 10/17/17 9:04 PM, David Holmes wrote: >> Hi Ioi, >> >> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>> >>> Summary: >>> >>> DumpLoadedClassList skips classes that can't be stored into the CDS >>> archive. For boot and platform loaders, only classes from the JRT image >>> are listed. >>> >>> The test for "is this class from JRT image" was insufficient. It >>> checked only classes whose source ends with "...../modules". For the >>> platform loader, which loads the graal classes, the source is in the >>> form "jrt:/jdk.internal.vm.compiler". >>> >>> The fix is to also check for a "jrt:" prefix. >> >> So ./share/classfile/classLoader.hpp: >> >> static bool is_jrt(const char* name) { return string_ends_with(name, MODULES_IMAGE_NAME); } >> >> is a badly named function or just plain broken? >> > > I've renamed all the is_jrt() functions to is_modules_image(), so it's clear as to what it does. > > See http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ The logic looks puzzling. For both NULL class loader and the PlatformClassLoader: The first if check at line 5938 excludes any classes not in the jimage. 5938 if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { 5939 skip = true; 5940 } The second if check at line 5942 skips any classes from the boot append class path loaded by the NULL class loader. The second check skips a subset of the classes that?s already skipped by the first check. The second check is redundant. 5942 if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { 5943 // For the boot loader, also skip all classes that are loaded from the appended entries 5944 skip = true; 5945 } What about the classes from -Xbootclasspath/a (not in --patch-module and --upgrade-module-path)? Thanks, Jiangli > > >>> I also restructured the code to make it easier to read -- the nesting of || and && is hard to follow. >>> >>> The original code had a subtle bug with the "!" operator: >>> >>> 5938 if (((class_loader == NULL && !ClassLoader::contains_append_entry(stream->source())) || >>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>> 5940 !ClassLoader::is_jrt(stream->source())) { >>> >>> So if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source()) was true, >>> the class would NOT be skipped. >>> >>> The new version removes the "!". >> >> So there should be an updated test to trigger the original bug. >> > > I added a test case, but it's in the closed repo for now (since the test case uses utilities in the closed AppCDS tests). This test will be opened as part of JDK-8185996 > > Thanks > - Ioi > >> Thanks, >> David >> >>> Thanks >>> - Ioi > From ioi.lam at oracle.com Thu Oct 19 04:50:51 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Wed, 18 Oct 2017 21:50:51 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: <732B9B1D-6593-4E2F-ACAC-0BD25C55FA43@oracle.com> References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> <732B9B1D-6593-4E2F-ACAC-0BD25C55FA43@oracle.com> Message-ID: <88a9a707-1158-f555-ffb0-653fe9bd9483@oracle.com> On 10/18/17 9:35 PM, Jiangli Zhou wrote: > Hi Ioi, > >> On Oct 18, 2017, at 4:54 PM, Ioi Lam > > wrote: >> >> >> >> On 10/17/17 9:04 PM, David Holmes wrote: >>> Hi Ioi, >>> >>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>> >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>> >>>> Summary: >>>> >>>> DumpLoadedClassList skips classes that can't be stored into the CDS >>>> archive. For boot and platform loaders, only classes from the JRT image >>>> are listed. >>>> >>>> The test for "is this class from JRT image" was insufficient. It >>>> checked only classes whose source ends with "...../modules". For the >>>> platform loader, which loads the graal classes, the source is in the >>>> form "jrt:/jdk.internal.vm.compiler". >>>> >>>> The fix is to also check for a "jrt:" prefix. >>> >>> So ./share/classfile/classLoader.hpp: >>> >>> static bool is_jrt(const char* name) { return string_ends_with(name, >>> MODULES_IMAGE_NAME); } >>> >>> is a badly named function or just plain broken? >>> >> >> I've renamed all the is_jrt() functions to is_modules_image(), so >> it's clear as to what it does. >> >> See >> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >> > > The logic looks puzzling. For both NULL class loader and the > PlatformClassLoader: > > The first if check at line 5938 excludes any classes not in the jimage. > > 5938 if (!ClassLoader::is_modules_image(stream->source()) && > strncmp(stream->source(), "jrt:", 4) != 0) { > 5939 skip = true; > 5940 } > The second if check at line 5942 skips any classes from the boot > append class path loaded by the NULL class loader. The second check > skips a subset of the classes that?s already skipped by the first > check. The second check is redundant. > You're right. The second check is redundant. I'll remove it. > 5942 if (class_loader == NULL && > ClassLoader::contains_append_entry(stream->source())) { > 5943 // For the boot loader, also skip all classes that are loaded > from the appended entries > 5944 skip = true; > 5945 } > What about the classes from -Xbootclasspath/a (not in --patch-module > and --upgrade-module-path)? > I am not sure what you mean here. Thanks - Ioi > Thanks, > Jiangli > >> >> >>>> I also restructured the code to make it easier to read -- the >>>> nesting of || and && is hard to follow. >>>> >>>> The original code had a subtle bug with the "!" operator: >>>> >>>> 5938?? if (((class_loader == NULL && >>>> !ClassLoader::contains_append_entry(stream->source())) || >>>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>>> 5940?????? !ClassLoader::is_jrt(stream->source())) { >>>> >>>> So if (class_loader == NULL && >>>> ClassLoader::contains_append_entry(stream->source()) was true, >>>> the class would NOT be skipped. >>>> >>>> The new version removes the "!". >>> >>> So there should be an updated test to trigger the original bug. >>> >> >> I added a test case, but it's in the closed repo for now (since the >> test case uses utilities in the closed AppCDS tests). This test will >> be opened as part of JDK-8185996 >> >> Thanks >> - Ioi >> >>> Thanks, >>> David >>> >>>> Thanks >>>> - Ioi >> > From mandy.chung at oracle.com Thu Oct 19 05:50:56 2017 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 18 Oct 2017 22:50:56 -0700 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <58eb212e-2ef7-03ce-f0c3-e02e1205c41a@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> <58eb212e-2ef7-03ce-f0c3-e02e1205c41a@oracle.com> Message-ID: On 10/18/17 9:09 PM, David Holmes wrote: > No problem with separate test, just wondering if it should be > somewhere else (with the other tests), or whether those other tests > will end up in the FindClass directory? The new test I added is in test/jdk/java/lang/ClassLoader/nativeLibrary which is about loading and unloading native library. The existing FindClass tests are tonga tests.? I am not sure if there are other FindClass tests under test/hotspot/jtreg but at least I don't find them yet. I think test/hotspot/runtime/jni/FindClass is a good location to me.? Do you have any better suggestion? Mandy From david.holmes at oracle.com Thu Oct 19 06:47:39 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 19 Oct 2017 16:47:39 +1000 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> <58eb212e-2ef7-03ce-f0c3-e02e1205c41a@oracle.com> Message-ID: <54e69f1b-24f5-fd2c-996f-4ff210b64442@oracle.com> On 19/10/2017 3:50 PM, mandy chung wrote: > On 10/18/17 9:09 PM, David Holmes wrote: >> No problem with separate test, just wondering if it should be >> somewhere else (with the other tests), or whether those other tests >> will end up in the FindClass directory? > > The new test I added is in test/jdk/java/lang/ClassLoader/nativeLibrary > which is about loading and unloading native library. > > The existing FindClass tests are tonga tests.? I am not sure if there > are other FindClass tests under test/hotspot/jtreg but at least I don't > find them yet. > > I think test/hotspot/runtime/jni/FindClass is a good location to me.? Do > you have any better suggestion? No that's fine. There's no single good location when tested functionality overlaps. Thanks, David > Mandy From coleen.phillimore at oracle.com Thu Oct 19 11:35:42 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 19 Oct 2017 07:35:42 -0400 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <54e69f1b-24f5-fd2c-996f-4ff210b64442@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> <58eb212e-2ef7-03ce-f0c3-e02e1205c41a@oracle.com> <54e69f1b-24f5-fd2c-996f-4ff210b64442@oracle.com> Message-ID: <997ec5df-a74e-524b-4e1d-25d928eb2a34@oracle.com> On 10/19/17 2:47 AM, David Holmes wrote: > On 19/10/2017 3:50 PM, mandy chung wrote: >> On 10/18/17 9:09 PM, David Holmes wrote: >>> No problem with separate test, just wondering if it should be >>> somewhere else (with the other tests), or whether those other tests >>> will end up in the FindClass directory? >> >> The new test I added is in >> test/jdk/java/lang/ClassLoader/nativeLibrary which is about loading >> and unloading native library. >> >> The existing FindClass tests are tonga tests.? I am not sure if there >> are other FindClass tests under test/hotspot/jtreg but at least I >> don't find them yet. >> >> I think test/hotspot/runtime/jni/FindClass is a good location to me.? >> Do you have any better suggestion? > > No that's fine. There's no single good location when tested > functionality overlaps. I agree.? I like having a sample in hotspot/runtime. Coleen > > Thanks, > David > >> Mandy From coleen.phillimore at oracle.com Thu Oct 19 11:39:35 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 19 Oct 2017 07:39:35 -0400 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> Message-ID: <269ab884-5969-7793-9a7a-5cee49c8431c@oracle.com> On 10/19/17 12:03 AM, mandy chung wrote: > > > On 10/18/17 5:26 PM, David Holmes wrote: >> Hi Mandy, >> >> On 19/10/2017 7:52 AM, mandy chung wrote: >>> The bug creeped in when I improved the code to use loader.is_null() >>> to set to system class loader per Coleen's suggestion.? It's my >>> oversight of the null loader case.? The is the version I pushed. >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >>> >>> The previous version was correct. >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ >> >> Yes I see what happened - and we all missed it. And there was no >> existing test it seems :( >> >>> I didn't go back to webrev.01.? The fix is an improvement to it. >> >> Okay, only concern I have is that this: >> >> 403?? oop loader = SystemDictionary::java_system_loader(); >> >> certainly looks like a "naked oop" - and the Java call may lead to GC >> occurring. >> > > Coleen also studied this and no naked oop (GC could happen in the java > method call but loader is set after it). > > I will look into making it a handle which may feel more secure. >> The test seems very lonely, where are the onLoad tests? >> > > The onLoad test is with JDK-8164512 which I plan to push to > jdk10/master but pending for this hotspot integration.? In this case, > I think it's good to have a separate test case (it's not the same as > the onLoad tests). > > BTW there is another onLoad test in the closed test. >> In Driver: >> >> 30? * @run main/othervm/native >> >> what does the "native" do? I don't see it documented. >> > > I have to find the documentation.? There are many hotspot tests using > native that builds native library. Yes, this is how I believe we link in the native library for the test.? This looks right to me. Mandy, I had a question about the "Native" test though, did you see that? Coleen > >> Regarding this comment in BootLoaderTest: >> >> // The JNI spec says FindClass returns NULL if class not found but >> // it also says that NCDFE is thrown if no definition can be found. >> // The current implementation throws NCDFE. >> >> The NCDFE is set pending by FindClass. If you return (or call into) >> java code then it will be thrown. But if you were remaining in native >> then you'd use the NULL return to determine success or failure (or >> check pending exception). >> > > Hmm.... let me take a closer look at it. > > thanks > Mandy >> Thanks, >> David >> >>> Mandy >>> >>> On 10/18/17 2:48 PM, David Holmes wrote: >>>> Hi Mandy, >>>> >>>> Do you still have the webrev for 8188052? I need to see how this >>>> got through given we looked at it so many times. :( >>>> >>>> Thanks, >>>> David >>>> >>>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>>> This is a regression caused by JDK-8188052. FindClass should only >>>>> use system class loader when there is no caller class or when it's >>>>> called from JNI_OnUnload. When there is a caller class, it should >>>>> either use its class loader or the one associated with JNI_Onload. >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>>> >>>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>>> >>>>> thanks >>>>> Mandy >>> > From yasuenag at gmail.com Thu Oct 19 11:51:19 2017 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Thu, 19 Oct 2017 20:51:19 +0900 Subject: [8u] RFR: 8189599: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize Message-ID: <1db9f08c-165d-3e2d-fc88-fbf3001de2cd@gmail.com> Hi all, I want to backport JDK-8087291 change to 8u because this issue will affect all Java 8 or later users. Could you review it? http://cr.openjdk.java.net/~ysuenaga/JDK-8189599/webrev.00/ The patch for jdk10/hs has been pushed: http://hg.openjdk.java.net/jdk10/hs/rev/dbd1f4f276ba This change will be sponsored by Poonam. Thanks, Yasumasa From coleen.phillimore at oracle.com Thu Oct 19 13:50:07 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 19 Oct 2017 09:50:07 -0400 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <269ab884-5969-7793-9a7a-5cee49c8431c@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> <269ab884-5969-7793-9a7a-5cee49c8431c@oracle.com> Message-ID: On 10/19/17 7:39 AM, coleen.phillimore at oracle.com wrote: > > > On 10/19/17 12:03 AM, mandy chung wrote: >> >> >> On 10/18/17 5:26 PM, David Holmes wrote: >>> Hi Mandy, >>> >>> On 19/10/2017 7:52 AM, mandy chung wrote: >>>> The bug creeped in when I improved the code to use loader.is_null() >>>> to set to system class loader per Coleen's suggestion.? It's my >>>> oversight of the null loader case.? The is the version I pushed. >>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >>>> >>>> The previous version was correct. >>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ >>> >>> Yes I see what happened - and we all missed it. And there was no >>> existing test it seems :( >>> >>>> I didn't go back to webrev.01.? The fix is an improvement to it. >>> >>> Okay, only concern I have is that this: >>> >>> 403?? oop loader = SystemDictionary::java_system_loader(); >>> >>> certainly looks like a "naked oop" - and the Java call may lead to >>> GC occurring. >>> >> >> Coleen also studied this and no naked oop (GC could happen in the >> java method call but loader is set after it). >> >> I will look into making it a handle which may feel more secure. >>> The test seems very lonely, where are the onLoad tests? >>> >> >> The onLoad test is with JDK-8164512 which I plan to push to >> jdk10/master but pending for this hotspot integration.? In this case, >> I think it's good to have a separate test case (it's not the same as >> the onLoad tests). >> >> BTW there is another onLoad test in the closed test. >>> In Driver: >>> >>> 30? * @run main/othervm/native >>> >>> what does the "native" do? I don't see it documented. >>> >> >> I have to find the documentation.? There are many hotspot tests using >> native that builds native library. > > Yes, this is how I believe we link in the native library for the > test.? This looks right to me. > Mandy, I had a question about the "Native" test though, did you see that? Sorry, never mind.? That was a different code review that I did yesterday. Coleen > > Coleen >> >>> Regarding this comment in BootLoaderTest: >>> >>> // The JNI spec says FindClass returns NULL if class not found but >>> // it also says that NCDFE is thrown if no definition can be found. >>> // The current implementation throws NCDFE. >>> >>> The NCDFE is set pending by FindClass. If you return (or call into) >>> java code then it will be thrown. But if you were remaining in >>> native then you'd use the NULL return to determine success or >>> failure (or check pending exception). >>> >> >> Hmm.... let me take a closer look at it. >> >> thanks >> Mandy >>> Thanks, >>> David >>> >>>> Mandy >>>> >>>> On 10/18/17 2:48 PM, David Holmes wrote: >>>>> Hi Mandy, >>>>> >>>>> Do you still have the webrev for 8188052? I need to see how this >>>>> got through given we looked at it so many times. :( >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>>>> This is a regression caused by JDK-8188052. FindClass should only >>>>>> use system class loader when there is no caller class or when >>>>>> it's called from JNI_OnUnload. When there is a caller class, it >>>>>> should either use its class loader or the one associated with >>>>>> JNI_Onload. >>>>>> >>>>>> Webrev: >>>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>>>> >>>>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>>>> >>>>>> thanks >>>>>> Mandy >>>> >> > From mandy.chung at oracle.com Thu Oct 19 15:53:43 2017 From: mandy.chung at oracle.com (mandy chung) Date: Thu, 19 Oct 2017 08:53:43 -0700 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> Message-ID: David, Coleen, I have revised the patch to use Handle (much simpler, thanks) and update the test to clear the exception: http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.01/index.html Thanks Mandy On 10/18/17 5:26 PM, David Holmes wrote: > Hi Mandy, > > On 19/10/2017 7:52 AM, mandy chung wrote: >> The bug creeped in when I improved the code to use loader.is_null() >> to set to system class loader per Coleen's suggestion.? It's my >> oversight of the null loader case.? The is the version I pushed. >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >> >> The previous version was correct. >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ > > Yes I see what happened - and we all missed it. And there was no > existing test it seems :( > >> I didn't go back to webrev.01.? The fix is an improvement to it. > > Okay, only concern I have is that this: > > 403?? oop loader = SystemDictionary::java_system_loader(); > > certainly looks like a "naked oop" - and the Java call may lead to GC > occurring. > > The test seems very lonely, where are the onLoad tests? > > In? Driver: > > 30? * @run main/othervm/native > > what does the "native" do? I don't see it documented. > > Regarding this comment in BootLoaderTest: > > // The JNI spec says FindClass returns NULL if class not found but > // it also says that NCDFE is thrown if no definition can be found. > // The current implementation throws NCDFE. > > The NCDFE is set pending by FindClass. If you return (or call into) > java code then it will be thrown. But if you were remaining in native > then you'd use the NULL return to determine success or failure (or > check pending exception). > > Thanks, > David > >> Mandy >> >> On 10/18/17 2:48 PM, David Holmes wrote: >>> Hi Mandy, >>> >>> Do you still have the webrev for 8188052? I need to see how this got >>> through given we looked at it so many times. :( >>> >>> Thanks, >>> David >>> >>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>> This is a regression caused by JDK-8188052. FindClass should only >>>> use system class loader when there is no caller class or when it's >>>> called from JNI_OnUnload. When there is a caller class, it should >>>> either use its class loader or the one associated with JNI_Onload. >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>> >>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>> >>>> thanks >>>> Mandy >> From jiangli.zhou at oracle.com Thu Oct 19 16:47:30 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Thu, 19 Oct 2017 09:47:30 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: <88a9a707-1158-f555-ffb0-653fe9bd9483@oracle.com> References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> <732B9B1D-6593-4E2F-ACAC-0BD25C55FA43@oracle.com> <88a9a707-1158-f555-ffb0-653fe9bd9483@oracle.com> Message-ID: > On Oct 18, 2017, at 9:50 PM, Ioi Lam wrote: > > > > On 10/18/17 9:35 PM, Jiangli Zhou wrote: >> Hi Ioi, >> >>> On Oct 18, 2017, at 4:54 PM, Ioi Lam > wrote: >>> >>> >>> >>> On 10/17/17 9:04 PM, David Holmes wrote: >>>> Hi Ioi, >>>> >>>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>>> >>>>> Summary: >>>>> >>>>> DumpLoadedClassList skips classes that can't be stored into the CDS >>>>> archive. For boot and platform loaders, only classes from the JRT image >>>>> are listed. >>>>> >>>>> The test for "is this class from JRT image" was insufficient. It >>>>> checked only classes whose source ends with "...../modules". For the >>>>> platform loader, which loads the graal classes, the source is in the >>>>> form "jrt:/jdk.internal.vm.compiler". >>>>> >>>>> The fix is to also check for a "jrt:" prefix. >>>> >>>> So ./share/classfile/classLoader.hpp: >>>> >>>> static bool is_jrt(const char* name) { return string_ends_with(name, MODULES_IMAGE_NAME); } >>>> >>>> is a badly named function or just plain broken? >>>> >>> >>> I've renamed all the is_jrt() functions to is_modules_image(), so it's clear as to what it does. >>> >>> See http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >> >> The logic looks puzzling. For both NULL class loader and the PlatformClassLoader: >> >> The first if check at line 5938 excludes any classes not in the jimage. >> >> 5938 if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { >> 5939 skip = true; >> 5940 } >> The second if check at line 5942 skips any classes from the boot append class path loaded by the NULL class loader. The second check skips a subset of the classes that?s already skipped by the first check. The second check is redundant. >> > You're right. The second check is redundant. I'll remove it. > >> 5942 if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { >> 5943 // For the boot loader, also skip all classes that are loaded from the appended entries >> 5944 skip = true; >> 5945 } >> What about the classes from -Xbootclasspath/a (not in --patch-module and --upgrade-module-path)? >> > I am not sure what you mean here. This excludes all classes from the -Xbootclasspath/a. However, classes from -Xbootclasspath/a can be archived and used at runtime properly. Thanks, Jiangli > > Thanks > - Ioi >> Thanks, >> Jiangli >> >>> >>> >>>>> I also restructured the code to make it easier to read -- the nesting of || and && is hard to follow. >>>>> >>>>> The original code had a subtle bug with the "!" operator: >>>>> >>>>> 5938 if (((class_loader == NULL && !ClassLoader::contains_append_entry(stream->source())) || >>>>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>>>> 5940 !ClassLoader::is_jrt(stream->source())) { >>>>> >>>>> So if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source()) was true, >>>>> the class would NOT be skipped. >>>>> >>>>> The new version removes the "!". >>>> >>>> So there should be an updated test to trigger the original bug. >>>> >>> >>> I added a test case, but it's in the closed repo for now (since the test case uses utilities in the closed AppCDS tests). This test will be opened as part of JDK-8185996 >>> >>> Thanks >>> - Ioi >>> >>>> Thanks, >>>> David >>>> >>>>> Thanks >>>>> - Ioi >>> >> > From coleen.phillimore at oracle.com Thu Oct 19 17:12:48 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 19 Oct 2017 13:12:48 -0400 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> Message-ID: <96831c9b-014d-f4ee-4150-d42af2228539@oracle.com> Looks great! Coleen On 10/19/17 11:53 AM, mandy chung wrote: > David, Coleen, > > I have revised the patch to use Handle (much simpler, thanks) and > update the test to clear the exception: > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.01/index.html > > > Thanks > Mandy > > On 10/18/17 5:26 PM, David Holmes wrote: >> Hi Mandy, >> >> On 19/10/2017 7:52 AM, mandy chung wrote: >>> The bug creeped in when I improved the code to use loader.is_null() >>> to set to system class loader per Coleen's suggestion.? It's my >>> oversight of the null loader case.? The is the version I pushed. >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >>> >>> The previous version was correct. >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ >> >> Yes I see what happened - and we all missed it. And there was no >> existing test it seems :( >> >>> I didn't go back to webrev.01.? The fix is an improvement to it. >> >> Okay, only concern I have is that this: >> >> 403?? oop loader = SystemDictionary::java_system_loader(); >> >> certainly looks like a "naked oop" - and the Java call may lead to GC >> occurring. >> >> The test seems very lonely, where are the onLoad tests? >> >> In? Driver: >> >> 30? * @run main/othervm/native >> >> what does the "native" do? I don't see it documented. >> >> Regarding this comment in BootLoaderTest: >> >> // The JNI spec says FindClass returns NULL if class not found but >> // it also says that NCDFE is thrown if no definition can be found. >> // The current implementation throws NCDFE. >> >> The NCDFE is set pending by FindClass. If you return (or call into) >> java code then it will be thrown. But if you were remaining in native >> then you'd use the NULL return to determine success or failure (or >> check pending exception). >> >> Thanks, >> David >> >>> Mandy >>> >>> On 10/18/17 2:48 PM, David Holmes wrote: >>>> Hi Mandy, >>>> >>>> Do you still have the webrev for 8188052? I need to see how this >>>> got through given we looked at it so many times. :( >>>> >>>> Thanks, >>>> David >>>> >>>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>>> This is a regression caused by JDK-8188052. FindClass should only >>>>> use system class loader when there is no caller class or when it's >>>>> called from JNI_OnUnload. When there is a caller class, it should >>>>> either use its class loader or the one associated with JNI_Onload. >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>>> >>>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>>> >>>>> thanks >>>>> Mandy >>> > From ioi.lam at oracle.com Thu Oct 19 17:55:48 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 19 Oct 2017 10:55:48 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> <732B9B1D-6593-4E2F-ACAC-0BD25C55FA43@oracle.com> <88a9a707-1158-f555-ffb0-653fe9bd9483@oracle.com> Message-ID: On 10/19/17 9:47 AM, Jiangli Zhou wrote: > >> On Oct 18, 2017, at 9:50 PM, Ioi Lam > > wrote: >> >> >> >> On 10/18/17 9:35 PM, Jiangli Zhou wrote: >>> Hi Ioi, >>> >>>> On Oct 18, 2017, at 4:54 PM, Ioi Lam >>> > wrote: >>>> >>>> >>>> >>>> On 10/17/17 9:04 PM, David Holmes wrote: >>>>> Hi Ioi, >>>>> >>>>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>>>> >>>>>> >>>>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>>>> >>>>>> Summary: >>>>>> >>>>>> DumpLoadedClassList skips classes that can't be stored into the CDS >>>>>> archive. For boot and platform loaders, only classes from the JRT >>>>>> image >>>>>> are listed. >>>>>> >>>>>> The test for "is this class from JRT image" was insufficient. It >>>>>> checked only classes whose source ends with "...../modules". For the >>>>>> platform loader, which loads the graal classes, the source is in the >>>>>> form "jrt:/jdk.internal.vm.compiler". >>>>>> >>>>>> The fix is to also check for a "jrt:" prefix. >>>>> >>>>> So ./share/classfile/classLoader.hpp: >>>>> >>>>> static bool is_jrt(const char* name) { return >>>>> string_ends_with(name, MODULES_IMAGE_NAME); } >>>>> >>>>> is a badly named function or just plain broken? >>>>> >>>> >>>> I've renamed all the is_jrt() functions to is_modules_image(), so >>>> it's clear as to what it does. >>>> >>>> See >>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >>>> >>> >>> The logic looks puzzling. For both NULL class loader and the >>> PlatformClassLoader: >>> >>> The first if check at line 5938 excludes any classes not in the jimage. >>> >>> 5938 if (!ClassLoader::is_modules_image(stream->source()) && >>> strncmp(stream->source(), "jrt:", 4) != 0) { >>> 5939 skip = true; >>> 5940 } >>> The second if check at line 5942 skips any classes from the boot >>> append class path loaded by the NULL class loader. The second check >>> skips a subset of the classes that?s already skipped by the first >>> check. The second check is redundant. >>> >> You're right. The second check is redundant. I'll remove it. >> >>> 5942 if (class_loader == NULL && >>> ClassLoader::contains_append_entry(stream->source())) { >>> 5943 // For the boot loader, also skip all classes that are loaded >>> from the appended entries >>> 5944 skip = true; >>> 5945 } >>> What about the classes from -Xbootclasspath/a (not in --patch-module >>> and --upgrade-module-path)? >>> >> I am not sure what you mean here. > > This excludes all classes from the -Xbootclasspath/a. However, classes > from -Xbootclasspath/a can be archived and used at runtime properly. > Hi Jiangli, You're right. I misunderstood the original comments about the treatment of the bootclasspath/a classes. Here's the updated code that should be easier to understand: ??????? bool skip = false; ??????? if (class_loader == NULL || SystemDictionary::is_platform_class_loader(class_loader)) { ????????? // For the boot and platform class loaders, skip classes that are not found in the ????????? // java runtime image, such as those found in the --patch-module entries. ????????? // These classes can't be loaded from the archive during runtime. ????????? if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { ??????????? skip = true; ????????? } ????????? if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { ??????????? // .. but don't skip the boot classes that are loaded from -Xbootclasspath/a ??????????? // as theycan be loaded from the archive during runtime. ??????????? skip = false; ????????? } ??????? } Thanks - Ioi > Thanks, > Jiangli > >> >> Thanks >> - Ioi >>> Thanks, >>> Jiangli >>> >>>> >>>> >>>>>> I also restructured the code to make it easier to read -- the >>>>>> nesting of || and && is hard to follow. >>>>>> >>>>>> The original code had a subtle bug with the "!" operator: >>>>>> >>>>>> 5938?? if (((class_loader == NULL && >>>>>> !ClassLoader::contains_append_entry(stream->source())) || >>>>>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>>>>> 5940 !ClassLoader::is_jrt(stream->source())) { >>>>>> >>>>>> So if (class_loader == NULL && >>>>>> ClassLoader::contains_append_entry(stream->source()) was true, >>>>>> the class would NOT be skipped. >>>>>> >>>>>> The new version removes the "!". >>>>> >>>>> So there should be an updated test to trigger the original bug. >>>>> >>>> >>>> I added a test case, but it's in the closed repo for now (since the >>>> test case uses utilities in the closed AppCDS tests). This test >>>> will be opened as part of JDK-8185996 >>>> >>>> Thanks >>>> - Ioi >>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> Thanks >>>>>> - Ioi >>>> >>> >> > From mandy.chung at oracle.com Thu Oct 19 18:11:03 2017 From: mandy.chung at oracle.com (mandy chung) Date: Thu, 19 Oct 2017 11:11:03 -0700 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <96831c9b-014d-f4ee-4150-d42af2228539@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <96831c9b-014d-f4ee-4150-d42af2228539@oracle.com> Message-ID: <9b029d2f-3c59-a341-c9d6-802d1e2508a7@oracle.com> Thanks Coleen.?? I renamed Driver.java to FindClassFromBoot.java which is a better test name than Driver (that should address David's comment).? No other change. http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.02/ I tested this with jdk core groups and hotspot group. Mandy On 10/19/17 10:12 AM, coleen.phillimore at oracle.com wrote: > Looks great! > Coleen > > On 10/19/17 11:53 AM, mandy chung wrote: >> David, Coleen, >> >> I have revised the patch to use Handle (much simpler, thanks) and >> update the test to clear the exception: >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.01/index.html >> >> >> Thanks >> Mandy >> >> On 10/18/17 5:26 PM, David Holmes wrote: >>> Hi Mandy, >>> >>> On 19/10/2017 7:52 AM, mandy chung wrote: >>>> The bug creeped in when I improved the code to use loader.is_null() >>>> to set to system class loader per Coleen's suggestion.? It's my >>>> oversight of the null loader case.? The is the version I pushed. >>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >>>> >>>> The previous version was correct. >>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ >>> >>> Yes I see what happened - and we all missed it. And there was no >>> existing test it seems :( >>> >>>> I didn't go back to webrev.01.? The fix is an improvement to it. >>> >>> Okay, only concern I have is that this: >>> >>> 403?? oop loader = SystemDictionary::java_system_loader(); >>> >>> certainly looks like a "naked oop" - and the Java call may lead to >>> GC occurring. >>> >>> The test seems very lonely, where are the onLoad tests? >>> >>> In? Driver: >>> >>> 30? * @run main/othervm/native >>> >>> what does the "native" do? I don't see it documented. >>> >>> Regarding this comment in BootLoaderTest: >>> >>> // The JNI spec says FindClass returns NULL if class not found but >>> // it also says that NCDFE is thrown if no definition can be found. >>> // The current implementation throws NCDFE. >>> >>> The NCDFE is set pending by FindClass. If you return (or call into) >>> java code then it will be thrown. But if you were remaining in >>> native then you'd use the NULL return to determine success or >>> failure (or check pending exception). >>> >>> Thanks, >>> David >>> >>>> Mandy >>>> >>>> On 10/18/17 2:48 PM, David Holmes wrote: >>>>> Hi Mandy, >>>>> >>>>> Do you still have the webrev for 8188052? I need to see how this >>>>> got through given we looked at it so many times. :( >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>>>> This is a regression caused by JDK-8188052. FindClass should only >>>>>> use system class loader when there is no caller class or when >>>>>> it's called from JNI_OnUnload. When there is a caller class, it >>>>>> should either use its class loader or the one associated with >>>>>> JNI_Onload. >>>>>> >>>>>> Webrev: >>>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>>>> >>>>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>>>> >>>>>> thanks >>>>>> Mandy >>>> >> > From lois.foltan at oracle.com Thu Oct 19 18:19:38 2017 From: lois.foltan at oracle.com (Lois Foltan) Date: Thu, 19 Oct 2017 14:19:38 -0400 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: <0a9de83e-b424-1f31-9025-9498fddafcf8@oracle.com> References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> <0a9de83e-b424-1f31-9025-9498fddafcf8@oracle.com> Message-ID: <5a52ada7-bba6-dfea-84c6-a3878419f404@oracle.com> On 10/18/2017 10:32 PM, Ioi Lam wrote: > > > On 10/18/17 6:26 PM, David Holmes wrote: >> Hi Ioi, >> >> On 19/10/2017 9:54 AM, Ioi Lam wrote: >>> >>> >>> On 10/17/17 9:04 PM, David Holmes wrote: >>>> Hi Ioi, >>>> >>>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>>> >>>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>>> >>>>> Summary: >>>>> >>>>> DumpLoadedClassList skips classes that can't be stored into the CDS >>>>> archive. For boot and platform loaders, only classes from the JRT >>>>> image >>>>> are listed. >>>>> >>>>> The test for "is this class from JRT image" was insufficient. It >>>>> checked only classes whose source ends with "...../modules". For the >>>>> platform loader, which loads the graal classes, the source is in the >>>>> form "jrt:/jdk.internal.vm.compiler". >>>>> >>>>> The fix is to also check for a "jrt:" prefix. >>>> >>>> So ./share/classfile/classLoader.hpp: >>>> >>>> static bool is_jrt(const char* name) { return >>>> string_ends_with(name, MODULES_IMAGE_NAME); } >>>> >>>> is a badly named function or just plain broken? >>>> >>> >>> I've renamed all the is_jrt() functions to is_modules_image(), so >>> it's clear as to what it does. >>> >>> See >>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >> >> >> >> Okay. But I'm still a bit confused about the expectations of >> is_modules_image() - where are these names defined? What does jrt: >> mean if not "in a module image" ?? Hi Ioi, I think it would be better to rename "is_jrt" to "is_java_runtime_image".? It has nothing to do with modules.? It has more to do with the JDK built as a "run-time" image vs. the JDK built as an exploded build. Thanks, Lois >> >> That aside your fix does what you needed, so looks fine. >> > > The problem we have is the ClassFileParser::source()? returns > "/lib/modules" for the boot loader (C code), and > "jrt:/java.compiler" for the platform loader (Java code). This should > probably be fixed, but I don't want to deal with that now :-( > > >>> >>> >>>>> I also restructured the code to make it easier to read -- the >>>>> nesting of || and && is hard to follow. >>>>> >>>>> The original code had a subtle bug with the "!" operator: >>>>> >>>>> 5938?? if (((class_loader == NULL && >>>>> !ClassLoader::contains_append_entry(stream->source())) || >>>>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>>>> 5940?????? !ClassLoader::is_jrt(stream->source())) { >>>>> >>>>> So if (class_loader == NULL && >>>>> ClassLoader::contains_append_entry(stream->source()) was true, >>>>> the class would NOT be skipped. >>>>> >>>>> The new version removes the "!". >>>> >>>> So there should be an updated test to trigger the original bug. >>>> >>> >>> I added a test case, but it's in the closed repo for now (since the >>> test case uses utilities in the closed AppCDS tests). This test will >>> be opened as part of JDK-8185996 >> >> Ok. >> > > Thanks > - Ioi > >> Thanks, >> David >> >>> Thanks >>> - Ioi >>> >>>> Thanks, >>>> David >>>> >>>>> Thanks >>>>> - Ioi >>> > From coleen.phillimore at oracle.com Thu Oct 19 18:22:04 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 19 Oct 2017 14:22:04 -0400 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <9b029d2f-3c59-a341-c9d6-802d1e2508a7@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <96831c9b-014d-f4ee-4150-d42af2228539@oracle.com> <9b029d2f-3c59-a341-c9d6-802d1e2508a7@oracle.com> Message-ID: <89aa0eeb-fd0f-4ce9-d6c3-7310e67462f2@oracle.com> Thanks Mandy!? That seems better. thanks, Coleen On 10/19/17 2:11 PM, mandy chung wrote: > Thanks Coleen.?? I renamed Driver.java to FindClassFromBoot.java which > is a better test name than Driver (that should address David's > comment).? No other change. > > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.02/ > > I tested this with jdk core groups and hotspot group. > > Mandy > > On 10/19/17 10:12 AM, coleen.phillimore at oracle.com wrote: >> Looks great! >> Coleen >> >> On 10/19/17 11:53 AM, mandy chung wrote: >>> David, Coleen, >>> >>> I have revised the patch to use Handle (much simpler, thanks) and >>> update the test to clear the exception: >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.01/index.html >>> >>> >>> Thanks >>> Mandy >>> >>> On 10/18/17 5:26 PM, David Holmes wrote: >>>> Hi Mandy, >>>> >>>> On 19/10/2017 7:52 AM, mandy chung wrote: >>>>> The bug creeped in when I improved the code to use >>>>> loader.is_null() to set to system class loader per Coleen's >>>>> suggestion.? It's my oversight of the null loader case.? The is >>>>> the version I pushed. >>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >>>>> >>>>> The previous version was correct. >>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ >>>> >>>> Yes I see what happened - and we all missed it. And there was no >>>> existing test it seems :( >>>> >>>>> I didn't go back to webrev.01.? The fix is an improvement to it. >>>> >>>> Okay, only concern I have is that this: >>>> >>>> 403?? oop loader = SystemDictionary::java_system_loader(); >>>> >>>> certainly looks like a "naked oop" - and the Java call may lead to >>>> GC occurring. >>>> >>>> The test seems very lonely, where are the onLoad tests? >>>> >>>> In? Driver: >>>> >>>> 30? * @run main/othervm/native >>>> >>>> what does the "native" do? I don't see it documented. >>>> >>>> Regarding this comment in BootLoaderTest: >>>> >>>> // The JNI spec says FindClass returns NULL if class not found but >>>> // it also says that NCDFE is thrown if no definition can be found. >>>> // The current implementation throws NCDFE. >>>> >>>> The NCDFE is set pending by FindClass. If you return (or call into) >>>> java code then it will be thrown. But if you were remaining in >>>> native then you'd use the NULL return to determine success or >>>> failure (or check pending exception). >>>> >>>> Thanks, >>>> David >>>> >>>>> Mandy >>>>> >>>>> On 10/18/17 2:48 PM, David Holmes wrote: >>>>>> Hi Mandy, >>>>>> >>>>>> Do you still have the webrev for 8188052? I need to see how this >>>>>> got through given we looked at it so many times. :( >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>>>>> This is a regression caused by JDK-8188052. FindClass should >>>>>>> only use system class loader when there is no caller class or >>>>>>> when it's called from JNI_OnUnload. When there is a caller >>>>>>> class, it should either use its class loader or the one >>>>>>> associated with JNI_Onload. >>>>>>> >>>>>>> Webrev: >>>>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>>>>> >>>>>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>>>>> >>>>>>> thanks >>>>>>> Mandy >>>>> >>> >> > From lois.foltan at oracle.com Thu Oct 19 18:36:06 2017 From: lois.foltan at oracle.com (Lois Foltan) Date: Thu, 19 Oct 2017 14:36:06 -0400 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: <5a52ada7-bba6-dfea-84c6-a3878419f404@oracle.com> References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> <0a9de83e-b424-1f31-9025-9498fddafcf8@oracle.com> <5a52ada7-bba6-dfea-84c6-a3878419f404@oracle.com> Message-ID: On 10/19/2017 2:19 PM, Lois Foltan wrote: > On 10/18/2017 10:32 PM, Ioi Lam wrote: > >> >> >> On 10/18/17 6:26 PM, David Holmes wrote: >>> Hi Ioi, >>> >>> On 19/10/2017 9:54 AM, Ioi Lam wrote: >>>> >>>> >>>> On 10/17/17 9:04 PM, David Holmes wrote: >>>>> Hi Ioi, >>>>> >>>>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>>>> >>>>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>>>> >>>>>> Summary: >>>>>> >>>>>> DumpLoadedClassList skips classes that can't be stored into the CDS >>>>>> archive. For boot and platform loaders, only classes from the JRT >>>>>> image >>>>>> are listed. >>>>>> >>>>>> The test for "is this class from JRT image" was insufficient. It >>>>>> checked only classes whose source ends with "...../modules". For the >>>>>> platform loader, which loads the graal classes, the source is in the >>>>>> form "jrt:/jdk.internal.vm.compiler". >>>>>> >>>>>> The fix is to also check for a "jrt:" prefix. >>>>> >>>>> So ./share/classfile/classLoader.hpp: >>>>> >>>>> static bool is_jrt(const char* name) { return >>>>> string_ends_with(name, MODULES_IMAGE_NAME); } >>>>> >>>>> is a badly named function or just plain broken? >>>>> >>>> >>>> I've renamed all the is_jrt() functions to is_modules_image(), so >>>> it's clear as to what it does. >>>> >>>> See >>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >>> >>> >>> >>> >>> Okay. But I'm still a bit confused about the expectations of >>> is_modules_image() - where are these names defined? What does jrt: >>> mean if not "in a module image" ?? > > Hi Ioi, > I think it would be better to rename "is_jrt" to > "is_java_runtime_image".? It has nothing to do with modules.? It has > more to do with the JDK built as a "run-time" image vs. the JDK built > as an exploded build. > Thanks, > Lois On 2nd look, I actually think is_modules_image() is probably a more accurate name.? So I'm good with your changes.? We are using ClassLoader::has_jrt_entry() to distinguish between a java runtime images vs. a exploded builds.? Sorry for the confusion. Lois > >>> >>> That aside your fix does what you needed, so looks fine. >>> >> >> The problem we have is the ClassFileParser::source()? returns >> "/lib/modules" for the boot loader (C code), and >> "jrt:/java.compiler" for the platform loader (Java code). This should >> probably be fixed, but I don't want to deal with that now :-( >> >> >>>> >>>> >>>>>> I also restructured the code to make it easier to read -- the >>>>>> nesting of || and && is hard to follow. >>>>>> >>>>>> The original code had a subtle bug with the "!" operator: >>>>>> >>>>>> 5938?? if (((class_loader == NULL && >>>>>> !ClassLoader::contains_append_entry(stream->source())) || >>>>>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>>>>> 5940?????? !ClassLoader::is_jrt(stream->source())) { >>>>>> >>>>>> So if (class_loader == NULL && >>>>>> ClassLoader::contains_append_entry(stream->source()) was true, >>>>>> the class would NOT be skipped. >>>>>> >>>>>> The new version removes the "!". >>>>> >>>>> So there should be an updated test to trigger the original bug. >>>>> >>>> >>>> I added a test case, but it's in the closed repo for now (since the >>>> test case uses utilities in the closed AppCDS tests). This test >>>> will be opened as part of JDK-8185996 >>> >>> Ok. >>> >> >> Thanks >> - Ioi >> >>> Thanks, >>> David >>> >>>> Thanks >>>> - Ioi >>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> Thanks >>>>>> - Ioi >>>> >> > From lois.foltan at oracle.com Thu Oct 19 18:50:45 2017 From: lois.foltan at oracle.com (Lois Foltan) Date: Thu, 19 Oct 2017 14:50:45 -0400 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> Message-ID: <8c21fbff-f1cc-e79e-cca6-c39e1f3568ee@oracle.com> Looks good Harold.? Thank you for fixing this! Lois On 10/16/2017 10:20 AM, harold seigel wrote: > Hi, > > Please review this new JDK-10 fix for bug JDK-8174954.? If the initial > resolution attempt fails with a LinkageError exception then the fix > saves the exception in the resolution_errors table and sets a flag in > the constant pool Cache, indicating the failure.? Subsequent attempts > to do the same resolution will see that the flag is set, retrieve the > exception from the resolution_errors table, and throw it, instead of > re-trying the resolution. > > The fix also prevents a race condition if two or more threads try to > do the same resolution concurrently.? When a thread tries to record > the result of its resolution attempt, it will check to see if another > thread recorded its result first.? If so, then it will use the result > recorded by the other thread.? Otherwise, it will record and use its > result.? Access to the resolution result is protected by the > 'resolved_references' lock. > > Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ > > JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 > > The fix was tested with the JCK Lang and VM tests, the JTreg hotspot, > java/io, java/lang, java/util, and other tests, the co-located NSK > tests, JPRT, with Mach5 tier2 - tier5 tests, and by hand to check race > condition handling. > > Thanks, Harold > From rkennke at redhat.com Thu Oct 19 19:14:39 2017 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 19 Oct 2017 21:14:39 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: References: Message-ID: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> Am 13.10.2017 um 03:20 schrieb David Holmes: > Hi Roman, > > Not a review as GC folk need to do that. > > On 13/10/2017 5:59 AM, Roman Kennke wrote: >> I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev because >> it touches both areas (i.e. the GC interface). >> >> Currently, all GC related argument processing is done in >> arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all sorts >> of GC specific methods etc. > > From a runtime perspective I like all the GC specific ifdefs and > settings moved out-of-line of the main argument processing. > > From a refactoring perspective I noticed that set_object_alignment() > no longer calls set_cms_values(). I presume setting it elsewhere is okay? I totally forgot to reply to this. What's important is that the CMS alignment values are set after set_object_alignment() figured out the object alignment. The patch moves that a little further down the road to the beginning of its GC specific argument processing, but from GC perspective should be the same. I looked thoroughly through the involved code paths and cannot see what could go wrong. I need one more review. GC folks? The current webrev is: http://cr.openjdk.java.net/~rkennke/8189171/webrev.01/ Thanks, Roman From harold.seigel at oracle.com Thu Oct 19 19:16:09 2017 From: harold.seigel at oracle.com (harold seigel) Date: Thu, 19 Oct 2017 15:16:09 -0400 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: <8c21fbff-f1cc-e79e-cca6-c39e1f3568ee@oracle.com> References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <8c21fbff-f1cc-e79e-cca6-c39e1f3568ee@oracle.com> Message-ID: Thanks for the review! Harold On 10/19/2017 2:50 PM, Lois Foltan wrote: > Looks good Harold.? Thank you for fixing this! > Lois > > On 10/16/2017 10:20 AM, harold seigel wrote: >> Hi, >> >> Please review this new JDK-10 fix for bug JDK-8174954.? If the >> initial resolution attempt fails with a LinkageError exception then >> the fix saves the exception in the resolution_errors table and sets a >> flag in the constant pool Cache, indicating the failure.? Subsequent >> attempts to do the same resolution will see that the flag is set, >> retrieve the exception from the resolution_errors table, and throw >> it, instead of re-trying the resolution. >> >> The fix also prevents a race condition if two or more threads try to >> do the same resolution concurrently.? When a thread tries to record >> the result of its resolution attempt, it will check to see if another >> thread recorded its result first.? If so, then it will use the result >> recorded by the other thread.? Otherwise, it will record and use its >> result.? Access to the resolution result is protected by the >> 'resolved_references' lock. >> >> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ >> >> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 >> >> The fix was tested with the JCK Lang and VM tests, the JTreg hotspot, >> java/io, java/lang, java/util, and other tests, the co-located NSK >> tests, JPRT, with Mach5 tier2 - tier5 tests, and by hand to check >> race condition handling. >> >> Thanks, Harold >> > From karen.kinnear at oracle.com Thu Oct 19 21:06:58 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Thu, 19 Oct 2017 17:06:58 -0400 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> Message-ID: <99DAE560-8B51-4B0A-9F84-0461771BC9C4@oracle.com> Harold, Looks good. Thank you for the new tests and for listing what you tested. thanks, Karen > On Oct 16, 2017, at 10:20 AM, harold seigel wrote: > > Hi, > > Please review this new JDK-10 fix for bug JDK-8174954. If the initial resolution attempt fails with a LinkageError exception then the fix saves the exception in the resolution_errors table and sets a flag in the constant pool Cache, indicating the failure. Subsequent attempts to do the same resolution will see that the flag is set, retrieve the exception from the resolution_errors table, and throw it, instead of re-trying the resolution. > > The fix also prevents a race condition if two or more threads try to do the same resolution concurrently. When a thread tries to record the result of its resolution attempt, it will check to see if another thread recorded its result first. If so, then it will use the result recorded by the other thread. Otherwise, it will record and use its result. Access to the resolution result is protected by the 'resolved_references' lock. > > Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ > > JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 > > The fix was tested with the JCK Lang and VM tests, the JTreg hotspot, java/io, java/lang, java/util, and other tests, the co-located NSK tests, JPRT, with Mach5 tier2 - tier5 tests, and by hand to check race condition handling. > > Thanks, Harold > From jiangli.zhou at oracle.com Thu Oct 19 21:57:23 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Thu, 19 Oct 2017 14:57:23 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> <732B9B1D-6593-4E2F-ACAC-0BD25C55FA43@oracle.com> <88a9a707-1158-f555-ffb0-653fe9bd9483@oracle.com> Message-ID: Hi Ioi, > On Oct 19, 2017, at 10:55 AM, Ioi Lam wrote: > > > > On 10/19/17 9:47 AM, Jiangli Zhou wrote: >> >>> On Oct 18, 2017, at 9:50 PM, Ioi Lam > wrote: >>> >>> >>> >>> On 10/18/17 9:35 PM, Jiangli Zhou wrote: >>>> Hi Ioi, >>>> >>>>> On Oct 18, 2017, at 4:54 PM, Ioi Lam > wrote: >>>>> >>>>> >>>>> >>>>> On 10/17/17 9:04 PM, David Holmes wrote: >>>>>> Hi Ioi, >>>>>> >>>>>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>>>>> >>>>>>> Summary: >>>>>>> >>>>>>> DumpLoadedClassList skips classes that can't be stored into the CDS >>>>>>> archive. For boot and platform loaders, only classes from the JRT image >>>>>>> are listed. >>>>>>> >>>>>>> The test for "is this class from JRT image" was insufficient. It >>>>>>> checked only classes whose source ends with "...../modules". For the >>>>>>> platform loader, which loads the graal classes, the source is in the >>>>>>> form "jrt:/jdk.internal.vm.compiler". >>>>>>> >>>>>>> The fix is to also check for a "jrt:" prefix. >>>>>> >>>>>> So ./share/classfile/classLoader.hpp: >>>>>> >>>>>> static bool is_jrt(const char* name) { return string_ends_with(name, MODULES_IMAGE_NAME); } >>>>>> >>>>>> is a badly named function or just plain broken? >>>>>> >>>>> >>>>> I've renamed all the is_jrt() functions to is_modules_image(), so it's clear as to what it does. >>>>> >>>>> See http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >>>> >>>> The logic looks puzzling. For both NULL class loader and the PlatformClassLoader: >>>> >>>> The first if check at line 5938 excludes any classes not in the jimage. >>>> >>>> 5938 if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { >>>> 5939 skip = true; >>>> 5940 } >>>> The second if check at line 5942 skips any classes from the boot append class path loaded by the NULL class loader. The second check skips a subset of the classes that?s already skipped by the first check. The second check is redundant. >>>> >>> You're right. The second check is redundant. I'll remove it. >>> >>>> 5942 if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { >>>> 5943 // For the boot loader, also skip all classes that are loaded from the appended entries >>>> 5944 skip = true; >>>> 5945 } >>>> What about the classes from -Xbootclasspath/a (not in --patch-module and --upgrade-module-path)? >>>> >>> I am not sure what you mean here. >> >> This excludes all classes from the -Xbootclasspath/a. However, classes from -Xbootclasspath/a can be archived and used at runtime properly. >> > > Hi Jiangli, > > You're right. I misunderstood the original comments about the treatment of the bootclasspath/a classes. Here's the updated code that should be easier to understand: > > bool skip = false; > if (class_loader == NULL || SystemDictionary::is_platform_class_loader(class_loader)) { > // For the boot and platform class loaders, skip classes that are not found in the > // java runtime image, such as those found in the --patch-module entries. > // These classes can't be loaded from the archive during runtime. > if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { > skip = true; > } > > if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { > // .. but don't skip the boot classes that are loaded from -Xbootclasspath/a > // as they can be loaded from the archive during runtime. > skip = false; > } > } Can you please check with a exploded build if the above also works? What is the stream->source() from classes from exploded build loaded by the NULL loader and the PlatformClassLoader? Thanks, Jiangli > > Thanks > - Ioi > > >> Thanks, >> Jiangli >> >>> >>> Thanks >>> - Ioi >>>> Thanks, >>>> Jiangli >>>> >>>>> >>>>> >>>>>>> I also restructured the code to make it easier to read -- the nesting of || and && is hard to follow. >>>>>>> >>>>>>> The original code had a subtle bug with the "!" operator: >>>>>>> >>>>>>> 5938 if (((class_loader == NULL && !ClassLoader::contains_append_entry(stream->source())) || >>>>>>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>>>>>> 5940 !ClassLoader::is_jrt(stream->source())) { >>>>>>> >>>>>>> So if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source()) was true, >>>>>>> the class would NOT be skipped. >>>>>>> >>>>>>> The new version removes the "!". >>>>>> >>>>>> So there should be an updated test to trigger the original bug. >>>>>> >>>>> >>>>> I added a test case, but it's in the closed repo for now (since the test case uses utilities in the closed AppCDS tests). This test will be opened as part of JDK-8185996 >>>>> >>>>> Thanks >>>>> - Ioi >>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>>> Thanks >>>>>>> - Ioi >>>>> >>>> >>> >> > From mandy.chung at oracle.com Thu Oct 19 23:05:38 2017 From: mandy.chung at oracle.com (mandy chung) Date: Thu, 19 Oct 2017 16:05:38 -0700 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <0b79bce9-1166-3e6a-1352-d5cdc3c91379@oracle.com> Message-ID: On 10/18/17 9:03 PM, mandy chung wrote: > >> >> 30? * @run main/othervm/native >> >> what does the "native" do? I don't see it documented. >> > > I have to find the documentation.? There are many hotspot tests using > native that builds native library. The jtreg tag spec [1] may be not up-to-date and I don't find the native tag. You can check https://bugs.openjdk.java.net/browse/CODETOOLS-7900892 that integrates this jtreg native support. Mandy [1] http://openjdk.java.net/jtreg/tag-spec.html From ioi.lam at oracle.com Thu Oct 19 23:06:07 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 19 Oct 2017 16:06:07 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> <732B9B1D-6593-4E2F-ACAC-0BD25C55FA43@oracle.com> <88a9a707-1158-f555-ffb0-653fe9bd9483@oracle.com> Message-ID: On 10/19/17 2:57 PM, Jiangli Zhou wrote: > Hi Ioi, > >> On Oct 19, 2017, at 10:55 AM, Ioi Lam > > wrote: >> >> >> >> On 10/19/17 9:47 AM, Jiangli Zhou wrote: >>> >>>> On Oct 18, 2017, at 9:50 PM, Ioi Lam >>> > wrote: >>>> >>>> >>>> >>>> On 10/18/17 9:35 PM, Jiangli Zhou wrote: >>>>> Hi Ioi, >>>>> >>>>>> On Oct 18, 2017, at 4:54 PM, Ioi Lam >>>>> > wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 10/17/17 9:04 PM, David Holmes wrote: >>>>>>> Hi Ioi, >>>>>>> >>>>>>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>>>>>> >>>>>>>> >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>>>>>> >>>>>>>> Summary: >>>>>>>> >>>>>>>> DumpLoadedClassList skips classes that can't be stored into the CDS >>>>>>>> archive. For boot and platform loaders, only classes from the >>>>>>>> JRT image >>>>>>>> are listed. >>>>>>>> >>>>>>>> The test for "is this class from JRT image" was insufficient. It >>>>>>>> checked only classes whose source ends with "...../modules". >>>>>>>> For the >>>>>>>> platform loader, which loads the graal classes, the source is >>>>>>>> in the >>>>>>>> form "jrt:/jdk.internal.vm.compiler". >>>>>>>> >>>>>>>> The fix is to also check for a "jrt:" prefix. >>>>>>> >>>>>>> So ./share/classfile/classLoader.hpp: >>>>>>> >>>>>>> static bool is_jrt(const char* name) { return >>>>>>> string_ends_with(name, MODULES_IMAGE_NAME); } >>>>>>> >>>>>>> is a badly named function or just plain broken? >>>>>>> >>>>>> >>>>>> I've renamed all the is_jrt() functions to is_modules_image(), so >>>>>> it's clear as to what it does. >>>>>> >>>>>> See >>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >>>>>> >>>>> >>>>> The logic looks puzzling. For both NULL class loader and the >>>>> PlatformClassLoader: >>>>> >>>>> The first if check at line 5938 excludes any classes not in the >>>>> jimage. >>>>> >>>>> 5938 if (!ClassLoader::is_modules_image(stream->source()) && >>>>> strncmp(stream->source(), "jrt:", 4) != 0) { >>>>> 5939 skip = true; >>>>> 5940 } >>>>> The second if check at line 5942 skips any classes from the boot >>>>> append class path loaded by the NULL class loader. The second >>>>> check skips a subset of the classes that?s already skipped by the >>>>> first check. The second check is redundant. >>>>> >>>> You're right. The second check is redundant. I'll remove it. >>>> >>>>> 5942 if (class_loader == NULL && >>>>> ClassLoader::contains_append_entry(stream->source())) { >>>>> 5943 // For the boot loader, also skip all classes that are loaded >>>>> from the appended entries >>>>> 5944 skip = true; >>>>> 5945 } >>>>> What about the classes from -Xbootclasspath/a (not in >>>>> --patch-module and --upgrade-module-path)? >>>>> >>>> I am not sure what you mean here. >>> >>> This excludes all classes from the -Xbootclasspath/a. However, >>> classes from -Xbootclasspath/a can be archived and used at runtime >>> properly. >>> >> >> Hi Jiangli, >> >> You're right. I misunderstood the original comments about the >> treatment of the bootclasspath/a classes. Here's the updated code >> that should be easier to understand: >> >> ??????? bool skip = false; >> ??????? if (class_loader == NULL || >> SystemDictionary::is_platform_class_loader(class_loader)) { >> ????????? // For the boot and platform class loaders, skip classes >> that are not found in the >> ????????? // java runtime image, such as those found in the >> --patch-module entries. >> ????????? // These classes can't be loaded from the archive during >> runtime. >> ????????? if (!ClassLoader::is_modules_image(stream->source()) && >> strncmp(stream->source(), "jrt:", 4) != 0) { >> ??????????? skip = true; >> ????????? } >> >> ????????? if (class_loader == NULL && >> ClassLoader::contains_append_entry(stream->source())) { >> ??????????? // .. but don't skip the boot classes that are loaded >> from -Xbootclasspath/a >> ??????????? // as theycan be loaded from the archive during runtime. >> ??????????? skip = false; >> ????????? } >> ??????? } > > Can you please check with a exploded build if the above also works? > What is the stream->source() from classes from exploded build loaded > by the NULL loader and the PlatformClassLoader? > Hi Jiangli, I checked the exploded build and all classes are skipped now, because they are not loaded from the $JAVA_HOME/lib/modules file. The stream->source looks like this: skip writing class sun/security/util/FilePermCompat from source /jdk/bld/cons/jdk/modules/java.base to classlist file However, I am not sure if this is relevant, since CDS is not supported with the exploded build. BTW, DumpLoadedClassList is used during the JDK build process to generate the classlist file. However, this uses the interim image, which is not an exploded build: open/make/GenerateLinkOptData.gmk:??? $(FIXPATH) $(INTERIM_IMAGE_DIR)/bin/java -XX:DumpLoadedClassList=$@ \ $ ./support/interim-image/bin/java -cp /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d:/jdk/cons/closed/test/hotspot/jtreg/runtime/AppCDS:/jdk/tmp/jtreg/work/classes/2/open/test/lib:/jdk/cons/open/test/lib:/java_re2/misc/promoted/jtreg/4.2/fcs/b09/binaries/jtreg/lib/javatest.jar:/jtreg/4.2/fcs/b09/binaries/jtreg/lib/jtreg.jar -XX:DumpLoadedClassList=app.list --patch-module=java.base=/jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/javabase.jar -Xbootclasspath/a:/jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/bootappend.jar -cp /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/app.jar ArrayListTest hello world. skip writing class java/lang/NewClass from source /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/javabase.jar to classlist file NewClass class java.lang.NewClass Foo class boot.append.Foo ioilinux ~/jdk/bld/cons$ wc app.list ? 701?? 701 22630 app.list Thanks - Ioi > Thanks, > Jiangli > >> >> Thanks >> - Ioi >> >> >>> Thanks, >>> Jiangli >>> >>>> >>>> Thanks >>>> - Ioi >>>>> Thanks, >>>>> Jiangli >>>>> >>>>>> >>>>>> >>>>>>>> I also restructured the code to make it easier to read -- the >>>>>>>> nesting of || and && is hard to follow. >>>>>>>> >>>>>>>> The original code had a subtle bug with the "!" operator: >>>>>>>> >>>>>>>> 5938?? if (((class_loader == NULL && >>>>>>>> !ClassLoader::contains_append_entry(stream->source())) || >>>>>>>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>>>>>>> 5940 !ClassLoader::is_jrt(stream->source())) { >>>>>>>> >>>>>>>> So if (class_loader == NULL && >>>>>>>> ClassLoader::contains_append_entry(stream->source()) was true, >>>>>>>> the class would NOT be skipped. >>>>>>>> >>>>>>>> The new version removes the "!". >>>>>>> >>>>>>> So there should be an updated test to trigger the original bug. >>>>>>> >>>>>> >>>>>> I added a test case, but it's in the closed repo for now (since >>>>>> the test case uses utilities in the closed AppCDS tests). This >>>>>> test will be opened as part of JDK-8185996 >>>>>> >>>>>> Thanks >>>>>> - Ioi >>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>>> Thanks >>>>>>>> - Ioi >>>>>> >>>>> >>>> >>> >> > From jiangli.zhou at Oracle.COM Fri Oct 20 00:02:33 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Thu, 19 Oct 2017 17:02:33 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> <732B9B1D-6593-4E2F-ACAC-0BD25C55FA43@oracle.com> <88a9a707-1158-f555-ffb0-653fe9bd9483@oracle.com> Message-ID: > On Oct 19, 2017, at 4:06 PM, Ioi Lam wrote: > > > > On 10/19/17 2:57 PM, Jiangli Zhou wrote: >> Hi Ioi, >> >>> On Oct 19, 2017, at 10:55 AM, Ioi Lam > wrote: >>> >>> >>> >>> On 10/19/17 9:47 AM, Jiangli Zhou wrote: >>>> >>>>> On Oct 18, 2017, at 9:50 PM, Ioi Lam > wrote: >>>>> >>>>> >>>>> >>>>> On 10/18/17 9:35 PM, Jiangli Zhou wrote: >>>>>> Hi Ioi, >>>>>> >>>>>>> On Oct 18, 2017, at 4:54 PM, Ioi Lam > wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 10/17/17 9:04 PM, David Holmes wrote: >>>>>>>> Hi Ioi, >>>>>>>> >>>>>>>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>>>>>>> >>>>>>>>> Summary: >>>>>>>>> >>>>>>>>> DumpLoadedClassList skips classes that can't be stored into the CDS >>>>>>>>> archive. For boot and platform loaders, only classes from the JRT image >>>>>>>>> are listed. >>>>>>>>> >>>>>>>>> The test for "is this class from JRT image" was insufficient. It >>>>>>>>> checked only classes whose source ends with "...../modules". For the >>>>>>>>> platform loader, which loads the graal classes, the source is in the >>>>>>>>> form "jrt:/jdk.internal.vm.compiler". >>>>>>>>> >>>>>>>>> The fix is to also check for a "jrt:" prefix. >>>>>>>> >>>>>>>> So ./share/classfile/classLoader.hpp: >>>>>>>> >>>>>>>> static bool is_jrt(const char* name) { return string_ends_with(name, MODULES_IMAGE_NAME); } >>>>>>>> >>>>>>>> is a badly named function or just plain broken? >>>>>>>> >>>>>>> >>>>>>> I've renamed all the is_jrt() functions to is_modules_image(), so it's clear as to what it does. >>>>>>> >>>>>>> See http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >>>>>> >>>>>> The logic looks puzzling. For both NULL class loader and the PlatformClassLoader: >>>>>> >>>>>> The first if check at line 5938 excludes any classes not in the jimage. >>>>>> >>>>>> 5938 if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { >>>>>> 5939 skip = true; >>>>>> 5940 } >>>>>> The second if check at line 5942 skips any classes from the boot append class path loaded by the NULL class loader. The second check skips a subset of the classes that?s already skipped by the first check. The second check is redundant. >>>>>> >>>>> You're right. The second check is redundant. I'll remove it. >>>>> >>>>>> 5942 if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { >>>>>> 5943 // For the boot loader, also skip all classes that are loaded from the appended entries >>>>>> 5944 skip = true; >>>>>> 5945 } >>>>>> What about the classes from -Xbootclasspath/a (not in --patch-module and --upgrade-module-path)? >>>>>> >>>>> I am not sure what you mean here. >>>> >>>> This excludes all classes from the -Xbootclasspath/a. However, classes from -Xbootclasspath/a can be archived and used at runtime properly. >>>> >>> >>> Hi Jiangli, >>> >>> You're right. I misunderstood the original comments about the treatment of the bootclasspath/a classes. Here's the updated code that should be easier to understand: >>> >>> bool skip = false; >>> if (class_loader == NULL || SystemDictionary::is_platform_class_loader(class_loader)) { >>> // For the boot and platform class loaders, skip classes that are not found in the >>> // java runtime image, such as those found in the --patch-module entries. >>> // These classes can't be loaded from the archive during runtime. >>> if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { >>> skip = true; >>> } >>> >>> if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { >>> // .. but don't skip the boot classes that are loaded from -Xbootclasspath/a >>> // as they can be loaded from the archive during runtime. >>> skip = false; >>> } >>> } >> >> Can you please check with a exploded build if the above also works? What is the stream->source() from classes from exploded build loaded by the NULL loader and the PlatformClassLoader? >> > > Hi Jiangli, > > I checked the exploded build and all classes are skipped now, because they are not loaded from the $JAVA_HOME/lib/modules file. The stream->source looks like this: > > skip writing class sun/security/util/FilePermCompat from source /jdk/bld/cons/jdk/modules/java.base to classlist file > > However, I am not sure if this is relevant, since CDS is not supported with the exploded build. > > BTW, DumpLoadedClassList is used during the JDK build process to generate the classlist file. However, this uses the interim image, which is not an exploded build: > > open/make/GenerateLinkOptData.gmk: $(FIXPATH) $(INTERIM_IMAGE_DIR)/bin/java -XX:DumpLoadedClassList=$@ \ > > > $ ./support/interim-image/bin/java -cp /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d:/jdk/cons/closed/test/hotspot/jtreg/runtime/AppCDS:/jdk/tmp/jtreg/work/classes/2/open/test/lib:/jdk/cons/open/test/lib:/java_re2/misc/promoted/jtreg/4.2/fcs/b09/binaries/jtreg/lib/javatest.jar:/jtreg/4.2/fcs/b09/binaries/jtreg/lib/jtreg.jar -XX:DumpLoadedClassList=app.list --patch-module=java.base=/jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/javabase.jar -Xbootclasspath/a:/jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/bootappend.jar -cp /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/app.jar ArrayListTest > hello world. > skip writing class java/lang/NewClass from source /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/javabase.jar to classlist file > NewClass > class java.lang.NewClass > Foo > class boot.append.Foo > ioilinux ~/jdk/bld/cons$ wc app.list > 701 701 22630 app.list That?s what I expected. Thanks for verifying it. Since the goal here is to generate a more ?precise? classlist for CDS and CDS does not support exploded build, it would make sense to print a warning and disable DumpLoadedClassList if ClassLoader::has_jrt_entry() is not true. Thanks, Jiangli > > > Thanks > - Ioi > >> Thanks, >> Jiangli >> >>> >>> Thanks >>> - Ioi >>> >>> >>>> Thanks, >>>> Jiangli >>>> >>>>> >>>>> Thanks >>>>> - Ioi >>>>>> Thanks, >>>>>> Jiangli >>>>>> >>>>>>> >>>>>>> >>>>>>>>> I also restructured the code to make it easier to read -- the nesting of || and && is hard to follow. >>>>>>>>> >>>>>>>>> The original code had a subtle bug with the "!" operator: >>>>>>>>> >>>>>>>>> 5938 if (((class_loader == NULL && !ClassLoader::contains_append_entry(stream->source())) || >>>>>>>>> 5939 SystemDictionary::is_platform_class_loader(class_loader)) && >>>>>>>>> 5940 !ClassLoader::is_jrt(stream->source())) { >>>>>>>>> >>>>>>>>> So if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source()) was true, >>>>>>>>> the class would NOT be skipped. >>>>>>>>> >>>>>>>>> The new version removes the "!". >>>>>>>> >>>>>>>> So there should be an updated test to trigger the original bug. >>>>>>>> >>>>>>> >>>>>>> I added a test case, but it's in the closed repo for now (since the test case uses utilities in the closed AppCDS tests). This test will be opened as part of JDK-8185996 >>>>>>> >>>>>>> Thanks >>>>>>> - Ioi >>>>>>> >>>>>>>> Thanks, >>>>>>>> David >>>>>>>> >>>>>>>>> Thanks >>>>>>>>> - Ioi >>>>>>> >>>>>> >>>>> >>>> >>> >> > From david.holmes at oracle.com Fri Oct 20 00:32:12 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Oct 2017 10:32:12 +1000 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <9b029d2f-3c59-a341-c9d6-802d1e2508a7@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <96831c9b-014d-f4ee-4150-d42af2228539@oracle.com> <9b029d2f-3c59-a341-c9d6-802d1e2508a7@oracle.com> Message-ID: <35953e4e-e20c-9c13-fbf6-4d045e007b7f@oracle.com> Hi Mandy, On 20/10/2017 4:11 AM, mandy chung wrote: > Thanks Coleen.?? I renamed Driver.java to FindClassFromBoot.java which > is a better test name than Driver (that should address David's > comment).? No other change. > > http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.02/ Sorry I missed the test updates till now. I don't think you needed to complicate the NCDFE handling at the native level. I would have just defined a negative test that expects to get NCDFE. (If the subsequent FindClass(env, "java/lang/NoClassDefFoundError"); itself fails (eg OOME) then you may crash on the instanceof call.) Up to you whether to proceed with current approach or not. Thanks, David > I tested this with jdk core groups and hotspot group. > > Mandy > > On 10/19/17 10:12 AM, coleen.phillimore at oracle.com wrote: >> Looks great! >> Coleen >> >> On 10/19/17 11:53 AM, mandy chung wrote: >>> David, Coleen, >>> >>> I have revised the patch to use Handle (much simpler, thanks) and >>> update the test to clear the exception: >>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.01/index.html >>> >>> >>> Thanks >>> Mandy >>> >>> On 10/18/17 5:26 PM, David Holmes wrote: >>>> Hi Mandy, >>>> >>>> On 19/10/2017 7:52 AM, mandy chung wrote: >>>>> The bug creeped in when I improved the code to use loader.is_null() >>>>> to set to system class loader per Coleen's suggestion.? It's my >>>>> oversight of the null loader case.? The is the version I pushed. >>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >>>>> >>>>> The previous version was correct. >>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ >>>> >>>> Yes I see what happened - and we all missed it. And there was no >>>> existing test it seems :( >>>> >>>>> I didn't go back to webrev.01.? The fix is an improvement to it. >>>> >>>> Okay, only concern I have is that this: >>>> >>>> 403?? oop loader = SystemDictionary::java_system_loader(); >>>> >>>> certainly looks like a "naked oop" - and the Java call may lead to >>>> GC occurring. >>>> >>>> The test seems very lonely, where are the onLoad tests? >>>> >>>> In? Driver: >>>> >>>> 30? * @run main/othervm/native >>>> >>>> what does the "native" do? I don't see it documented. >>>> >>>> Regarding this comment in BootLoaderTest: >>>> >>>> // The JNI spec says FindClass returns NULL if class not found but >>>> // it also says that NCDFE is thrown if no definition can be found. >>>> // The current implementation throws NCDFE. >>>> >>>> The NCDFE is set pending by FindClass. If you return (or call into) >>>> java code then it will be thrown. But if you were remaining in >>>> native then you'd use the NULL return to determine success or >>>> failure (or check pending exception). >>>> >>>> Thanks, >>>> David >>>> >>>>> Mandy >>>>> >>>>> On 10/18/17 2:48 PM, David Holmes wrote: >>>>>> Hi Mandy, >>>>>> >>>>>> Do you still have the webrev for 8188052? I need to see how this >>>>>> got through given we looked at it so many times. :( >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>>>>> This is a regression caused by JDK-8188052. FindClass should only >>>>>>> use system class loader when there is no caller class or when >>>>>>> it's called from JNI_OnUnload. When there is a caller class, it >>>>>>> should either use its class loader or the one associated with >>>>>>> JNI_Onload. >>>>>>> >>>>>>> Webrev: >>>>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>>>>> >>>>>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>>>>> >>>>>>> thanks >>>>>>> Mandy >>>>> >>> >> > From mandy.chung at oracle.com Fri Oct 20 00:40:34 2017 From: mandy.chung at oracle.com (mandy chung) Date: Thu, 19 Oct 2017 17:40:34 -0700 Subject: Review Request JDK-8189193: FindClass mistakenly uses system class loader when the initiating loader is bootstrap loader In-Reply-To: <35953e4e-e20c-9c13-fbf6-4d045e007b7f@oracle.com> References: <6def7a58-1877-4b67-4dda-b54994af013a@oracle.com> <033b7e85-8c35-0d73-62df-a73b36acc0f9@oracle.com> <96831c9b-014d-f4ee-4150-d42af2228539@oracle.com> <9b029d2f-3c59-a341-c9d6-802d1e2508a7@oracle.com> <35953e4e-e20c-9c13-fbf6-4d045e007b7f@oracle.com> Message-ID: <32d4f20d-91e7-aa44-683d-1ecb461e0cbb@oracle.com> On 10/19/17 5:32 PM, David Holmes wrote: > Hi Mandy, > > On 20/10/2017 4:11 AM, mandy chung wrote: >> Thanks Coleen.?? I renamed Driver.java to FindClassFromBoot.java >> which is a better test name than Driver (that should address David's >> comment).? No other change. >> >> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.02/ > > Sorry I missed the test updates till now. I don't think you needed to > complicate the NCDFE handling at the native level. I would have just > defined a negative test that expects to get NCDFE. (If the subsequent > FindClass(env, "java/lang/NoClassDefFoundError"); itself fails (eg > OOME) then you may crash on the instanceof call.) > > Up to you whether to proceed with current approach or not. > I actually like the current approach that BootNativeLibrary::findClass to return null if class not found.? In fact this was my initial version until I realized FindClass throws NCDFE as well.? I don't find the native code complicated. This test is a othervm test and so OOME should be very unlikely. I will push as is. Thanks Mandy > Thanks, > David > >> I tested this with jdk core groups and hotspot group. >> >> Mandy >> >> On 10/19/17 10:12 AM, coleen.phillimore at oracle.com wrote: >>> Looks great! >>> Coleen >>> >>> On 10/19/17 11:53 AM, mandy chung wrote: >>>> David, Coleen, >>>> >>>> I have revised the patch to use Handle (much simpler, thanks) and >>>> update the test to clear the exception: >>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.01/index.html >>>> >>>> >>>> Thanks >>>> Mandy >>>> >>>> On 10/18/17 5:26 PM, David Holmes wrote: >>>>> Hi Mandy, >>>>> >>>>> On 19/10/2017 7:52 AM, mandy chung wrote: >>>>>> The bug creeped in when I improved the code to use >>>>>> loader.is_null() to set to system class loader per Coleen's >>>>>> suggestion.? It's my oversight of the null loader case.? The is >>>>>> the version I pushed. >>>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.02/ >>>>>> >>>>>> The previous version was correct. >>>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8188052/webrev.01/ >>>>> >>>>> Yes I see what happened - and we all missed it. And there was no >>>>> existing test it seems :( >>>>> >>>>>> I didn't go back to webrev.01. The fix is an improvement to it. >>>>> >>>>> Okay, only concern I have is that this: >>>>> >>>>> 403?? oop loader = SystemDictionary::java_system_loader(); >>>>> >>>>> certainly looks like a "naked oop" - and the Java call may lead to >>>>> GC occurring. >>>>> >>>>> The test seems very lonely, where are the onLoad tests? >>>>> >>>>> In? Driver: >>>>> >>>>> 30? * @run main/othervm/native >>>>> >>>>> what does the "native" do? I don't see it documented. >>>>> >>>>> Regarding this comment in BootLoaderTest: >>>>> >>>>> // The JNI spec says FindClass returns NULL if class not found but >>>>> // it also says that NCDFE is thrown if no definition can be found. >>>>> // The current implementation throws NCDFE. >>>>> >>>>> The NCDFE is set pending by FindClass. If you return (or call >>>>> into) java code then it will be thrown. But if you were remaining >>>>> in native then you'd use the NULL return to determine success or >>>>> failure (or check pending exception). >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> Mandy >>>>>> >>>>>> On 10/18/17 2:48 PM, David Holmes wrote: >>>>>>> Hi Mandy, >>>>>>> >>>>>>> Do you still have the webrev for 8188052? I need to see how this >>>>>>> got through given we looked at it so many times. :( >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>> On 19/10/2017 7:30 AM, mandy chung wrote: >>>>>>>> This is a regression caused by JDK-8188052. FindClass should >>>>>>>> only use system class loader when there is no caller class or >>>>>>>> when it's called from JNI_OnUnload. When there is a caller >>>>>>>> class, it should either use its class loader or the one >>>>>>>> associated with JNI_Onload. >>>>>>>> >>>>>>>> Webrev: >>>>>>>> http://cr.openjdk.java.net/~mchung/jdk10/webrevs/8189193/webrev.00/ >>>>>>>> >>>>>>>> >>>>>>>> JBS:https://bugs.openjdk.java.net/browse/JDK-8189193 >>>>>>>> >>>>>>>> thanks >>>>>>>> Mandy >>>>>> >>>> >>> >> From david.holmes at oracle.com Fri Oct 20 05:15:33 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 20 Oct 2017 15:15:33 +1000 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> Message-ID: <2827d94c-1744-86da-b7bc-d67554cb3519@oracle.com> Hi Harold, On 19/10/2017 12:20 AM, harold seigel wrote: > Hi David, > > Thanks for looking at this. > > Please see inline comments. > > Thanks, Harold > > > On 10/17/2017 10:22 PM, David Holmes wrote: >> Hi Harold, >> >> On 17/10/2017 12:20 AM, harold seigel wrote: >>> Hi, >>> >>> Please review this new JDK-10 fix for bug JDK-8174954.? If the >>> initial resolution attempt fails with a LinkageError exception then >>> the fix saves the exception in the resolution_errors table and sets a >>> flag in the constant pool Cache, indicating the failure.? Subsequent >>> attempts to do the same resolution will see that the flag is set, >>> retrieve the exception from the resolution_errors table, and throw >>> it, instead of re-trying the resolution. >> >> I'm a little confused. The fix seems all about indy, which seems to be >> the topic of: >> >> ?JDK-8174942 Bootstrap Method Called Multiple Times Despite Initial >> Resolution Failure >> >> whereas this bug seems to be about a more general problem. Is this fix >> addressing both issues?? > Thanks for noticing this.? This fix addresses both bugs.? The sample > program for 8174942 is one of the test cases for this fix.? I'll close > 8174942 as a duplicate. I'm still somewhat confused as it seems that all the substantive changes here have the word "indy" in them. What part of the change actually fixes the non-indy issue tested by MethodAccessReadTwice.java? Thanks, David >> >> BTW there's no reason for JDK-8174954 to remain confidential. > Agreed.? I opened the bug. >> >> Thanks, >> David >> >>> The fix also prevents a race condition if two or more threads try to >>> do the same resolution concurrently. When a thread tries to record >>> the result of its resolution attempt, it will check to see if another >>> thread recorded its result first.? If so, then it will use the result >>> recorded by the other thread.? Otherwise, it will record and use its >>> result. Access to the resolution result is protected by the >>> 'resolved_references' lock. >>> >>> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ >>> >>> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 >>> >>> The fix was tested with the JCK Lang and VM tests, the JTreg hotspot, >>> java/io, java/lang, java/util, and other tests, the co-located NSK >>> tests, JPRT, with Mach5 tier2 - tier5 tests, and by hand to check >>> race condition handling. >>> >>> Thanks, Harold >>> > From john.r.rose at oracle.com Fri Oct 20 05:48:10 2017 From: john.r.rose at oracle.com (John Rose) Date: Thu, 19 Oct 2017 22:48:10 -0700 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: <2827d94c-1744-86da-b7bc-d67554cb3519@oracle.com> References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> <2827d94c-1744-86da-b7bc-d67554cb3519@oracle.com> Message-ID: <3DEE01DC-F7F8-4940-A689-51168238CA6E@oracle.com> The resolved_references array never has a null value in it except before resolution. Therefore you can use a cas instead of the more expensive lock to race and set the array elements. Why use a lock for this purpose? > On Oct 19, 2017, at 10:15 PM, David Holmes wrote: > > Hi Harold, > >> On 19/10/2017 12:20 AM, harold seigel wrote: >> Hi David, >> Thanks for looking at this. >> Please see inline comments. >> Thanks, Harold >>> On 10/17/2017 10:22 PM, David Holmes wrote: >>> Hi Harold, >>> >>>> On 17/10/2017 12:20 AM, harold seigel wrote: >>>> Hi, >>>> >>>> Please review this new JDK-10 fix for bug JDK-8174954. If the initial resolution attempt fails with a LinkageError exception then the fix saves the exception in the resolution_errors table and sets a flag in the constant pool Cache, indicating the failure. Subsequent attempts to do the same resolution will see that the flag is set, retrieve the exception from the resolution_errors table, and throw it, instead of re-trying the resolution. >>> >>> I'm a little confused. The fix seems all about indy, which seems to be the topic of: >>> >>> JDK-8174942 Bootstrap Method Called Multiple Times Despite Initial Resolution Failure >>> >>> whereas this bug seems to be about a more general problem. Is this fix addressing both issues?? >> Thanks for noticing this. This fix addresses both bugs. The sample program for 8174942 is one of the test cases for this fix. I'll close 8174942 as a duplicate. > > I'm still somewhat confused as it seems that all the substantive changes here have the word "indy" in them. What part of the change actually fixes the non-indy issue tested by MethodAccessReadTwice.java? > > Thanks, > David >>> >>> BTW there's no reason for JDK-8174954 to remain confidential. >> Agreed. I opened the bug. >>> >>> Thanks, >>> David >>> >>>> The fix also prevents a race condition if two or more threads try to do the same resolution concurrently. When a thread tries to record the result of its resolution attempt, it will check to see if another thread recorded its result first. If so, then it will use the result recorded by the other thread. Otherwise, it will record and use its result. Access to the resolution result is protected by the 'resolved_references' lock. >>>> >>>> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ >>>> >>>> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 >>>> >>>> The fix was tested with the JCK Lang and VM tests, the JTreg hotspot, java/io, java/lang, java/util, and other tests, the co-located NSK tests, JPRT, with Mach5 tier2 - tier5 tests, and by hand to check race condition handling. >>>> >>>> Thanks, Harold >>>> From harold.seigel at oracle.com Fri Oct 20 11:53:54 2017 From: harold.seigel at oracle.com (harold seigel) Date: Fri, 20 Oct 2017 07:53:54 -0400 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: <99DAE560-8B51-4B0A-9F84-0461771BC9C4@oracle.com> References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <99DAE560-8B51-4B0A-9F84-0461771BC9C4@oracle.com> Message-ID: <61ac2b54-5bb6-f224-b01a-db6e0f76e848@oracle.com> Thanks for the review! Harold On 10/19/2017 5:06 PM, Karen Kinnear wrote: > Harold, > > Looks good. > Thank you for the new tests and for listing what you tested. > > thanks, > Karen > >> On Oct 16, 2017, at 10:20 AM, harold seigel wrote: >> >> Hi, >> >> Please review this new JDK-10 fix for bug JDK-8174954. If the initial resolution attempt fails with a LinkageError exception then the fix saves the exception in the resolution_errors table and sets a flag in the constant pool Cache, indicating the failure. Subsequent attempts to do the same resolution will see that the flag is set, retrieve the exception from the resolution_errors table, and throw it, instead of re-trying the resolution. >> >> The fix also prevents a race condition if two or more threads try to do the same resolution concurrently. When a thread tries to record the result of its resolution attempt, it will check to see if another thread recorded its result first. If so, then it will use the result recorded by the other thread. Otherwise, it will record and use its result. Access to the resolution result is protected by the 'resolved_references' lock. >> >> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ >> >> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 >> >> The fix was tested with the JCK Lang and VM tests, the JTreg hotspot, java/io, java/lang, java/util, and other tests, the co-located NSK tests, JPRT, with Mach5 tier2 - tier5 tests, and by hand to check race condition handling. >> >> Thanks, Harold >> From erik.osterlund at oracle.com Fri Oct 20 12:18:49 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Fri, 20 Oct 2017 14:18:49 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> Message-ID: <59E9E9A9.7020306@oracle.com> Hi Roman, I would like to keep the CollectedHeap as the facade interface to the GC, like it is today, instead of having a new GC class making it two facade interfaces. Of course, the CollectedHeap may only be used as a facade after it has been created. And now we are dealing with code before it was created during the bootstrapping process. So what I would like to have here is a class specifically for that, rather than another GC facade interface. I think this is mostly an exercise of finding a better name for the class you call "GC". Now I am not an expert in coming up with good names myself, but some name that indicates this is being used for flag processing would be good. Perhaps GCArgumentProcessor. The benefit of calling it something like that is that if a GC requires argument processing later in the bootstrapping, possibly after CollectedHeap has been created, it can still be used for this purpose without having to feel weird about it. How would you feel about that? Does it seem to make sense? Thanks, /Erik On 2017-10-19 21:14, Roman Kennke wrote: > Am 13.10.2017 um 03:20 schrieb David Holmes: >> Hi Roman, >> >> Not a review as GC folk need to do that. >> >> On 13/10/2017 5:59 AM, Roman Kennke wrote: >>> I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev >>> because it touches both areas (i.e. the GC interface). >>> >>> Currently, all GC related argument processing is done in >>> arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all >>> sorts of GC specific methods etc. >> >> From a runtime perspective I like all the GC specific ifdefs and >> settings moved out-of-line of the main argument processing. >> >> From a refactoring perspective I noticed that set_object_alignment() >> no longer calls set_cms_values(). I presume setting it elsewhere is >> okay? > I totally forgot to reply to this. > > What's important is that the CMS alignment values are set after > set_object_alignment() figured out the object alignment. The patch > moves that a little further down the road to the beginning of its GC > specific argument processing, but from GC perspective should be the > same. I looked thoroughly through the involved code paths and cannot > see what could go wrong. > > I need one more review. GC folks? The current webrev is: > http://cr.openjdk.java.net/~rkennke/8189171/webrev.01/ > > > Thanks, > Roman > From rkennke at redhat.com Fri Oct 20 12:25:13 2017 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 20 Oct 2017 14:25:13 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <59E9E9A9.7020306@oracle.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> Message-ID: <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> Hi Erik, Yes that is fine. Keep in mind that I'm planning to put heap creation into that class too. I was thinking maybe to simply reverse the current naming: name the all-static 'factory factory' 'GC', and thus call GC::initialize() and GC::factory() or such, and the main interface GCFactory or CollectedHeapFactory. What do you think? Roman > I would like to keep the CollectedHeap as the facade interface to the > GC, like it is today, instead of having a new GC class making it two > facade interfaces. > Of course, the CollectedHeap may only be used as a facade after it has > been created. And now we are dealing with code before it was created > during the bootstrapping process. > > So what I would like to have here is a class specifically for that, > rather than another GC facade interface. I think this is mostly an > exercise of finding a better name for the class you call "GC". Now I > am not an expert in coming up with good names myself, but some name > that indicates this is being used for flag processing would be good. > Perhaps GCArgumentProcessor. The benefit of calling it something like > that is that if a GC requires argument processing later in the > bootstrapping, possibly after CollectedHeap has been created, it can > still be used for this purpose without having to feel weird about it. > > How would you feel about that? Does it seem to make sense? > > Thanks, > /Erik > > On 2017-10-19 21:14, Roman Kennke wrote: >> Am 13.10.2017 um 03:20 schrieb David Holmes: >>> Hi Roman, >>> >>> Not a review as GC folk need to do that. >>> >>> On 13/10/2017 5:59 AM, Roman Kennke wrote: >>>> I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev >>>> because it touches both areas (i.e. the GC interface). >>>> >>>> Currently, all GC related argument processing is done in >>>> arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all >>>> sorts of GC specific methods etc. >>> >>> From a runtime perspective I like all the GC specific ifdefs and >>> settings moved out-of-line of the main argument processing. >>> >>> From a refactoring perspective I noticed that set_object_alignment() >>> no longer calls set_cms_values(). I presume setting it elsewhere is >>> okay? >> I totally forgot to reply to this. >> >> What's important is that the CMS alignment values are set after >> set_object_alignment() figured out the object alignment. The patch >> moves that a little further down the road to the beginning of its GC >> specific argument processing, but from GC perspective should be the >> same. I looked thoroughly through the involved code paths and cannot >> see what could go wrong. >> >> I need one more review. GC folks? The current webrev is: >> http://cr.openjdk.java.net/~rkennke/8189171/webrev.01/ >> >> >> Thanks, >> Roman >> > From harold.seigel at oracle.com Fri Oct 20 13:37:05 2017 From: harold.seigel at oracle.com (harold seigel) Date: Fri, 20 Oct 2017 09:37:05 -0400 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: <2827d94c-1744-86da-b7bc-d67554cb3519@oracle.com> References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> <2827d94c-1744-86da-b7bc-d67554cb3519@oracle.com> Message-ID: <5c14a878-9dbd-3538-7fd9-1a1067c992e1@oracle.com> Hi David, MethodAccessReadTwice.java is an indy test.? Lines 120 - 153 test that the same invokedynamic instruction gets the same result.? The invokedynamic instruction gets generated for the string concatenation at line 31 of file .../p5/c5.java: 24 package p5; 25 26 import java.lang.Module; 27 import p2.c2; 28 29 public class c5 { 30 public void method5(p2.c2 param) { 31 System.out.println("In c5's method5 with param = " + param); 32 } 33 34 public void methodAddReadEdge(Module m) { 35 // Add a read edge from p5/c5's module (first_mod) to second_mod 36 c5.class.getModule().addReads(m); 37 } 38 } The indy resolution throws an IAE because package p5's module cannot read package p2's module. The second test case, at lines 156 - 161 of MethodAccessReadTwice.java, similarly tests invokedynamic instructions.? In this case, they are generated for the string concatenations at lines 32 and 43 of file .../p7/c7.java. This problem does not exist for non-indy instructions because, when their resolutions fail, they update the constant pool.? Indy cannot do this because, unlike other instructions, different indy instructions that use the same constant pool index are allowed to get different results.? Hence, the fix is indy only. Thanks, Harold On 10/20/2017 1:15 AM, David Holmes wrote: > Hi Harold, > > On 19/10/2017 12:20 AM, harold seigel wrote: >> Hi David, >> >> Thanks for looking at this. >> >> Please see inline comments. >> >> Thanks, Harold >> >> >> On 10/17/2017 10:22 PM, David Holmes wrote: >>> Hi Harold, >>> >>> On 17/10/2017 12:20 AM, harold seigel wrote: >>>> Hi, >>>> >>>> Please review this new JDK-10 fix for bug JDK-8174954.? If the >>>> initial resolution attempt fails with a LinkageError exception then >>>> the fix saves the exception in the resolution_errors table and sets >>>> a flag in the constant pool Cache, indicating the failure.? >>>> Subsequent attempts to do the same resolution will see that the >>>> flag is set, retrieve the exception from the resolution_errors >>>> table, and throw it, instead of re-trying the resolution. >>> >>> I'm a little confused. The fix seems all about indy, which seems to >>> be the topic of: >>> >>> ?JDK-8174942 Bootstrap Method Called Multiple Times Despite Initial >>> Resolution Failure >>> >>> whereas this bug seems to be about a more general problem. Is this >>> fix addressing both issues?? >> Thanks for noticing this.? This fix addresses both bugs.? The sample >> program for 8174942 is one of the test cases for this fix.? I'll >> close 8174942 as a duplicate. > > I'm still somewhat confused as it seems that all the substantive > changes here have the word "indy" in them. What part of the change > actually fixes the non-indy issue tested by MethodAccessReadTwice.java? > > Thanks, > David >>> >>> BTW there's no reason for JDK-8174954 to remain confidential. >> Agreed.? I opened the bug. >>> >>> Thanks, >>> David >>> >>>> The fix also prevents a race condition if two or more threads try >>>> to do the same resolution concurrently. When a thread tries to >>>> record the result of its resolution attempt, it will check to see >>>> if another thread recorded its result first.? If so, then it will >>>> use the result recorded by the other thread.? Otherwise, it will >>>> record and use its result. Access to the resolution result is >>>> protected by the 'resolved_references' lock. >>>> >>>> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ >>>> >>>> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 >>>> >>>> The fix was tested with the JCK Lang and VM tests, the JTreg >>>> hotspot, java/io, java/lang, java/util, and other tests, the >>>> co-located NSK tests, JPRT, with Mach5 tier2 - tier5 tests, and by >>>> hand to check race condition handling. >>>> >>>> Thanks, Harold >>>> >> From erik.osterlund at oracle.com Fri Oct 20 13:54:53 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Fri, 20 Oct 2017 15:54:53 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> Message-ID: <59EA002D.1050605@oracle.com> Hi Roman, I'm usually not one to really mind names too much, but I don't believe some bootstrapping code for the GC should hog the name "GC". Instead of GC::initialize(), I'm thinking a static GCArgumentProcessor::initialize() and instead of GC::factory(), I am thinking a static CollectedHeapFactory::create(). Or something like that. The reason is that I think the name GC is too general for the bootstraping-only problems its code touches. Hope I am making sense. Thanks, /Erik On 2017-10-20 14:25, Roman Kennke wrote: > Hi Erik, > > Yes that is fine. > > Keep in mind that I'm planning to put heap creation into that class too. > > I was thinking maybe to simply reverse the current naming: name the > all-static 'factory factory' 'GC', and thus call GC::initialize() and > GC::factory() or such, and the main interface GCFactory or > CollectedHeapFactory. What do you think? > > Roman > > >> I would like to keep the CollectedHeap as the facade interface to the >> GC, like it is today, instead of having a new GC class making it two >> facade interfaces. >> Of course, the CollectedHeap may only be used as a facade after it >> has been created. And now we are dealing with code before it was >> created during the bootstrapping process. >> >> So what I would like to have here is a class specifically for that, >> rather than another GC facade interface. I think this is mostly an >> exercise of finding a better name for the class you call "GC". Now I >> am not an expert in coming up with good names myself, but some name >> that indicates this is being used for flag processing would be good. >> Perhaps GCArgumentProcessor. The benefit of calling it something like >> that is that if a GC requires argument processing later in the >> bootstrapping, possibly after CollectedHeap has been created, it can >> still be used for this purpose without having to feel weird about it. >> >> How would you feel about that? Does it seem to make sense? >> >> Thanks, >> /Erik >> >> On 2017-10-19 21:14, Roman Kennke wrote: >>> Am 13.10.2017 um 03:20 schrieb David Holmes: >>>> Hi Roman, >>>> >>>> Not a review as GC folk need to do that. >>>> >>>> On 13/10/2017 5:59 AM, Roman Kennke wrote: >>>>> I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev >>>>> because it touches both areas (i.e. the GC interface). >>>>> >>>>> Currently, all GC related argument processing is done in >>>>> arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all >>>>> sorts of GC specific methods etc. >>>> >>>> From a runtime perspective I like all the GC specific ifdefs and >>>> settings moved out-of-line of the main argument processing. >>>> >>>> From a refactoring perspective I noticed that >>>> set_object_alignment() no longer calls set_cms_values(). I presume >>>> setting it elsewhere is okay? >>> I totally forgot to reply to this. >>> >>> What's important is that the CMS alignment values are set after >>> set_object_alignment() figured out the object alignment. The patch >>> moves that a little further down the road to the beginning of its GC >>> specific argument processing, but from GC perspective should be the >>> same. I looked thoroughly through the involved code paths and cannot >>> see what could go wrong. >>> >>> I need one more review. GC folks? The current webrev is: >>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.01/ >>> >>> >>> Thanks, >>> Roman >>> >> > From zgu at redhat.com Fri Oct 20 14:00:08 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 20 Oct 2017 10:00:08 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information Message-ID: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> Up to now, there is little visibility into how metaspaces are used, and by whom. This proposed enhancement gives NMT the ability to report metaspace usages on per-classloader level, via a new NMT command "metadata" For example: 2496: Metaspaces: Metadata space: reserved= 8192KB committed= 5888KB Class space: reserved= 1048576KB committed= 640KB Per-classloader metadata: ClassLoader: for anonymous class Metadata capacity= 5.00KB used= 1.16KB free= 3.84KB waste= 0.05KB Class data capacity= 1.00KB used= 0.59KB free= 0.41KB waste= 0.00KB .... ClassLoader: Metadata capacity= 5640.00KB used= 5607.42KB free= 32.58KB waste= 0.46KB Class data capacity= 520.00KB used= 500.94KB free= 19.06KB waste= 0.02KB Bug: https://bugs.openjdk.java.net/browse/JDK-8189688 Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html Test: hotspot_tier1_runtime, fastdebug and release on x64 Linux. Thanks, -Zhengyu From harold.seigel at oracle.com Fri Oct 20 14:17:56 2017 From: harold.seigel at oracle.com (harold seigel) Date: Fri, 20 Oct 2017 10:17:56 -0400 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: <3DEE01DC-F7F8-4940-A689-51168238CA6E@oracle.com> References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> <2827d94c-1744-86da-b7bc-d67554cb3519@oracle.com> <3DEE01DC-F7F8-4940-A689-51168238CA6E@oracle.com> Message-ID: Hi John, I used a lock because, if resolution fails, the thread checks if another thread's resolution either simultaneously succeeded or failed.? I didn't see how to safely check both success and failure by another thread without using a lock. The possibility of successful resolution by one thread while another thread's resolution failed with a LinkageError exception is unlikely (probably requiring a perfectly timed change to the module graph).? Could the check for success be done outside of the lock, after the cas?? The resolution_error array element could then be cleared, if need be. Thanks, Harold On 10/20/2017 1:48 AM, John Rose wrote: > The resolved_references array never has a null value in it except before resolution. Therefore you can use a cas instead of the more expensive lock to race and set the array elements. Why use a lock for this purpose? > >> On Oct 19, 2017, at 10:15 PM, David Holmes wrote: >> >> Hi Harold, >> >>> On 19/10/2017 12:20 AM, harold seigel wrote: >>> Hi David, >>> Thanks for looking at this. >>> Please see inline comments. >>> Thanks, Harold >>>> On 10/17/2017 10:22 PM, David Holmes wrote: >>>> Hi Harold, >>>> >>>>> On 17/10/2017 12:20 AM, harold seigel wrote: >>>>> Hi, >>>>> >>>>> Please review this new JDK-10 fix for bug JDK-8174954. If the initial resolution attempt fails with a LinkageError exception then the fix saves the exception in the resolution_errors table and sets a flag in the constant pool Cache, indicating the failure. Subsequent attempts to do the same resolution will see that the flag is set, retrieve the exception from the resolution_errors table, and throw it, instead of re-trying the resolution. >>>> I'm a little confused. The fix seems all about indy, which seems to be the topic of: >>>> >>>> JDK-8174942 Bootstrap Method Called Multiple Times Despite Initial Resolution Failure >>>> >>>> whereas this bug seems to be about a more general problem. Is this fix addressing both issues?? >>> Thanks for noticing this. This fix addresses both bugs. The sample program for 8174942 is one of the test cases for this fix. I'll close 8174942 as a duplicate. >> I'm still somewhat confused as it seems that all the substantive changes here have the word "indy" in them. What part of the change actually fixes the non-indy issue tested by MethodAccessReadTwice.java? >> >> Thanks, >> David >>>> BTW there's no reason for JDK-8174954 to remain confidential. >>> Agreed. I opened the bug. >>>> Thanks, >>>> David >>>> >>>>> The fix also prevents a race condition if two or more threads try to do the same resolution concurrently. When a thread tries to record the result of its resolution attempt, it will check to see if another thread recorded its result first. If so, then it will use the result recorded by the other thread. Otherwise, it will record and use its result. Access to the resolution result is protected by the 'resolved_references' lock. >>>>> >>>>> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ >>>>> >>>>> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 >>>>> >>>>> The fix was tested with the JCK Lang and VM tests, the JTreg hotspot, java/io, java/lang, java/util, and other tests, the co-located NSK tests, JPRT, with Mach5 tier2 - tier5 tests, and by hand to check race condition handling. >>>>> >>>>> Thanks, Harold >>>>> From ioi.lam at oracle.com Fri Oct 20 20:05:35 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Fri, 20 Oct 2017 13:05:35 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> <732B9B1D-6593-4E2F-ACAC-0BD25C55FA43@oracle.com> <88a9a707-1158-f555-ffb0-653fe9bd9483@oracle.com> Message-ID: <10cc7022-8ce8-e41f-ea24-e9911a4c836e@oracle.com> On 10/19/17 5:02 PM, Jiangli Zhou wrote: > >> On Oct 19, 2017, at 4:06 PM, Ioi Lam > > wrote: >> >> >> >> On 10/19/17 2:57 PM, Jiangli Zhou wrote: >>> Hi Ioi, >>> >>>> On Oct 19, 2017, at 10:55 AM, Ioi Lam >>> > wrote: >>>> >>>> >>>> >>>> On 10/19/17 9:47 AM, Jiangli Zhou wrote: >>>>> >>>>>> On Oct 18, 2017, at 9:50 PM, Ioi Lam >>>>> > wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 10/18/17 9:35 PM, Jiangli Zhou wrote: >>>>>>> Hi Ioi, >>>>>>> >>>>>>>> On Oct 18, 2017, at 4:54 PM, Ioi Lam >>>>>>> > wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 10/17/17 9:04 PM, David Holmes wrote: >>>>>>>>> Hi Ioi, >>>>>>>>> >>>>>>>>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>>>>>>>> >>>>>>>>>> Summary: >>>>>>>>>> >>>>>>>>>> DumpLoadedClassList skips classes that can't be stored into >>>>>>>>>> the CDS >>>>>>>>>> archive. For boot and platform loaders, only classes from the >>>>>>>>>> JRT image >>>>>>>>>> are listed. >>>>>>>>>> >>>>>>>>>> The test for "is this class from JRT image" was insufficient. It >>>>>>>>>> checked only classes whose source ends with "...../modules". >>>>>>>>>> For the >>>>>>>>>> platform loader, which loads the graal classes, the source is >>>>>>>>>> in the >>>>>>>>>> form "jrt:/jdk.internal.vm.compiler". >>>>>>>>>> >>>>>>>>>> The fix is to also check for a "jrt:" prefix. >>>>>>>>> >>>>>>>>> So ./share/classfile/classLoader.hpp: >>>>>>>>> >>>>>>>>> static bool is_jrt(const char* name) { return >>>>>>>>> string_ends_with(name, MODULES_IMAGE_NAME); } >>>>>>>>> >>>>>>>>> is a badly named function or just plain broken? >>>>>>>>> >>>>>>>> >>>>>>>> I've renamed all the is_jrt() functions to is_modules_image(), >>>>>>>> so it's clear as to what it does. >>>>>>>> >>>>>>>> See >>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >>>>>>>> >>>>>>> >>>>>>> The logic looks puzzling. For both NULL class loader and the >>>>>>> PlatformClassLoader: >>>>>>> >>>>>>> The first if check at line 5938 excludes any classes not in the >>>>>>> jimage. >>>>>>> >>>>>>> 5938 if (!ClassLoader::is_modules_image(stream->source()) && >>>>>>> strncmp(stream->source(), "jrt:", 4) != 0) { >>>>>>> 5939 skip = true; >>>>>>> 5940 } >>>>>>> The second if check at line 5942 skips any classes from the boot >>>>>>> append class path loaded by the NULL class loader. The second >>>>>>> check skips a subset of the classes that?s already skipped by >>>>>>> the first check. The second check is redundant. >>>>>>> >>>>>> You're right. The second check is redundant. I'll remove it. >>>>>> >>>>>>> 5942 if (class_loader == NULL && >>>>>>> ClassLoader::contains_append_entry(stream->source())) { >>>>>>> 5943 // For the boot loader, also skip all classes that are >>>>>>> loaded from the appended entries >>>>>>> 5944 skip = true; >>>>>>> 5945 } >>>>>>> What about the classes from -Xbootclasspath/a (not in >>>>>>> --patch-module and --upgrade-module-path)? >>>>>>> >>>>>> I am not sure what you mean here. >>>>> >>>>> This excludes all classes from the -Xbootclasspath/a. However, >>>>> classes from -Xbootclasspath/a can be archived and used at runtime >>>>> properly. >>>>> >>>> >>>> Hi Jiangli, >>>> >>>> You're right. I misunderstood the original comments about the >>>> treatment of the bootclasspath/a classes. Here's the updated code >>>> that should be easier to understand: >>>> >>>> ??????? bool skip = false; >>>> ??????? if (class_loader == NULL || >>>> SystemDictionary::is_platform_class_loader(class_loader)) { >>>> ????????? // For the boot and platform class loaders, skip classes >>>> that are not found in the >>>> ????????? // java runtime image, such as those found in the >>>> --patch-module entries. >>>> ????????? // These classes can't be loaded from the archive during >>>> runtime. >>>> ????????? if (!ClassLoader::is_modules_image(stream->source()) && >>>> strncmp(stream->source(), "jrt:", 4) != 0) { >>>> ??????????? skip = true; >>>> ????????? } >>>> >>>> ????????? if (class_loader == NULL && >>>> ClassLoader::contains_append_entry(stream->source())) { >>>> ??????????? // .. but don't skip the boot classes that are loaded >>>> from -Xbootclasspath/a >>>> ??????????? // as theycan be loaded from the archive during runtime. >>>> ??????????? skip = false; >>>> ????????? } >>>> ??????? } >>> >>> Can you please check with a exploded build if the above also works? >>> What is the stream->source() from classes from exploded build loaded >>> by the NULL loader and the PlatformClassLoader? >>> >> >> Hi Jiangli, >> >> I checked the exploded build and all classes are skipped now, because >> they are not loaded from the $JAVA_HOME/lib/modules file. The >> stream->source looks like this: >> >> skip writing class sun/security/util/FilePermCompat from source >> /jdk/bld/cons/jdk/modules/java.base to classlist file >> >> However, I am not sure if this is relevant, since CDS is not >> supported with the exploded build. >> >> BTW, DumpLoadedClassList is used during the JDK build process to >> generate the classlist file. However, this uses the interim image, >> which is not an exploded build: >> >> open/make/GenerateLinkOptData.gmk:??? $(FIXPATH) >> $(INTERIM_IMAGE_DIR)/bin/java -XX:DumpLoadedClassList=$@ \ >> >> >> $ ./support/interim-image/bin/java -cp >> /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d:/jdk/cons/closed/test/hotspot/jtreg/runtime/AppCDS:/jdk/tmp/jtreg/work/classes/2/open/test/lib:/jdk/cons/open/test/lib:/java_re2/misc/promoted/jtreg/4.2/fcs/b09/binaries/jtreg/lib/javatest.jar:/jtreg/4.2/fcs/b09/binaries/jtreg/lib/jtreg.jar >> -XX:DumpLoadedClassList=app.list >> --patch-module=java.base=/jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/javabase.jar >> -Xbootclasspath/a:/jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/bootappend.jar >> -cp >> /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/app.jar >> ArrayListTest >> hello world. >> skip writing class java/lang/NewClass from source >> /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/javabase.jar >> to classlist file >> NewClass >> class java.lang.NewClass >> Foo >> class boot.append.Foo >> ioilinux ~/jdk/bld/cons$ wc app.list >> ? 701?? 701 22630 app.list > > > That?s what I expected. Thanks for verifying it. Since the goal here > is to generate a more ?precise? classlist for CDS and CDS does not > support exploded build, it would make sense to print a warning and > disable DumpLoadedClassList if?ClassLoader::has_jrt_entry() is not true. > Hi Jiangli, Thanks for the suggestion. I've added the warning as you suggested: ??? if (DumpLoadedClassList != NULL && stream->source() != NULL && classlist_file->is_open()) { !???? if (!ClassLoader::has_jrt_entry()) { !?? ??? warning("DumpLoadedClassList and CDS are not supported in exploded build"); !?????? DumpLoadedClassList = NULL; !???? } else if (SystemDictionaryShared::is_sharing_possible(_loader_data) && ????????? _host_klass == NULL) { ??????? // Only dump the classes that can be stored into CDS archive. ??????? // Anonymous classes such as generated LambdaForm classes are also not included. ??????? oop class_loader = _loader_data->class_loader(); ??????? ResourceMark rm(THREAD); ??????? bool skip = false; ??????? if (class_loader == NULL || SystemDictionary::is_platform_class_loader(class_loader)) { ????????? // For the boot and platform class loaders, skip classes that are not found in the ????????? // java runtime image, such as those found in the --patch-module entries. ????????? // These classes can't be loaded from the archive during runtime. ????????? if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { ??????????? skip = true; ????????? } ????????? if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { ??????????? // .. but don't skip the boot classes that are loaded from -Xbootclasspath/a ??????????? // as they can be loaded from the archive during runtime. ??????????? skip = false; ????????? } ??????? } ??????? if (skip) { ????????? tty->print_cr("skip writing class %s from source %s to classlist file", ??????????? _class_name->as_C_string(), stream->source()); ??????? } else { ????????? classlist_file->print_cr("%s", _class_name->as_C_string()); ????????? classlist_file->flush(); ??????? } ????? } ??? } This message is printed at run time for the exploded build: Java HotSpot(TM) 64-Bit Server VM warning: DumpLoadedClassList and CDS are not supported in exploded build What do you think? - Ioi From jiangli.zhou at oracle.com Fri Oct 20 20:13:38 2017 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Fri, 20 Oct 2017 13:13:38 -0700 Subject: RFR (XS) 8185160 -XX:DumpLoadedClassList omits graal classes In-Reply-To: <10cc7022-8ce8-e41f-ea24-e9911a4c836e@oracle.com> References: <78eacd8a-183b-3930-32f5-5e66997203ac@oracle.com> <0add3787-c774-588e-5f99-9d2b877a7374@oracle.com> <732B9B1D-6593-4E2F-ACAC-0BD25C55FA43@oracle.com> <88a9a707-1158-f555-ffb0-653fe9bd9483@oracle.com> <10cc7022-8ce8-e41f-ea24-e9911a4c836e@oracle.com> Message-ID: > On Oct 20, 2017, at 1:05 PM, Ioi Lam wrote: > > > > On 10/19/17 5:02 PM, Jiangli Zhou wrote: >> >>> On Oct 19, 2017, at 4:06 PM, Ioi Lam > wrote: >>> >>> >>> >>> On 10/19/17 2:57 PM, Jiangli Zhou wrote: >>>> Hi Ioi, >>>> >>>>> On Oct 19, 2017, at 10:55 AM, Ioi Lam > wrote: >>>>> >>>>> >>>>> >>>>> On 10/19/17 9:47 AM, Jiangli Zhou wrote: >>>>>> >>>>>>> On Oct 18, 2017, at 9:50 PM, Ioi Lam > wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 10/18/17 9:35 PM, Jiangli Zhou wrote: >>>>>>>> Hi Ioi, >>>>>>>> >>>>>>>>> On Oct 18, 2017, at 4:54 PM, Ioi Lam > wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 10/17/17 9:04 PM, David Holmes wrote: >>>>>>>>>> Hi Ioi, >>>>>>>>>> >>>>>>>>>> On 18/10/2017 1:52 PM, Ioi Lam wrote: >>>>>>>>>>> http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v01/ >>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8185160 >>>>>>>>>>> >>>>>>>>>>> Summary: >>>>>>>>>>> >>>>>>>>>>> DumpLoadedClassList skips classes that can't be stored into the CDS >>>>>>>>>>> archive. For boot and platform loaders, only classes from the JRT image >>>>>>>>>>> are listed. >>>>>>>>>>> >>>>>>>>>>> The test for "is this class from JRT image" was insufficient. It >>>>>>>>>>> checked only classes whose source ends with "...../modules". For the >>>>>>>>>>> platform loader, which loads the graal classes, the source is in the >>>>>>>>>>> form "jrt:/jdk.internal.vm.compiler". >>>>>>>>>>> >>>>>>>>>>> The fix is to also check for a "jrt:" prefix. >>>>>>>>>> >>>>>>>>>> So ./share/classfile/classLoader.hpp: >>>>>>>>>> >>>>>>>>>> static bool is_jrt(const char* name) { return string_ends_with(name, MODULES_IMAGE_NAME); } >>>>>>>>>> >>>>>>>>>> is a badly named function or just plain broken? >>>>>>>>>> >>>>>>>>> >>>>>>>>> I've renamed all the is_jrt() functions to is_modules_image(), so it's clear as to what it does. >>>>>>>>> >>>>>>>>> See http://cr.openjdk.java.net/~iklam/jdk10/8185160-dump-class-list-omits-graal.v02/ >>>>>>>> >>>>>>>> The logic looks puzzling. For both NULL class loader and the PlatformClassLoader: >>>>>>>> >>>>>>>> The first if check at line 5938 excludes any classes not in the jimage. >>>>>>>> >>>>>>>> 5938 if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { >>>>>>>> 5939 skip = true; >>>>>>>> 5940 } >>>>>>>> The second if check at line 5942 skips any classes from the boot append class path loaded by the NULL class loader. The second check skips a subset of the classes that?s already skipped by the first check. The second check is redundant. >>>>>>>> >>>>>>> You're right. The second check is redundant. I'll remove it. >>>>>>> >>>>>>>> 5942 if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { >>>>>>>> 5943 // For the boot loader, also skip all classes that are loaded from the appended entries >>>>>>>> 5944 skip = true; >>>>>>>> 5945 } >>>>>>>> What about the classes from -Xbootclasspath/a (not in --patch-module and --upgrade-module-path)? >>>>>>>> >>>>>>> I am not sure what you mean here. >>>>>> >>>>>> This excludes all classes from the -Xbootclasspath/a. However, classes from -Xbootclasspath/a can be archived and used at runtime properly. >>>>>> >>>>> >>>>> Hi Jiangli, >>>>> >>>>> You're right. I misunderstood the original comments about the treatment of the bootclasspath/a classes. Here's the updated code that should be easier to understand: >>>>> >>>>> bool skip = false; >>>>> if (class_loader == NULL || SystemDictionary::is_platform_class_loader(class_loader)) { >>>>> // For the boot and platform class loaders, skip classes that are not found in the >>>>> // java runtime image, such as those found in the --patch-module entries. >>>>> // These classes can't be loaded from the archive during runtime. >>>>> if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { >>>>> skip = true; >>>>> } >>>>> >>>>> if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { >>>>> // .. but don't skip the boot classes that are loaded from -Xbootclasspath/a >>>>> // as they can be loaded from the archive during runtime. >>>>> skip = false; >>>>> } >>>>> } >>>> >>>> Can you please check with a exploded build if the above also works? What is the stream->source() from classes from exploded build loaded by the NULL loader and the PlatformClassLoader? >>>> >>> >>> Hi Jiangli, >>> >>> I checked the exploded build and all classes are skipped now, because they are not loaded from the $JAVA_HOME/lib/modules file. The stream->source looks like this: >>> >>> skip writing class sun/security/util/FilePermCompat from source /jdk/bld/cons/jdk/modules/java.base to classlist file >>> >>> However, I am not sure if this is relevant, since CDS is not supported with the exploded build. >>> >>> BTW, DumpLoadedClassList is used during the JDK build process to generate the classlist file. However, this uses the interim image, which is not an exploded build: >>> >>> open/make/GenerateLinkOptData.gmk: $(FIXPATH) $(INTERIM_IMAGE_DIR)/bin/java -XX:DumpLoadedClassList=$@ \ >>> >>> >>> $ ./support/interim-image/bin/java -cp /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d:/jdk/cons/closed/test/hotspot/jtreg/runtime/AppCDS:/jdk/tmp/jtreg/work/classes/2/open/test/lib:/jdk/cons/open/test/lib:/java_re2/misc/promoted/jtreg/4.2/fcs/b09/binaries/jtreg/lib/javatest.jar:/jtreg/4.2/fcs/b09/binaries/jtreg/lib/jtreg.jar -XX:DumpLoadedClassList=app.list --patch-module=java.base=/jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/javabase.jar -Xbootclasspath/a:/jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/bootappend.jar -cp /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/app.jar ArrayListTest >>> hello world. >>> skip writing class java/lang/NewClass from source /jdk/tmp/jtreg/work/classes/2/runtime/AppCDS/DumpClassList.d/javabase.jar to classlist file >>> NewClass >>> class java.lang.NewClass >>> Foo >>> class boot.append.Foo >>> ioilinux ~/jdk/bld/cons$ wc app.list >>> 701 701 22630 app.list >> >> >> That?s what I expected. Thanks for verifying it. Since the goal here is to generate a more ?precise? classlist for CDS and CDS does not support exploded build, it would make sense to print a warning and disable DumpLoadedClassList if ClassLoader::has_jrt_entry() is not true. >> > > Hi Jiangli, > > Thanks for the suggestion. I've added the warning as you suggested: > > > if (DumpLoadedClassList != NULL && stream->source() != NULL && classlist_file->is_open()) { > ! if (!ClassLoader::has_jrt_entry()) { > ! warning("DumpLoadedClassList and CDS are not supported in exploded build"); > ! DumpLoadedClassList = NULL; > ! } else if (SystemDictionaryShared::is_sharing_possible(_loader_data) && > _host_klass == NULL) { > // Only dump the classes that can be stored into CDS archive. > // Anonymous classes such as generated LambdaForm classes are also not included. > oop class_loader = _loader_data->class_loader(); > ResourceMark rm(THREAD); > bool skip = false; > if (class_loader == NULL || SystemDictionary::is_platform_class_loader(class_loader)) { > // For the boot and platform class loaders, skip classes that are not found in the > // java runtime image, such as those found in the --patch-module entries. > // These classes can't be loaded from the archive during runtime. > if (!ClassLoader::is_modules_image(stream->source()) && strncmp(stream->source(), "jrt:", 4) != 0) { > skip = true; > } > > if (class_loader == NULL && ClassLoader::contains_append_entry(stream->source())) { > // .. but don't skip the boot classes that are loaded from -Xbootclasspath/a > // as they can be loaded from the archive during runtime. > skip = false; > } > } > if (skip) { > tty->print_cr("skip writing class %s from source %s to classlist file", > _class_name->as_C_string(), stream->source()); > } else { > classlist_file->print_cr("%s", _class_name->as_C_string()); > classlist_file->flush(); > } > } > } > > > This message is printed at run time for the exploded build: > > Java HotSpot(TM) 64-Bit Server VM warning: DumpLoadedClassList and CDS are not supported in exploded build > > What do you think? Looks good. Thanks, Jiangli > > - Ioi > > From john.r.rose at oracle.com Sat Oct 21 00:24:35 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 20 Oct 2017 17:24:35 -0700 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> <2827d94c-1744-86da-b7bc-d67554cb3519@oracle.com> <3DEE01DC-F7F8-4940-A689-51168238CA6E@oracle.com> Message-ID: On Oct 20, 2017, at 7:17 AM, harold seigel wrote: > > I used a lock because, if resolution fails, the thread checks if another thread's resolution either simultaneously succeeded or failed. I didn't see how to safely check both success and failure by another thread without using a lock. Right; we would need a two-word cas to update both states atomically, using cas. > The possibility of successful resolution by one thread while another thread's resolution failed with a LinkageError exception is unlikely (probably requiring a perfectly timed change to the module graph). This is a real possibility with indy/condy, since BSMs can misbehave. > Could the check for success be done outside of the lock, after the cas? The resolution_error array element could then be cleared, if need be. Something clever like that might be possible, but I don't see it yet. (I was hopeful!) I think this suggestion can cause the erring thread to throw while the winning thread sets a result. The resolution_error element might be acted on by the losing thread, if no message gets to it by the time it decides what to do. (Another thread might be doing a slow-motion cas at the same moment.) Basically, the resolved_refs array is null if either of two states holds: Not yet resolved, or resolved in error. (The interpreter scurries into the runtime if it finds a null, and DTRT.) This invariant makes it true that, as long as nobody ever sets the RR element to non-null, everybody can agree on a final state if they use a lock. I think if a non-null value is cas-ed in, and another thread is getting a failure, There is one other way to make things work lock-free: Represent the resolution states (none, done, error) more fully in the RR word that gets cas-ed. This would require an interpreter change to DTRT if the RR word indicates the error state. The indication could be simple sentinel value, an error_sentinel like the null_sentinel. If the interpreter sees error_sentinel, it would call into the runtime, which would then extract the error from a table and throw it. Thanks for the helpful discussion. ? John From john.r.rose at oracle.com Sat Oct 21 00:28:00 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 20 Oct 2017 17:28:00 -0700 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> <2827d94c-1744-86da-b7bc-d67554cb3519@oracle.com> <3DEE01DC-F7F8-4940-A689-51168238CA6E@oracle.com> Message-ID: <95177DE6-C6F9-46EB-8F4E-9CD9AF26D22D@oracle.com> On Oct 20, 2017, at 5:24 PM, John Rose wrote: > > I think if a non-null value is cas-ed in, and another thread is getting a failure, (Please disregard!) From david.holmes at oracle.com Sun Oct 22 06:17:26 2017 From: david.holmes at oracle.com (David Holmes) Date: Sun, 22 Oct 2017 16:17:26 +1000 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: <5c14a878-9dbd-3538-7fd9-1a1067c992e1@oracle.com> References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> <2827d94c-1744-86da-b7bc-d67554cb3519@oracle.com> <5c14a878-9dbd-3538-7fd9-1a1067c992e1@oracle.com> Message-ID: Hi Harold, On 20/10/2017 11:37 PM, harold seigel wrote: > Hi David, > > MethodAccessReadTwice.java is an indy test.? Lines 120 - 153 test that > the same invokedynamic instruction gets the same result.? The > invokedynamic instruction gets generated for the string concatenation at > line 31 of file .../p5/c5.java: Okay. It was totally not at all evident from the bug report and the test code that the underlying problem was actually related to indy due to the string concatenation. Thanks, David > > 24 package p5; > 25 > 26 import java.lang.Module; > 27 import p2.c2; > 28 > 29 public class c5 { > 30 public void method5(p2.c2 param) { > 31 System.out.println("In c5's method5 with param = " + param); > 32 } > 33 > 34 public void methodAddReadEdge(Module m) { > 35 // Add a read edge from p5/c5's module (first_mod) to second_mod > 36 c5.class.getModule().addReads(m); > 37 } > 38 } > > The indy resolution throws an IAE because package p5's module cannot > read package p2's module. > > The second test case, at lines 156 - 161 of MethodAccessReadTwice.java, > similarly tests invokedynamic instructions.? In this case, they are > generated for the string concatenations at lines 32 and 43 of file > .../p7/c7.java. > > This problem does not exist for non-indy instructions because, when > their resolutions fail, they update the constant pool.? Indy cannot do > this because, unlike other instructions, different indy instructions > that use the same constant pool index are allowed to get different > results.? Hence, the fix is indy only. > > Thanks, Harold > > On 10/20/2017 1:15 AM, David Holmes wrote: >> Hi Harold, >> >> On 19/10/2017 12:20 AM, harold seigel wrote: >>> Hi David, >>> >>> Thanks for looking at this. >>> >>> Please see inline comments. >>> >>> Thanks, Harold >>> >>> >>> On 10/17/2017 10:22 PM, David Holmes wrote: >>>> Hi Harold, >>>> >>>> On 17/10/2017 12:20 AM, harold seigel wrote: >>>>> Hi, >>>>> >>>>> Please review this new JDK-10 fix for bug JDK-8174954.? If the >>>>> initial resolution attempt fails with a LinkageError exception then >>>>> the fix saves the exception in the resolution_errors table and sets >>>>> a flag in the constant pool Cache, indicating the failure. >>>>> Subsequent attempts to do the same resolution will see that the >>>>> flag is set, retrieve the exception from the resolution_errors >>>>> table, and throw it, instead of re-trying the resolution. >>>> >>>> I'm a little confused. The fix seems all about indy, which seems to >>>> be the topic of: >>>> >>>> ?JDK-8174942 Bootstrap Method Called Multiple Times Despite Initial >>>> Resolution Failure >>>> >>>> whereas this bug seems to be about a more general problem. Is this >>>> fix addressing both issues?? >>> Thanks for noticing this.? This fix addresses both bugs.? The sample >>> program for 8174942 is one of the test cases for this fix.? I'll >>> close 8174942 as a duplicate. >> >> I'm still somewhat confused as it seems that all the substantive >> changes here have the word "indy" in them. What part of the change >> actually fixes the non-indy issue tested by MethodAccessReadTwice.java? >> >> Thanks, >> David >>>> >>>> BTW there's no reason for JDK-8174954 to remain confidential. >>> Agreed.? I opened the bug. >>>> >>>> Thanks, >>>> David >>>> >>>>> The fix also prevents a race condition if two or more threads try >>>>> to do the same resolution concurrently. When a thread tries to >>>>> record the result of its resolution attempt, it will check to see >>>>> if another thread recorded its result first.? If so, then it will >>>>> use the result recorded by the other thread.? Otherwise, it will >>>>> record and use its result. Access to the resolution result is >>>>> protected by the 'resolved_references' lock. >>>>> >>>>> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ >>>>> >>>>> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 >>>>> >>>>> The fix was tested with the JCK Lang and VM tests, the JTreg >>>>> hotspot, java/io, java/lang, java/util, and other tests, the >>>>> co-located NSK tests, JPRT, with Mach5 tier2 - tier5 tests, and by >>>>> hand to check race condition handling. >>>>> >>>>> Thanks, Harold >>>>> >>> > From dmitry.samersoff at bell-sw.com Sun Oct 22 13:46:57 2017 From: dmitry.samersoff at bell-sw.com (Dmitry Samersoff) Date: Sun, 22 Oct 2017 16:46:57 +0300 Subject: RFR(S): JDK-8163011 AArch64: NMT detail stack trace cleanup In-Reply-To: <87196637-2b40-8c33-bd1f-f0990e1a6884@redhat.com> References: <81369691-46f1-a0f0-c3f8-8bc17ae94e7d@bell-sw.com> <87196637-2b40-8c33-bd1f-f0990e1a6884@redhat.com> Message-ID: <10b05aac-19de-9c95-8025-a9942af08f53@bell-sw.com> Andrew, Please, see: http://cr.openjdk.java.net/~dsamersoff/JDK-8163011/aarch64_only/webrev.02/ -Dmitry On 10/17/2017 03:14 PM, Andrew Haley wrote: > On 17/10/17 09:31, dmitry.samersov wrote: >> Please, review a minimal, aarch64 only patch that makes aarch64 behavior >> similar to x86 one. > > It'll want a proper bug id; and please rename the local variable to fp. > From david.holmes at oracle.com Mon Oct 23 03:47:15 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 23 Oct 2017 13:47:15 +1000 Subject: Unification of jni_.h In-Reply-To: References: Message-ID: Hi Magnus, On 18/10/2017 11:10 PM, Magnus Ihse Bursie wrote: > In the context of JDK-8167078 (Duplicate header files in hotspot and > jdk), I was looking at unifying the platform specific extensions to > jni.h, the jni_md.h files between hotspot and java.base. > > The main difference here is that the hotspot jni_* files are divided > into individual files based on cpu, while the java.base versions are > divided according to operating system (which, in turn, implicates > compiler). On a second look, however, it turns out to not be so > problematic -- a hotspot file like jni_x86.h contains basically > #ifdef WIN32 > ... // do windows stuff > #else > ... // do posix/macos stuff > #endif > > and the two blocks match very well what the java.base versions are doing > for the different operating systems. > > In fact, the OS (and compiler) division seems more natural, since there > is much redundancy in the hotspot files, and the java.base OS-based > versions are much simpler. > > I'm proposing to remove the hotspot CPU-based files and just replace > them with the java.base versions. However, a few differences crept out. > I'd like to discuss them before proceeding. > > For jni_aarch64.h: There's a windows (or rather, "not unix") part of the > jni_aarch64.h that I do not believe have ever been used. Nevertheless, > it contains "typedef int jint" rather than "typedef long jint", which > the java.base windows version uses. Since I've never heard of aarch64 > building on Windows, I presume this is irrelevant. Likely just copied from the x86 version at some point. There is no Windows-aarch64 support in the OpenJDK. > For jni_aarch64.h and jni_sparc.h: The unix version will match with the > java.base version as long as _LP64 is defined. This should be just fine, > since we define that when building hotspot/the JDK, and the java.base > version that is exported by the JDK on linux/aarch64 and solaris/sparc > has always required the _LP64 define. Yep 64-bit only. > (In fact, the macos and unix versions in java.base only contain trivial > differences. The macos version should probably be removed in favor of a > single unix version.) > > For jni_x86.h: There windows part suffers the same problem as for > aarch64, but here it's potentially problematic. The hotspot version uses > "typedef int jint" while the java.base version uses "typedef long jint". > If this *really* were a problem, we would probably have gotten reports > from JNI code unable to work on Windows, so I assume this turns out to > be the same datatype in the end. Don't know if it's due to luck, > compiler flags or by definition. Windows is either ILP32 or LLP64 - in both cases int and long are the same and 32-bits. > > For jni_s390.h: Here's the most problematic version. The exported > java.base version uses: > #ifdef _LP64 > typedef long jlong; > #else > typedef long long jlong; > #endif > but s390 uses: > typedef long int jlong; > > My best assumption here is that we're only really using s390x (the > 64-bit version) and that "long int" is functionally equivalent to > "long". Or that jni is broken on s390 and nobody really noticed. > > Also, s390 uses a simplified version of the gcc attribute dance used for > JNIEXPORT/JNIIMPORT. I think the s390 version is just not really > updated, and that using the newer in java.base is harmless. Can't comment on s390. > For jni_arm.h: Finally, the jni_arm.h (the 32-bit formerly closed Oracle > port), the JNIEXPORT/JNIIMPORT is different, by defining > __attribute__((externally_visible)). This might have been relevant due > to compile/link time symbol processing for that platform, but > technically it should probably not have been connected to the platform > per se, but rather to the compilation procedure. Since the arm-32 > platform is kind of abandoned right now, I propose to modify the > compilation to be more standard, if this is required, rather than > keeping these attributes. As Dean stated this likely related to LTO. And unfortunately attention was not paid to the comment: // Note: please do not change these without also changing jni_md.h in the JDK // repository possibly due to open/closed issues back in time. But I'd say we would want the hotspot version so that nothing that would previously work is now broken. Thanks, David > > /Magnus From david.holmes at oracle.com Mon Oct 23 04:18:58 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 23 Oct 2017 14:18:58 +1000 Subject: (XXS) RFR: 8189776: Remove dead code in jvm.cpp: force_verify_field_access Message-ID: <5f03e75b-19ea-27d8-fe50-d03371052f05@oracle.com> Trivial dead code cleanup. webrev: http://cr.openjdk.java.net/~dholmes/8189776/webrev/ Bug: https://bugs.openjdk.java.net/browse/JDK-8189776 JDK-8057777 deleted the old unused JVM_AllocateNewObject interface, which was the sole caller of the local force_verify_field_access method. However force_verify_field_access was not removed. Cleaning this up simplifies the nestmate work I'm doing (JEP 181). Thanks, David From claes.redestad at oracle.com Mon Oct 23 06:51:43 2017 From: claes.redestad at oracle.com (Claes Redestad) Date: Mon, 23 Oct 2017 08:51:43 +0200 Subject: (XXS) RFR: 8189776: Remove dead code in jvm.cpp: force_verify_field_access In-Reply-To: <5f03e75b-19ea-27d8-fe50-d03371052f05@oracle.com> References: <5f03e75b-19ea-27d8-fe50-d03371052f05@oracle.com> Message-ID: <55fede90-8491-d2df-0161-44d8f72f2f5f@oracle.com> On 2017-10-23 06:18, David Holmes wrote: > Trivial dead code cleanup. Looks good! /Claes From david.holmes at oracle.com Mon Oct 23 07:11:40 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 23 Oct 2017 17:11:40 +1000 Subject: (XXS) RFR: 8189776: Remove dead code in jvm.cpp: force_verify_field_access In-Reply-To: <55fede90-8491-d2df-0161-44d8f72f2f5f@oracle.com> References: <5f03e75b-19ea-27d8-fe50-d03371052f05@oracle.com> <55fede90-8491-d2df-0161-44d8f72f2f5f@oracle.com> Message-ID: <63e3857f-b856-be5c-9a8f-0fc8b5c31457@oracle.com> Thanks Claes! David On 23/10/2017 4:51 PM, Claes Redestad wrote: > On 2017-10-23 06:18, David Holmes wrote: >> Trivial dead code cleanup. > > Looks good! > > /Claes From aph at redhat.com Mon Oct 23 07:53:06 2017 From: aph at redhat.com (Andrew Haley) Date: Mon, 23 Oct 2017 08:53:06 +0100 Subject: RFR(S): JDK-8163011 AArch64: NMT detail stack trace cleanup In-Reply-To: <10b05aac-19de-9c95-8025-a9942af08f53@bell-sw.com> References: <81369691-46f1-a0f0-c3f8-8bc17ae94e7d@bell-sw.com> <87196637-2b40-8c33-bd1f-f0990e1a6884@redhat.com> <10b05aac-19de-9c95-8025-a9942af08f53@bell-sw.com> Message-ID: <5ce3ace4-9bed-643a-6998-7656ac1d5a21@redhat.com> On 22/10/17 14:46, Dmitry Samersoff wrote: > Please, see: > > http://cr.openjdk.java.net/~dsamersoff/JDK-8163011/aarch64_only/webrev.02/ Thanks. This is OK. It's an ugly hack, but it's not our ugly hack. :-) -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From martin.doerr at sap.com Mon Oct 23 09:33:46 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Mon, 23 Oct 2017 09:33:46 +0000 Subject: Unification of jni_.h In-Reply-To: References: Message-ID: <7dac704ecd5047de983226604ea71c74@sap.com> Hi, we only support 64 bit on s390. Seems like the code could be cleaned up or updated. Thanks for looking into it. Best regards, Martin -----Original Message----- From: hotspot-runtime-dev [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On Behalf Of David Holmes Sent: Montag, 23. Oktober 2017 05:47 To: Magnus Ihse Bursie ; hotspot-runtime-dev at openjdk.java.net; build-dev Subject: Re: Unification of jni_.h Hi Magnus, On 18/10/2017 11:10 PM, Magnus Ihse Bursie wrote: > In the context of JDK-8167078 (Duplicate header files in hotspot and > jdk), I was looking at unifying the platform specific extensions to > jni.h, the jni_md.h files between hotspot and java.base. > > The main difference here is that the hotspot jni_* files are divided > into individual files based on cpu, while the java.base versions are > divided according to operating system (which, in turn, implicates > compiler). On a second look, however, it turns out to not be so > problematic -- a hotspot file like jni_x86.h contains basically > #ifdef WIN32 > ... // do windows stuff > #else > ... // do posix/macos stuff > #endif > > and the two blocks match very well what the java.base versions are doing > for the different operating systems. > > In fact, the OS (and compiler) division seems more natural, since there > is much redundancy in the hotspot files, and the java.base OS-based > versions are much simpler. > > I'm proposing to remove the hotspot CPU-based files and just replace > them with the java.base versions. However, a few differences crept out. > I'd like to discuss them before proceeding. > > For jni_aarch64.h: There's a windows (or rather, "not unix") part of the > jni_aarch64.h that I do not believe have ever been used. Nevertheless, > it contains "typedef int jint" rather than "typedef long jint", which > the java.base windows version uses. Since I've never heard of aarch64 > building on Windows, I presume this is irrelevant. Likely just copied from the x86 version at some point. There is no Windows-aarch64 support in the OpenJDK. > For jni_aarch64.h and jni_sparc.h: The unix version will match with the > java.base version as long as _LP64 is defined. This should be just fine, > since we define that when building hotspot/the JDK, and the java.base > version that is exported by the JDK on linux/aarch64 and solaris/sparc > has always required the _LP64 define. Yep 64-bit only. > (In fact, the macos and unix versions in java.base only contain trivial > differences. The macos version should probably be removed in favor of a > single unix version.) > > For jni_x86.h: There windows part suffers the same problem as for > aarch64, but here it's potentially problematic. The hotspot version uses > "typedef int jint" while the java.base version uses "typedef long jint". > If this *really* were a problem, we would probably have gotten reports > from JNI code unable to work on Windows, so I assume this turns out to > be the same datatype in the end. Don't know if it's due to luck, > compiler flags or by definition. Windows is either ILP32 or LLP64 - in both cases int and long are the same and 32-bits. > > For jni_s390.h: Here's the most problematic version. The exported > java.base version uses: > #ifdef _LP64 > typedef long jlong; > #else > typedef long long jlong; > #endif > but s390 uses: > typedef long int jlong; > > My best assumption here is that we're only really using s390x (the > 64-bit version) and that "long int" is functionally equivalent to > "long". Or that jni is broken on s390 and nobody really noticed. > > Also, s390 uses a simplified version of the gcc attribute dance used for > JNIEXPORT/JNIIMPORT. I think the s390 version is just not really > updated, and that using the newer in java.base is harmless. Can't comment on s390. > For jni_arm.h: Finally, the jni_arm.h (the 32-bit formerly closed Oracle > port), the JNIEXPORT/JNIIMPORT is different, by defining > __attribute__((externally_visible)). This might have been relevant due > to compile/link time symbol processing for that platform, but > technically it should probably not have been connected to the platform > per se, but rather to the compilation procedure. Since the arm-32 > platform is kind of abandoned right now, I propose to modify the > compilation to be more standard, if this is required, rather than > keeping these attributes. As Dean stated this likely related to LTO. And unfortunately attention was not paid to the comment: // Note: please do not change these without also changing jni_md.h in the JDK // repository possibly due to open/closed issues back in time. But I'd say we would want the hotspot version so that nothing that would previously work is now broken. Thanks, David > > /Magnus From coleen.phillimore at oracle.com Mon Oct 23 12:10:38 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 23 Oct 2017 08:10:38 -0400 Subject: Unification of jni_.h In-Reply-To: <7dac704ecd5047de983226604ea71c74@sap.com> References: <7dac704ecd5047de983226604ea71c74@sap.com> Message-ID: With my change that I'm working on for JDK-8189610, I'm cleaning up the jni_platform files also.??? They were mostly the same with some minor differences that we're working through. Martin, when I get this working here, can you test out your platforms on it? thanks, Coleen On 10/23/17 5:33 AM, Doerr, Martin wrote: > Hi, > > we only support 64 bit on s390. Seems like the code could be cleaned up or updated. Thanks for looking into it. > > Best regards, > Martin > > > -----Original Message----- > From: hotspot-runtime-dev [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On Behalf Of David Holmes > Sent: Montag, 23. Oktober 2017 05:47 > To: Magnus Ihse Bursie ; hotspot-runtime-dev at openjdk.java.net; build-dev > Subject: Re: Unification of jni_.h > > Hi Magnus, > > On 18/10/2017 11:10 PM, Magnus Ihse Bursie wrote: >> In the context of JDK-8167078 (Duplicate header files in hotspot and >> jdk), I was looking at unifying the platform specific extensions to >> jni.h, the jni_md.h files between hotspot and java.base. >> >> The main difference here is that the hotspot jni_* files are divided >> into individual files based on cpu, while the java.base versions are >> divided according to operating system (which, in turn, implicates >> compiler). On a second look, however, it turns out to not be so >> problematic -- a hotspot file like jni_x86.h contains basically >> #ifdef WIN32 >> ... // do windows stuff >> #else >> ... // do posix/macos stuff >> #endif >> >> and the two blocks match very well what the java.base versions are doing >> for the different operating systems. >> >> In fact, the OS (and compiler) division seems more natural, since there >> is much redundancy in the hotspot files, and the java.base OS-based >> versions are much simpler. >> >> I'm proposing to remove the hotspot CPU-based files and just replace >> them with the java.base versions. However, a few differences crept out. >> I'd like to discuss them before proceeding. >> >> For jni_aarch64.h: There's a windows (or rather, "not unix") part of the >> jni_aarch64.h that I do not believe have ever been used. Nevertheless, >> it contains "typedef int jint" rather than "typedef long jint", which >> the java.base windows version uses. Since I've never heard of aarch64 >> building on Windows, I presume this is irrelevant. > Likely just copied from the x86 version at some point. There is no > Windows-aarch64 support in the OpenJDK. > >> For jni_aarch64.h and jni_sparc.h: The unix version will match with the >> java.base version as long as _LP64 is defined. This should be just fine, >> since we define that when building hotspot/the JDK, and the java.base >> version that is exported by the JDK on linux/aarch64 and solaris/sparc >> has always required the _LP64 define. > Yep 64-bit only. > >> (In fact, the macos and unix versions in java.base only contain trivial >> differences. The macos version should probably be removed in favor of a >> single unix version.) >> >> For jni_x86.h: There windows part suffers the same problem as for >> aarch64, but here it's potentially problematic. The hotspot version uses >> "typedef int jint" while the java.base version uses "typedef long jint". >> If this *really* were a problem, we would probably have gotten reports >> from JNI code unable to work on Windows, so I assume this turns out to >> be the same datatype in the end. Don't know if it's due to luck, >> compiler flags or by definition. > Windows is either ILP32 or LLP64 - in both cases int and long are the > same and 32-bits. > >> For jni_s390.h: Here's the most problematic version. The exported >> java.base version uses: >> #ifdef _LP64 >> typedef long jlong; >> #else >> typedef long long jlong; >> #endif >> but s390 uses: >> typedef long int jlong; >> >> My best assumption here is that we're only really using s390x (the >> 64-bit version) and that "long int" is functionally equivalent to >> "long". Or that jni is broken on s390 and nobody really noticed. >> >> Also, s390 uses a simplified version of the gcc attribute dance used for >> JNIEXPORT/JNIIMPORT. I think the s390 version is just not really >> updated, and that using the newer in java.base is harmless. > Can't comment on s390. > >> For jni_arm.h: Finally, the jni_arm.h (the 32-bit formerly closed Oracle >> port), the JNIEXPORT/JNIIMPORT is different, by defining >> __attribute__((externally_visible)). This might have been relevant due >> to compile/link time symbol processing for that platform, but >> technically it should probably not have been connected to the platform >> per se, but rather to the compilation procedure. Since the arm-32 >> platform is kind of abandoned right now, I propose to modify the >> compilation to be more standard, if this is required, rather than >> keeping these attributes. > As Dean stated this likely related to LTO. And unfortunately attention > was not paid to the comment: > > // Note: please do not change these without also changing jni_md.h in > the JDK > // repository > > possibly due to open/closed issues back in time. But I'd say we would > want the hotspot version so that nothing that would previously work is > now broken. > > Thanks, > David > >> /Magnus From thomas.stuefe at gmail.com Mon Oct 23 13:01:15 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 23 Oct 2017 15:01:15 +0200 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> Message-ID: Hi Zhengyu, this is a nice addition! ------ General remarks: We already have a multitude of printing functions in metaspace.cpp, in MetaspaceAux I find: void MetaspaceAux::print_on(outputStream* out) and void MetaspaceAux::print_on(outputStream* out, Metaspace::MetadataType mdtype) void MetaspaceAux::dump(outputStream* out) There even exists a cleanup issue: https://bugs.openjdk.java.net/browse/JDK-8185034. I am not happy that we add yet another printing function to this bunch, which basically does the same. Is there any way to reuse the existing functions? If not, could we at least for now rename print_metadata() to something like print_metadata_for_nmt(), to tell it apart from print_on etc? And mark them for later cleanup and consolidation? ---------- About the numbers: I am not sure who this output is for. Customers analyzing metaspace shortage or hotspot developers debugging the metaspace allocation? Depending on the answer I would change the output somewhat. If this it is for customers, the numbers are too much and the names are misleading. Especially the per-classloader numbers: "Metadata capacity=%10.2f%s used=%10.2f%s free=%10.2f%s waste=%10.2f%s" If we only want to answer the question "how much metaspace does a classloader use up?", either "capacity" or "used" should be sufficient. It does not really matter much which, really, and for most cases the numbers should be very similar. In this case I would omit "free" and "waste", because those are implementation details of the metaspace chunk allocation and - provided the implementation is correct - provide no useful information. "free" just means "space left in current chunk" which is a implementation detail. Once the chunk is used up, a new one is aquired. "waste" is only one tiny part of how memory can be wasted in the current metaspace implementation. It does not include waste in retired VirtualspaceNodes or in free chunks idling in free lists. The latter would be an important addition, regardless if this is for customers or for us. Chunks idling in freelists can make a huge impact, and unfortunately this may happen in perfectly valid cases. See e.g. our JEP " https://bugs.openjdk.java.net/browse/JDK-8166690". We have already a printing routine for free chunks, in ChunkManager::print_all_chunkmanagers(). Could you add this to your output? ------------- Other than above notes, the changes seem straighforward, I did not see anything wrong. Minor nits: - PrintCLDMetaspaceInfoClosure::_out, _scale and _unit could all be constant (with _out being a outputStream* const). - We also do not need to provide scale *and* unit as argument, one can be deduced from the other. This would also prevent mismatches. I did not look closely at locking issues, I assume MetaspaceAux::print_metadata() is supposed to run only in Safepoints? Thanks & Kind Regards, Thomas On Fri, Oct 20, 2017 at 4:00 PM, Zhengyu Gu wrote: > Up to now, there is little visibility into how metaspaces are used, and by > whom. > > This proposed enhancement gives NMT the ability to report metaspace usages > on per-classloader level, via a new NMT command "metadata" > > > For example: > > 2496: > Metaspaces: > Metadata space: reserved= 8192KB committed= 5888KB > Class space: reserved= 1048576KB committed= 640KB > > Per-classloader metadata: > > ClassLoader: for anonymous class > Metadata capacity= 5.00KB used= 1.16KB free= 3.84KB > waste= 0.05KB > Class data capacity= 1.00KB used= 0.59KB free= 0.41KB > waste= 0.00KB > > .... > > ClassLoader: > Metadata capacity= 5640.00KB used= 5607.42KB free= 32.58KB > waste= 0.46KB > Class data capacity= 520.00KB used= 500.94KB free= 19.06KB > waste= 0.02KB > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8189688 > Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html > > Test: > > hotspot_tier1_runtime, fastdebug and release on x64 Linux. > > Thanks, > > -Zhengyu > > > From martin.doerr at sap.com Mon Oct 23 13:05:16 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Mon, 23 Oct 2017 13:05:16 +0000 Subject: Unification of jni_.h In-Reply-To: References: <7dac704ecd5047de983226604ea71c74@sap.com> Message-ID: Hi Coleen, sure, we can test it when a webrev is available. Best regards, Martin -----Original Message----- From: hotspot-runtime-dev [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On Behalf Of coleen.phillimore at oracle.com Sent: Montag, 23. Oktober 2017 14:11 To: hotspot-runtime-dev at openjdk.java.net Subject: Re: Unification of jni_.h With my change that I'm working on for JDK-8189610, I'm cleaning up the jni_platform files also.??? They were mostly the same with some minor differences that we're working through. Martin, when I get this working here, can you test out your platforms on it? thanks, Coleen On 10/23/17 5:33 AM, Doerr, Martin wrote: > Hi, > > we only support 64 bit on s390. Seems like the code could be cleaned up or updated. Thanks for looking into it. > > Best regards, > Martin > > > -----Original Message----- > From: hotspot-runtime-dev [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On Behalf Of David Holmes > Sent: Montag, 23. Oktober 2017 05:47 > To: Magnus Ihse Bursie ; hotspot-runtime-dev at openjdk.java.net; build-dev > Subject: Re: Unification of jni_.h > > Hi Magnus, > > On 18/10/2017 11:10 PM, Magnus Ihse Bursie wrote: >> In the context of JDK-8167078 (Duplicate header files in hotspot and >> jdk), I was looking at unifying the platform specific extensions to >> jni.h, the jni_md.h files between hotspot and java.base. >> >> The main difference here is that the hotspot jni_* files are divided >> into individual files based on cpu, while the java.base versions are >> divided according to operating system (which, in turn, implicates >> compiler). On a second look, however, it turns out to not be so >> problematic -- a hotspot file like jni_x86.h contains basically >> #ifdef WIN32 >> ... // do windows stuff >> #else >> ... // do posix/macos stuff >> #endif >> >> and the two blocks match very well what the java.base versions are doing >> for the different operating systems. >> >> In fact, the OS (and compiler) division seems more natural, since there >> is much redundancy in the hotspot files, and the java.base OS-based >> versions are much simpler. >> >> I'm proposing to remove the hotspot CPU-based files and just replace >> them with the java.base versions. However, a few differences crept out. >> I'd like to discuss them before proceeding. >> >> For jni_aarch64.h: There's a windows (or rather, "not unix") part of the >> jni_aarch64.h that I do not believe have ever been used. Nevertheless, >> it contains "typedef int jint" rather than "typedef long jint", which >> the java.base windows version uses. Since I've never heard of aarch64 >> building on Windows, I presume this is irrelevant. > Likely just copied from the x86 version at some point. There is no > Windows-aarch64 support in the OpenJDK. > >> For jni_aarch64.h and jni_sparc.h: The unix version will match with the >> java.base version as long as _LP64 is defined. This should be just fine, >> since we define that when building hotspot/the JDK, and the java.base >> version that is exported by the JDK on linux/aarch64 and solaris/sparc >> has always required the _LP64 define. > Yep 64-bit only. > >> (In fact, the macos and unix versions in java.base only contain trivial >> differences. The macos version should probably be removed in favor of a >> single unix version.) >> >> For jni_x86.h: There windows part suffers the same problem as for >> aarch64, but here it's potentially problematic. The hotspot version uses >> "typedef int jint" while the java.base version uses "typedef long jint". >> If this *really* were a problem, we would probably have gotten reports >> from JNI code unable to work on Windows, so I assume this turns out to >> be the same datatype in the end. Don't know if it's due to luck, >> compiler flags or by definition. > Windows is either ILP32 or LLP64 - in both cases int and long are the > same and 32-bits. > >> For jni_s390.h: Here's the most problematic version. The exported >> java.base version uses: >> #ifdef _LP64 >> typedef long jlong; >> #else >> typedef long long jlong; >> #endif >> but s390 uses: >> typedef long int jlong; >> >> My best assumption here is that we're only really using s390x (the >> 64-bit version) and that "long int" is functionally equivalent to >> "long". Or that jni is broken on s390 and nobody really noticed. >> >> Also, s390 uses a simplified version of the gcc attribute dance used for >> JNIEXPORT/JNIIMPORT. I think the s390 version is just not really >> updated, and that using the newer in java.base is harmless. > Can't comment on s390. > >> For jni_arm.h: Finally, the jni_arm.h (the 32-bit formerly closed Oracle >> port), the JNIEXPORT/JNIIMPORT is different, by defining >> __attribute__((externally_visible)). This might have been relevant due >> to compile/link time symbol processing for that platform, but >> technically it should probably not have been connected to the platform >> per se, but rather to the compilation procedure. Since the arm-32 >> platform is kind of abandoned right now, I propose to modify the >> compilation to be more standard, if this is required, rather than >> keeping these attributes. > As Dean stated this likely related to LTO. And unfortunately attention > was not paid to the comment: > > // Note: please do not change these without also changing jni_md.h in > the JDK > // repository > > possibly due to open/closed issues back in time. But I'd say we would > want the hotspot version so that nothing that would previously work is > now broken. > > Thanks, > David > >> /Magnus From harold.seigel at oracle.com Mon Oct 23 13:31:35 2017 From: harold.seigel at oracle.com (harold seigel) Date: Mon, 23 Oct 2017 09:31:35 -0400 Subject: RFR 8174954: Parameter target type is allowed access after a module read edge or a package export has occurred after failed resolution In-Reply-To: References: <175262e1-5b1d-bd23-d9c7-28c24096c62a@oracle.com> <0f1ef2db-00b4-8642-d5cc-d78683a4a9b1@oracle.com> <2827d94c-1744-86da-b7bc-d67554cb3519@oracle.com> <5c14a878-9dbd-3538-7fd9-1a1067c992e1@oracle.com> Message-ID: Hi David, I'll add comments to the test to make it clearer that the IAE exception occurs when executing the invokedynamic instruction for the string concat. Thanks, Harold On 10/22/2017 2:17 AM, David Holmes wrote: > Hi Harold, > > On 20/10/2017 11:37 PM, harold seigel wrote: >> Hi David, >> >> MethodAccessReadTwice.java is an indy test.? Lines 120 - 153 test >> that the same invokedynamic instruction gets the same result.? The >> invokedynamic instruction gets generated for the string concatenation >> at line 31 of file .../p5/c5.java: > > Okay. It was totally not at all evident from the bug report and the > test code that the underlying problem was actually related to indy due > to the string concatenation. > > Thanks, > David > >> >> ?????? 24 package p5; >> ?????? 25 >> ?????? 26 import java.lang.Module; >> ?????? 27 import p2.c2; >> ?????? 28 >> ?????? 29 public class c5 { >> ?????? 30???? public void method5(p2.c2 param) { >> ?????? 31???????? System.out.println("In c5's method5 with param = " >> + param); >> ?????? 32???? } >> ?????? 33 >> ?????? 34???? public void methodAddReadEdge(Module m) { >> ?????? 35???????? // Add a read edge from p5/c5's module (first_mod) >> to second_mod >> ?????? 36???????? c5.class.getModule().addReads(m); >> ?????? 37???? } >> ?????? 38 } >> >> The indy resolution throws an IAE because package p5's module cannot >> read package p2's module. >> >> The second test case, at lines 156 - 161 of >> MethodAccessReadTwice.java, similarly tests invokedynamic >> instructions.? In this case, they are generated for the string >> concatenations at lines 32 and 43 of file .../p7/c7.java. >> >> This problem does not exist for non-indy instructions because, when >> their resolutions fail, they update the constant pool. Indy cannot do >> this because, unlike other instructions, different indy instructions >> that use the same constant pool index are allowed to get different >> results.? Hence, the fix is indy only. >> >> Thanks, Harold >> >> On 10/20/2017 1:15 AM, David Holmes wrote: >>> Hi Harold, >>> >>> On 19/10/2017 12:20 AM, harold seigel wrote: >>>> Hi David, >>>> >>>> Thanks for looking at this. >>>> >>>> Please see inline comments. >>>> >>>> Thanks, Harold >>>> >>>> >>>> On 10/17/2017 10:22 PM, David Holmes wrote: >>>>> Hi Harold, >>>>> >>>>> On 17/10/2017 12:20 AM, harold seigel wrote: >>>>>> Hi, >>>>>> >>>>>> Please review this new JDK-10 fix for bug JDK-8174954. If the >>>>>> initial resolution attempt fails with a LinkageError exception >>>>>> then the fix saves the exception in the resolution_errors table >>>>>> and sets a flag in the constant pool Cache, indicating the >>>>>> failure.? Subsequent attempts to do the same resolution will see >>>>>> that the flag is set, retrieve the exception from the >>>>>> resolution_errors table, and throw it, instead of re-trying the >>>>>> resolution. >>>>> >>>>> I'm a little confused. The fix seems all about indy, which seems >>>>> to be the topic of: >>>>> >>>>> ?JDK-8174942 Bootstrap Method Called Multiple Times Despite >>>>> Initial Resolution Failure >>>>> >>>>> whereas this bug seems to be about a more general problem. Is this >>>>> fix addressing both issues?? >>>> Thanks for noticing this.? This fix addresses both bugs. The sample >>>> program for 8174942 is one of the test cases for this fix.? I'll >>>> close 8174942 as a duplicate. >>> >>> I'm still somewhat confused as it seems that all the substantive >>> changes here have the word "indy" in them. What part of the change >>> actually fixes the non-indy issue tested by MethodAccessReadTwice.java? >>> >>> Thanks, >>> David >>>>> >>>>> BTW there's no reason for JDK-8174954 to remain confidential. >>>> Agreed.? I opened the bug. >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> The fix also prevents a race condition if two or more threads try >>>>>> to do the same resolution concurrently. When a thread tries to >>>>>> record the result of its resolution attempt, it will check to see >>>>>> if another thread recorded its result first.? If so, then it will >>>>>> use the result recorded by the other thread.? Otherwise, it will >>>>>> record and use its result. Access to the resolution result is >>>>>> protected by the 'resolved_references' lock. >>>>>> >>>>>> Open webrev: >>>>>> http://cr.openjdk.java.net/~hseigel/bug_8174954.1/webrev/ >>>>>> >>>>>> JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8174954 >>>>>> >>>>>> The fix was tested with the JCK Lang and VM tests, the JTreg >>>>>> hotspot, java/io, java/lang, java/util, and other tests, the >>>>>> co-located NSK tests, JPRT, with Mach5 tier2 - tier5 tests, and >>>>>> by hand to check race condition handling. >>>>>> >>>>>> Thanks, Harold >>>>>> >>>> >> From zgu at redhat.com Mon Oct 23 14:03:34 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 23 Oct 2017 10:03:34 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> Message-ID: <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> Hi Thomas, Thanks for the review. Please see comments inline. On 10/23/2017 09:01 AM, Thomas St?fe wrote: > Hi Zhengyu, > > this is a nice addition! > > ------ > > General remarks: > > We already have a multitude of printing functions in metaspace.cpp, in > MetaspaceAux I find: > > void MetaspaceAux::print_on(outputStream* out) > > and > > void MetaspaceAux::print_on(outputStream* out, Metaspace::MetadataType > mdtype) > void MetaspaceAux::dump(outputStream* out) > > There even exists a cleanup issue: > https://bugs.openjdk.java.net/browse/JDK-8185034. > > I am not happy that we add yet another printing function to this bunch, > which basically does the same. Is there any way to reuse the existing > functions? > > If not, could we at least for now rename print_metadata() to something > like print_metadata_for_nmt(), to tell it apart from print_on etc? And > mark them for later cleanup and consolidation? Yes, I noticed the existing print_on functions and thought about using them. However, none of them matches NMT convention, e.g. NMT uses "=" between label and value. I will rename print_metadata to print_metadata_for_nmt. > > ---------- > > About the numbers: > > I am not sure who this output is for. Customers analyzing metaspace > shortage or hotspot developers debugging the metaspace allocation? > Depending on the answer I would change the output somewhat. > > If this it is for customers, the numbers are too much and the names are > misleading. Especially the per-classloader numbers: > > "Metadata capacity=%10.2f%s used=%10.2f%s free=%10.2f%s waste=%10.2f%s" > > If we only want to answer the question "how much metaspace does a > classloader use up?", either "capacity" or "used" should be sufficient. > It does not really matter much which, really, and for most cases the > numbers should be very similar. > In this case I would omit "free" and "waste", because those are > implementation details of the metaspace chunk allocation and - provided > the implementation is correct - provide no useful information. "free" > just means "space left in current chunk" which is a implementation > detail. Once the chunk is used up, a new one is aquired. "waste" is only > one tiny part of how memory can be wasted in the current metaspace > implementation. It does not include waste in retired VirtualspaceNodes > or in free chunks idling in free lists. > As far as I know, NMT is largely used by JVM developers. "free" and "waste" information, actually, is very important for us to understand some of issues with current implementation. (e.g. https://bugs.openjdk.java.net/browse/JDK-8187338) This patch is derived from our investigation of memory cost for classes, these numbers play important roles. > The latter would be an important addition, regardless if this is for > customers or for us. Chunks idling in freelists can make a huge impact, > and unfortunately this may happen in perfectly valid cases. See e.g. our > JEP "https://bugs.openjdk.java.net/browse/JDK-8166690". We have already > a printing routine for free chunks, in > ChunkManager::print_all_chunkmanagers(). Could you add this to your output? Make sense! Could you recommend what data and format you would like to see? > > ------------- > > Other than above notes, the changes seem straighforward, I did not see > anything wrong. Minor nits: > > - PrintCLDMetaspaceInfoClosure::_out, _scale and _unit could all be > constant (with _out being a outputStream* const). > - We also do not need to provide scale *and* unit as argument, one can > be deduced from the other. This would also prevent mismatches.We do need scale here, since nmt command line can set scale and we do have to honor that. However, passing unit is ugly! I don't want to introduce inverse dependence (metaspace -> nmt), which is also ugly. Probably should move scale mapping code from NMTUtil to a common util? > > I did not look closely at locking issues, I assume > MetaspaceAux::print_metadata() is supposed to run only in Safepoints? No. sum_xxx_xxx_in_use queries are guarded by locks. Thanks, -Zhengyu > > > Thanks & Kind Regards, Thomas > > > > > > > On Fri, Oct 20, 2017 at 4:00 PM, Zhengyu Gu > wrote: > > Up to now, there is little visibility into how metaspaces are used, > and by whom. > > This proposed enhancement gives NMT the ability to report metaspace > usages on per-classloader level, via a new NMT command "metadata" > > > For example: > > 2496: > Metaspaces: > Metadata space: reserved= 8192KB committed= 5888KB > Class space: reserved= 1048576KB committed= 640KB > > Per-classloader metadata: > > ClassLoader: for anonymous class > Metadata capacity= 5.00KB used= 1.16KB free= > 3.84KB waste= 0.05KB > Class data capacity= 1.00KB used= 0.59KB free= > 0.41KB waste= 0.00KB > > .... > > ClassLoader: > Metadata capacity= 5640.00KB used= 5607.42KB free= > 32.58KB waste= 0.46KB > Class data capacity= 520.00KB used= 500.94KB free= > 19.06KB waste= 0.02KB > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8189688 > > Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html > > > Test: > > hotspot_tier1_runtime, fastdebug and release on x64 Linux. > > Thanks, > > -Zhengyu > > > From thomas.stuefe at gmail.com Mon Oct 23 14:40:55 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 23 Oct 2017 16:40:55 +0200 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> Message-ID: Hi Zhengyu, On Mon, Oct 23, 2017 at 4:03 PM, Zhengyu Gu wrote: > Hi Thomas, > > Thanks for the review. Please see comments inline. > > On 10/23/2017 09:01 AM, Thomas St?fe wrote: > >> Hi Zhengyu, >> >> this is a nice addition! >> >> ------ >> >> General remarks: >> >> We already have a multitude of printing functions in metaspace.cpp, in >> MetaspaceAux I find: >> >> void MetaspaceAux::print_on(outputStream* out) >> >> and >> >> void MetaspaceAux::print_on(outputStream* out, Metaspace::MetadataType >> mdtype) >> void MetaspaceAux::dump(outputStream* out) >> >> There even exists a cleanup issue: https://bugs.openjdk.java.net/ >> browse/JDK-8185034. >> >> I am not happy that we add yet another printing function to this bunch, >> which basically does the same. Is there any way to reuse the existing >> functions? >> >> If not, could we at least for now rename print_metadata() to something >> like print_metadata_for_nmt(), to tell it apart from print_on etc? And mark >> them for later cleanup and consolidation? >> > > Yes, I noticed the existing print_on functions and thought about using > them. However, none of them matches NMT convention, e.g. NMT uses "=" > between label and value. > > I will rename print_metadata to print_metadata_for_nmt. > > Thank you. > > >> ---------- >> >> About the numbers: >> >> I am not sure who this output is for. Customers analyzing metaspace >> shortage or hotspot developers debugging the metaspace allocation? >> Depending on the answer I would change the output somewhat. >> >> If this it is for customers, the numbers are too much and the names are >> misleading. Especially the per-classloader numbers: >> >> "Metadata capacity=%10.2f%s used=%10.2f%s free=%10.2f%s waste=%10.2f%s" >> >> If we only want to answer the question "how much metaspace does a >> classloader use up?", either "capacity" or "used" should be sufficient. It >> does not really matter much which, really, and for most cases the numbers >> should be very similar. >> > > In this case I would omit "free" and "waste", because those are >> implementation details of the metaspace chunk allocation and - provided the >> implementation is correct - provide no useful information. "free" just >> means "space left in current chunk" which is a implementation detail. Once >> the chunk is used up, a new one is aquired. "waste" is only one tiny part >> of how memory can be wasted in the current metaspace implementation. It >> does not include waste in retired VirtualspaceNodes or in free chunks >> idling in free lists. >> >> As far as I know, NMT is largely used by JVM developers. > > "free" and "waste" information, actually, is very important for us to > understand some of issues with current implementation. > (e.g. https://bugs.openjdk.java.net/browse/JDK-8187338) > > This patch is derived from our investigation of memory cost for classes, > these numbers play important roles. > > Okay. So this is important for understanding cases where we have a large number of miniscule class loaders, each one only having a few numbers of chunks. In this case, would it be useful to have a running total of "free" and "waste", too? Or even better, also average and peak values of "free" and "waste" (to tell apart cases where you have one outlier vs cases where just all metaspaces waste a lot of memory). Just out of curiosity, I looked at http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt and it seems that "free" is the problem, not "waste"? So I guess what happens is that all the little classloaders allocate just enough memory to push them over "_small_chunk_limit=4", so they allocate the first medium chunk, from which then they take a small bite and leave the rest laying around? > The latter would be an important addition, regardless if this is for >> customers or for us. Chunks idling in freelists can make a huge impact, and >> unfortunately this may happen in perfectly valid cases. See e.g. our JEP " >> https://bugs.openjdk.java.net/browse/JDK-8166690". We have already a >> printing routine for free chunks, in ChunkManager::print_all_chunkmanagers(). >> Could you add this to your output? >> > > Make sense! Could you recommend what data and format you would like to see? > > > Would not ChunkManager::print_all_chunkmanagers() be sufficient? > >> ------------- >> >> Other than above notes, the changes seem straighforward, I did not see >> anything wrong. Minor nits: >> >> - PrintCLDMetaspaceInfoClosure::_out, _scale and _unit could all be >> constant (with _out being a outputStream* const). >> - We also do not need to provide scale *and* unit as argument, one can be >> deduced from the other. This would also prevent mismatches.We do need scale >> here, since nmt command line can set scale and we do >> > have to honor that. > > However, passing unit is ugly! I don't want to introduce inverse > dependence (metaspace -> nmt), which is also ugly. > > Yes, that would be regrettable. > Probably should move scale mapping code from NMTUtil to a common util? > > > That would be possible, these functions (scale_from_name etc) are simple enough to be moved to a generic file. However, if you pass scale, deducing the string that goes with it is trivial and I would have no qualms doing this by hand in metaspace.cpp. But it is a matter of taste. > >> I did not look closely at locking issues, I assume >> MetaspaceAux::print_metadata() is supposed to run only in Safepoints? >> > > No. sum_xxx_xxx_in_use queries are guarded by locks. > > Thanks, > > -Zhengyu > > Thanks, Thomas > >> >> Thanks & Kind Regards, Thomas >> >> >> >> >> >> >> On Fri, Oct 20, 2017 at 4:00 PM, Zhengyu Gu > zgu at redhat.com>> wrote: >> >> Up to now, there is little visibility into how metaspaces are used, >> and by whom. >> >> This proposed enhancement gives NMT the ability to report metaspace >> usages on per-classloader level, via a new NMT command "metadata" >> >> >> For example: >> >> 2496: >> Metaspaces: >> Metadata space: reserved= 8192KB committed= 5888KB >> Class space: reserved= 1048576KB committed= 640KB >> >> Per-classloader metadata: >> >> ClassLoader: for anonymous class >> Metadata capacity= 5.00KB used= 1.16KB free= >> 3.84KB waste= 0.05KB >> Class data capacity= 1.00KB used= 0.59KB free= >> 0.41KB waste= 0.00KB >> >> .... >> >> ClassLoader: >> Metadata capacity= 5640.00KB used= 5607.42KB free= >> 32.58KB waste= 0.46KB >> Class data capacity= 520.00KB used= 500.94KB free= >> 19.06KB waste= 0.02KB >> >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >> >> >> Test: >> >> hotspot_tier1_runtime, fastdebug and release on x64 Linux. >> >> Thanks, >> >> -Zhengyu >> >> >> >> From adinn at redhat.com Mon Oct 23 14:52:12 2017 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 23 Oct 2017 15:52:12 +0100 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> Message-ID: On 23/10/17 15:40, Thomas St?fe wrote: > Okay. So this is important for understanding cases where we have a large > number of miniscule class loaders, each one only having a few numbers of > chunks. In this case, would it be useful to have a running total of "free" > and "waste", too? Or even better, also average and peak values of "free" > and "waste" (to tell apart cases where you have one outlier vs cases where > just all metaspaces waste a lot of memory). > > Just out of curiosity, I looked at > http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt and it seems > that "free" is the problem, not "waste"? So I guess what happens is that > all the little classloaders allocate just enough memory to push them over > "_small_chunk_limit=4", so they allocate the first medium chunk, from which > then they take a small bite and leave the rest laying around? Yes, that is pretty how it works. Only the problem is twice as bad when you have separate class metadata and non-class metadata (i.e. when using compressed class oops which is by far the most common case) since this invariably wastes more space. Similar behaviour happens when an anonymous class is created because of a lambda, except that you only get one small allocation before a medium chunk is created. So, for small lambdas, you almost always end up using around 50% of 2 1MB chunks and, for more complex lambdas, around 50% of a 1MB chunk plus less than 50% of a 1MB + 4MB chunk. So, the space waste for lambdas is really quite spectacular. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From thomas.stuefe at gmail.com Mon Oct 23 15:03:07 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 23 Oct 2017 17:03:07 +0200 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> Message-ID: On Mon, Oct 23, 2017 at 4:52 PM, Andrew Dinn wrote: > On 23/10/17 15:40, Thomas St?fe wrote: > > Okay. So this is important for understanding cases where we have a large > > number of miniscule class loaders, each one only having a few numbers of > > chunks. In this case, would it be useful to have a running total of > "free" > > and "waste", too? Or even better, also average and peak values of "free" > > and "waste" (to tell apart cases where you have one outlier vs cases > where > > just all metaspaces waste a lot of memory). > > > > Just out of curiosity, I looked at > > http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt and it seems > > that "free" is the problem, not "waste"? So I guess what happens is that > > all the little classloaders allocate just enough memory to push them over > > "_small_chunk_limit=4", so they allocate the first medium chunk, from > which > > then they take a small bite and leave the rest laying around? > > Yes, that is pretty how it works. Only the problem is twice as bad when > you have separate class metadata and non-class metadata (i.e. when using > compressed class oops which is by far the most common case) since this > invariably wastes more space. > > :( In this case, is the performance gain of CompressedClassPointers even worth it? > Similar behaviour happens when an anonymous class is created because of > a lambda, except that you only get one small allocation before a medium > chunk is created. So, for small lambdas, you almost always end up using > around 50% of 2 1MB chunks and, for more complex lambdas, around 50% of > a 1MB chunk plus less than 50% of a 1MB + 4MB chunk. So, the space waste > for lambdas is really quite spectacular. > > Ouch. I am confused about the sizes though . Are we talking about the chunk sizes in metaspace.cpp? Because they are way smaller, with medium chunks being 32/64K? Or are these large chunks a RedHat addition? Best Regards, Thomas > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > From zgu at redhat.com Mon Oct 23 15:03:07 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 23 Oct 2017 11:03:07 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> Message-ID: <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> > > > Okay. So this is important for understanding cases where we have a large > number of miniscule class loaders, each one only having a few numbers of > chunks. In this case, would it be useful to have a running total of > "free" and "waste", too? Or even better, also average and peak values of > "free" and "waste" (to tell apart cases where you have one outlier vs > cases where just all metaspaces waste a lot of memory). > Just out of curiosity, I looked at > http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt and it seems > that "free" is the problem, not "waste"? So I guess what happens is that > all the little classloaders allocate just enough memory to push them > over "_small_chunk_limit=4", so they allocate the first medium chunk, > from which then they take a small bite and leave the rest laying around? > Yes. The free space of anonymous classes should be counted as waste, in my opinion. I am not sure if everyone agrees, so I took the summary line out of this patch. I would be more than happy to restore the summary line, if you find it is useful :-) > The latter would be an important addition, regardless if this is > for customers or for us. Chunks idling in freelists can make a > huge impact, and unfortunately this may happen in perfectly > valid cases. See e.g. our JEP > "https://bugs.openjdk.java.net/browse/JDK-8166690 > ". We have > already a printing routine for free chunks, in > ChunkManager::print_all_chunkmanagers(). Could you add this to > your output? > > > Make sense! Could you recommend what data and format you would like > to see? > > > > Would not ChunkManager::print_all_chunkmanagers() be sufficient? Okay, I might need to tweak output format. Thanks, -Zhengyu > > > ------------- > > Other than above notes, the changes seem straighforward, I did > not see anything wrong. Minor nits: > > - PrintCLDMetaspaceInfoClosure::_out, _scale and _unit could all > be constant (with _out being a outputStream* const). > - We also do not need to provide scale *and* unit as argument, > one can be deduced from the other. This would also prevent > mismatches.We do need scale here, since nmt command line can set > scale and we do > > have to honor that. > > However, passing unit is ugly! I don't want to introduce inverse > dependence (metaspace -> nmt), which is also ugly. > > > Yes, that would be regrettable. > > Probably should move scale mapping code from NMTUtil to a common util? > > > > That would be possible, these functions (scale_from_name etc) are simple > enough to be moved to a generic file. However, if you pass scale, > deducing the string that goes with it is trivial and I would have no > qualms doing this by hand in metaspace.cpp. But it is a matter of taste. > > > I did not look closely at locking issues, I assume > MetaspaceAux::print_metadata() is supposed to run only in > Safepoints? > > > No. sum_xxx_xxx_in_use queries are guarded by locks. > > Thanks, > > -Zhengyu > > > Thanks, Thomas > > > > Thanks & Kind Regards, Thomas > > > > > > > On Fri, Oct 20, 2017 at 4:00 PM, Zhengyu Gu >> wrote: > > Up to now, there is little visibility into how metaspaces > are used, > and by whom. > > This proposed enhancement gives NMT the ability to report > metaspace > usages on per-classloader level, via a new NMT command > "metadata" > > > For example: > > 2496: > Metaspaces: > Metadata space: reserved= 8192KB committed= 5888KB > Class space: reserved= 1048576KB committed= 640KB > > Per-classloader metadata: > > ClassLoader: for anonymous class > Metadata capacity= 5.00KB used= 1.16KB > free= 3.84KB waste= 0.05KB > Class data capacity= 1.00KB used= 0.59KB > free= 0.41KB waste= 0.00KB > > .... > > ClassLoader: > Metadata capacity= 5640.00KB used= 5607.42KB > free= 32.58KB waste= 0.46KB > Class data capacity= 520.00KB used= 500.94KB > free= 19.06KB waste= 0.02KB > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8189688 > > > > Webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html > > > > > > Test: > > hotspot_tier1_runtime, fastdebug and release on x64 Linux. > > Thanks, > > -Zhengyu > > > > From thomas.stuefe at gmail.com Mon Oct 23 15:08:40 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 23 Oct 2017 17:08:40 +0200 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> Message-ID: On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu wrote: > >> >> Okay. So this is important for understanding cases where we have a large >> number of miniscule class loaders, each one only having a few numbers of >> chunks. In this case, would it be useful to have a running total of "free" >> and "waste", too? Or even better, also average and peak values of "free" >> and "waste" (to tell apart cases where you have one outlier vs cases where >> just all metaspaces waste a lot of memory). >> Just out of curiosity, I looked at http://cr.openjdk.java.net/~zg >> u/cld_metaspace/wildfly.txt and it seems that "free" is the problem, >> not "waste"? So I guess what happens is that all the little classloaders >> allocate just enough memory to push them over "_small_chunk_limit=4", so >> they allocate the first medium chunk, from which then they take a small >> bite and leave the rest laying around? >> >> Yes. The free space of anonymous classes should be counted as waste, in > my opinion. I am not sure if everyone agrees, so I took the summary line > out of this patch. > > I would be more than happy to restore the summary line, if you find it is > useful :-) > > I agree with you. Typically class loaders stop loading at some point, especially these tine ones, and often will not resume loading. So, effectivly, the space is wasted. It still helps to distinguish "free" in the current chunks from "free" in the other chunks to tell this situation apart from a situation where we have just a bug in metaspace chunk handling preventing us to use up our chunks. > > The latter would be an important addition, regardless if this is >> for customers or for us. Chunks idling in freelists can make a >> huge impact, and unfortunately this may happen in perfectly >> valid cases. See e.g. our JEP >> "https://bugs.openjdk.java.net/browse/JDK-8166690 >> ". We have >> already a printing routine for free chunks, in >> ChunkManager::print_all_chunkmanagers(). Could you add this to >> your output? >> >> >> Make sense! Could you recommend what data and format you would like >> to see? >> >> >> >> Would not ChunkManager::print_all_chunkmanagers() be sufficient? >> > Okay, I might need to tweak output format. > > Fine by me. Nothing depends on it yet. Thanks, Thomas > Thanks, > > -Zhengyu > > >> >> ------------- >> >> Other than above notes, the changes seem straighforward, I did >> not see anything wrong. Minor nits: >> >> - PrintCLDMetaspaceInfoClosure::_out, _scale and _unit could all >> be constant (with _out being a outputStream* const). >> - We also do not need to provide scale *and* unit as argument, >> one can be deduced from the other. This would also prevent >> mismatches.We do need scale here, since nmt command line can set >> scale and we do >> >> have to honor that. >> >> However, passing unit is ugly! I don't want to introduce inverse >> dependence (metaspace -> nmt), which is also ugly. >> >> >> Yes, that would be regrettable. >> >> Probably should move scale mapping code from NMTUtil to a common util? >> >> >> >> That would be possible, these functions (scale_from_name etc) are simple >> enough to be moved to a generic file. However, if you pass scale, deducing >> the string that goes with it is trivial and I would have no qualms doing >> this by hand in metaspace.cpp. But it is a matter of taste. >> >> >> I did not look closely at locking issues, I assume >> MetaspaceAux::print_metadata() is supposed to run only in >> Safepoints? >> >> >> No. sum_xxx_xxx_in_use queries are guarded by locks. >> >> Thanks, >> >> -Zhengyu >> >> >> Thanks, Thomas >> >> >> >> Thanks & Kind Regards, Thomas >> >> >> >> >> >> >> On Fri, Oct 20, 2017 at 4:00 PM, Zhengyu Gu > > >> >> wrote: >> >> Up to now, there is little visibility into how metaspaces >> are used, >> and by whom. >> >> This proposed enhancement gives NMT the ability to report >> metaspace >> usages on per-classloader level, via a new NMT command >> "metadata" >> >> >> For example: >> >> 2496: >> Metaspaces: >> Metadata space: reserved= 8192KB committed= >> 5888KB >> Class space: reserved= 1048576KB committed= >> 640KB >> >> Per-classloader metadata: >> >> ClassLoader: for anonymous class >> Metadata capacity= 5.00KB used= 1.16KB >> free= 3.84KB waste= 0.05KB >> Class data capacity= 1.00KB used= 0.59KB >> free= 0.41KB waste= 0.00KB >> >> .... >> >> ClassLoader: >> Metadata capacity= 5640.00KB used= 5607.42KB >> free= 32.58KB waste= 0.46KB >> Class data capacity= 520.00KB used= 500.94KB >> free= 19.06KB waste= 0.02KB >> >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> Webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> > >> >> Test: >> >> hotspot_tier1_runtime, fastdebug and release on x64 Linux. >> >> Thanks, >> >> -Zhengyu >> >> >> >> >> From rkennke at redhat.com Mon Oct 23 15:19:10 2017 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 23 Oct 2017 17:19:10 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <59EA002D.1050605@oracle.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> Message-ID: <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> Hi Erik, thanks for your suggestions. So I renamed 'GC' -> 'CollectedHeapFactory' (and subclasses likewise) and 'GCFactory' to 'GCArgumentProcessor'. This will make even more sense once I added heap creation to CollectedHeapFactory (JDK-8189389). Differential: http://cr.openjdk.java.net/~rkennke/8189171/webrev.02.diff/ Full webrev: http://cr.openjdk.java.net/~rkennke/8189171/webrev.02/ Better? Roman > Hi Roman, > > I'm usually not one to really mind names too much, but I don't believe > some bootstrapping code for the GC should hog the name "GC". > > Instead of GC::initialize(), I'm thinking a static > GCArgumentProcessor::initialize() and instead of GC::factory(), I am > thinking a static CollectedHeapFactory::create(). Or something like that. > > The reason is that I think the name GC is too general for the > bootstraping-only problems its code touches. Hope I am making sense. > > Thanks, > /Erik > > On 2017-10-20 14:25, Roman Kennke wrote: >> Hi Erik, >> >> Yes that is fine. >> >> Keep in mind that I'm planning to put heap creation into that class too. >> >> I was thinking maybe to simply reverse the current naming: name the >> all-static 'factory factory' 'GC', and thus call GC::initialize() and >> GC::factory() or such, and the main interface GCFactory or >> CollectedHeapFactory. What do you think? >> >> Roman >> >> >>> I would like to keep the CollectedHeap as the facade interface to >>> the GC, like it is today, instead of having a new GC class making it >>> two facade interfaces. >>> Of course, the CollectedHeap may only be used as a facade after it >>> has been created. And now we are dealing with code before it was >>> created during the bootstrapping process. >>> >>> So what I would like to have here is a class specifically for that, >>> rather than another GC facade interface. I think this is mostly an >>> exercise of finding a better name for the class you call "GC". Now I >>> am not an expert in coming up with good names myself, but some name >>> that indicates this is being used for flag processing would be good. >>> Perhaps GCArgumentProcessor. The benefit of calling it something >>> like that is that if a GC requires argument processing later in the >>> bootstrapping, possibly after CollectedHeap has been created, it can >>> still be used for this purpose without having to feel weird about it. >>> >>> How would you feel about that? Does it seem to make sense? >>> >>> Thanks, >>> /Erik >>> >>> On 2017-10-19 21:14, Roman Kennke wrote: >>>> Am 13.10.2017 um 03:20 schrieb David Holmes: >>>>> Hi Roman, >>>>> >>>>> Not a review as GC folk need to do that. >>>>> >>>>> On 13/10/2017 5:59 AM, Roman Kennke wrote: >>>>>> I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev >>>>>> because it touches both areas (i.e. the GC interface). >>>>>> >>>>>> Currently, all GC related argument processing is done in >>>>>> arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all >>>>>> sorts of GC specific methods etc. >>>>> >>>>> From a runtime perspective I like all the GC specific ifdefs and >>>>> settings moved out-of-line of the main argument processing. >>>>> >>>>> From a refactoring perspective I noticed that >>>>> set_object_alignment() no longer calls set_cms_values(). I presume >>>>> setting it elsewhere is okay? >>>> I totally forgot to reply to this. >>>> >>>> What's important is that the CMS alignment values are set after >>>> set_object_alignment() figured out the object alignment. The patch >>>> moves that a little further down the road to the beginning of its >>>> GC specific argument processing, but from GC perspective should be >>>> the same. I looked thoroughly through the involved code paths and >>>> cannot see what could go wrong. >>>> >>>> I need one more review. GC folks? The current webrev is: >>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.01/ >>>> >>>> >>>> Thanks, >>>> Roman >>>> >>> >> > From tomas.kalibera at gmail.com Tue Oct 10 09:36:27 2017 From: tomas.kalibera at gmail.com (Tomas Kalibera) Date: Tue, 10 Oct 2017 11:36:27 +0200 Subject: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? Message-ID: Dear hotspot developers, I am one of the core R developers and what has been causing us a lot of trouble is the 2M cap for stack size of the initial thread on Linux. In short, R has its own stack overflow checking, the stack size is taken from rlimit, and R needs a lot of stack (often more than 2M). Via rJava package, R uses Java, so it initializes Java via JNI. As a workaround for some old Linux problem, hotspot caps the stack at 2M by inserting guard pages. Consequently, R crashes when the recursion gets too deep, and the crash is ugly as R's stack overflow checking does not now about the shrinkage of the stack. Users have no chance of diagnosing what's wrong and it is causing us trouble increasingly more lately, as we have more recursive calls (JIT implemented in R) and as newer gcc versions produce bigger frames from our interpreter loop. The problem is still present in Java 9. We're so hopeless that I implemented a workaround for rJava where we fill up the stack just enough before initializing the JVM so that is_initial_thread returns false even for the initial thread, so the guard page is not inserted. This is probably something we will use for now - or is there a simpler workaround? (the code is here, but probably you get the idea just from the description of it: https://github.com/s-u/rJava/pull/102/files) As you surely know well, the 2M cap comes from a bug report: http://bugs.java.com/view_bug.do?bug_id=4466587 And there is a cleanup of this code discussed in http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html If I understand correctly the discussion, this is the new version for Java 10 - is that correct? http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html Now, if I am reading this patch correctly, I think that the 2M cap is gone. There is a new 8M cap, which however only applies when the rlimit stack size is unlimited, which is something we could probably live with. Is this a correct interpretation of the code? If so, we should be fine for Java 10. Even though, having an option to initialize the JVM without stack overflow checking for the initial thread might be even better... for cases like ours, when we have our own stack overflow checking. Thanks, Tomas From adinn at redhat.com Mon Oct 23 15:29:19 2017 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 23 Oct 2017 16:29:19 +0100 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> Message-ID: On 23/10/17 16:03, Thomas St?fe wrote: > I am confused about the sizes though?. Are we talking about the chunk > sizes in metaspace.cpp? Because they are way smaller, with medium chunks > being 32/64K? Or are these large chunks a RedHat addition? Oops, apologies -- wrong units! Yes, this is indeed chunk sizes but I meant to say 1KB and 4KB chunks (MBs would be truly dreadful :-). The relevant definitions in metaspace.cpp are: enum ChunkSizes { // in words. ClassSpecializedChunk = 128, SpecializedChunk = 128, ClassSmallChunk = 256, SmallChunk = 512, ClassMediumChunk = 4 * K, MediumChunk = 8 * K }; Anonymous classes (used for the anon classes used to back lambdas) initially allocate a ClassSpecializedChunk and a SpecializedChunk i.e. 2 x 128 * 8 bytes = 2 x 1KB. If you have something that goes beyond the most basic lambda then you may well also allocate an extra SmallChunk = 512 * 8 bytes = 4KB. We experimented (with both real and synthetic examples) and found a high proportion of cases where 6KB chunks was being allocated but around 50% was left unused as free tail ends of these 2/3 chunks. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From sgehwolf at redhat.com Mon Oct 23 16:16:58 2017 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Mon, 23 Oct 2017 18:16:58 +0200 Subject: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? In-Reply-To: References: Message-ID: <1508775418.3888.12.camel@redhat.com> On Tue, 2017-10-10 at 11:36 +0200, Tomas Kalibera wrote: > Dear hotspot developers, > > I am one of the core R developers and what has been causing us a lot of? > trouble is the 2M cap for stack size of the initial thread on Linux. In? > short, R has its own stack overflow checking, the stack size is taken? > from rlimit, and R needs a lot of stack (often more than 2M). Via rJava? > package, R uses Java, so it initializes Java via JNI. As a workaround? > for some old Linux problem, hotspot caps the stack at 2M by inserting? > guard pages. Consequently, R crashes when the recursion gets too deep,? > and the crash is ugly as R's stack overflow checking does not now about? > the shrinkage of the stack. Users have no chance of diagnosing what's? > wrong and it is causing us trouble increasingly more lately, as we have? > more recursive calls (JIT implemented in R) and as newer gcc versions? > produce bigger frames from our interpreter loop. The problem is still? > present in Java 9. > > We're so hopeless that I implemented a workaround for rJava where we? > fill up the stack just enough before initializing the JVM so that? > is_initial_thread returns false even for the initial thread, so the? > guard page is not inserted. This is probably something we will use for? > now - or is there a simpler workaround? (the code is here, but probably? > you get the idea just from the description of it:? > https://github.com/s-u/rJava/pull/102/files) > > As you surely know well, the 2M cap comes from a bug report:? > http://bugs.java.com/view_bug.do?bug_id=4466587 > And there is a cleanup of this code discussed in > http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html > > If I understand correctly the discussion, this is the new version for? > Java 10 - is that correct? > http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html > > Now, if I am reading this patch correctly, I think that the 2M cap is? > gone. There is a new 8M cap, which however only applies when the rlimit? > stack size is unlimited, which is something we could probably live with.? > Is this a correct interpretation of the code? If so, we should be fine? > for Java 10. > > Even though, having an option to initialize the JVM without stack? > overflow checking for the initial thread might be even better... for? > cases like ours, when we have our own stack overflow checking. This maybe what you are looking for: https://bugs.openjdk.java.net/browse/JDK-8189170 Cheers, Severin From martin.doerr at sap.com Mon Oct 23 17:25:17 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Mon, 23 Oct 2017 17:25:17 +0000 Subject: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? In-Reply-To: References: Message-ID: Hi everbody, after reading several mails about guard page problems (and quite some own bad experience), I wonder if they are really worth all the pain. Is the performance benefit from stack banging so great, that it justifies the tremendous complexity? If not, I'd vote for replacing them by explicit checks. Loading a limit from the Thread and performing a comparison doesn't sound so expensive. It could be the case that some platforms need banging (e.g. Windows due to on demand committed stack), but probably not all. Am I missing anything? Best regards, Martin -----Original Message----- From: hotspot-runtime-dev [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On Behalf Of Tomas Kalibera Sent: Dienstag, 10. Oktober 2017 11:36 To: hotspot-runtime-dev at openjdk.java.net Subject: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? Dear hotspot developers, I am one of the core R developers and what has been causing us a lot of trouble is the 2M cap for stack size of the initial thread on Linux. In short, R has its own stack overflow checking, the stack size is taken from rlimit, and R needs a lot of stack (often more than 2M). Via rJava package, R uses Java, so it initializes Java via JNI. As a workaround for some old Linux problem, hotspot caps the stack at 2M by inserting guard pages. Consequently, R crashes when the recursion gets too deep, and the crash is ugly as R's stack overflow checking does not now about the shrinkage of the stack. Users have no chance of diagnosing what's wrong and it is causing us trouble increasingly more lately, as we have more recursive calls (JIT implemented in R) and as newer gcc versions produce bigger frames from our interpreter loop. The problem is still present in Java 9. We're so hopeless that I implemented a workaround for rJava where we fill up the stack just enough before initializing the JVM so that is_initial_thread returns false even for the initial thread, so the guard page is not inserted. This is probably something we will use for now - or is there a simpler workaround? (the code is here, but probably you get the idea just from the description of it: https://github.com/s-u/rJava/pull/102/files) As you surely know well, the 2M cap comes from a bug report: http://bugs.java.com/view_bug.do?bug_id=4466587 And there is a cleanup of this code discussed in http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html If I understand correctly the discussion, this is the new version for Java 10 - is that correct? http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html Now, if I am reading this patch correctly, I think that the 2M cap is gone. There is a new 8M cap, which however only applies when the rlimit stack size is unlimited, which is something we could probably live with. Is this a correct interpretation of the code? If so, we should be fine for Java 10. Even though, having an option to initialize the JVM without stack overflow checking for the initial thread might be even better... for cases like ours, when we have our own stack overflow checking. Thanks, Tomas From poonam.bajaj at oracle.com Mon Oct 23 17:54:58 2017 From: poonam.bajaj at oracle.com (Poonam Parhar) Date: Mon, 23 Oct 2017 10:54:58 -0700 Subject: [8u] RFR: 8189599: InitialBootClassLoaderMetaspaceSize and CompressedClassSpaceSize should be checked consistent from MaxMetaspaceSize In-Reply-To: <1db9f08c-165d-3e2d-fc88-fbf3001de2cd@gmail.com> References: <1db9f08c-165d-3e2d-fc88-fbf3001de2cd@gmail.com> Message-ID: <43ceeb98-0f52-6852-1c87-454465476f0d@oracle.com> Hello Yasumasa, The changes look good! Thanks, Poonam On 10/19/2017 4:51 AM, Yasumasa Suenaga wrote: > Hi all, > > I want to backport JDK-8087291 change to 8u because this issue will > affect all Java 8 or later users. > Could you review it? > > ? http://cr.openjdk.java.net/~ysuenaga/JDK-8189599/webrev.00/ > > The patch for jdk10/hs has been pushed: > > ? http://hg.openjdk.java.net/jdk10/hs/rev/dbd1f4f276ba > > This change will be sponsored by Poonam. > > > Thanks, > > Yasumasa From thomas.stuefe at gmail.com Mon Oct 23 18:03:00 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 23 Oct 2017 20:03:00 +0200 Subject: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? In-Reply-To: <1508775418.3888.12.camel@redhat.com> References: <1508775418.3888.12.camel@redhat.com> Message-ID: On Mon, Oct 23, 2017 at 6:16 PM, Severin Gehwolf wrote: > On Tue, 2017-10-10 at 11:36 +0200, Tomas Kalibera wrote: > > Dear hotspot developers, > > > > I am one of the core R developers and what has been causing us a lot of > > trouble is the 2M cap for stack size of the initial thread on Linux. In > > short, R has its own stack overflow checking, the stack size is taken > > from rlimit, and R needs a lot of stack (often more than 2M). Via rJava > > package, R uses Java, so it initializes Java via JNI. As a workaround > > for some old Linux problem, hotspot caps the stack at 2M by inserting > > guard pages. Consequently, R crashes when the recursion gets too deep, > > and the crash is ugly as R's stack overflow checking does not now about > > the shrinkage of the stack. Users have no chance of diagnosing what's > > wrong and it is causing us trouble increasingly more lately, as we have > > more recursive calls (JIT implemented in R) and as newer gcc versions > > produce bigger frames from our interpreter loop. The problem is still > > present in Java 9. > > > > We're so hopeless that I implemented a workaround for rJava where we > > fill up the stack just enough before initializing the JVM so that > > is_initial_thread returns false even for the initial thread, so the > > guard page is not inserted. This is probably something we will use for > > now - or is there a simpler workaround? (the code is here, but probably > > you get the idea just from the description of it: > > https://github.com/s-u/rJava/pull/102/files) > > > > As you surely know well, the 2M cap comes from a bug report: > > http://bugs.java.com/view_bug.do?bug_id=4466587 > > And there is a cleanup of this code discussed in > > http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size- > option-Xss-is-ignored-td292960.html > > > > If I understand correctly the discussion, this is the new version for > > Java 10 - is that correct? > > http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/ > os/linux/vm/os_linux.cpp.udiff.html > > > > Now, if I am reading this patch correctly, I think that the 2M cap is > > gone. There is a new 8M cap, which however only applies when the rlimit > > stack size is unlimited, which is something we could probably live with. > > Is this a correct interpretation of the code? If so, we should be fine > > for Java 10. > > > > Even though, having an option to initialize the JVM without stack > > overflow checking for the initial thread might be even better... for > > cases like ours, when we have our own stack overflow checking. > > This maybe what you are looking for: > https://bugs.openjdk.java.net/browse/JDK-8189170 > > Cheers, > Severin > We also had a lengthy discussion about this, see: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/024859.html Best Regards, Thomas From zgu at redhat.com Mon Oct 23 18:19:49 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 23 Oct 2017 14:19:49 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> Message-ID: Hi Thomas, I added chunkmanager statistics to the output. However, I did not revive per-classload summary line. Had Second thought, that NMT probably should just report facts and leave interpretation to users. Updated webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html Thanks, -Zhengyu On 10/23/2017 11:08 AM, Thomas St?fe wrote: > > > On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu > wrote: > > > > Okay. So this is important for understanding cases where we have > a large number of miniscule class loaders, each one only having > a few numbers of chunks. In this case, would it be useful to > have a running total of "free" and "waste", too? Or even better, > also average and peak values of "free" and "waste" (to tell > apart cases where you have one outlier vs cases where just all > metaspaces waste a lot of memory). > Just out of curiosity, I looked at > http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt > and > it seems that "free" is the problem, not "waste"? So I guess > what happens is that all the little classloaders allocate just > enough memory to push them over "_small_chunk_limit=4", so they > allocate the first medium chunk, from which then they take a > small bite and leave the rest laying around? > > Yes. The free space of anonymous classes should be counted as waste, > in my opinion. I am not sure if everyone agrees, so I took the > summary line out of this patch. > > I would be more than happy to restore the summary line, if you find > it is useful :-) > > > I agree with you. Typically class loaders stop loading at some point, > especially these tine ones, and often will not resume loading. So, > effectivly, the space is wasted. It still helps to distinguish "free" in > the current chunks from "free" in the other chunks to tell this > situation apart from a situation where we have just a bug in metaspace > chunk handling preventing us to use up our chunks. > > > The latter would be an important addition, regardless > if this is > for customers or for us. Chunks idling in freelists can > make a > huge impact, and unfortunately this may happen in perfectly > valid cases. See e.g. our JEP > "https://bugs.openjdk.java.net/browse/JDK-8166690 > > >". We have > already a printing routine for free chunks, in > ChunkManager::print_all_chunkmanagers(). Could you add > this to > your output? > > > Make sense! Could you recommend what data and format you > would like > to see? > > > > Would not ChunkManager::print_all_chunkmanagers() be sufficient? > > Okay, I might need to tweak output format. > > > Fine by me. Nothing depends on it yet. > > Thanks, Thomas > > Thanks, > > -Zhengyu > > > > ------------- > > Other than above notes, the changes seem > straighforward, I did > not see anything wrong. Minor nits: > > - PrintCLDMetaspaceInfoClosure::_out, _scale and _unit > could all > be constant (with _out being a outputStream* const). > - We also do not need to provide scale *and* unit as > argument, > one can be deduced from the other. This would also prevent > mismatches.We do need scale here, since nmt command > line can set > scale and we do > > have to honor that. > > However, passing unit is ugly! I don't want to introduce > inverse > dependence (metaspace -> nmt), which is also ugly. > > > Yes, that would be regrettable. > > Probably should move scale mapping code from NMTUtil to a > common util? > > > > That would be possible, these functions (scale_from_name etc) > are simple enough to be moved to a generic file. However, if you > pass scale, deducing the string that goes with it is trivial and > I would have no qualms doing this by hand in metaspace.cpp. But > it is a matter of taste. > > > I did not look closely at locking issues, I assume > MetaspaceAux::print_metadata() is supposed to run only in > Safepoints? > > > No. sum_xxx_xxx_in_use queries are guarded by locks. > > Thanks, > > -Zhengyu > > > Thanks, Thomas > > > > Thanks & Kind Regards, Thomas > > > > > > > On Fri, Oct 20, 2017 at 4:00 PM, Zhengyu Gu > > > > > > >>> wrote: > > Up to now, there is little visibility into how > metaspaces > are used, > and by whom. > > This proposed enhancement gives NMT the ability to > report > metaspace > usages on per-classloader level, via a new NMT command > "metadata" > > > For example: > > 2496: > Metaspaces: > Metadata space: reserved= 8192KB > committed= 5888KB > Class space: reserved= 1048576KB > committed= 640KB > > Per-classloader metadata: > > ClassLoader: for anonymous class > Metadata capacity= 5.00KB used= 1.16KB > free= 3.84KB waste= 0.05KB > Class data capacity= 1.00KB used= 0.59KB > free= 0.41KB waste= 0.00KB > > .... > > ClassLoader: > Metadata capacity= 5640.00KB used= 5607.42KB > free= 32.58KB waste= 0.46KB > Class data capacity= 520.00KB used= 500.94KB > free= 19.06KB waste= 0.02KB > > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8189688 > > > > > >> > Webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html > > > > > > > > >> > > Test: > > hotspot_tier1_runtime, fastdebug and release on > x64 Linux. > > Thanks, > > -Zhengyu > > > > > From thomas.stuefe at gmail.com Mon Oct 23 18:31:57 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 23 Oct 2017 20:31:57 +0200 Subject: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? In-Reply-To: References: Message-ID: Rereading I found my first answer a bit shortish and I wish to elaborate a bit on my answer, even though David is probably in a better position to do so. We had a discussion about this on hotspot.runtime ( http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/thread.html#24859), AFAIK David plans to implement the switch to disable guard stack creation on the primordial thread. Both jdk9 and jdk10 already contain David's fix for 8170307, which changes the clamp-down size for the primordial thread on Linux to 8M. Latest versions are here: http://hg.openjdk.java.net/jdk9/hs/hotspot/file/c68024d52834/src/os/linux/vm/os_linux.cpp http://hg.openjdk.java.net/jdk10/hs/file/1aecd400f2fa/src/hotspot/os/linux/os_linux.cpp Kind Regards, Thomas On Tue, Oct 10, 2017 at 11:36 AM, Tomas Kalibera wrote: > Dear hotspot developers, > > I am one of the core R developers and what has been causing us a lot of > trouble is the 2M cap for stack size of the initial thread on Linux. In > short, R has its own stack overflow checking, the stack size is taken from > rlimit, and R needs a lot of stack (often more than 2M). Via rJava package, > R uses Java, so it initializes Java via JNI. As a workaround for some old > Linux problem, hotspot caps the stack at 2M by inserting guard pages. > Consequently, R crashes when the recursion gets too deep, and the crash is > ugly as R's stack overflow checking does not now about the shrinkage of the > stack. Users have no chance of diagnosing what's wrong and it is causing us > trouble increasingly more lately, as we have more recursive calls (JIT > implemented in R) and as newer gcc versions produce bigger frames from our > interpreter loop. The problem is still present in Java 9. > > We're so hopeless that I implemented a workaround for rJava where we fill > up the stack just enough before initializing the JVM so that > is_initial_thread returns false even for the initial thread, so the guard > page is not inserted. This is probably something we will use for now - or > is there a simpler workaround? (the code is here, but probably you get the > idea just from the description of it: https://github.com/s-u/rJava/p > ull/102/files) > > As you surely know well, the 2M cap comes from a bug report: > http://bugs.java.com/view_bug.do?bug_id=4466587 > And there is a cleanup of this code discussed in > http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-opt > ion-Xss-is-ignored-td292960.html > > If I understand correctly the discussion, this is the new version for Java > 10 - is that correct? > http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os > /linux/vm/os_linux.cpp.udiff.html > > Now, if I am reading this patch correctly, I think that the 2M cap is > gone. There is a new 8M cap, which however only applies when the rlimit > stack size is unlimited, which is something we could probably live with. Is > this a correct interpretation of the code? If so, we should be fine for > Java 10. > > Even though, having an option to initialize the JVM without stack overflow > checking for the initial thread might be even better... for cases like > ours, when we have our own stack overflow checking. > > Thanks, > Tomas > > From tomas.kalibera at gmail.com Mon Oct 23 19:32:35 2017 From: tomas.kalibera at gmail.com (Tomas Kalibera) Date: Mon, 23 Oct 2017 21:32:35 +0200 Subject: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? In-Reply-To: References: Message-ID: <542ce2d2-80a4-dee4-c0ee-b7615132ab36@gmail.com> Hi Thomas, thanks for the summary, I've read the thread about the CSR and I am happy with the planned new option to disable the guard pages on the primordial thread. For older versions of Java that won't have that option, we can use the workaround in R/rJava we have now. In Java 9 we do get the guard pages inside the primordial thread stack as well, but it is because of -Xss and the default thread stack size, not because the 2M cap which is no longer in Java 9. The workaround still works, though, as long as it fills up the stack above -Xss (which is by default <2M). Best Tomas On 10/23/2017 08:31 PM, Thomas St?fe wrote: > Rereading I found my first answer a bit shortish and I wish to > elaborate a bit on my answer, even though David is probably in a > better position to do so. > > We had a discussion about this on hotspot.runtime > (http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/thread.html#24859), > AFAIK David plans to implement the switch to disable guard stack > creation on the primordial thread. > > Both jdk9 and jdk10 already contain David's fix for 8170307, which > changes the clamp-down size for the primordial thread on Linux to 8M. > > Latest versions are here: > http://hg.openjdk.java.net/jdk9/hs/hotspot/file/c68024d52834/src/os/linux/vm/os_linux.cpp > http://hg.openjdk.java.net/jdk10/hs/file/1aecd400f2fa/src/hotspot/os/linux/os_linux.cpp > > > Kind Regards, Thomas > > On Tue, Oct 10, 2017 at 11:36 AM, Tomas Kalibera > > wrote: > > Dear hotspot developers, > > I am one of the core R developers and what has been causing us a > lot of trouble is the 2M cap for stack size of the initial thread > on Linux. In short, R has its own stack overflow checking, the > stack size is taken from rlimit, and R needs a lot of stack (often > more than 2M). Via rJava package, R uses Java, so it initializes > Java via JNI. As a workaround for some old Linux problem, hotspot > caps the stack at 2M by inserting guard pages. Consequently, R > crashes when the recursion gets too deep, and the crash is ugly as > R's stack overflow checking does not now about the shrinkage of > the stack. Users have no chance of diagnosing what's wrong and it > is causing us trouble increasingly more lately, as we have more > recursive calls (JIT implemented in R) and as newer gcc versions > produce bigger frames from our interpreter loop. The problem is > still present in Java 9. > > We're so hopeless that I implemented a workaround for rJava where > we fill up the stack just enough before initializing the JVM so > that is_initial_thread returns false even for the initial thread, > so the guard page is not inserted. This is probably something we > will use for now - or is there a simpler workaround? (the code is > here, but probably you get the idea just from the description of > it: https://github.com/s-u/rJava/pull/102/files > ) > > As you surely know well, the 2M cap comes from a bug report: > http://bugs.java.com/view_bug.do?bug_id=4466587 > > And there is a cleanup of this code discussed in > http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html > > > If I understand correctly the discussion, this is the new version > for Java 10 - is that correct? > http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html > > > Now, if I am reading this patch correctly, I think that the 2M cap > is gone. There is a new 8M cap, which however only applies when > the rlimit stack size is unlimited, which is something we could > probably live with. Is this a correct interpretation of the code? > If so, we should be fine for Java 10. > > Even though, having an option to initialize the JVM without stack > overflow checking for the initial thread might be even better... > for cases like ours, when we have our own stack overflow checking. > > Thanks, > Tomas > > From david.holmes at oracle.com Mon Oct 23 21:50:31 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 24 Oct 2017 07:50:31 +1000 Subject: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? In-Reply-To: References: Message-ID: <0cda4508-6d07-ded1-476b-6674c8cbcf8a@oracle.com> Please Note that Tomas's email was sent October 10 and only just appeared on the mailing list (presumably he was not subscribed at the time). Since then this has all been discussed and an additional workaround in the pipeline through: https://bugs.openjdk.java.net/browse/JDK-8189170 Thanks, David On 24/10/2017 4:31 AM, Thomas St?fe wrote: > Rereading I found my first answer a bit shortish and I wish to elaborate > a bit on my answer, even though David is probably in a better position > to do so. > > We had a discussion about this on hotspot.runtime > (http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2017-October/thread.html#24859), > AFAIK David plans to implement the switch to disable guard stack > creation on the primordial thread. > > Both jdk9 and jdk10 already contain David's fix for 8170307, which > changes the clamp-down size for the primordial thread on Linux to 8M. > > Latest versions are here: > http://hg.openjdk.java.net/jdk9/hs/hotspot/file/c68024d52834/src/os/linux/vm/os_linux.cpp > http://hg.openjdk.java.net/jdk10/hs/file/1aecd400f2fa/src/hotspot/os/linux/os_linux.cpp > > > Kind Regards, Thomas > > On Tue, Oct 10, 2017 at 11:36 AM, Tomas Kalibera > > wrote: > > Dear hotspot developers, > > I am one of the core R developers and what has been causing us a lot > of trouble is the 2M cap for stack size of the initial thread on > Linux. In short, R has its own stack overflow checking, the stack > size is taken from rlimit, and R needs a lot of stack (often more > than 2M). Via rJava package, R uses Java, so it initializes Java via > JNI. As a workaround for some old Linux problem, hotspot caps the > stack at 2M by inserting guard pages. Consequently, R crashes when > the recursion gets too deep, and the crash is ugly as R's stack > overflow checking does not now about the shrinkage of the stack. > Users have no chance of diagnosing what's wrong and it is causing us > trouble increasingly more lately, as we have more recursive calls > (JIT implemented in R) and as newer gcc versions produce bigger > frames from our interpreter loop. The problem is still present in > Java 9. > > We're so hopeless that I implemented a workaround for rJava where we > fill up the stack just enough before initializing the JVM so that > is_initial_thread returns false even for the initial thread, so the > guard page is not inserted. This is probably something we will use > for now - or is there a simpler workaround? (the code is here, but > probably you get the idea just from the description of it: > https://github.com/s-u/rJava/pull/102/files > ) > > As you surely know well, the 2M cap comes from a bug report: > http://bugs.java.com/view_bug.do?bug_id=4466587 > > And there is a cleanup of this code discussed in > http://openjdk.5641.n7.nabble.com/RFR-8170307-Stack-size-option-Xss-is-ignored-td292960.html > > > If I understand correctly the discussion, this is the new version > for Java 10 - is that correct? > http://cr.openjdk.java.net/~dholmes/8170307/webrev.v2/src/os/linux/vm/os_linux.cpp.udiff.html > > > Now, if I am reading this patch correctly, I think that the 2M cap > is gone. There is a new 8M cap, which however only applies when the > rlimit stack size is unlimited, which is something we could probably > live with. Is this a correct interpretation of the code? If so, we > should be fine for Java 10. > > Even though, having an option to initialize the JVM without stack > overflow checking for the initial thread might be even better... for > cases like ours, when we have our own stack overflow checking. > > Thanks, > Tomas > > From coleen.phillimore at oracle.com Mon Oct 23 22:29:25 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 23 Oct 2017 18:29:25 -0400 Subject: RFR (trivial) 8189794: Assert in InstanceKlass::cast called from Exceptions::new_exceptions Message-ID: <13c677b0-14e4-534f-d1eb-a4f57653d505@oracle.com> Summary: Fix call to InstanceKlass::cast to only be after verifying class is non-null. Also fixed an unnecessary InstanceKlass::cast found while looking for similar instances of this bug.? I didn't find this same bug anywhere else.? Tested with internal vm.gc.locker tests, and with JPRT which is equivalent to tier1 only faster at the moment. open webrev at http://cr.openjdk.java.net/~coleenp/8189794.01/webrev bug link https://bugs.openjdk.java.net/browse/JDK-8189794 Thanks, Coleen From serguei.spitsyn at oracle.com Mon Oct 23 22:58:56 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 23 Oct 2017 15:58:56 -0700 Subject: (XXS) RFR: 8189776: Remove dead code in jvm.cpp: force_verify_field_access In-Reply-To: <5f03e75b-19ea-27d8-fe50-d03371052f05@oracle.com> References: <5f03e75b-19ea-27d8-fe50-d03371052f05@oracle.com> Message-ID: <6ba467d1-78f8-ab00-b173-d34b0f898997@oracle.com> Hi David, It looks good. Thanks, Serguei On 10/22/17 21:18, David Holmes wrote: > Trivial dead code cleanup. > > webrev: http://cr.openjdk.java.net/~dholmes/8189776/webrev/ > > Bug: https://bugs.openjdk.java.net/browse/JDK-8189776 > > JDK-8057777 deleted the old unused JVM_AllocateNewObject interface, > which was the sole caller of the local force_verify_field_access > method. However force_verify_field_access was not removed. > > Cleaning this up simplifies the nestmate work I'm doing (JEP 181). > > Thanks, > David From david.holmes at oracle.com Mon Oct 23 23:11:09 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 24 Oct 2017 09:11:09 +1000 Subject: RFR (trivial) 8189794: Assert in InstanceKlass::cast called from Exceptions::new_exceptions In-Reply-To: <13c677b0-14e4-534f-d1eb-a4f57653d505@oracle.com> References: <13c677b0-14e4-534f-d1eb-a4f57653d505@oracle.com> Message-ID: <2c30d93c-7a0d-c63f-a7d3-a2cd46801908@oracle.com> Looks good! Thanks, David On 24/10/2017 8:29 AM, coleen.phillimore at oracle.com wrote: > Summary: Fix call to InstanceKlass::cast to only be after verifying > class is non-null. > > Also fixed an unnecessary InstanceKlass::cast found while looking for > similar instances of this bug.? I didn't find this same bug anywhere > else.? Tested with internal vm.gc.locker tests, and with JPRT which is > equivalent to tier1 only faster at the moment. > > open webrev at http://cr.openjdk.java.net/~coleenp/8189794.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8189794 > > Thanks, > Coleen From serguei.spitsyn at oracle.com Tue Oct 24 01:57:02 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Mon, 23 Oct 2017 18:57:02 -0700 Subject: RFR (trivial) 8189794: Assert in InstanceKlass::cast called from Exceptions::new_exceptions In-Reply-To: <13c677b0-14e4-534f-d1eb-a4f57653d505@oracle.com> References: <13c677b0-14e4-534f-d1eb-a4f57653d505@oracle.com> Message-ID: <38f5dd33-e8e7-64f2-ed96-ff778d27dca9@oracle.com> Hi Coleen, Looks good. Thanks, Serguei On 10/23/17 15:29, coleen.phillimore at oracle.com wrote: > Summary: Fix call to InstanceKlass::cast to only be after verifying > class is non-null. > > Also fixed an unnecessary InstanceKlass::cast found while looking for > similar instances of this bug.? I didn't find this same bug anywhere > else.? Tested with internal vm.gc.locker tests, and with JPRT which is > equivalent to tier1 only faster at the moment. > > open webrev at http://cr.openjdk.java.net/~coleenp/8189794.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8189794 > > Thanks, > Coleen From jini.george at oracle.com Tue Oct 24 06:35:10 2017 From: jini.george at oracle.com (Jini George) Date: Tue, 24 Oct 2017 12:05:10 +0530 Subject: RFR: SA: JDK-8189798: SA cleanup - part 1 Message-ID: <18501902-23db-de6c-b83d-640cd33df836@oracle.com> Hello, As a part of SA next, I am working on writing a test case which compares the fields and the types of the fields of the SA java classes with the corresponding entries in the vmStructs tables. This, to some extent, would help in preventing errors in SA due to the changes in hotspot. As a precursor to this, I am in the process of making some cleanup related changes (mostly in SA). I plan to have the changes done in parts. For this webrev, most of the changes are for: 1. Avoiding having some values being redefined in SA. Instead have those exported through vmStructs, and read it in SA. (CompactibleFreeListSpace::_min_chunk_size_in_bytes, CompactibleFreeListSpace::IndexSetSize) Redefinition of hotspot values in SA makes SA error prone, when the value gets altered in hotspot and the corresponding modification gets missed out in SA. 2. To remove some unused code (JNIid.java). 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. 4. Modify variable names in SA and hotspot to match the counterpart names, so that the comparison of the fields become easier. Most of the changes belong to this group. Could I please get reviews done for these precursor changes ? JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ Thank you, Jini. From jini.george at oracle.com Tue Oct 24 07:31:37 2017 From: jini.george at oracle.com (Jini George) Date: Tue, 24 Oct 2017 13:01:37 +0530 Subject: RFR: SA: JDK-8189798: SA cleanup - part 1 In-Reply-To: <18501902-23db-de6c-b83d-640cd33df836@oracle.com> References: <18501902-23db-de6c-b83d-640cd33df836@oracle.com> Message-ID: Adding hotspot-dev too. Thanks, Jini. On 10/24/2017 12:05 PM, Jini George wrote: > Hello, > > As a part of SA next, I am working on writing a test case which compares > the fields and the types of the fields of the SA java classes with the > corresponding entries in the vmStructs tables. This, to some extent, > would help in preventing errors in SA due to the changes in hotspot. As > a precursor to this, I am in the process of making some cleanup related > changes (mostly in SA). I plan to have the changes done in parts. For > this webrev, most of the changes are for: > > 1. Avoiding having some values being redefined in SA. Instead have those > exported through vmStructs, and read it in SA. > (CompactibleFreeListSpace::_min_chunk_size_in_bytes, > CompactibleFreeListSpace::IndexSetSize) > > Redefinition of hotspot values in SA makes SA error prone, when the > value gets altered in hotspot and the corresponding modification gets > missed out in SA. > > 2. To remove some unused code (JNIid.java). > 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. > 4. Modify variable names in SA and hotspot to match the counterpart > names, so that the comparison of the fields become easier. Most of the > changes belong to this group. > > Could I please get reviews done for these precursor changes ? > > JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 > webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ > > Thank you, > Jini. > From thomas.stuefe at gmail.com Tue Oct 24 07:41:10 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 24 Oct 2017 09:41:10 +0200 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> Message-ID: Hi Zhengyu, On Mon, Oct 23, 2017 at 8:19 PM, Zhengyu Gu wrote: > Hi Thomas, > > I added chunkmanager statistics to the output. > > However, I did not revive per-classload summary line. Had Second thought, > that NMT probably should just report facts and leave interpretation to > users. > > Not sure whether we misunderstood each other. I was thinking of something in the line of: <<<< .... ClassLoader: jdk/internal/reflect/DelegatingClassLoader Metadata: capacity: 5.00KB used: 3.32KB free: 1.68KB waste: 0.00KB Class data: capacity: 1.00KB used: 0.54KB free: 0.46KB waste: 0.00KB ClassLoader: for anonymous class Metadata: capacity: 1.00KB used: 1.00KB free: 0.00KB waste: 0.00KB Class data: capacity: 1.00KB used: 0.64KB free: 0.36KB waste: 0.00KB .... Summary: XX class loaders encountered, total capacity: xx, total waste: xx. Peak usage by class loader: xxxx >>>> These numbers would not be interpretation, just a convenience to the reader to get a clear picture. It even may be useful to separate the output between basic and detailed mode, and in basic mode omit all the single class loaders and just print the summary line. > Updated webrev: http://cr.openjdk.java.net/~zg > u/8189688/webrev.01/index.html > > metaspace.hpp: I would have preferred to keep scale_unit file local static instead of exposing it via MetaspaceAux, because it does not really have anything to do with metaspace. Otherwise ok. --- metaspace.cpp - ChunkManager::print_statistics(): thanks for reusing this function! -> Scale can only be ever 1, K, M, G, yes? So, could you add an assert at the start of the function, and a comment at the prototype or function head? -> Also, now that ChunkManager::print_statistics() takes a scale, could you please change the printout to use floats like you did in PrintCLDMetaspaceInfoClosure::print_metaspace()? - I am concerned about races in PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). Maybe my understanding is limited. We bail if cld->is_unloading. But nothing prevents a ClassLoaderDataGraph from starting to unload a ClassLoaderData and tearing down the Metaspace after we started printing it in another thread, right? Also, I do not see any fences. So, I wonder whether PrintCLDMetaspaceInfoClosure should run in lock protection. And then, I wonder if it would be good to separate data gathering and printing. We print to a (at least in principle) unknown output stream, which may be doing slow File IO or even Network IO. Doing this under lock protection seems not a good idea (but I see other places where this happens too). For an example, see ChunkManager::get_statistic() vs ChunkManager::print_statistic(). Could you do the same, have a data gathering step where you collect your quadrupel in a variable sized list of structures in memory, and print it out in a separate step? I realize though that the effort would be larger than for what we did in ChunkManager::get_statistic(), where we only fill a structure. ------ vm_operations.hpp - VM_PrintMetadata : do we still need _unit? > Thanks, > > -Zhengyu > Thanks! Thomas > > > On 10/23/2017 11:08 AM, Thomas St?fe wrote: > >> >> >> On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu > zgu at redhat.com>> wrote: >> >> >> >> Okay. So this is important for understanding cases where we have >> a large number of miniscule class loaders, each one only having >> a few numbers of chunks. In this case, would it be useful to >> have a running total of "free" and "waste", too? Or even better, >> also average and peak values of "free" and "waste" (to tell >> apart cases where you have one outlier vs cases where just all >> metaspaces waste a lot of memory). >> Just out of curiosity, I looked at >> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >> and >> it seems that "free" is the problem, not "waste"? So I guess >> what happens is that all the little classloaders allocate just >> enough memory to push them over "_small_chunk_limit=4", so they >> allocate the first medium chunk, from which then they take a >> small bite and leave the rest laying around? >> >> Yes. The free space of anonymous classes should be counted as waste, >> in my opinion. I am not sure if everyone agrees, so I took the >> summary line out of this patch. >> >> I would be more than happy to restore the summary line, if you find >> it is useful :-) >> >> >> I agree with you. Typically class loaders stop loading at some point, >> especially these tine ones, and often will not resume loading. So, >> effectivly, the space is wasted. It still helps to distinguish "free" in >> the current chunks from "free" in the other chunks to tell this situation >> apart from a situation where we have just a bug in metaspace chunk handling >> preventing us to use up our chunks. >> >> >> The latter would be an important addition, regardless >> if this is >> for customers or for us. Chunks idling in freelists can >> make a >> huge impact, and unfortunately this may happen in >> perfectly >> valid cases. See e.g. our JEP >> "https://bugs.openjdk.java.net/browse/JDK-8166690 >> >> > >". We have >> already a printing routine for free chunks, in >> ChunkManager::print_all_chunkmanagers(). Could you add >> this to >> your output? >> >> >> Make sense! Could you recommend what data and format you >> would like >> to see? >> >> >> >> Would not ChunkManager::print_all_chunkmanagers() be sufficient? >> >> Okay, I might need to tweak output format. >> >> >> Fine by me. Nothing depends on it yet. >> >> Thanks, Thomas >> >> Thanks, >> >> -Zhengyu >> >> >> >> ------------- >> >> Other than above notes, the changes seem >> straighforward, I did >> not see anything wrong. Minor nits: >> >> - PrintCLDMetaspaceInfoClosure::_out, _scale and _unit >> could all >> be constant (with _out being a outputStream* const). >> - We also do not need to provide scale *and* unit as >> argument, >> one can be deduced from the other. This would also >> prevent >> mismatches.We do need scale here, since nmt command >> line can set >> scale and we do >> >> have to honor that. >> >> However, passing unit is ugly! I don't want to introduce >> inverse >> dependence (metaspace -> nmt), which is also ugly. >> >> >> Yes, that would be regrettable. >> >> Probably should move scale mapping code from NMTUtil to a >> common util? >> >> >> >> That would be possible, these functions (scale_from_name etc) >> are simple enough to be moved to a generic file. However, if you >> pass scale, deducing the string that goes with it is trivial and >> I would have no qualms doing this by hand in metaspace.cpp. But >> it is a matter of taste. >> >> >> I did not look closely at locking issues, I assume >> MetaspaceAux::print_metadata() is supposed to run only >> in >> Safepoints? >> >> >> No. sum_xxx_xxx_in_use queries are guarded by locks. >> >> Thanks, >> >> -Zhengyu >> >> >> Thanks, Thomas >> >> >> >> Thanks & Kind Regards, Thomas >> >> >> >> >> >> >> On Fri, Oct 20, 2017 at 4:00 PM, Zhengyu Gu >> >> > >> >> >> >>> wrote: >> >> Up to now, there is little visibility into how >> metaspaces >> are used, >> and by whom. >> >> This proposed enhancement gives NMT the ability to >> report >> metaspace >> usages on per-classloader level, via a new NMT >> command >> "metadata" >> >> >> For example: >> >> 2496: >> Metaspaces: >> Metadata space: reserved= 8192KB >> committed= 5888KB >> Class space: reserved= 1048576KB >> committed= 640KB >> >> Per-classloader metadata: >> >> ClassLoader: for anonymous class >> Metadata capacity= 5.00KB used= >> 1.16KB >> free= 3.84KB waste= 0.05KB >> Class data capacity= 1.00KB used= >> 0.59KB >> free= 0.41KB waste= 0.00KB >> >> .... >> >> ClassLoader: >> Metadata capacity= 5640.00KB used= >> 5607.42KB >> free= 32.58KB waste= 0.46KB >> Class data capacity= 520.00KB used= >> 500.94KB >> free= 19.06KB waste= 0.02KB >> >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > >> > >> >> Webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> > >> > gu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> >> >> >> Test: >> >> hotspot_tier1_runtime, fastdebug and release on >> x64 Linux. >> >> Thanks, >> >> -Zhengyu >> >> >> >> >> >> From thomas.stuefe at gmail.com Tue Oct 24 09:11:59 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 24 Oct 2017 11:11:59 +0200 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> Message-ID: Hi Andrew, On Mon, Oct 23, 2017 at 5:29 PM, Andrew Dinn wrote: > On 23/10/17 16:03, Thomas St?fe wrote: > > I am confused about the sizes though . Are we talking about the chunk > > sizes in metaspace.cpp? Because they are way smaller, with medium chunks > > being 32/64K? Or are these large chunks a RedHat addition? > Oops, apologies -- wrong units! > > Yes, this is indeed chunk sizes but I meant to say 1KB and 4KB chunks > (MBs would be truly dreadful :-). That makes more sense :) > The relevant definitions in > metaspace.cpp are: > > enum ChunkSizes { // in words. > ClassSpecializedChunk = 128, > SpecializedChunk = 128, > ClassSmallChunk = 256, > SmallChunk = 512, > ClassMediumChunk = 4 * K, > MediumChunk = 8 * K > }; > > Anonymous classes (used for the anon classes used to back lambdas) > initially allocate a ClassSpecializedChunk and a SpecializedChunk i.e. 2 > x 128 * 8 bytes = 2 x 1KB. If you have something that goes beyond the > most basic lambda then you may well also allocate an extra SmallChunk = > 512 * 8 bytes = 4KB. We experimented (with both real and synthetic > examples) and found a high proportion of cases where 6KB chunks was > being allocated but around 50% was left unused as free tail ends of > these 2/3 chunks. > > regards, > > > Last year I analyzed some OOM-in-classspace situations. I saw this problem, but was disregarding it as something which would have diminishing effect with increased number of loaded classes. But now I see that this is indeed a real world problem. (The actual problem in my case turned out to be the class space being flooded with small chunks, which were free because its creating class loaders have been unloaded, but due to the unability of the metaspace coding to reuse chunks in a different size were not reused. We added a patch in our fork to split-and-merge free metaspace chunks on demand to deal with this problem.) My first impulse was to say "Well, lets make the _small_chunk_limit configurable via command line", but I think that is a short sighted stop gap. A better solution may be to make _small_chunk_limit a per-classloader-setting, so that class loaders which we know will only be used for some anonymous classes can be started with a high _small_chunk_limit to never let them allocate medium chunks. I wonder if another solution could be some sort of stealing mechanism: Periodically walk all class loaders, and, if they have half-eaten medium chunks laying around they did not touch for a while, split that chunk into n small chunks and return n-1 chunks to the free list. This only works though if the "used" portion of the medium chunk does not surpass the size of one small chunk, because the portion in use cannot be split into multiple chunks. Considering that the size ratio small:medium is 1:16, this may not be that useful. As you say, usually the medium chunks are used up about half way. Kind Regards, Thomas > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > From aph at redhat.com Tue Oct 24 09:23:51 2017 From: aph at redhat.com (Andrew Haley) Date: Tue, 24 Oct 2017 10:23:51 +0100 Subject: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? In-Reply-To: References: Message-ID: <9a6106cc-50ad-9431-9064-8b2b825951e7@redhat.com> On 23/10/17 18:25, Doerr, Martin wrote: > after reading several mails about guard page problems (and quite > some own bad experience), I wonder if they are really worth all the > pain. Is the performance benefit from stack banging so great, that > it justifies the tremendous complexity? > > If not, I'd vote for replacing them by explicit checks. Loading a > limit from the Thread and performing a comparison doesn't sound so > expensive. > > It could be the case that some platforms need banging (e.g. Windows > due to on demand committed stack), but probably not all. > Am I missing anything? I think you should try it and tell us the results. If we have power-of-2 stack sizes it can be done without any loads at all, but that's probably too restrictive, and we've had quite enough tricksiness. It's hard to beat the speed of a simple store, though. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From martin.doerr at sap.com Tue Oct 24 10:59:59 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Tue, 24 Oct 2017 10:59:59 +0000 Subject: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? In-Reply-To: <9a6106cc-50ad-9431-9064-8b2b825951e7@redhat.com> References: <9a6106cc-50ad-9431-9064-8b2b825951e7@redhat.com> Message-ID: <3ef4f6f626854d75a69bd908eaf01904@sap.com> Hi Andrew, thanks for your reply. I guess that the performance impact will be negligible if we have enough inlining (which reduces the number of stack checks). Reducing complexity and saving stack space sounds more valuable to me. I may consider making experiments when I find some time for it. Best regards, Martin -----Original Message----- From: Andrew Haley [mailto:aph at redhat.com] Sent: Dienstag, 24. Oktober 2017 11:24 To: Doerr, Martin ; Tomas Kalibera ; hotspot-runtime-dev at openjdk.java.net Subject: Re: JVM inserting guard pages into stack of initial thread - will that stop with Java 10? On 23/10/17 18:25, Doerr, Martin wrote: > after reading several mails about guard page problems (and quite > some own bad experience), I wonder if they are really worth all the > pain. Is the performance benefit from stack banging so great, that > it justifies the tremendous complexity? > > If not, I'd vote for replacing them by explicit checks. Loading a > limit from the Thread and performing a comparison doesn't sound so > expensive. > > It could be the case that some platforms need banging (e.g. Windows > due to on demand committed stack), but probably not all. > Am I missing anything? I think you should try it and tell us the results. If we have power-of-2 stack sizes it can be done without any loads at all, but that's probably too restrictive, and we've had quite enough tricksiness. It's hard to beat the speed of a simple store, though. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From coleen.phillimore at oracle.com Tue Oct 24 12:24:12 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 24 Oct 2017 08:24:12 -0400 Subject: RFR (trivial) 8189794: Assert in InstanceKlass::cast called from Exceptions::new_exceptions In-Reply-To: <38f5dd33-e8e7-64f2-ed96-ff778d27dca9@oracle.com> References: <13c677b0-14e4-534f-d1eb-a4f57653d505@oracle.com> <38f5dd33-e8e7-64f2-ed96-ff778d27dca9@oracle.com> Message-ID: Thank you Serguei and David. Coleen On 10/23/17 9:57 PM, serguei.spitsyn at oracle.com wrote: > Hi Coleen, > > Looks good. > > Thanks, > Serguei > > > On 10/23/17 15:29, coleen.phillimore at oracle.com wrote: >> Summary: Fix call to InstanceKlass::cast to only be after verifying >> class is non-null. >> >> Also fixed an unnecessary InstanceKlass::cast found while looking for >> similar instances of this bug.? I didn't find this same bug anywhere >> else.? Tested with internal vm.gc.locker tests, and with JPRT which >> is equivalent to tier1 only faster at the moment. >> >> open webrev at http://cr.openjdk.java.net/~coleenp/8189794.01/webrev >> bug link https://bugs.openjdk.java.net/browse/JDK-8189794 >> >> Thanks, >> Coleen > From zgu at redhat.com Tue Oct 24 15:08:13 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 24 Oct 2017 11:08:13 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> Message-ID: Hi Thomas, > > Not sure whether we misunderstood each other. I was thinking of > something in the line of: > > <<<< > .... > ClassLoader: jdk/internal/reflect/DelegatingClassLoader > Metadata: > capacity: 5.00KB used: 3.32KB free: 1.68KB waste: 0.00KB > Class data: > capacity: 1.00KB used: 0.54KB free: 0.46KB waste: 0.00KB > > ClassLoader: for anonymous class > Metadata: > capacity: 1.00KB used: 1.00KB free: 0.00KB waste: 0.00KB > Class data: > capacity: 1.00KB used: 0.64KB free: 0.36KB waste: 0.00KB > .... > > Summary: > XX class loaders encountered, total capacity: xx, total waste: xx. > > Peak usage by class loader: xxxx > >>>> Added summary lines: Total class loaders= 56 capacity= 6378.00KB used= 6205.08KB free= 172.92KB waste= 1.44KB For anonymous classes= 54 capacity= 212.00KB used= 96.95KB free= 115.05KB waste= 0.94KB > > These numbers would not be interpretation, just a convenience to the > reader to get a clear picture. > > It even may be useful to separate the output between basic and detailed > mode, and in basic mode omit all the single class loaders and just print > the summary line. > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html > > > > > metaspace.hpp: > > I would have preferred to keep scale_unit file local static instead of > exposing it via MetaspaceAux, because it does not really have anything > to do with metaspace. Fixed > > Otherwise ok. > > --- > > metaspace.cpp > > - ChunkManager::print_statistics(): thanks for reusing this function! > -> Scale can only be ever 1, K, M, G, yes? So, could you add an > assert at the start of the function, and a comment at the prototype or > function head? > -> Also, now that ChunkManager::print_statistics() takes a scale, > could you please change the printout to use floats like you did in > PrintCLDMetaspaceInfoClosure::print_metaspace()? Done. Chunkmanager (non-class): 0 specialized (128 bytes) chunks, total 0.00KB 0 small (512 bytes) chunks, total 0.00KB 0 medium (8192 bytes) chunks, total 0.00KB 0 humongous chunks, total 0.00KB total size: 0.00KB. Chunkmanager (class): 0 specialized (128 bytes) chunks, total 0.00KB 0 small (256 bytes) chunks, total 0.00KB 0 medium (4096 bytes) chunks, total 0.00KB 0 humongous chunks, total 0.00KB total size: 0.00KB. > > - I am concerned about races in > PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). Maybe my > understanding is limited. We bail if cld->is_unloading. But nothing > prevents a ClassLoaderDataGraph from starting to unload a > ClassLoaderData and tearing down the Metaspace after we started printing > it in another thread, right? Also, I do not see any fences. > > So, I wonder whether PrintCLDMetaspaceInfoClosure should run in lock > protection. And then, I wonder if it would be good to separate data > gathering and printing. We print to a (at least in principle) unknown > output stream, which may be doing slow File IO or even Network IO. Doing > this under lock protection seems not a good idea (but I see other places > where this happens too). > > For an example, see ChunkManager::get_statistic() vs > ChunkManager::print_statistic(). Could you do the same, have a data > gathering step where you collect your > quadrupel in a variable sized list of structures in memory, and print it > out in a separate step? I realize though that the effort would be larger > than for what we did in ChunkManager::get_statistic(), where we only > fill a structure. > Unlike other NMT queries, this query is executed at a safepoint by VMThread, so it should be okay. Updated webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ Thanks, -Zhengyu > ------ > > vm_operations.hpp > > - VM_PrintMetadata : do we still need _unit? > > > Thanks, > > -Zhengyu > > > > Thanks! > > Thomas > > > > On 10/23/2017 11:08 AM, Thomas St?fe wrote: > > > > On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu >> wrote: > > > > Okay. So this is important for understanding cases > where we have > a large number of miniscule class loaders, each one > only having > a few numbers of chunks. In this case, would it be > useful to > have a running total of "free" and "waste", too? Or > even better, > also average and peak values of "free" and "waste" (to tell > apart cases where you have one outlier vs cases where > just all > metaspaces waste a lot of memory). > Just out of curiosity, I looked at > http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt > > > > and > it seems that "free" is the problem, not "waste"? So I > guess > what happens is that all the little classloaders > allocate just > enough memory to push them over "_small_chunk_limit=4", > so they > allocate the first medium chunk, from which then they > take a > small bite and leave the rest laying around? > > Yes. The free space of anonymous classes should be counted > as waste, > in my opinion. I am not sure if everyone agrees, so I took the > summary line out of this patch. > > I would be more than happy to restore the summary line, if > you find > it is useful :-) > > > I agree with you. Typically class loaders stop loading at some > point, especially these tine ones, and often will not resume > loading. So, effectivly, the space is wasted. It still helps to > distinguish "free" in the current chunks from "free" in the > other chunks to tell this situation apart from a situation where > we have just a bug in metaspace chunk handling preventing us to > use up our chunks. > > > The latter would be an important addition, > regardless > if this is > for customers or for us. Chunks idling in > freelists can > make a > huge impact, and unfortunately this may happen > in perfectly > valid cases. See e.g. our JEP > > "https://bugs.openjdk.java.net/browse/JDK-8166690 > > > > > > >>". We have > already a printing routine for free chunks, in > ChunkManager::print_all_chunkmanagers(). Could > you add > this to > your output? > > > Make sense! Could you recommend what data and > format you > would like > to see? > > > > Would not ChunkManager::print_all_chunkmanagers() be > sufficient? > > Okay, I might need to tweak output format. > > > Fine by me. Nothing depends on it yet. > > Thanks, Thomas > > Thanks, > > -Zhengyu > > > > ------------- > > Other than above notes, the changes seem > straighforward, I did > not see anything wrong. Minor nits: > > - PrintCLDMetaspaceInfoClosure::_out, _scale > and _unit > could all > be constant (with _out being a outputStream* > const). > - We also do not need to provide scale *and* > unit as > argument, > one can be deduced from the other. This would > also prevent > mismatches.We do need scale here, since nmt > command > line can set > scale and we do > > have to honor that. > > However, passing unit is ugly! I don't want to > introduce > inverse > dependence (metaspace -> nmt), which is also ugly. > > > Yes, that would be regrettable. > > Probably should move scale mapping code from > NMTUtil to a > common util? > > > > That would be possible, these functions > (scale_from_name etc) > are simple enough to be moved to a generic file. > However, if you > pass scale, deducing the string that goes with it is > trivial and > I would have no qualms doing this by hand in > metaspace.cpp. But > it is a matter of taste. > > > I did not look closely at locking issues, I assume > MetaspaceAux::print_metadata() is supposed to > run only in > Safepoints? > > > No. sum_xxx_xxx_in_use queries are guarded by locks. > > Thanks, > > -Zhengyu > > > Thanks, Thomas > > > > Thanks & Kind Regards, Thomas > > > > > > > On Fri, Oct 20, 2017 at 4:00 PM, Zhengyu Gu > > > > > >> > > > > > > >>>> wrote: > > Up to now, there is little visibility > into how > metaspaces > are used, > and by whom. > > This proposed enhancement gives NMT the > ability to > report > metaspace > usages on per-classloader level, via a > new NMT command > "metadata" > > > For example: > > 2496: > Metaspaces: > Metadata space: reserved= 8192KB > committed= 5888KB > Class space: reserved= 1048576KB > committed= 640KB > > Per-classloader metadata: > > ClassLoader: for anonymous class > Metadata capacity= 5.00KB > used= 1.16KB > free= 3.84KB waste= 0.05KB > Class data capacity= 1.00KB > used= 0.59KB > free= 0.41KB waste= 0.00KB > > .... > > ClassLoader: > Metadata capacity= 5640.00KB > used= 5607.42KB > free= 32.58KB waste= 0.46KB > Class data capacity= 520.00KB > used= 500.94KB > free= 19.06KB waste= 0.02KB > > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8189688 > > > > > > >> > > > > > > > >>> > Webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html > > > > > > > > >> > > > > > > > > > >>> > > Test: > > hotspot_tier1_runtime, fastdebug and > release on > x64 Linux. > > Thanks, > > -Zhengyu > > > > > > From thomas.schatzl at oracle.com Tue Oct 24 15:38:45 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 24 Oct 2017 17:38:45 +0200 Subject: RFR(M): 8171181: Supporting heap allocation on alternative memory devices In-Reply-To: References: Message-ID: <1508859525.3150.26.camel@oracle.com> Hi Kishor, initial review using the webrevs, and the comment thread in hotspot- runtime-dev (readded to cc because questions were left unanswered there). The code also touches runtime code quite a bit, so I would appreciate comments by the runtime team. On Thu, 2017-10-19 at 15:00 +0000, Kharbas, Kishor wrote: > Hi Sangheon, > ? > Thanks for the review and comments. > I created two webrevs: > webrev.07 : Is the rebase of webrev.06 on jdk10??? (http://cr.openjdk > .java.net/~kkharbas/8171181/webrev.07/) > webrev.08 : Is incremental webrev over 07.????????????? (http://cr.op > enjdk.java.net/~kkharbas/8171181/webrev.08/) > ? Some first comments on the webrev follow: - could you please provide both a full and incremental webrev for your changes? The former help the reviewers late to the party, the incremental ones the people already working with you. - the patch has not been rebased to jdk10/hs as Sangheon suggested. Probably you rebased to jdk10/jdk10? os_posix.cpp: - os::create_file_for_heap: the malloc allocates one byte too little, creating an automatic write outside the buffer at the strcat() method. - os::create_file_for_heap does not use the size parameter. Please remove. - in several places: improve the output of the error messages to have meaning for the user. Not just "malloc failed"; there were already some suggestions in the referenced thread at ?http://mail.openjdk.java.net/p ipermail/hotspot-runtime-dev/2017-March/022733.html . Same for the asserts, and particularly for them - they will implicitly terminate the VM, so e.g. a "fchmod error" is definitely too little. At least error no/output of os::strerror() etc. would be required for quick diagnosis. I read in some of the emails that you intend to let the caller print a good error message. I fear that the caller might not have the necessary information any more to do that in a useful way. - in that same thread there has also been the question about the need for blocking the signals for the process that has gone unanswered. - in that same thread there has also been the question about why the code sets permissions to 0600 manually; this is because of glibc 2.0.6 compatibility. Glibc 2.06 has been released 1997-12-29. I looked a bit through glibc requirements for the Oracle JDK, and while I could not find exact requirements, some posts indicate the need for GLIBC_2.4 at minimum (for jdk7) which is from 2003. I recommend removing this, it seems unnecessary and completely outdated. - os::malloc should be paired with os::free in os::create_file_for_heap() (2x) - I am not sure whether the methods that deal with mapping memory to a file should have "dax" in their name. The code seems to be pretty generically map memory into a file. - I am not sure if the current approach to FS errors after mkstemp is very nice. Please at least print a meaningful warning message to the log. - please also specify the meaning of the return value for os::create_file_for_heap() in the documentation. - there is a missing "p" in the name of os::reserve_mmaped_memory(). - os::util_posix_fallocate(): s/continous/continuous - os::map_memory_to_dax_file(): according to man pages, posix_fallocate does not set errno, but the code checking its return (or util_posix_fallocate()) expects it to do so. - os::attempt_reserve_memory_at(), in the call to pd_attempt_reserve_memory_at() for AIX, in the third parameter, please use the exact name of the parameter in the comment. - os::attempt_reserve_memory_at(), the second "if" needs a blank before the bracket. - os::attempt_reserve_memory_at(), in the comment, s/mmemory/memory - in that same method, the #if..#endif block should be aligned like other similar blocks. - in that same method, the "else" should be on the same line as the closing bracket of the if-body. os_windows.cpp: os::create_file_for_heap(): - same bug about length of allocation the _alloca call. Not completely sure about why use alloca (or not use alloca in os_posix.cpp). - this version calls os::native_path(). It seems strange to not do that as well in the others (although it does nothing). - os::reserve_memory_aligned(): an "else" should be moved into the same line as the closing bracket. universe.cpp: - Universe::reserve_heap(): I do not think this functionality has anything to do with compressed oops, so the log message should not have the coops tag. The log message could/should have more information about the file location. I also recommend using "Java heap" instead of "Heap" here, as the latter is ambiguous. virtualspace.cpp: - in failed_to_reserve_as_requested: "else" indentation. Also, the "else { if (..." could be shortened to "} else if (..." to avoid the extra indentation level. (2x) - ReservedSpace::initialize(): s/upto/up to - ReservedSpace::initialize(): remove the coops tag here as well. Also, the info message might be more similar to the comment above where it says that large page support is up to the file system, and the flag ignored. - the code contains the snippet if (_fd_for_heap != -1) { if (!os::unmap_memory... } else { if (!os::release_memory... } at least twice. Maybe this code snippet could be extracted into a new method. - (ignore if you want) ReservedHeapSpace::ReservedHeapSpace, at the end - maybe the condition is easier to read if it looks like: if (_fd_for_heap != -1) { os::close(_fd_for_heap); } than it is now. - virtualspace.hpp: in the changed constructor signature, please add a comment about what backing_fs_for_heap actually means (and what happens if set). arguments.cpp: - Arguments::finalize_vm_init_args: maybe at the place where the change adds the warning/info message about NUMA support being specific to the file system; also add the same warning about UseLargePages that is located elsewhere. Like "UseXXXX may not be supported in some specific file system implementations." - or just ignore as Andrew said in the other email thread. Note that I am not sure that the selected place is the correct place for such warning about incompatible flags (and maybe UseNUMA/UseLargePages should be forcibly disabled here? But then again, it does not hurt just having it enabled?). I will let the runtime team comment as a lot of that argument processing changed in jdk9. Maybe Arguments::check_vm_args_consistency() is better. There is similar code in Arguments::adjust_after_os()... os.cpp: - os::reserve_memory: "else" indentation - os::reserve_memory: I do not follow that comment - it explains the code as written, not why, and what map_memory_to_dax_file() has to do with AIX and SHM. It uses an outdated method name, and also s/your/our. Probably the whole comment can be removed. os.hpp: - indentation of the new #if defined clause - here I may probably be speaking wrongly on behalf of the runtime team, but os.hpp, as an abstraction of the OS should probably not have platform specific ifdefs in it. Thanks, Thomas > In webrev08 I have addressed all the comments, except for ones below: > ? > --------------------------------------- > src/os/aix/vm/os_aix.cpp > > 2514 char* os::pd_attempt_reserve_memory_at(size_t bytes, char* > requested_addr, bool use_SHM) { > - Question. Why os_aix has additional parameter at > pd_attemp_reserve_memory_at()? Probably only AIX has shmated memory > implementation? > ???????? A: Yes that?s correct. > ? > -------------------------------------- > 137?????? log_debug(gc, heap, coops)("UseLargePages can't be set with > HeapDir option."); > - Is it enough to print log message instead of warning message? i.e. > Don't we need something at arguments.cpp:3656, similar to NUMA flags? > ?????? A: If don?t disable UseLargePages like UseNUMA because large > pages can be used for other allocation such as CodeCache. > ? > ------------------------------------ > ? > 603 ReservedHeapSpace::ReservedHeapSpace(size_t size, size_t > alignment, bool large, const char* backing_fs_for_heap) > - ReservedSpace has '_backing_fd' but the constructor doesn't take it > as a parameter and only ReservedHeapSpace uses it. This seems not > ideal, couldn't make it better? I know actual logic is at > ReservedSpace so it is not convenient to add _backing_fs_for_heap at > ReservedHeapSapce. > ????? A: ReservedHeapSpace depends on few functions in ReservedSpace > such as initialize(), release(). So instead of passing it as > argument, I set it as a propert of ReservedSpace. > ? > ----------------------------------- > 1663 > - You removed os::attempt_reserve_memory_at() from os.cpp and split > into os_posix.cpp and os_windows.cpp. > ? But I think you should remain os::attempt_reserve_memory_at() at > os.cpp and implement os specific stuffs at each os_xxx.cpp files for > pd_xxx. Of cource move MemTracker function call as well. > ??????? A: I do it this way to reduce the redundant code, If I > implement in OS specific files in pd_xxx(), the code to replace > reserved mapping with file mapping > (replace_existing_mapping_with_dax_file_mapping()) will be redundant. > ???????????? Still if you feel I will do the change and see how it > looks. > ? > Regards > Kishor > ? > ? > From: sangheon.kim [mailto:sangheon.kim at oracle.com]? > Sent: Tuesday, September 26, 2017 3:18 PM > To: Kharbas, Kishor ; 'hotspot-gc-dev at openj > dk.java.net' > Subject: Re: RFR(M): 8171181: Supporting heap allocation on > alternative memory devices > ? > Hi Kishor, > > On 07/20/2017 06:34 PM, Kharbas, Kishor wrote: > I have a new version of this patch at http://cr.openjdk.java.net/~kkh > arbas/8171181/webrev.06/ > ? > This version has been tested on Windows, Linux, Solaris and Mac OS. I > could not get access to AIX for testing. > I used tmpfs to test the functionality. Cases that were tested were. > 1.?????? Allocation of heap using file mapping when ?XX:HeapDir= > option is used. > 2.?????? Creation of nameless temporary file for Heap allocation > which prevents access to file using its name. > 3.?????? Correct deletion and freeing up of space allocated for file > under different exit conditions. > 4.?????? Error handling when path specified is not present, heap size > is more than size of file system, etc. > Overall seems good. > I tried to list some missing part. > > 1. Please rebase with consolidated repository. (jdk10/hs) > 2. Build failure on Solaris. > ??? (Sorry no build error log file, as I tested a few weeks ago, it > is deleted) > 3. Have you tested with various heap reserve path using > HeapBaseMinAddress flag? i.e. to test code path of > ReservedHeapSpace::try_reserve_heap() and try_reserve_range(). > 4. How about adding HeapDir allocation success message? e.g. > gc+heap+coops=info > 5. Adding JTREG test. Probably we would need to verify this > allocation is succeeded via #4 added allocation success message. > 6. CSR (Compatibility & Specification Review). At some point, you > need to file another CR of 'CSR' type to add a new flag of 'HeapDir'. > 7. It will be much appreciated if you provide incremental webrev. I > think 06(this version) vs. 07(rebase version) would be hard to get. > Probably from 08 version. > > Here's my comments. > ----------------------------- > src/os/aix/vm/os_aix.cpp > > 2514 char* os::pd_attempt_reserve_memory_at(size_t bytes, char* > requested_addr, bool use_SHM) { > - Question. Why os_aix has additional parameter at > pd_attemp_reserve_memory_at()? Probably only AIX has shmated memory > implementation? > > ----------------------------- > src/os/posix/vm/os_posix.cpp > > 148?? char *fullname = (char*)::malloc(strlen(dir) + > sizeof(name_template)); > - Use os::malloc() > > 196?? int flags; > 197? > 198?? flags = MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS; > - Combining 196 and 198 seems better to me. > > 200???? assert((uintptr_t)requested_addr % os::Linux::page_size() == > 0, "unaligned address"); > - Linux dependency on posix file which makes build error on Solaris. > Probably os::vm_page_size(). > > 207?? addr = (char*)::mmap(requested_addr, bytes, PROT_NONE, > 208???? flags, -1, 0); > - Missing some spaces? Alignment seems odd to me. > > 226???? if (ret == -1) > - Probably you wanted to add handling code? If not, just return ret. > > 252?? if (addr == MAP_FAILED || (base != NULL && addr != base)) { > 253???? if (addr != MAP_FAILED) { > 254?????? if (!os::release_memory(addr, size)) { > 255???????? warning("Could not release memory on unsuccessful file > mapping"); > 256?????? } > 257???? } > 258???? return NULL; > 259?? } > - Splitting MAP_FAILED case and another gives better readability to > me. But this is your call. > > 269? > - Extra line. > > 284?? if (result != NULL && file_desc != -1) { > 285???? if (replace_existing_mapping_with_dax_file_mapping(result, > bytes, file_desc) == NULL) { > 286?????? vm_exit_during_initialization(err_msg("Error in mapping > Java heap at the given filesystem directory")); > 287???? } > 288???? > MemTracker::record_virtual_memory_reserve_and_commit((address)result, > bytes, CALLER_PC); > 289???? return result; > 290?? } > 291?? if (result != NULL) { > 292???? MemTracker::record_virtual_memory_reserve((address)result, > bytes, CALLER_PC); > 293?? } > - Combining line 284 and 291 seems better to me. > 284?? if (result != NULL) { > ??????? if (file_desc != -1) { > ????????? if (replace_existing_mapping_with_dax_file_mapping(result, > bytes, file_desc) == NULL) { > ??????????? vm_exit_during_initialization(err_msg("Error in mapping > Java heap at the given filesystem directory")); > ????????? } > ????????? > MemTracker::record_virtual_memory_reserve_and_commit((address)result, > bytes, CALLER_PC); > ??????? } else { > ????????? MemTracker::record_virtual_memory_reserve((address)result, > bytes, CALLER_PC); > ??????? } > ????? } > ????? return result; > > ----------------------------- > src/os/windows/vm/os_windows.cpp > 3141 // if 'base' is not NULL, function will return NULL if it cannot > get 'base' > - Start with uppercase. > > 3142 // > - This line seems redundant. > > 3151?????? vm_exit_during_initialization(err_msg("Could not allocate > sufficient disk space for heap")); > - heap -> Java heap (same as line 3153) > > 3168?? assert(base != NULL, "base cannot be NULL"); > - 'base' -> 'Base' or 'Base address'. > > 3172? > - Redundant line. > > 3230???? } > 3231???? else { > -> } else { > > 3278?? return reserve_memory(bytes, requested_addr, 0); > - Is it correct to use '0' not '-1'? It would be better to explain > why we use hard-coded value here. > > ----------------------------- > src/share/vm/memory/universe.cpp > - No comments > > ----------------------------- > src/share/vm/memory/virtualspace.cpp > - copyright update > > 74??????????????????????????????????????????? const size_t size, bool > special, bool is_file_mapped= false) > - Need space between 'is_file_mapped' and '='. > > 92?????????? fatal("os::release_memory failed"); > - Typo, 'os::unmap_memory failed'. > > 129?? // If there is a backing file directory for this VirtualSpace > then whether > - This is not VirtualSpace. Probably just 'space'. > > 130?? // large pages are allocated is upto the filesystem the dir > resides in. > - 'dir'? Probably 'backing file for Java heap'. > > 137?????? log_debug(gc, heap, coops)("UseLargePages can't be set with > HeapDir option."); > - Is it enough to print log message instead of warning message? i.e. > Don't we need something at arguments.cpp:3656, similar to NUMA flags? > > 191???????? // unmap_memory will do extra work esp. in Windows > - esp. -> especially > > 282?????? } > 283?????? else { > -> } else { > > 346?? // If there is a backing file directory for this VirtualSpace > then whether > - Again this is not VirtualSpace, so just 'space'. > > 352???? if (UseLargePages && (!FLAG_IS_DEFAULT(UseLargePages) || > 353?????? !FLAG_IS_DEFAULT(LargePageSizeInBytes))) { > - Wrong alignment at line 353. Consider to make same as line 380. > > 603 ReservedHeapSpace::ReservedHeapSpace(size_t size, size_t > alignment, bool large, const char* backing_fs_for_heap) > - ReservedSpace has '_backing_fd' but the constructor doesn't take it > as a parameter and only ReservedHeapSpace uses it. This seems not > ideal, couldn't make it better? I know actual logic is at > ReservedSpace so it is not convenient to add _backing_fs_for_heap at > ReservedHeapSapce. > > ----------------------------- > src/share/vm/memory/virtualspace.hpp > 40?? int??? _backing_fd; > - I would prefer to have better name to describe.? > ? e.g. as command-line option name is 'HeapDir', _heap_fd or > _fd_for_heap(similar to below)? > > 115?? ReservedHeapSpace(size_t size, size_t forced_base_alignment, > bool large, const char* backingFSforHeap = NULL); > - Snake case. How about 'fs_for_heap' or 'heap_fs'? > > ----------------------------- > src/share/vm/runtime/arguments.cpp > 3655???? FLAG_SET_CMDLINE(bool, UseNUMA, false); > - (questions) Don't need to add a warning message for > UseLargePagesSame=true as commented virtualspace.cpp:137? > > ----------------------------- > src/share/vm/runtime/globals.hpp > - No comments > > ----------------------------- > src/share/vm/runtime/os.cpp > > 1632? > - Extra line. > > 1642?? } > 1643?? else { > -> } else { > > 1663 > - You removed os::attempt_reserve_memory_at() from os.cpp and split > into os_posix.cpp and os_windows.cpp. > ? But I think you should remain os::attempt_reserve_memory_at() at > os.cpp and implement os specific stuffs at each os_xxx.cpp files for > pd_xxx. Of cource move MemTracker function call as well. > > ----------------------------- > src/share/vm/runtime/os.hpp > > 349?? // replace existing reserved memory with file mapping > - Start with uppercase letter. > > Thanks, > Sangheon > > > > ? > - Kishor > ? > From: Kharbas, Kishor? > Sent: Tuesday, July 11, 2017 6:40 PM > To: 'hotspot-gc-dev at openjdk.java.net' t> > Cc: Kharbas, Kishor > Subject: RFR(M): 8171181: Supporting heap allocation on alternative > memory devices > ? > Greetings, > ? > I have an updated patch for JEP https://bugs.openjdk.java.net/browse/ > JDK-8171181 at http://cr.openjdk.java.net/~kkharbas/8171181/webrev.05 > This patch fixes the bugs pointed earlier and other suggestions to > make the code less intrusive. > ? > I have also sent this to ?hotspot-runtime-dev? mailing list (included > below). > ? > I would appreciate comments and feedback. > ? > Thanks > Kishor > ? > From: Kharbas, Kishor? > Sent: Monday, July 10, 2017 1:53 PM > To: hotspot-runtime-dev at openjdk.java.net > Cc: Kharbas, Kishor > Subject: RFR(M): 8171181: Supporting heap allocation on alternative > memory devices > ? > Hello all! > ? > I have an updated patch for https://bugs.openjdk.java.net/browse/JDK- > 8171181 at http://cr.openjdk.java.net/~kkharbas/8171181/webrev.05 > I have lost the old email chain so had to start a fresh one. The > archived conversation can be found at - http://mail.openjdk.java.net/ > pipermail/hotspot-runtime-dev/2017-March/022733.html > ? > 1.?????? I have worked on all the comments and fixed the bugs. Mainly > bugs fixed are related to sigprocmask() and changed the > implementation such that ?fd? is not passed all the way down the call > stack. Thus minimizing function signature changes. > ? > 2.?????? Patch supports all OS?es. Consolidated all Posix compliant > OS?s implementation in os_posix.cpp. > ? > 3.?????? The patch is tested on Windows and Linux. Working on testing > it on other OS?es. > ? > Let me know if this version looks clean and correct. > ? > Thanks > Kishor > ? From thomas.stuefe at gmail.com Tue Oct 24 16:08:18 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 24 Oct 2017 18:08:18 +0200 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> Message-ID: Hi Zhengyu, - VM_PrintMetadata still has the _unit member, but I see it nowhere used. - I would have split the summary between class and non-class area, that may have been more useful. But this is a matter of taste, I leave it up to you. All else looks fine to me now. I do not need another review. Thanks for your work, this will come in handy! ..Thomas On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu wrote: > Hi Thomas, > > >> Not sure whether we misunderstood each other. I was thinking of something >> in the line of: >> >> <<<< >> .... >> ClassLoader: jdk/internal/reflect/DelegatingClassLoader >> Metadata: >> capacity: 5.00KB used: 3.32KB free: 1.68KB waste: 0.00KB >> Class data: >> capacity: 1.00KB used: 0.54KB free: 0.46KB waste: 0.00KB >> >> ClassLoader: for anonymous class >> Metadata: >> capacity: 1.00KB used: 1.00KB free: 0.00KB waste: 0.00KB >> Class data: >> capacity: 1.00KB used: 0.64KB free: 0.36KB waste: 0.00KB >> .... >> >> Summary: >> XX class loaders encountered, total capacity: xx, total waste: xx. >> >> Peak usage by class loader: xxxx >> >>>> >> > Added summary lines: > > Total class loaders= 56 capacity= 6378.00KB used= 6205.08KB > free= 172.92KB waste= 1.44KB > For anonymous classes= 54 capacity= 212.00KB used= 96.95KB > free= 115.05KB waste= 0.94KB > > > > >> These numbers would not be interpretation, just a convenience to the >> reader to get a clear picture. >> >> It even may be useful to separate the output between basic and detailed >> mode, and in basic mode omit all the single class loaders and just print >> the summary line. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >> >> >> >> >> metaspace.hpp: >> >> I would have preferred to keep scale_unit file local static instead of >> exposing it via MetaspaceAux, because it does not really have anything to >> do with metaspace. >> > Fixed > > >> Otherwise ok. >> >> --- >> >> metaspace.cpp >> >> - ChunkManager::print_statistics(): thanks for reusing this function! >> -> Scale can only be ever 1, K, M, G, yes? So, could you add an >> assert at the start of the function, and a comment at the prototype or >> function head? >> -> Also, now that ChunkManager::print_statistics() takes a scale, >> could you please change the printout to use floats like you did in >> PrintCLDMetaspaceInfoClosure::print_metaspace()? >> > Done. > > > Chunkmanager (non-class): > 0 specialized (128 bytes) chunks, total 0.00KB > 0 small (512 bytes) chunks, total 0.00KB > 0 medium (8192 bytes) chunks, total 0.00KB > 0 humongous chunks, total 0.00KB > total size: 0.00KB. > Chunkmanager (class): > 0 specialized (128 bytes) chunks, total 0.00KB > 0 small (256 bytes) chunks, total 0.00KB > 0 medium (4096 bytes) chunks, total 0.00KB > 0 humongous chunks, total 0.00KB > total size: 0.00KB. > > >> - I am concerned about races in PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* >> cld). Maybe my understanding is limited. We bail if cld->is_unloading. But >> nothing prevents a ClassLoaderDataGraph from starting to unload a >> ClassLoaderData and tearing down the Metaspace after we started printing it >> in another thread, right? Also, I do not see any fences. >> >> So, I wonder whether PrintCLDMetaspaceInfoClosure should run in lock >> protection. And then, I wonder if it would be good to separate data >> gathering and printing. We print to a (at least in principle) unknown >> output stream, which may be doing slow File IO or even Network IO. Doing >> this under lock protection seems not a good idea (but I see other places >> where this happens too). >> >> For an example, see ChunkManager::get_statistic() vs >> ChunkManager::print_statistic(). Could you do the same, have a data >> gathering step where you collect your quadrupel >> in a variable sized list of structures in memory, and print it out in a >> separate step? I realize though that the effort would be larger than for >> what we did in ChunkManager::get_statistic(), where we only fill a >> structure. >> >> > Unlike other NMT queries, this query is executed at a safepoint by > VMThread, so it should be okay. > > Updated webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ > > Thanks, > > -Zhengyu > > > ------ >> >> vm_operations.hpp >> >> - VM_PrintMetadata : do we still need _unit? >> >> >> Thanks, >> >> -Zhengyu >> >> >> >> Thanks! >> >> Thomas >> >> >> >> On 10/23/2017 11:08 AM, Thomas St?fe wrote: >> >> >> >> On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu > > >> wrote: >> >> >> >> Okay. So this is important for understanding cases >> where we have >> a large number of miniscule class loaders, each one >> only having >> a few numbers of chunks. In this case, would it be >> useful to >> have a running total of "free" and "waste", too? Or >> even better, >> also average and peak values of "free" and "waste" (to >> tell >> apart cases where you have one outlier vs cases where >> just all >> metaspaces waste a lot of memory). >> Just out of curiosity, I looked at >> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >> >> > gu/cld_metaspace/wildfly.txt >> > and >> it seems that "free" is the problem, not "waste"? So I >> guess >> what happens is that all the little classloaders >> allocate just >> enough memory to push them over "_small_chunk_limit=4", >> so they >> allocate the first medium chunk, from which then they >> take a >> small bite and leave the rest laying around? >> >> Yes. The free space of anonymous classes should be counted >> as waste, >> in my opinion. I am not sure if everyone agrees, so I took >> the >> summary line out of this patch. >> >> I would be more than happy to restore the summary line, if >> you find >> it is useful :-) >> >> >> I agree with you. Typically class loaders stop loading at some >> point, especially these tine ones, and often will not resume >> loading. So, effectivly, the space is wasted. It still helps to >> distinguish "free" in the current chunks from "free" in the >> other chunks to tell this situation apart from a situation where >> we have just a bug in metaspace chunk handling preventing us to >> use up our chunks. >> >> >> The latter would be an important addition, >> regardless >> if this is >> for customers or for us. Chunks idling in >> freelists can >> make a >> huge impact, and unfortunately this may happen >> in perfectly >> valid cases. See e.g. our JEP >> "https://bugs.openjdk.java.net >> /browse/JDK-8166690 >> >> > > >> > /browse/JDK-8166690 >> >> > >>". We have >> already a printing routine for free chunks, in >> ChunkManager::print_all_chunkmanagers(). Could >> you add >> this to >> your output? >> >> >> Make sense! Could you recommend what data and >> format you >> would like >> to see? >> >> >> >> Would not ChunkManager::print_all_chunkmanagers() be >> sufficient? >> >> Okay, I might need to tweak output format. >> >> >> Fine by me. Nothing depends on it yet. >> >> Thanks, Thomas >> >> Thanks, >> >> -Zhengyu >> >> >> >> ------------- >> >> Other than above notes, the changes seem >> straighforward, I did >> not see anything wrong. Minor nits: >> >> - PrintCLDMetaspaceInfoClosure::_out, _scale >> and _unit >> could all >> be constant (with _out being a outputStream* >> const). >> - We also do not need to provide scale *and* >> unit as >> argument, >> one can be deduced from the other. This would >> also prevent >> mismatches.We do need scale here, since nmt >> command >> line can set >> scale and we do >> >> have to honor that. >> >> However, passing unit is ugly! I don't want to >> introduce >> inverse >> dependence (metaspace -> nmt), which is also ugly. >> >> >> Yes, that would be regrettable. >> >> Probably should move scale mapping code from >> NMTUtil to a >> common util? >> >> >> >> That would be possible, these functions >> (scale_from_name etc) >> are simple enough to be moved to a generic file. >> However, if you >> pass scale, deducing the string that goes with it is >> trivial and >> I would have no qualms doing this by hand in >> metaspace.cpp. But >> it is a matter of taste. >> >> >> I did not look closely at locking issues, I >> assume >> MetaspaceAux::print_metadata() is supposed to >> run only in >> Safepoints? >> >> >> No. sum_xxx_xxx_in_use queries are guarded by locks. >> >> Thanks, >> >> -Zhengyu >> >> >> Thanks, Thomas >> >> >> >> Thanks & Kind Regards, Thomas >> >> >> >> >> >> >> On Fri, Oct 20, 2017 at 4:00 PM, Zhengyu Gu >> >> > >> >> >> >> >> > >> >> >> >>>> wrote: >> >> Up to now, there is little visibility >> into how >> metaspaces >> are used, >> and by whom. >> >> This proposed enhancement gives NMT the >> ability to >> report >> metaspace >> usages on per-classloader level, via a >> new NMT command >> "metadata" >> >> >> For example: >> >> 2496: >> Metaspaces: >> Metadata space: reserved= 8192KB >> committed= 5888KB >> Class space: reserved= 1048576KB >> committed= 640KB >> >> Per-classloader metadata: >> >> ClassLoader: for anonymous class >> Metadata capacity= 5.00KB >> used= 1.16KB >> free= 3.84KB waste= 0.05KB >> Class data capacity= 1.00KB >> used= 0.59KB >> free= 0.41KB waste= 0.00KB >> >> .... >> >> ClassLoader: >> Metadata capacity= 5640.00KB >> used= 5607.42KB >> free= 32.58KB waste= 0.46KB >> Class data capacity= 520.00KB >> used= 500.94KB >> free= 19.06KB waste= 0.02KB >> >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > /browse/JDK-8189688 >> >> > >> >> > /browse/JDK-8189688 >> >> > > >> > /browse/JDK-8189688 >> >> > >>> >> Webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> > >> > gu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> >> >> < >> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> > >> > gu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> >>> >> >> Test: >> >> hotspot_tier1_runtime, fastdebug and >> release on >> x64 Linux. >> >> Thanks, >> >> -Zhengyu >> >> >> >> >> >> >> From zgu at redhat.com Tue Oct 24 18:04:46 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 24 Oct 2017 14:04:46 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> Message-ID: Hi Thomas, On 10/24/2017 12:08 PM, Thomas St?fe wrote: > Hi Zhengyu, > > - VM_PrintMetadata still has the _unit member, but I see it nowhere used. > > - I would have split the summary between class and non-class area, that > may have been more useful. But this is a matter of taste, I leave it up > to you. > Agree, should go little further. Summary: Total class loaders= 754 capacity= 67528.00KB used= 61216.88KB free= 9453.11KB waste= 38.73KB Metadata capacity= 58780.00KB used= 54053.45KB free= 4726.55KB waste= 36.38KB Class data capacity= 8748.00KB used= 7163.43KB free= 1584.57KB waste= 2.35KB For anonymous classes= 580 capacity= 2732.00KB used= 1065.06KB free= 1666.94KB waste= 16.27KB Metadata capacity= 2152.00KB used= 738.46KB free= 1413.54KB waste= 16.27KB Class data capacity= 580.00KB used= 326.60KB free= 253.40KB waste= 0.00KB Updated webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ Thanks, -Zhengyu > All else looks fine to me now. I do not need another review. > > Thanks for your work, this will come in handy! > > ..Thomas > > On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu > wrote: > > Hi Thomas, > > > Not sure whether we misunderstood each other. I was thinking of > something in the line of: > > <<<< > .... > ClassLoader: jdk/internal/reflect/DelegatingClassLoader > Metadata: > capacity: 5.00KB used: 3.32KB free: 1.68KB > waste: 0.00KB > Class data: > capacity: 1.00KB used: 0.54KB free: 0.46KB > waste: 0.00KB > > ClassLoader: for anonymous class > Metadata: > capacity: 1.00KB used: 1.00KB free: 0.00KB > waste: 0.00KB > Class data: > capacity: 1.00KB used: 0.64KB free: 0.36KB > waste: 0.00KB > .... > > Summary: > XX class loaders encountered, total capacity: xx, total waste: xx. > > Peak usage by class loader: xxxx > >>>> > > Added summary lines: > > Total class loaders= 56 capacity= 6378.00KB used= > 6205.08KB free= 172.92KB waste= 1.44KB > For anonymous classes= 54 capacity= 212.00KB used= 96.95KB > free= 115.05KB waste= 0.94KB > > > > > These numbers would not be interpretation, just a convenience to > the reader to get a clear picture. > > It even may be useful to separate the output between basic and > detailed mode, and in basic mode omit all the single class > loaders and just print the summary line. > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html > > > > > > > > metaspace.hpp: > > I would have preferred to keep scale_unit file local static > instead of exposing it via MetaspaceAux, because it does not > really have anything to do with metaspace. > > Fixed > > > Otherwise ok. > > --- > > metaspace.cpp > > - ChunkManager::print_statistics(): thanks for reusing this > function! > -> Scale can only be ever 1, K, M, G, yes? So, could you > add an assert at the start of the function, and a comment at the > prototype or function head? > -> Also, now that ChunkManager::print_statistics() takes a > scale, could you please change the printout to use floats like > you did in PrintCLDMetaspaceInfoClosure::print_metaspace()? > > Done. > > > Chunkmanager (non-class): > 0 specialized (128 bytes) chunks, total 0.00KB > 0 small (512 bytes) chunks, total 0.00KB > 0 medium (8192 bytes) chunks, total 0.00KB > 0 humongous chunks, total 0.00KB > total size: 0.00KB. > Chunkmanager (class): > 0 specialized (128 bytes) chunks, total 0.00KB > 0 small (256 bytes) chunks, total 0.00KB > 0 medium (4096 bytes) chunks, total 0.00KB > 0 humongous chunks, total 0.00KB > total size: 0.00KB. > > > - I am concerned about races in > PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). > Maybe my understanding is limited. We bail if cld->is_unloading. > But nothing prevents a ClassLoaderDataGraph from starting to > unload a ClassLoaderData and tearing down the Metaspace after we > started printing it in another thread, right? Also, I do not see > any fences. > > So, I wonder whether PrintCLDMetaspaceInfoClosure should run in > lock protection. And then, I wonder if it would be good to > separate data gathering and printing. We print to a (at least in > principle) unknown output stream, which may be doing slow File > IO or even Network IO. Doing this under lock protection seems > not a good idea (but I see other places where this happens too). > > For an example, see ChunkManager::get_statistic() vs > ChunkManager::print_statistic(). Could you do the same, have a > data gathering step where you collect your > quadrupel in a variable sized list of > structures in memory, and print it out in a separate step? I > realize though that the effort would be larger than for what we > did in ChunkManager::get_statistic(), where we only fill a > structure. > > > Unlike other NMT queries, this query is executed at a safepoint by > VMThread, so it should be okay. > > Updated webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ > > > Thanks, > > -Zhengyu > > > ------ > > vm_operations.hpp > > - VM_PrintMetadata : do we still need _unit? > > > Thanks, > > -Zhengyu > > > > Thanks! > > Thomas > > > > On 10/23/2017 11:08 AM, Thomas St?fe wrote: > > > > On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu > > > > > >>> wrote: > > > > Okay. So this is important for understanding cases > where we have > a large number of miniscule class loaders, > each one > only having > a few numbers of chunks. In this case, would it be > useful to > have a running total of "free" and "waste", > too? Or > even better, > also average and peak values of "free" and > "waste" (to tell > apart cases where you have one outlier vs > cases where > just all > metaspaces waste a lot of memory). > Just out of curiosity, I looked at > http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt > > > > > > > > >> and > it seems that "free" is the problem, not > "waste"? So I > guess > what happens is that all the little classloaders > allocate just > enough memory to push them over > "_small_chunk_limit=4", > so they > allocate the first medium chunk, from which > then they > take a > small bite and leave the rest laying around? > > Yes. The free space of anonymous classes should be > counted > as waste, > in my opinion. I am not sure if everyone agrees, > so I took the > summary line out of this patch. > > I would be more than happy to restore the summary > line, if > you find > it is useful :-) > > > I agree with you. Typically class loaders stop loading > at some > point, especially these tine ones, and often will not > resume > loading. So, effectivly, the space is wasted. It still > helps to > distinguish "free" in the current chunks from "free" in the > other chunks to tell this situation apart from a > situation where > we have just a bug in metaspace chunk handling > preventing us to > use up our chunks. > > > The latter would be an important > addition, > regardless > if this is > for customers or for us. Chunks idling in > freelists can > make a > huge impact, and unfortunately this > may happen > in perfectly > valid cases. See e.g. our JEP > > "https://bugs.openjdk.java.net/browse/JDK-8166690 > > > > > > >> > > > > > > > >>>". We have > already a printing routine for free > chunks, in > > ChunkManager::print_all_chunkmanagers(). Could > you add > this to > your output? > > > Make sense! Could you recommend what data and > format you > would like > to see? > > > > Would not > ChunkManager::print_all_chunkmanagers() be > sufficient? > > Okay, I might need to tweak output format. > > > Fine by me. Nothing depends on it yet. > > Thanks, Thomas > > Thanks, > > -Zhengyu > > > > ------------- > > Other than above notes, the changes seem > straighforward, I did > not see anything wrong. Minor nits: > > - PrintCLDMetaspaceInfoClosure::_out, > _scale > and _unit > could all > be constant (with _out being a > outputStream* > const). > - We also do not need to provide > scale *and* > unit as > argument, > one can be deduced from the other. > This would > also prevent > mismatches.We do need scale here, > since nmt > command > line can set > scale and we do > > have to honor that. > > However, passing unit is ugly! I don't > want to > introduce > inverse > dependence (metaspace -> nmt), which is > also ugly. > > > Yes, that would be regrettable. > > Probably should move scale mapping code from > NMTUtil to a > common util? > > > > That would be possible, these functions > (scale_from_name etc) > are simple enough to be moved to a generic file. > However, if you > pass scale, deducing the string that goes with > it is > trivial and > I would have no qualms doing this by hand in > metaspace.cpp. But > it is a matter of taste. > > > I did not look closely at locking > issues, I assume > MetaspaceAux::print_metadata() is > supposed to > run only in > Safepoints? > > > No. sum_xxx_xxx_in_use queries are > guarded by locks. > > Thanks, > > -Zhengyu > > > Thanks, Thomas > > > > Thanks & Kind Regards, Thomas > > > > > > > On Fri, Oct 20, 2017 at 4:00 PM, > Zhengyu Gu > > > > > >> > > > > >>> > > > > > >> > > > > > >>>>> wrote: > > Up to now, there is little > visibility > into how > metaspaces > are used, > and by whom. > > This proposed enhancement gives > NMT the > ability to > report > metaspace > usages on per-classloader level, > via a > new NMT command > "metadata" > > > For example: > > 2496: > Metaspaces: > Metadata space: reserved= > 8192KB > committed= 5888KB > Class space: reserved= > 1048576KB > committed= 640KB > > Per-classloader metadata: > > ClassLoader: for anonymous class > Metadata capacity= 5.00KB > used= 1.16KB > free= 3.84KB waste= 0.05KB > Class data capacity= 1.00KB > used= 0.59KB > free= 0.41KB waste= 0.00KB > > .... > > ClassLoader: > Metadata capacity= 5640.00KB > used= 5607.42KB > free= 32.58KB waste= 0.46KB > Class data capacity= 520.00KB > used= 500.94KB > free= 19.06KB waste= 0.02KB > > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8189688 > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>>> > Webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html > > > > > > > > >> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > >>>> > > Test: > > hotspot_tier1_runtime, > fastdebug and > release on > x64 Linux. > > Thanks, > > -Zhengyu > > > > > > > From Derek.White at cavium.com Tue Oct 24 21:39:45 2017 From: Derek.White at cavium.com (White, Derek) Date: Tue, 24 Oct 2017 21:39:45 +0000 Subject: RFR(S): JDK-8163011 AArch64: NMT detail stack trace cleanup In-Reply-To: <5ce3ace4-9bed-643a-6998-7656ac1d5a21@redhat.com> References: <81369691-46f1-a0f0-c3f8-8bc17ae94e7d@bell-sw.com> <87196637-2b40-8c33-bd1f-f0990e1a6884@redhat.com> <10b05aac-19de-9c95-8025-a9942af08f53@bell-sw.com> <5ce3ace4-9bed-643a-6998-7656ac1d5a21@redhat.com> Message-ID: Hi Dmitry, Looks good to me! - Derek > -----Original Message----- > From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- > bounces at openjdk.java.net] On Behalf Of Andrew Haley > Sent: Monday, October 23, 2017 3:53 AM > To: Dmitry Samersoff ; hotspot-runtime- > dev at openjdk.java.net > Subject: Re: RFR(S): JDK-8163011 AArch64: NMT detail stack trace cleanup > > On 22/10/17 14:46, Dmitry Samersoff wrote: > > > Please, see: > > > > http://cr.openjdk.java.net/~dsamersoff/JDK- > 8163011/aarch64_only/webrev.02/ > > Thanks. This is OK. It's an ugly hack, but it's not our ugly hack. :-) > > -- > Andrew Haley > Java Platform Lead Engineer > Red Hat UK Ltd. > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From thomas.stuefe at gmail.com Wed Oct 25 04:51:46 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 25 Oct 2017 06:51:46 +0200 Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace fragmentation Message-ID: Hi all, could I please have your thoughts and reviews for this enhancement: Issue: https://bugs.openjdk.java.net/browse/JDK-8189864 Webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8189864- metaspace-map/webrev.00/webrev/ At SAP, we added a something we call a metaspace map to the metaspace analysis functions. This is a very simple ASCII print of the metaspace layout. It shows the composition of the VirtualSpaceNodes on a chunk basis. It facilitates examining fragmentation and has been proven useful when experimenting with the metaspace allocator. This change adds the map printing code and invokes it if a Metaspace OOM occurs and -Xlog:gc+metaspace+freelist=debug is active (in this case, we already print out a bunch of things). We also may consider adding this to NMT or as a jcmd addition, but I did not want to overload the patch. Example output looks like this (mind that this will only make sense with a monospaced font). Dots indicate starts of chunks. Letters indicate chunk type - (H)umongous, (M)edium, (S)mall (X)specialized - with lower case letters indicating a free chunk, upper case letters indicating a chunk in use. 0x0000000100000000: ...... xxxxxxHHHHHHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH 0x0000000100020000: HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH 0x0000000100040000: HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH 0x0000000100060000: . . . . . . . . . . . . . . SSSSSSSSSSSSMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMSSSSSSSSMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 0x0000000100080000: . . . . MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 0x00000001000a0000: . . . . MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 0x00000001000c0000: . . . . MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 0x00000001000e0000: . . . . MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 0x0000000100100000: . . . . . . . . . . . . . . . . . . . . . . . . MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMSSSSMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMSS SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS 0x0000000100120000: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS Thank you! Thomas From thomas.stuefe at gmail.com Wed Oct 25 05:00:50 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 25 Oct 2017 07:00:50 +0200 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> Message-ID: Hi Zhengyu, On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu wrote: > Hi Thomas, > > On 10/24/2017 12:08 PM, Thomas St?fe wrote: > >> Hi Zhengyu, >> >> - VM_PrintMetadata still has the _unit member, but I see it nowhere used. >> >> - I would have split the summary between class and non-class area, that >> may have been more useful. But this is a matter of taste, I leave it up to >> you. >> >> Agree, should go little further. > > Summary: > Total class loaders= 754 capacity= 67528.00KB used= 61216.88KB > free= 9453.11KB waste= 38.73KB > Metadata capacity= 58780.00KB used= 54053.45KB > free= 4726.55KB waste= 36.38KB > Class data capacity= 8748.00KB used= 7163.43KB > free= 1584.57KB waste= 2.35KB > > For anonymous classes= 580 capacity= 2732.00KB used= 1065.06KB > free= 1666.94KB waste= 16.27KB > Metadata capacity= 2152.00KB used= 738.46KB > free= 1413.54KB waste= 16.27KB > Class data capacity= 580.00KB used= 326.60KB > free= 253.40KB waste= 0.00KB > > > Nice, thanks :) > Updated webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ > > All is well. Note that you could reduce the footprint of your change somewhat by defining a structure like this: struct numbers_t { size_t cap; size_t used; size_t free; size_t waste; } and replacing the many members in PrintCLDMetaspaceInfoClosure with instances of this structure: numbers_t total; numbers_t total_class; numbers_t total_anon; numbers_t total_anon_class; Depending on how far you go, your code could deflate quite a bit. Give the structure a default ctor and your large initializer list goes away; give it a printing function and you reduce print-summary() to four function calls; with something like an numbers_t::add(size_t cap, size_t used, size_t free, size_t waste) you can reduce the additions in PrintCLDMetaspaceInfoClosure::print_metaspace() to four lines. Just a suggestion, I leave it up to you if you do this. Lines 3108 and 3129 miss each a space before BytesPerWord. Change looks fine otherwise. Thanks, Thomas Thanks, > > -Zhengyu > > > All else looks fine to me now. I do not need another review. >> >> Thanks for your work, this will come in handy! >> >> ..Thomas >> >> On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu > zgu at redhat.com>> wrote: >> >> Hi Thomas, >> >> >> Not sure whether we misunderstood each other. I was thinking of >> something in the line of: >> >> <<<< >> .... >> ClassLoader: jdk/internal/reflect/DelegatingClassLoader >> Metadata: >> capacity: 5.00KB used: 3.32KB free: 1.68KB >> waste: 0.00KB >> Class data: >> capacity: 1.00KB used: 0.54KB free: 0.46KB >> waste: 0.00KB >> >> ClassLoader: for anonymous class >> Metadata: >> capacity: 1.00KB used: 1.00KB free: 0.00KB >> waste: 0.00KB >> Class data: >> capacity: 1.00KB used: 0.64KB free: 0.36KB >> waste: 0.00KB >> .... >> >> Summary: >> XX class loaders encountered, total capacity: xx, total waste: xx. >> >> Peak usage by class loader: xxxx >> >>>> >> >> Added summary lines: >> >> Total class loaders= 56 capacity= 6378.00KB used= >> 6205.08KB free= 172.92KB waste= 1.44KB >> For anonymous classes= 54 capacity= 212.00KB used= 96.95KB >> free= 115.05KB waste= 0.94KB >> >> >> >> >> These numbers would not be interpretation, just a convenience to >> the reader to get a clear picture. >> >> It even may be useful to separate the output between basic and >> detailed mode, and in basic mode omit all the single class >> loaders and just print the summary line. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >> >> > gu/8189688/webrev.01/index.html >> > >> >> >> >> metaspace.hpp: >> >> I would have preferred to keep scale_unit file local static >> instead of exposing it via MetaspaceAux, because it does not >> really have anything to do with metaspace. >> >> Fixed >> >> >> Otherwise ok. >> >> --- >> >> metaspace.cpp >> >> - ChunkManager::print_statistics(): thanks for reusing this >> function! >> -> Scale can only be ever 1, K, M, G, yes? So, could you >> add an assert at the start of the function, and a comment at the >> prototype or function head? >> -> Also, now that ChunkManager::print_statistics() takes a >> scale, could you please change the printout to use floats like >> you did in PrintCLDMetaspaceInfoClosure::print_metaspace()? >> >> Done. >> >> >> Chunkmanager (non-class): >> 0 specialized (128 bytes) chunks, total 0.00KB >> 0 small (512 bytes) chunks, total 0.00KB >> 0 medium (8192 bytes) chunks, total 0.00KB >> 0 humongous chunks, total 0.00KB >> total size: 0.00KB. >> Chunkmanager (class): >> 0 specialized (128 bytes) chunks, total 0.00KB >> 0 small (256 bytes) chunks, total 0.00KB >> 0 medium (4096 bytes) chunks, total 0.00KB >> 0 humongous chunks, total 0.00KB >> total size: 0.00KB. >> >> >> - I am concerned about races in >> PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >> Maybe my understanding is limited. We bail if cld->is_unloading. >> But nothing prevents a ClassLoaderDataGraph from starting to >> unload a ClassLoaderData and tearing down the Metaspace after we >> started printing it in another thread, right? Also, I do not see >> any fences. >> >> So, I wonder whether PrintCLDMetaspaceInfoClosure should run in >> lock protection. And then, I wonder if it would be good to >> separate data gathering and printing. We print to a (at least in >> principle) unknown output stream, which may be doing slow File >> IO or even Network IO. Doing this under lock protection seems >> not a good idea (but I see other places where this happens too). >> >> For an example, see ChunkManager::get_statistic() vs >> ChunkManager::print_statistic(). Could you do the same, have a >> data gathering step where you collect your >> quadrupel in a variable sized list of >> structures in memory, and print it out in a separate step? I >> realize though that the effort would be larger than for what we >> did in ChunkManager::get_statistic(), where we only fill a >> structure. >> >> >> Unlike other NMT queries, this query is executed at a safepoint by >> VMThread, so it should be okay. >> >> Updated webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >> >> >> Thanks, >> >> -Zhengyu >> >> >> ------ >> >> vm_operations.hpp >> >> - VM_PrintMetadata : do we still need _unit? >> >> >> Thanks, >> >> -Zhengyu >> >> >> >> Thanks! >> >> Thomas >> >> >> >> On 10/23/2017 11:08 AM, Thomas St?fe wrote: >> >> >> >> On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu >> >> > >> >> >>> wrote: >> >> >> >> Okay. So this is important for understanding >> cases >> where we have >> a large number of miniscule class loaders, >> each one >> only having >> a few numbers of chunks. In this case, would it >> be >> useful to >> have a running total of "free" and "waste", >> too? Or >> even better, >> also average and peak values of "free" and >> "waste" (to tell >> apart cases where you have one outlier vs >> cases where >> just all >> metaspaces waste a lot of memory). >> Just out of curiosity, I looked at >> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >> >> > gu/cld_metaspace/wildfly.txt >> > >> > gu/cld_metaspace/wildfly.txt >> >> > gu/cld_metaspace/wildfly.txt >> >> >> and >> it seems that "free" is the problem, not >> "waste"? So I >> guess >> what happens is that all the little classloaders >> allocate just >> enough memory to push them over >> "_small_chunk_limit=4", >> so they >> allocate the first medium chunk, from which >> then they >> take a >> small bite and leave the rest laying around? >> >> Yes. The free space of anonymous classes should be >> counted >> as waste, >> in my opinion. I am not sure if everyone agrees, >> so I took the >> summary line out of this patch. >> >> I would be more than happy to restore the summary >> line, if >> you find >> it is useful :-) >> >> >> I agree with you. Typically class loaders stop loading >> at some >> point, especially these tine ones, and often will not >> resume >> loading. So, effectivly, the space is wasted. It still >> helps to >> distinguish "free" in the current chunks from "free" in >> the >> other chunks to tell this situation apart from a >> situation where >> we have just a bug in metaspace chunk handling >> preventing us to >> use up our chunks. >> >> >> The latter would be an important >> addition, >> regardless >> if this is >> for customers or for us. Chunks idling >> in >> freelists can >> make a >> huge impact, and unfortunately this >> may happen >> in perfectly >> valid cases. See e.g. our JEP >> " >> https://bugs.openjdk.java.net/browse/JDK-8166690 >> >> > > >> > /browse/JDK-8166690 >> >> > >> >> < >> https://bugs.openjdk.java.net/browse/JDK-8166690 >> >> > > >> > /browse/JDK-8166690 >> >> > >>>". We have >> already a printing routine for free >> chunks, in >> ChunkManager::print_all_chunkmanagers(). >> Could >> you add >> this to >> your output? >> >> >> Make sense! Could you recommend what data >> and >> format you >> would like >> to see? >> >> >> >> Would not >> ChunkManager::print_all_chunkmanagers() be >> sufficient? >> >> Okay, I might need to tweak output format. >> >> >> Fine by me. Nothing depends on it yet. >> >> Thanks, Thomas >> >> Thanks, >> >> -Zhengyu >> >> >> >> ------------- >> >> Other than above notes, the changes >> seem >> straighforward, I did >> not see anything wrong. Minor nits: >> >> - PrintCLDMetaspaceInfoClosure::_out, >> _scale >> and _unit >> could all >> be constant (with _out being a >> outputStream* >> const). >> - We also do not need to provide >> scale *and* >> unit as >> argument, >> one can be deduced from the other. >> This would >> also prevent >> mismatches.We do need scale here, >> since nmt >> command >> line can set >> scale and we do >> >> have to honor that. >> >> However, passing unit is ugly! I don't >> want to >> introduce >> inverse >> dependence (metaspace -> nmt), which is >> also ugly. >> >> >> Yes, that would be regrettable. >> >> Probably should move scale mapping code >> from >> NMTUtil to a >> common util? >> >> >> >> That would be possible, these functions >> (scale_from_name etc) >> are simple enough to be moved to a generic file. >> However, if you >> pass scale, deducing the string that goes with >> it is >> trivial and >> I would have no qualms doing this by hand in >> metaspace.cpp. But >> it is a matter of taste. >> >> >> I did not look closely at locking >> issues, I assume >> MetaspaceAux::print_metadata() is >> supposed to >> run only in >> Safepoints? >> >> >> No. sum_xxx_xxx_in_use queries are >> guarded by locks. >> >> Thanks, >> >> -Zhengyu >> >> >> Thanks, Thomas >> >> >> >> Thanks & Kind Regards, Thomas >> >> >> >> >> >> >> On Fri, Oct 20, 2017 at 4:00 PM, >> Zhengyu Gu >> >> > >> >> >> >> > > > >> >> >>> >> >> > >> >> >> >> >> > > > >> >> >>>>> wrote: >> >> Up to now, there is little >> visibility >> into how >> metaspaces >> are used, >> and by whom. >> >> This proposed enhancement gives >> NMT the >> ability to >> report >> metaspace >> usages on per-classloader level, >> via a >> new NMT command >> "metadata" >> >> >> For example: >> >> 2496: >> Metaspaces: >> Metadata space: reserved= >> 8192KB >> committed= 5888KB >> Class space: reserved= >> 1048576KB >> committed= 640KB >> >> Per-classloader metadata: >> >> ClassLoader: for anonymous class >> Metadata capacity= >> 5.00KB >> used= 1.16KB >> free= 3.84KB waste= 0.05KB >> Class data capacity= >> 1.00KB >> used= 0.59KB >> free= 0.41KB waste= 0.00KB >> >> .... >> >> ClassLoader: >> Metadata capacity= >> 5640.00KB >> used= 5607.42KB >> free= 32.58KB waste= >> 0.46KB >> Class data capacity= >> 520.00KB >> used= 500.94KB >> free= 19.06KB waste= >> 0.02KB >> >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > /browse/JDK-8189688 >> >> > >> >> < >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > /browse/JDK-8189688 >> >> > >>> >> < >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > /browse/JDK-8189688 >> >> > >> >> < >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > /browse/JDK-8189688 >> >> > >>>> >> Webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> > >> > gu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> >> >> < >> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> > >> > gu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> >>> >> < >> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> > >> > gu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> >> >> < >> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> > >> > gu/8189688/webrev.00/index.html >> >> > gu/8189688/webrev.00/index.html >> > >>>>> >> >> Test: >> >> hotspot_tier1_runtime, >> fastdebug and >> release on >> x64 Linux. >> >> Thanks, >> >> -Zhengyu >> >> >> >> >> >> >> >> From david.holmes at oracle.com Wed Oct 25 12:53:06 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 25 Oct 2017 22:53:06 +1000 Subject: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be exited by java In-Reply-To: References: Message-ID: <0218889e-b99f-8e6e-f1c1-97e16afe13a3@oracle.com> Hi Adam, On 18/10/2017 9:17 PM, Adam Farley8 wrote: > Hi All, > > Updated summary, based on the responses: I think this is a good summary. Thanks. I can file a bug for this, but it will take some work to see how to fit this into the existing specifications and file CSR requests for those changes. This won't make 18.3 (or whatever it gets called). You will need a "champion" to help flesh this out in full and work with you on those CSR requests. I can't volunteer to do that at this time. > 1) Exit(0) (during VM startup) is a big bug because it imitates > successful completion of external cpp code accessing JNI methods. > 2) One solution for (1) is to specify a new return code for JNI. > 3) The supplied code (diff) generates, facilitates, and handles that > return code for the exit(0) scenario: -agentlib:jdwp=help > 4) The exit(0) problem (in general) is worth fixing, however we may > choose not to support the use of this new code in the jdwp example case. > 5) The supplied test confirms that the supplied code works (run via > unzip, and then bash TestStart.sh bin dir>). > 6) To implement this new return code, plus the code that handles it, we > would need to follow the CSR process to modify the JNI spec. > 7) To implement the jdwp scenario fix, if we choose to support it at > all, we would also need to use the CSR process the JVM TI spec. > 8) To address all of the worst instances of exit(0), we would need to > search for exit(0) and raise a bug for each significant one (or group). > 9) To solve (8) in one bug would be a lot of work, arguably too much > work for a single bug. > 10) If the new return code is identified as the appropriate solution to > this problem, we will need to agree on the right name and return code > value. > > Also, here's a shortlist of the main questions I recall being raised > here, plus answers people have given. > > A) What are the potential solutions for the exit(0) problem? > ? ? i) New JNI Return Code. > ? ? ii) Remove the info-only options, at least via the JNI, and return > a standard error code if they are used. > ? ? iii) Remove the info-only options, at least via the JNI, and filter > them out if they are used. > ? ? iv) Retain existing behaviour, and document a need for the user to > filter out help options before starting the VM via the JNI. > > B) What are the criteria for the "Best" solution? > ? ? i) It must prevent "exit(0)" calls. > ? ? ii) It must be proven to work. > ? ? iii) It should require minimal (none, ideally) behaviour change > from the java.exe user. > ? ? iv) It should allow the external cpp code accessing the JNI to > complete normally, without being prematurely terminated. > > C) Which solutions meet the (B) criteria? > ? ? i) Both the "new return code" and the "remove info-only options" > solutions meet the (B) criteria. > > D) Is it right to have any "Do X and then exit." arguments at all, and > (if no) what would be the alternatives? > ? ?i) ? Given the VM is a loadable shared library the answer should be No. We should not have any "Do X and then exit" arguments. The alternatives would be a mix of: - added Dcmds to "Do X" - added launcher smarts to recognize options that need a Dcmd and "then exit". I find the notion of "help" options problematic for a shared library. But I don't have a good answer for how else to document things. That all said I think this ship has sailed and we're unlikely to want to invest the time and effort in trying to clean this up this way. Thanks, David > Best Regards > > Adam Farley > > P.S. As per Alan and David's emails, the exit(#) references have been > removed entirely, as discussing them alongside the original exit(0) > problem risks scope creep. > > This bug, if raised, should only cover the exit(0) cases. I believe we > have consensus here. > > > > From: David Holmes > To: Adam Farley8 , Alan Bateman > , core-libs-dev at openjdk.java.net, > hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com > Date: 13/10/2017 14:31 > Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be > exited by java > ------------------------------------------------------------------------ > > > > Hi Adam, > > On 13/10/2017 10:16 PM, Adam Farley8 wrote: > > Hi All, > > > > Here's a summary of the email below (which is intended, partly, as a > > summary of the emails before it). > > > > Let me know if you agree/disagree with any of these points. > > > > 1) Exit(#) during vm startup is a bug because it should return a code > > regardless of the state of the VM. > > Yes it's a bug but not one that is likely to be addressed in any > foreseeable timeframe. There are simply too many "exit on error" paths. > If we were to start using C++ exceptions within the VM that might > provide a way to quickly get back to the CreateJavaVM routine where we > could return an error code - but that is itself a major project that has > barely even been discussed AFAIK. (Compiler folk have talked about it > because compiler paths are fairly self-contained - though that was > before Graal and AOT.) > > > 2) Exit(0) is an *especially* big bug because it imitates successful > > completion of external cpp code accessing JNI methods. > > 3) One solution is to specify a new return code for JNI. > > A solution for 2) yes. > > > 4) The supplied code (diff) generates, facilitates, and handles that > > return code for the exit(0) scenario: -agentlib:jdwp=help > > 5) The supplied test confirms that the supplied code works (run via > > unzip, and then bash TestStart.sh > bin dir>). > > 6) To implement this new return code, plus the code that handles it, we > > would need to follow the CSR process. > > 7) To implement the fix for the scenario used as an example of the new > > return code's use, we would need to modify the JVM TI spec. > > Yes you have demonstrated a potential solution for the agent case. The > question is, is it the right solution? Is it a worthwhile solution? (As > I've said I'd prefer not to have any "do something then exit" VM > arguments.) And can we make it fit into the existing specs without > contorting things too much. > > I still think it easier/preferable for whatever loads the VM to filter > out the VM args that trigger this behaviour. I mean if you pass > -Xshare:dump you don't have any right to expect a functioning VM after > JNI_CreateJavaVM returns - at least I don't think so. Just don't do it. > Yes it is an imperfection in the invocation API, but life isn't perfect. > > > 8) To address all of the worst instances of exit(#), we would need to > > search for exit(#) and raise a bug for each significant one (or group). > > 9) To solve (8) in one bug would be a lot of work, arguably too much > > work for a single bug. > > This is simply impractical. You may be able to pick off a few > low-hanging cases, but that won't really make any practical difference. > > > 10) If the new return code is chosen as the appropriate solution to this > > problem, we may need to choose a better name for the return code. > > > > Is this a fair assessment of the current state of the debate? > > It's a fair summary of your position and proposal. > > Cheers, > David > > > I'm on IRC every weekday from 9am-5pm (4pm fridays) BST (GMT+1) if > > anyone wants to discuss this in real-time on the openjdk channel. > > > > Best Regards > > > > Adam Farley > > > > > > > > -- Previous Email -- > > > > Hi David, Alan, > > > > You are right in that the changes to HotSpot would be nontrivial. > > I see a number of places in (e.g.) arguments.cpp that seem to > > exit in the same manner as Xlog (such as -Xinternalversion). > > > > I would advise ploughing through the CSR process to alter the > > JNI spec, and simultaneously identify some key paths that can > > be raised as bugs. That way, when people have time to address > > these issues, the mechanism to handle a silent exit is already > > in place. > > > > The JDWP fix can be raised separately as one of these bugs, if > > it would make things simpler. > > > > As for the name, JNI_SILENT_EXIT is a placeholder, and can be > > readily changed. Do you have any suggestions? > > > > Lastly, in an ideal world, the VM initialisation should never exit(#). It > > should return a return code that tells the caller something, pass or > > fail, messy or tidy. That way, if someone is using the JNI as part of > > something bigger (like a database or a web server), one of these > > scenarios is just a bug, rather than a world-ender like exit(#). > > > > And now for the individual messages. :) > > > > David: Having help data returned by the launcher seems like a > > good way to avoid exit(0) calls, but I'm not sure how we'd prevent > > a JNI-caller using those options. Ultimately, to be sure, we'd have > > to remove the logic for those options, centralise the data to better > > enable launcher access, and add some logic in there so it can find > > any other help data (e.g. from the jdwp agent library). I feel this would > > be a bigger task than adding the new return code and changing the > > vm, plus it wouldn't provide for any non-help scenarios where the > > vm wants to shut down without error during initialisation. > > > > Alan: I should mention that the silent exit solution is already in > > use in the OpenJ9 VM. Not all of the exit paths have been > > resolved, but many have. > > > > The code is open and can be found here: > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_eclipse_openj9&d=DwIC-g&c=jf_iaSHvJObTbx-siA1ZOg&r=P5m8KWUXJf-CeVJc0hDGD9AQ2LkcXDC0PMV9ntVw5Ho&m=c_McLiDSKtlzF_gNlWcAvBTNbDqyGHyW325GY3_3QkU&s=0QVowxRejNRAXI0Vv5whKQFWGTO36XVICmPYCG8EqIU&e= > > > > And though the silent exit code is disabled for the time being, it > > can be re-enabled by entering this class: > > > > runtime/vm/jvminit.c > > > > and altering line 2343 ( ctrl-f for exit(1) if it's not there). > > > > I won't paste the full code here in case people are concerned > > about contamination, but I would assert that this code (and the > > associated vm files) prove that the concept is possible. > > > > Note that that code should not be enabled until after we've > > integrated the code that can handle a silent exit. > > > > Best Regards > > > > Adam Farley > > > > P.S. Thank you both for your efforts on this. :) > > > > > > > > From: David Holmes > > To: Alan Bateman , Adam Farley8 > > , core-libs-dev at openjdk.java.net, > > hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com > > Date: 15/09/2017 12:03 > > Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be > > exited by java > > ------------------------------------------------------------------------ > > > > > > > > > > > > On 15/09/2017 8:17 PM, Alan Bateman wrote: > > ?> On 15/09/2017 02:47, David Holmes wrote: > > ?>> Hi Adam, > > ?>> > > ?>> I am still very much torn over this one. I think the idea of > > ?>> print-and-exit flags for a potentially hosted library like the JVM is > > ?>> just wrong - we should never have done that, but we did. Fixing that > > ?>> by moving the flags to the launcher is far from trivial**. Endorsing > > ?>> and encouraging these sorts of flag by adding JNI support seems to be > > ?>> sending the wrong message. > > ?>> > > ?>> ** I can envisage a "help xxx" Dcmd that can read back the info from > > ?>> the VM. The launcher can send the Dcmd, print the output and > exit. The > > ?>> launcher would not need to know what the xxx values mean, but would > > ?>> have to intercept the existing ones. > > ?>> > > ?>> Another option is just to be aware of these flags (are there more > than > > ?>> jdwp and Xlog?) and deal with them specially in your custom > launcher - > > ?>> either filter them out and ignore them, or else launch the VM in its > > ?>> own process to respond to them. > > ?>> > > ?>> Any changes to the JNI specification need to go through the CSR > process. > > ?> Yes, it would require an update to the JNI spec, also a change to the > > ?> JVM TI spec where Agent_OnLoad returning a non-0 value is specified to > > ?> terminates the VM. The name and value needs discussion too, esp. > as the > > ?> JNI spec uses negative values for failure. > > ?> > > ?> In any case, I'm also torn over this one as it's a corner case that is > > ?> only interesting for custom launchers that load agents with > options that > > ?> print usage messages. It wouldn't be hard to have the Agent_OnLoad > > ?> specify a printf hook that the agent could use for output although > there > > ?> are complications with agents such as JDWP that also announce their > > ?> transport end point. Beyond that there is still the issue of the > custom > > ?> launcher that would need to know to destroy the VM without > reporting an > > ?> error. > > ?> > > ?> So what happened to the more meaty part to this which is fixing the > > ?> various cases in HotSpot that terminate the process during > > ?> initialization? I would expect some progress could be made on those > > ?> cases while trying to decide whether to rev the JNI and JVM TI > specs to > > ?> cover the help case. > > > > Trying to eliminate the vm_exit_during_initialization paths in hotspot > > is a huge undertaking IMHO. > > > > David > > > > ?> > > ?> -Alan > > > > Unless stated otherwise above: > > IBM United Kingdom Limited - Registered in England and Wales with number > > 741598. > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > PO6 3AU > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From adam.farley at uk.ibm.com Wed Oct 25 15:23:34 2017 From: adam.farley at uk.ibm.com (Adam Farley8) Date: Wed, 25 Oct 2017 16:23:34 +0100 Subject: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be exited by java In-Reply-To: References: Message-ID: Thanks David :) Please let me know what the bug number is when you have time. Everyone else: Any volunteers to champion this? I'm on IRC 9-5 from monday-friday if anyone wants to discuss it. Best Regards Adam Farley P.S. Here's the summary including David's addition, minus the chevrons: 1) Exit(0) (during VM startup) is a big bug because it imitates successful completion of external cpp code accessing JNI methods. 2) One solution for (1) is to specify a new return code for JNI. 3) The supplied code (diff) generates, facilitates, and handles that return code for the exit(0) scenario: -agentlib:jdwp=help 4) The exit(0) problem (in general) is worth fixing, however we may choose not to support the use of this new code in the jdwp example case. 5) The supplied test confirms that the supplied code works (run via unzip, and then bash TestStart.sh ). 6) To implement this new return code, plus the code that handles it, we would need to follow the CSR process to modify the JNI spec. 7) To implement the jdwp scenario fix, if we choose to support it at all, we would also need to use the CSR process the JVM TI spec. 8) To address all of the worst instances of exit(0), we would need to search for exit(0) and raise a bug for each significant one (or group). 9) To solve (8) in one bug would be a lot of work, arguably too much work for a single bug. 10) If the new return code is identified as the appropriate solution to this problem, we will need to agree on the right name and return code value. Also, here's a shortlist of the main questions I recall being raised here, plus answers people have given. A) What are the potential solutions for the exit(0) problem? i) New JNI Return Code. ii) Remove the info-only options, at least via the JNI, and return a standard error code if they are used. iii) Remove the info-only options, at least via the JNI, and filter them out if they are used. iv) Retain existing behaviour, and document a need for the user to filter out help options before starting the VM via the JNI. B) What are the criteria for the "Best" solution? i) It must prevent "exit(0)" calls. ii) It must be proven to work. iii) It should require minimal (none, ideally) behaviour change from the java.exe user. iv) It should allow the external cpp code accessing the JNI to complete normally, without being prematurely terminated. C) Which solutions meet the (B) criteria? i) Both the "new return code" and the "remove info-only options" solutions meet the (B) criteria. D) Is it right to have any "Do X and then exit." arguments at all, and (if no) what would be the alternatives? i) Given the VM is a loadable shared library the answer should be No. We should not have any "Do X and then exit" arguments. The alternatives would be a mix of: - added Dcmds to "Do X" - added launcher smarts to recognize options that need a Dcmd and "then exit". Note from David: I find the notion of "help" options problematic for a shared library. But I don't have a good answer for how else to document things. That all said I think this ship has sailed and we're unlikely to want to invest the time and effort in trying to clean this up this way. From: David Holmes To: Adam Farley8 Cc: Alan Bateman , core-libs-dev at openjdk.java.net, hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com Date: 25/10/2017 13:53 Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be exited by java Hi Adam, On 18/10/2017 9:17 PM, Adam Farley8 wrote: > Hi All, > > Updated summary, based on the responses: I think this is a good summary. Thanks. I can file a bug for this, but it will take some work to see how to fit this into the existing specifications and file CSR requests for those changes. This won't make 18.3 (or whatever it gets called). You will need a "champion" to help flesh this out in full and work with you on those CSR requests. I can't volunteer to do that at this time. > 1) Exit(0) (during VM startup) is a big bug because it imitates > successful completion of external cpp code accessing JNI methods. > 2) One solution for (1) is to specify a new return code for JNI. > 3) The supplied code (diff) generates, facilitates, and handles that > return code for the exit(0) scenario: -agentlib:jdwp=help > 4) The exit(0) problem (in general) is worth fixing, however we may > choose not to support the use of this new code in the jdwp example case. > 5) The supplied test confirms that the supplied code works (run via > unzip, and then bash TestStart.sh bin dir>). > 6) To implement this new return code, plus the code that handles it, we > would need to follow the CSR process to modify the JNI spec. > 7) To implement the jdwp scenario fix, if we choose to support it at > all, we would also need to use the CSR process the JVM TI spec. > 8) To address all of the worst instances of exit(0), we would need to > search for exit(0) and raise a bug for each significant one (or group). > 9) To solve (8) in one bug would be a lot of work, arguably too much > work for a single bug. > 10) If the new return code is identified as the appropriate solution to > this problem, we will need to agree on the right name and return code > value. > > Also, here's a shortlist of the main questions I recall being raised > here, plus answers people have given. > > A) What are the potential solutions for the exit(0) problem? > i) New JNI Return Code. > ii) Remove the info-only options, at least via the JNI, and return > a standard error code if they are used. > iii) Remove the info-only options, at least via the JNI, and filter > them out if they are used. > iv) Retain existing behaviour, and document a need for the user to > filter out help options before starting the VM via the JNI. > > B) What are the criteria for the "Best" solution? > i) It must prevent "exit(0)" calls. > ii) It must be proven to work. > iii) It should require minimal (none, ideally) behaviour change > from the java.exe user. > iv) It should allow the external cpp code accessing the JNI to > complete normally, without being prematurely terminated. > > C) Which solutions meet the (B) criteria? > i) Both the "new return code" and the "remove info-only options" > solutions meet the (B) criteria. > > D) Is it right to have any "Do X and then exit." arguments at all, and > (if no) what would be the alternatives? > i) ? Given the VM is a loadable shared library the answer should be No. We should not have any "Do X and then exit" arguments. The alternatives would be a mix of: - added Dcmds to "Do X" - added launcher smarts to recognize options that need a Dcmd and "then exit". I find the notion of "help" options problematic for a shared library. But I don't have a good answer for how else to document things. That all said I think this ship has sailed and we're unlikely to want to invest the time and effort in trying to clean this up this way. Thanks, David > Best Regards > > Adam Farley > > P.S. As per Alan and David's emails, the exit(#) references have been > removed entirely, as discussing them alongside the original exit(0) > problem risks scope creep. > > This bug, if raised, should only cover the exit(0) cases. I believe we > have consensus here. > > > > From: David Holmes > To: Adam Farley8 , Alan Bateman > , core-libs-dev at openjdk.java.net, > hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com > Date: 13/10/2017 14:31 > Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be > exited by java > ------------------------------------------------------------------------ > > > > Hi Adam, > > On 13/10/2017 10:16 PM, Adam Farley8 wrote: > > Hi All, > > > > Here's a summary of the email below (which is intended, partly, as a > > summary of the emails before it). > > > > Let me know if you agree/disagree with any of these points. > > > > 1) Exit(#) during vm startup is a bug because it should return a code > > regardless of the state of the VM. > > Yes it's a bug but not one that is likely to be addressed in any > foreseeable timeframe. There are simply too many "exit on error" paths. > If we were to start using C++ exceptions within the VM that might > provide a way to quickly get back to the CreateJavaVM routine where we > could return an error code - but that is itself a major project that has > barely even been discussed AFAIK. (Compiler folk have talked about it > because compiler paths are fairly self-contained - though that was > before Graal and AOT.) > > > 2) Exit(0) is an *especially* big bug because it imitates successful > > completion of external cpp code accessing JNI methods. > > 3) One solution is to specify a new return code for JNI. > > A solution for 2) yes. > > > 4) The supplied code (diff) generates, facilitates, and handles that > > return code for the exit(0) scenario: -agentlib:jdwp=help > > 5) The supplied test confirms that the supplied code works (run via > > unzip, and then bash TestStart.sh > bin dir>). > > 6) To implement this new return code, plus the code that handles it, we > > would need to follow the CSR process. > > 7) To implement the fix for the scenario used as an example of the new > > return code's use, we would need to modify the JVM TI spec. > > Yes you have demonstrated a potential solution for the agent case. The > question is, is it the right solution? Is it a worthwhile solution? (As > I've said I'd prefer not to have any "do something then exit" VM > arguments.) And can we make it fit into the existing specs without > contorting things too much. > > I still think it easier/preferable for whatever loads the VM to filter > out the VM args that trigger this behaviour. I mean if you pass > -Xshare:dump you don't have any right to expect a functioning VM after > JNI_CreateJavaVM returns - at least I don't think so. Just don't do it. > Yes it is an imperfection in the invocation API, but life isn't perfect. > > > 8) To address all of the worst instances of exit(#), we would need to > > search for exit(#) and raise a bug for each significant one (or group). > > 9) To solve (8) in one bug would be a lot of work, arguably too much > > work for a single bug. > > This is simply impractical. You may be able to pick off a few > low-hanging cases, but that won't really make any practical difference. > > > 10) If the new return code is chosen as the appropriate solution to this > > problem, we may need to choose a better name for the return code. > > > > Is this a fair assessment of the current state of the debate? > > It's a fair summary of your position and proposal. > > Cheers, > David > > > I'm on IRC every weekday from 9am-5pm (4pm fridays) BST (GMT+1) if > > anyone wants to discuss this in real-time on the openjdk channel. > > > > Best Regards > > > > Adam Farley > > > > > > > > -- Previous Email -- > > > > Hi David, Alan, > > > > You are right in that the changes to HotSpot would be nontrivial. > > I see a number of places in (e.g.) arguments.cpp that seem to > > exit in the same manner as Xlog (such as -Xinternalversion). > > > > I would advise ploughing through the CSR process to alter the > > JNI spec, and simultaneously identify some key paths that can > > be raised as bugs. That way, when people have time to address > > these issues, the mechanism to handle a silent exit is already > > in place. > > > > The JDWP fix can be raised separately as one of these bugs, if > > it would make things simpler. > > > > As for the name, JNI_SILENT_EXIT is a placeholder, and can be > > readily changed. Do you have any suggestions? > > > > Lastly, in an ideal world, the VM initialisation should never exit(#). It > > should return a return code that tells the caller something, pass or > > fail, messy or tidy. That way, if someone is using the JNI as part of > > something bigger (like a database or a web server), one of these > > scenarios is just a bug, rather than a world-ender like exit(#). > > > > And now for the individual messages. :) > > > > David: Having help data returned by the launcher seems like a > > good way to avoid exit(0) calls, but I'm not sure how we'd prevent > > a JNI-caller using those options. Ultimately, to be sure, we'd have > > to remove the logic for those options, centralise the data to better > > enable launcher access, and add some logic in there so it can find > > any other help data (e.g. from the jdwp agent library). I feel this would > > be a bigger task than adding the new return code and changing the > > vm, plus it wouldn't provide for any non-help scenarios where the > > vm wants to shut down without error during initialisation. > > > > Alan: I should mention that the silent exit solution is already in > > use in the OpenJ9 VM. Not all of the exit paths have been > > resolved, but many have. > > > > The code is open and can be found here: > < https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_eclipse_openj9&d=DwIC-g&c=jf_iaSHvJObTbx-siA1ZOg&r=P5m8KWUXJf-CeVJc0hDGD9AQ2LkcXDC0PMV9ntVw5Ho&m=c_McLiDSKtlzF_gNlWcAvBTNbDqyGHyW325GY3_3QkU&s=0QVowxRejNRAXI0Vv5whKQFWGTO36XVICmPYCG8EqIU&e= > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_eclipse_openj9&d=DwIC-g&c=jf_iaSHvJObTbx-siA1ZOg&r=P5m8KWUXJf-CeVJc0hDGD9AQ2LkcXDC0PMV9ntVw5Ho&m=c_McLiDSKtlzF_gNlWcAvBTNbDqyGHyW325GY3_3QkU&s=0QVowxRejNRAXI0Vv5whKQFWGTO36XVICmPYCG8EqIU&e= > > > > And though the silent exit code is disabled for the time being, it > > can be re-enabled by entering this class: > > > > runtime/vm/jvminit.c > > > > and altering line 2343 ( ctrl-f for exit(1) if it's not there). > > > > I won't paste the full code here in case people are concerned > > about contamination, but I would assert that this code (and the > > associated vm files) prove that the concept is possible. > > > > Note that that code should not be enabled until after we've > > integrated the code that can handle a silent exit. > > > > Best Regards > > > > Adam Farley > > > > P.S. Thank you both for your efforts on this. :) > > > > > > > > From: David Holmes > > To: Alan Bateman , Adam Farley8 > > , core-libs-dev at openjdk.java.net, > > hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com > > Date: 15/09/2017 12:03 > > Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be > > exited by java > > ------------------------------------------------------------------------ > > > > > > > > > > > > On 15/09/2017 8:17 PM, Alan Bateman wrote: > > > On 15/09/2017 02:47, David Holmes wrote: > > >> Hi Adam, > > >> > > >> I am still very much torn over this one. I think the idea of > > >> print-and-exit flags for a potentially hosted library like the JVM is > > >> just wrong - we should never have done that, but we did. Fixing that > > >> by moving the flags to the launcher is far from trivial**. Endorsing > > >> and encouraging these sorts of flag by adding JNI support seems to be > > >> sending the wrong message. > > >> > > >> ** I can envisage a "help xxx" Dcmd that can read back the info from > > >> the VM. The launcher can send the Dcmd, print the output and > exit. The > > >> launcher would not need to know what the xxx values mean, but would > > >> have to intercept the existing ones. > > >> > > >> Another option is just to be aware of these flags (are there more > than > > >> jdwp and Xlog?) and deal with them specially in your custom > launcher - > > >> either filter them out and ignore them, or else launch the VM in its > > >> own process to respond to them. > > >> > > >> Any changes to the JNI specification need to go through the CSR > process. > > > Yes, it would require an update to the JNI spec, also a change to the > > > JVM TI spec where Agent_OnLoad returning a non-0 value is specified to > > > terminates the VM. The name and value needs discussion too, esp. > as the > > > JNI spec uses negative values for failure. > > > > > > In any case, I'm also torn over this one as it's a corner case that is > > > only interesting for custom launchers that load agents with > options that > > > print usage messages. It wouldn't be hard to have the Agent_OnLoad > > > specify a printf hook that the agent could use for output although > there > > > are complications with agents such as JDWP that also announce their > > > transport end point. Beyond that there is still the issue of the > custom > > > launcher that would need to know to destroy the VM without > reporting an > > > error. > > > > > > So what happened to the more meaty part to this which is fixing the > > > various cases in HotSpot that terminate the process during > > > initialization? I would expect some progress could be made on those > > > cases while trying to decide whether to rev the JNI and JVM TI > specs to > > > cover the help case. > > > > Trying to eliminate the vm_exit_during_initialization paths in hotspot > > is a huge undertaking IMHO. > > > > David > > > > > > > > -Alan > > > > Unless stated otherwise above: > > IBM United Kingdom Limited - Registered in England and Wales with number > > 741598. > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > PO6 3AU > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From erik.osterlund at oracle.com Wed Oct 25 15:26:56 2017 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 25 Oct 2017 17:26:56 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> Message-ID: <59F0AD40.3020307@oracle.com> Hi Roman, Definitely looks better. So the static GCArgumentProcessor has a singleton GCCollectedHeapFactory that is created during GCArgumentProcessor::initialize(). That makes sense to me, but I think it can be even better. First off - sorry for bikeshedding a bit here. To save you from having to update the code due to my bikeshedding, I have prepared a patch to show you what I meant, and you can tell me whether you agree or not. Full webrev: http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ Incremental over yours: http://cr.openjdk.java.net/~eosterlund/8189171/webrev.02_03/ My changes are the following: I collapsed the all-static GCArgumentProcessor and the CollectedHeapFactory into the same class called GCArguments (the natural GC helper for Arguments). It's a singleton object that deals with GC arguments stuff. GCArguments has some static functions to create_instance(), creating the singleton GCArguments object, and after you have created the singleton instance, you access it through GCArguments::instance(), and it will point to a GC-specific GCArguments, e.g. G1Arguments, CMSArguments, etc. The main reason why I think it's better to have a GC-specific GCArguments class compared to a CollectedHeapFactory class, is that I would like to eventually move all arguments processing into the GCArguments class, regardless of where (before or after CollectedHeap exists) in the boot strapping process. For example, today some argument stuff is dumped in the arguments.cpp file, and other stuff is dumped in CollectorPolicy::initialize_flags which is called later when the heap exists where it seemingly reads values out from flags, into internal variables, then changes them to what they should be and write the values back to the flags, and subsequently asserts that the internal and flag variants have the same value. Seems to me like we could in a subsequent refactoring move some of that kind of stuff into GCArguments instead, so that the argument setting logic is contained in the same class rather than split all over the place. But that seems like a separate RFE. Other than that, noticed some precompile header include was missing, the GCArgumentProcessor::initialize() function returned jint but it always returned JNI_OK, and called fatal() if something went wrong, instead of logging an error and returning JNI_ERR. And now that these types are collapsed, I moved initialize_flags_global into the abstract initial_flags, allowing GCs to override it completely if necessary, but in practice having them now call the super initial_flags(). So yeah, went ahead and fixed that for you. As for your subsequent patch creating the heap, that will just be a virtual call into the specific GCArguments class. I hope you agree with my proposal. It seemed easier for me to provide a patch describing what I'm thinking than to write it down in text. Thanks, /Erik On 2017-10-23 17:19, Roman Kennke wrote: > Hi Erik, > > thanks for your suggestions. So I renamed 'GC' -> > 'CollectedHeapFactory' (and subclasses likewise) and 'GCFactory' to > 'GCArgumentProcessor'. This will make even more sense once I added > heap creation to CollectedHeapFactory (JDK-8189389). > > Differential: > http://cr.openjdk.java.net/~rkennke/8189171/webrev.02.diff/ > > > Full webrev: > http://cr.openjdk.java.net/~rkennke/8189171/webrev.02/ > > > Better? > > Roman > >> Hi Roman, >> >> I'm usually not one to really mind names too much, but I don't >> believe some bootstrapping code for the GC should hog the name "GC". >> >> Instead of GC::initialize(), I'm thinking a static >> GCArgumentProcessor::initialize() and instead of GC::factory(), I am >> thinking a static CollectedHeapFactory::create(). Or something like >> that. >> >> The reason is that I think the name GC is too general for the >> bootstraping-only problems its code touches. Hope I am making sense. >> >> Thanks, >> /Erik >> >> On 2017-10-20 14:25, Roman Kennke wrote: >>> Hi Erik, >>> >>> Yes that is fine. >>> >>> Keep in mind that I'm planning to put heap creation into that class >>> too. >>> >>> I was thinking maybe to simply reverse the current naming: name the >>> all-static 'factory factory' 'GC', and thus call GC::initialize() >>> and GC::factory() or such, and the main interface GCFactory or >>> CollectedHeapFactory. What do you think? >>> >>> Roman >>> >>> >>>> I would like to keep the CollectedHeap as the facade interface to >>>> the GC, like it is today, instead of having a new GC class making >>>> it two facade interfaces. >>>> Of course, the CollectedHeap may only be used as a facade after it >>>> has been created. And now we are dealing with code before it was >>>> created during the bootstrapping process. >>>> >>>> So what I would like to have here is a class specifically for that, >>>> rather than another GC facade interface. I think this is mostly an >>>> exercise of finding a better name for the class you call "GC". Now >>>> I am not an expert in coming up with good names myself, but some >>>> name that indicates this is being used for flag processing would be >>>> good. Perhaps GCArgumentProcessor. The benefit of calling it >>>> something like that is that if a GC requires argument processing >>>> later in the bootstrapping, possibly after CollectedHeap has been >>>> created, it can still be used for this purpose without having to >>>> feel weird about it. >>>> >>>> How would you feel about that? Does it seem to make sense? >>>> >>>> Thanks, >>>> /Erik >>>> >>>> On 2017-10-19 21:14, Roman Kennke wrote: >>>>> Am 13.10.2017 um 03:20 schrieb David Holmes: >>>>>> Hi Roman, >>>>>> >>>>>> Not a review as GC folk need to do that. >>>>>> >>>>>> On 13/10/2017 5:59 AM, Roman Kennke wrote: >>>>>>> I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev >>>>>>> because it touches both areas (i.e. the GC interface). >>>>>>> >>>>>>> Currently, all GC related argument processing is done in >>>>>>> arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all >>>>>>> sorts of GC specific methods etc. >>>>>> >>>>>> From a runtime perspective I like all the GC specific ifdefs and >>>>>> settings moved out-of-line of the main argument processing. >>>>>> >>>>>> From a refactoring perspective I noticed that >>>>>> set_object_alignment() no longer calls set_cms_values(). I >>>>>> presume setting it elsewhere is okay? >>>>> I totally forgot to reply to this. >>>>> >>>>> What's important is that the CMS alignment values are set after >>>>> set_object_alignment() figured out the object alignment. The patch >>>>> moves that a little further down the road to the beginning of its >>>>> GC specific argument processing, but from GC perspective should be >>>>> the same. I looked thoroughly through the involved code paths and >>>>> cannot see what could go wrong. >>>>> >>>>> I need one more review. GC folks? The current webrev is: >>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.01/ >>>>> >>>>> >>>>> Thanks, >>>>> Roman >>>>> >>>> >>> >> > From rkennke at redhat.com Wed Oct 25 15:54:23 2017 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 25 Oct 2017 17:54:23 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: <59F0AD40.3020307@oracle.com> References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> Message-ID: Hi Erik, thank you for your suggestions and changes. I like it: if you look at webrev.00 you will notice that my original proposal had both the static stuff and the virtual stuff in the same class too. So from there it's mostly the renaming exercise :-) So let me just put your changes up for review (again), if you don't mind: Full webrev: http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ Incremental over webrev.02: http://cr.openjdk.java.net/~eosterlund/8189171/webrev.02_03/ So I suppose we can count it by reviewed ok by you? Then it requires one more review to go in, right? Thanks, Roman > Hi Roman, > > Definitely looks better. So the static GCArgumentProcessor has a > singleton GCCollectedHeapFactory that is created during > GCArgumentProcessor::initialize(). > That makes sense to me, but I think it can be even better. > > First off - sorry for bikeshedding a bit here. > > To save you from having to update the code due to my bikeshedding, I > have prepared a patch to show you what I meant, and you can tell me > whether you agree or not. > > Full webrev: > http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ > > Incremental over yours: > http://cr.openjdk.java.net/~eosterlund/8189171/webrev.02_03/ > > My changes are the following: > > I collapsed the all-static GCArgumentProcessor and the > CollectedHeapFactory into the same class called GCArguments (the > natural GC helper for Arguments). It's a singleton object that deals > with GC arguments stuff. > > GCArguments has some static functions to create_instance(), creating > the singleton GCArguments object, and after you have created the > singleton instance, you access it through GCArguments::instance(), and > it will point to a GC-specific GCArguments, e.g. G1Arguments, > CMSArguments, etc. > > The main reason why I think it's better to have a GC-specific > GCArguments class compared to a CollectedHeapFactory class, is that I > would like to eventually move all arguments processing into the > GCArguments class, regardless of where (before or after CollectedHeap > exists) in the boot strapping process. For example, today some > argument stuff is dumped in the arguments.cpp file, and other stuff is > dumped in CollectorPolicy::initialize_flags which is called later when > the heap exists where it seemingly reads values out from flags, into > internal variables, then changes them to what they should be and write > the values back to the flags, and subsequently asserts that the > internal and flag variants have the same value. Seems to me like we > could in a subsequent refactoring move some of that kind of stuff into > GCArguments instead, so that the argument setting logic is contained > in the same class rather than split all over the place. But that seems > like a separate RFE. > > Other than that, noticed some precompile header include was missing, > the GCArgumentProcessor::initialize() function returned jint but it > always returned JNI_OK, and called fatal() if something went wrong, > instead of logging an error and returning JNI_ERR. And now that these > types are collapsed, I moved initialize_flags_global into the abstract > initial_flags, allowing GCs to override it completely if necessary, > but in practice having them now call the super initial_flags(). So > yeah, went ahead and fixed that for you. > > As for your subsequent patch creating the heap, that will just be a > virtual call into the specific GCArguments class. > > I hope you agree with my proposal. It seemed easier for me to provide > a patch describing what I'm thinking than to write it down in text. > > Thanks, > /Erik > > On 2017-10-23 17:19, Roman Kennke wrote: >> Hi Erik, >> >> thanks for your suggestions. So I renamed 'GC' -> >> 'CollectedHeapFactory' (and subclasses likewise) and 'GCFactory' to >> 'GCArgumentProcessor'. This will make even more sense once I added >> heap creation to CollectedHeapFactory (JDK-8189389). >> >> Differential: >> http://cr.openjdk.java.net/~rkennke/8189171/webrev.02.diff/ >> >> >> Full webrev: >> http://cr.openjdk.java.net/~rkennke/8189171/webrev.02/ >> >> >> Better? >> >> Roman >> >>> Hi Roman, >>> >>> I'm usually not one to really mind names too much, but I don't >>> believe some bootstrapping code for the GC should hog the name "GC". >>> >>> Instead of GC::initialize(), I'm thinking a static >>> GCArgumentProcessor::initialize() and instead of GC::factory(), I am >>> thinking a static CollectedHeapFactory::create(). Or something like >>> that. >>> >>> The reason is that I think the name GC is too general for the >>> bootstraping-only problems its code touches. Hope I am making sense. >>> >>> Thanks, >>> /Erik >>> >>> On 2017-10-20 14:25, Roman Kennke wrote: >>>> Hi Erik, >>>> >>>> Yes that is fine. >>>> >>>> Keep in mind that I'm planning to put heap creation into that class >>>> too. >>>> >>>> I was thinking maybe to simply reverse the current naming: name the >>>> all-static 'factory factory' 'GC', and thus call GC::initialize() >>>> and GC::factory() or such, and the main interface GCFactory or >>>> CollectedHeapFactory. What do you think? >>>> >>>> Roman >>>> >>>> >>>>> I would like to keep the CollectedHeap as the facade interface to >>>>> the GC, like it is today, instead of having a new GC class making >>>>> it two facade interfaces. >>>>> Of course, the CollectedHeap may only be used as a facade after it >>>>> has been created. And now we are dealing with code before it was >>>>> created during the bootstrapping process. >>>>> >>>>> So what I would like to have here is a class specifically for >>>>> that, rather than another GC facade interface. I think this is >>>>> mostly an exercise of finding a better name for the class you call >>>>> "GC". Now I am not an expert in coming up with good names myself, >>>>> but some name that indicates this is being used for flag >>>>> processing would be good. Perhaps GCArgumentProcessor. The benefit >>>>> of calling it something like that is that if a GC requires >>>>> argument processing later in the bootstrapping, possibly after >>>>> CollectedHeap has been created, it can still be used for this >>>>> purpose without having to feel weird about it. >>>>> >>>>> How would you feel about that? Does it seem to make sense? >>>>> >>>>> Thanks, >>>>> /Erik >>>>> >>>>> On 2017-10-19 21:14, Roman Kennke wrote: >>>>>> Am 13.10.2017 um 03:20 schrieb David Holmes: >>>>>>> Hi Roman, >>>>>>> >>>>>>> Not a review as GC folk need to do that. >>>>>>> >>>>>>> On 13/10/2017 5:59 AM, Roman Kennke wrote: >>>>>>>> I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev >>>>>>>> because it touches both areas (i.e. the GC interface). >>>>>>>> >>>>>>>> Currently, all GC related argument processing is done in >>>>>>>> arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all >>>>>>>> sorts of GC specific methods etc. >>>>>>> >>>>>>> From a runtime perspective I like all the GC specific ifdefs and >>>>>>> settings moved out-of-line of the main argument processing. >>>>>>> >>>>>>> From a refactoring perspective I noticed that >>>>>>> set_object_alignment() no longer calls set_cms_values(). I >>>>>>> presume setting it elsewhere is okay? >>>>>> I totally forgot to reply to this. >>>>>> >>>>>> What's important is that the CMS alignment values are set after >>>>>> set_object_alignment() figured out the object alignment. The >>>>>> patch moves that a little further down the road to the beginning >>>>>> of its GC specific argument processing, but from GC perspective >>>>>> should be the same. I looked thoroughly through the involved code >>>>>> paths and cannot see what could go wrong. >>>>>> >>>>>> I need one more review. GC folks? The current webrev is: >>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.01/ >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Roman >>>>>> >>>>> >>>> >>> >> > From erik.osterlund at oracle.com Wed Oct 25 16:11:45 2017 From: erik.osterlund at oracle.com (Erik Osterlund) Date: Wed, 25 Oct 2017 18:11:45 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> Message-ID: Hi Roman, > On 25 Oct 2017, at 17:54, Roman Kennke wrote: > > Hi Erik, > > thank you for your suggestions and changes. > > I like it: if you look at webrev.00 you will notice that my original proposal had both the static stuff and the virtual stuff in the same class too. So from there it's mostly the renaming exercise :-) Yeah. Sounds like we are aligned. > So let me just put your changes up for review (again), if you don't mind: > > Full webrev: > http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ > > Incremental over webrev.02: > http://cr.openjdk.java.net/~eosterlund/8189171/webrev.02_03/ > > So I suppose we can count it by reviewed ok by you? Yes. Looks fantastic. > Then it requires one more review to go in, right? Yup. Thanks, /Erik > Thanks, Roman > > >> Hi Roman, >> >> Definitely looks better. So the static GCArgumentProcessor has a singleton GCCollectedHeapFactory that is created during GCArgumentProcessor::initialize(). >> That makes sense to me, but I think it can be even better. >> >> First off - sorry for bikeshedding a bit here. >> >> To save you from having to update the code due to my bikeshedding, I have prepared a patch to show you what I meant, and you can tell me whether you agree or not. >> >> Full webrev: >> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >> >> Incremental over yours: >> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.02_03/ >> >> My changes are the following: >> >> I collapsed the all-static GCArgumentProcessor and the CollectedHeapFactory into the same class called GCArguments (the natural GC helper for Arguments). It's a singleton object that deals with GC arguments stuff. >> >> GCArguments has some static functions to create_instance(), creating the singleton GCArguments object, and after you have created the singleton instance, you access it through GCArguments::instance(), and it will point to a GC-specific GCArguments, e.g. G1Arguments, CMSArguments, etc. >> >> The main reason why I think it's better to have a GC-specific GCArguments class compared to a CollectedHeapFactory class, is that I would like to eventually move all arguments processing into the GCArguments class, regardless of where (before or after CollectedHeap exists) in the boot strapping process. For example, today some argument stuff is dumped in the arguments.cpp file, and other stuff is dumped in CollectorPolicy::initialize_flags which is called later when the heap exists where it seemingly reads values out from flags, into internal variables, then changes them to what they should be and write the values back to the flags, and subsequently asserts that the internal and flag variants have the same value. Seems to me like we could in a subsequent refactoring move some of that kind of stuff into GCArguments instead, so that the argument setting logic is contained in the same class rather than split all over the place. But that seems like a separate RFE. >> >> Other than that, noticed some precompile header include was missing, the GCArgumentProcessor::initialize() function returned jint but it always returned JNI_OK, and called fatal() if something went wrong, instead of logging an error and returning JNI_ERR. And now that these types are collapsed, I moved initialize_flags_global into the abstract initial_flags, allowing GCs to override it completely if necessary, but in practice having them now call the super initial_flags(). So yeah, went ahead and fixed that for you. >> >> As for your subsequent patch creating the heap, that will just be a virtual call into the specific GCArguments class. >> >> I hope you agree with my proposal. It seemed easier for me to provide a patch describing what I'm thinking than to write it down in text. >> >> Thanks, >> /Erik >> >>> On 2017-10-23 17:19, Roman Kennke wrote: >>> Hi Erik, >>> >>> thanks for your suggestions. So I renamed 'GC' -> 'CollectedHeapFactory' (and subclasses likewise) and 'GCFactory' to 'GCArgumentProcessor'. This will make even more sense once I added heap creation to CollectedHeapFactory (JDK-8189389). >>> >>> Differential: >>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.02.diff/ >>> >>> Full webrev: >>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.02/ >>> >>> Better? >>> >>> Roman >>> >>>> Hi Roman, >>>> >>>> I'm usually not one to really mind names too much, but I don't believe some bootstrapping code for the GC should hog the name "GC". >>>> >>>> Instead of GC::initialize(), I'm thinking a static GCArgumentProcessor::initialize() and instead of GC::factory(), I am thinking a static CollectedHeapFactory::create(). Or something like that. >>>> >>>> The reason is that I think the name GC is too general for the bootstraping-only problems its code touches. Hope I am making sense. >>>> >>>> Thanks, >>>> /Erik >>>> >>>>> On 2017-10-20 14:25, Roman Kennke wrote: >>>>> Hi Erik, >>>>> >>>>> Yes that is fine. >>>>> >>>>> Keep in mind that I'm planning to put heap creation into that class too. >>>>> >>>>> I was thinking maybe to simply reverse the current naming: name the all-static 'factory factory' 'GC', and thus call GC::initialize() and GC::factory() or such, and the main interface GCFactory or CollectedHeapFactory. What do you think? >>>>> >>>>> Roman >>>>> >>>>> >>>>>> I would like to keep the CollectedHeap as the facade interface to the GC, like it is today, instead of having a new GC class making it two facade interfaces. >>>>>> Of course, the CollectedHeap may only be used as a facade after it has been created. And now we are dealing with code before it was created during the bootstrapping process. >>>>>> >>>>>> So what I would like to have here is a class specifically for that, rather than another GC facade interface. I think this is mostly an exercise of finding a better name for the class you call "GC". Now I am not an expert in coming up with good names myself, but some name that indicates this is being used for flag processing would be good. Perhaps GCArgumentProcessor. The benefit of calling it something like that is that if a GC requires argument processing later in the bootstrapping, possibly after CollectedHeap has been created, it can still be used for this purpose without having to feel weird about it. >>>>>> >>>>>> How would you feel about that? Does it seem to make sense? >>>>>> >>>>>> Thanks, >>>>>> /Erik >>>>>> >>>>>>> On 2017-10-19 21:14, Roman Kennke wrote: >>>>>>>> Am 13.10.2017 um 03:20 schrieb David Holmes: >>>>>>>> Hi Roman, >>>>>>>> >>>>>>>> Not a review as GC folk need to do that. >>>>>>>> >>>>>>>>> On 13/10/2017 5:59 AM, Roman Kennke wrote: >>>>>>>>> I'm posting it to both hotspot-runtime-dev and hotspot-gc-dev because it touches both areas (i.e. the GC interface). >>>>>>>>> >>>>>>>>> Currently, all GC related argument processing is done in arguments.cpp, littering it with #ifdef INCLUDE_ALL_GCS and all sorts of GC specific methods etc. >>>>>>>> >>>>>>>> From a runtime perspective I like all the GC specific ifdefs and settings moved out-of-line of the main argument processing. >>>>>>>> >>>>>>>> From a refactoring perspective I noticed that set_object_alignment() no longer calls set_cms_values(). I presume setting it elsewhere is okay? >>>>>>> I totally forgot to reply to this. >>>>>>> >>>>>>> What's important is that the CMS alignment values are set after set_object_alignment() figured out the object alignment. The patch moves that a little further down the road to the beginning of its GC specific argument processing, but from GC perspective should be the same. I looked thoroughly through the involved code paths and cannot see what could go wrong. >>>>>>> >>>>>>> I need one more review. GC folks? The current webrev is: >>>>>>> http://cr.openjdk.java.net/~rkennke/8189171/webrev.01/ >>>>>>> >>>>>>> Thanks, >>>>>>> Roman >>>>>>> >>>>>> >>>>> >>>> >>> >> > From calvin.cheung at oracle.com Wed Oct 25 19:48:48 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Wed, 25 Oct 2017 12:48:48 -0700 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <59E52F99.8090903@oracle.com> References: <59E52F99.8090903@oracle.com> Message-ID: <59F0EAA0.5020203@oracle.com> I've reworked this fix by using the Unicode version of open (wopen) to handle path name longer than max path with the path prefixed to indicate an extended length path as described in [1]. The Unicode version of stat (wstat) doesn't work well with long path [2]. So FindFirstFileW will be used.The data in WIN32_FIND_DATA returned from FindFirstFileW needs to be transferred to the stat structure since the caller expects a return stat structure and other platforms return a stat structure. In classLoader.cpp, calculate the size of buffer required instead of limiting it to JVM_MAXPATHLEN. In os_windows.cpp, dynamically allocate buffers in os::open and os::stat. updated webrev: http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ It passed hs-tier2 testing using mach5. thanks, Calvin [1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#MAX_PATH [2] https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral On 10/16/17, 3:15 PM, Calvin Cheung wrote: > JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 > > Adding a warning message if the full path or the directory length > exceeds MAX_PATH on windows. > > Example warning messages. > > 1) The full path exceeds MAX_PATH: > > Java HotSpot(TM) 64-Bit Server VM warning: construct full path name > failed: total length 270 exceeds max length of 260 > dir > T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > length 259 > name Hello.class length 11 > > 2) The directory path exceeds MAX_PATH: > > Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: path > length 265 exceeds max length 260 > path > T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx > > webrev: > http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ > > Testing: > JPRT (including the new test) > > thanks, > Calvin From david.holmes at oracle.com Thu Oct 26 02:12:34 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 26 Oct 2017 12:12:34 +1000 Subject: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be exited by java In-Reply-To: References: Message-ID: Filed: https://bugs.openjdk.java.net/browse/JDK-8190187 "JVM options that cause the VM to "do X then exit" are incompatible with the JNI Invocation API" David On 26/10/2017 1:23 AM, Adam Farley8 wrote: > Thanks David :) > > Please let me know what the bug number is when you have time. > > Everyone else: Any volunteers to champion this? > > I'm on IRC 9-5 from monday-friday if anyone wants to discuss it. > > Best Regards > > Adam Farley > > P.S. Here's the summary including David's addition, minus the chevrons: > > 1) Exit(0) (during VM startup) is a big bug because it imitates > successful completion of external cpp code accessing JNI methods. > 2) One solution for (1) is to specify a new return code for JNI. > 3) The supplied code (diff) generates, facilitates, and handles that > return code for the exit(0) scenario: -agentlib:jdwp=help > 4) The exit(0) problem (in general) is worth fixing, however we may > choose not to support the use of this new code in the jdwp example case. > 5) The supplied test confirms that the supplied code works (run via > unzip, and then bash TestStart.sh bin dir>). > 6) To implement this new return code, plus the code that handles it, we > would need to follow the CSR process to modify the JNI spec. > 7) To implement the jdwp scenario fix, if we choose to support it at > all, we would also need to use the CSR process the JVM TI spec. > 8) To address all of the worst instances of exit(0), we would need to > search for exit(0) and raise a bug for each significant one (or group). > 9) To solve (8) in one bug would be a lot of work, arguably too much > work for a single bug. > 10) If the new return code is identified as the appropriate solution to > this problem, we will need to agree on the right name and return code > value. > > Also, here's a shortlist of the main questions I recall being raised > here, plus answers people have given. > > A) What are the potential solutions for the exit(0) problem? > i) New JNI Return Code. > ii) Remove the info-only options, at least via the JNI, and return a > standard error code if they are used. > iii) Remove the info-only options, at least via the JNI, and filter them > out if they are used. > iv) Retain existing behaviour, and document a need for the user to > filter out help options before starting the VM via the JNI. > > B) What are the criteria for the "Best" solution? > i) It must prevent "exit(0)" calls. > ii) It must be proven to work. > iii) It should require minimal (none, ideally) behaviour change from the > java.exe user. > iv) It should allow the external cpp code accessing the JNI to complete > normally, without being prematurely terminated. > > C) Which solutions meet the (B) criteria? > i) Both the "new return code" and the "remove info-only options" > solutions meet the (B) criteria. > > D) Is it right to have any "Do X and then exit." arguments at all, and > (if no) what would be the alternatives? > ? ?i) Given the VM is a loadable shared library the answer should be > No. We should not have any "Do X and then exit" > ? ? arguments. The alternatives would be a mix of: > ? ? ? ?- added Dcmds to "Do X" > ? ? ? ?- added launcher smarts to recognize options that need a Dcmd > and "then exit". > > ? ? ? ?Note from David: > ? ? I find the notion of "help" options problematic for a shared > library. But I don't have a good answer for how else > ? ? to document things. > > ? ? ? ?That all said I think this ship has sailed and we're unlikely to > want to invest the time and effort in trying to clean > ? ? this up this way. > > > > From: David Holmes > To: Adam Farley8 > Cc: Alan Bateman , > core-libs-dev at openjdk.java.net, hotspot-runtime-dev at openjdk.java.net, > thomas.stuefe at gmail.com > Date: 25/10/2017 13:53 > Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be > exited by java > ------------------------------------------------------------------------ > > > > Hi Adam, > > On 18/10/2017 9:17 PM, Adam Farley8 wrote: >> Hi All, >> >> Updated summary, based on the responses: > > I think this is a good summary. Thanks. > > I can file a bug for this, but it will take some work to see how to fit > this into the existing specifications and file CSR requests for those > changes. This won't make 18.3 (or whatever it gets called). You will > need a "champion" to help flesh this out in full and work with you on > those CSR requests. I can't volunteer to do that at this time. > >> 1) Exit(0) (during VM startup) is a big bug because it imitates >> successful completion of external cpp code accessing JNI methods. >> 2) One solution for (1) is to specify a new return code for JNI. >> 3) The supplied code (diff) generates, facilitates, and handles that >> return code for the exit(0) scenario: -agentlib:jdwp=help >> 4) The exit(0) problem (in general) is worth fixing, however we may >> choose not to support the use of this new code in the jdwp example case. >> 5) The supplied test confirms that the supplied code works (run via >> unzip, and then bash TestStart.sh > bin dir>). >> 6) To implement this new return code, plus the code that handles it, we >> would need to follow the CSR process to modify the JNI spec. >> 7) To implement the jdwp scenario fix, if we choose to support it at >> all, we would also need to use the CSR process the JVM TI spec. >> 8) To address all of the worst instances of exit(0), we would need to >> search for exit(0) and raise a bug for each significant one (or group). >> 9) To solve (8) in one bug would be a lot of work, arguably too much >> work for a single bug. >> 10) If the new return code is identified as the appropriate solution to >> this problem, we will need to agree on the right name and return code >> value. >> >> Also, here's a shortlist of the main questions I recall being raised >> here, plus answers people have given. >> >> A) What are the potential solutions for the exit(0) problem? >> ?? ? i) New JNI Return Code. >> ?? ? ii) Remove the info-only options, at least via the JNI, and return >> a standard error code if they are used. >> ?? ? iii) Remove the info-only options, at least via the JNI, and filter >> them out if they are used. >> ?? ? iv) Retain existing behaviour, and document a need for the user to >> filter out help options before starting the VM via the JNI. >> >> B) What are the criteria for the "Best" solution? >> ?? ? i) It must prevent "exit(0)" calls. >> ?? ? ii) It must be proven to work. >> ?? ? iii) It should require minimal (none, ideally) behaviour change >> from the java.exe user. >> ?? ? iv) It should allow the external cpp code accessing the JNI to >> complete normally, without being prematurely terminated. >> >> C) Which solutions meet the (B) criteria? >> ?? ? i) Both the "new return code" and the "remove info-only options" >> solutions meet the (B) criteria. >> >> D) Is it right to have any "Do X and then exit." arguments at all, and >> (if no) what would be the alternatives? >> ?? ?i) ? > > Given the VM is a loadable shared library the answer should be No. We > should not have any "Do X and then exit" arguments. The alternatives > would be a mix of: > - added Dcmds to "Do X" > - added launcher smarts to recognize options that need a Dcmd and "then > exit". > > I find the notion of "help" options problematic for a shared library. > But I don't have a good answer for how else to document things. > > That all said I think this ship has sailed and we're unlikely to want to > invest the time and effort in trying to clean this up this way. > > Thanks, > David > >> Best Regards >> >> Adam Farley >> >> P.S. As per Alan and David's emails, the exit(#) references have been >> removed entirely, as discussing them alongside the original exit(0) >> problem risks scope creep. >> >> This bug, if raised, should only cover the exit(0) cases. I believe we >> have consensus here. >> >> >> >> From: David Holmes >> To: Adam Farley8 , Alan Bateman >> , core-libs-dev at openjdk.java.net, >> hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com >> Date: 13/10/2017 14:31 >> Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be >> exited by java >> ------------------------------------------------------------------------ >> >> >> >> Hi Adam, >> >> On 13/10/2017 10:16 PM, Adam Farley8 wrote: >> ?> Hi All, >> ?> >> ?> Here's a summary of the email below (which is intended, partly, as a >> ?> summary of the emails before it). >> ?> >> ?> Let me know if you agree/disagree with any of these points. >> ?> >> ?> 1) Exit(#) during vm startup is a bug because it should return a code >> ?> regardless of the state of the VM. >> >> Yes it's a bug but not one that is likely to be addressed in any >> foreseeable timeframe. There are simply too many "exit on error" paths. >> If we were to start using C++ exceptions within the VM that might >> provide a way to quickly get back to the CreateJavaVM routine where we >> could return an error code - but that is itself a major project that has >> barely even been discussed AFAIK. (Compiler folk have talked about it >> because compiler paths are fairly self-contained - though that was >> before Graal and AOT.) >> >> ?> 2) Exit(0) is an *especially* big bug because it imitates successful >> ?> completion of external cpp code accessing JNI methods. >> ?> 3) One solution is to specify a new return code for JNI. >> >> A solution for 2) yes. >> >> ?> 4) The supplied code (diff) generates, facilitates, and handles that >> ?> return code for the exit(0) scenario: -agentlib:jdwp=help >> ?> 5) The supplied test confirms that the supplied code works (run via >> ?> unzip, and then bash TestStart.sh > ?> bin dir>). >> ?> 6) To implement this new return code, plus the code that handles it, we >> ?> would need to follow the CSR process. >> ?> 7) To implement the fix for the scenario used as an example of the new >> ?> return code's use, we would need to modify the JVM TI spec. >> >> Yes you have demonstrated a potential solution for the agent case. The >> question is, is it the right solution? Is it a worthwhile solution? (As >> I've said I'd prefer not to have any "do something then exit" VM >> arguments.) And can we make it fit into the existing specs without >> contorting things too much. >> >> I still think it easier/preferable for whatever loads the VM to filter >> out the VM args that trigger this behaviour. I mean if you pass >> -Xshare:dump you don't have any right to expect a functioning VM after >> JNI_CreateJavaVM returns - at least I don't think so. Just don't do it. >> Yes it is an imperfection in the invocation API, but life isn't perfect. >> >> ?> 8) To address all of the worst instances of exit(#), we would need to >> ?> search for exit(#) and raise a bug for each significant one (or group). >> ?> 9) To solve (8) in one bug would be a lot of work, arguably too much >> ?> work for a single bug. >> >> This is simply impractical. You may be able to pick off a few >> low-hanging cases, but that won't really make any practical difference. >> >> ?> 10) If the new return code is chosen as the appropriate solution to this >> ?> problem, we may need to choose a better name for the return code. >> ?> >> ?> Is this a fair assessment of the current state of the debate? >> >> It's a fair summary of your position and proposal. >> >> Cheers, >> David >> >> ?> I'm on IRC every weekday from 9am-5pm (4pm fridays) BST (GMT+1) if >> ?> anyone wants to discuss this in real-time on the openjdk channel. >> ?> >> ?> Best Regards >> ?> >> ?> Adam Farley >> ?> >> ?> >> ?> >> ?> -- Previous Email -- >> ?> >> ?> Hi David, Alan, >> ?> >> ?> You are right in that the changes to HotSpot would be nontrivial. >> ?> I see a number of places in (e.g.) arguments.cpp that seem to >> ?> exit in the same manner as Xlog (such as -Xinternalversion). >> ?> >> ?> I would advise ploughing through the CSR process to alter the >> ?> JNI spec, and simultaneously identify some key paths that can >> ?> be raised as bugs. That way, when people have time to address >> ?> these issues, the mechanism to handle a silent exit is already >> ?> in place. >> ?> >> ?> The JDWP fix can be raised separately as one of these bugs, if >> ?> it would make things simpler. >> ?> >> ?> As for the name, JNI_SILENT_EXIT is a placeholder, and can be >> ?> readily changed. Do you have any suggestions? >> ?> >> ?> Lastly, in an ideal world, the VM initialisation should never exit(#). It >> ?> should return a return code that tells the caller something, pass or >> ?> fail, messy or tidy. That way, if someone is using the JNI as part of >> ?> something bigger (like a database or a web server), one of these >> ?> scenarios is just a bug, rather than a world-ender like exit(#). >> ?> >> ?> And now for the individual messages. :) >> ?> >> ?> David: Having help data returned by the launcher seems like a >> ?> good way to avoid exit(0) calls, but I'm not sure how we'd prevent >> ?> a JNI-caller using those options. Ultimately, to be sure, we'd have >> ?> to remove the logic for those options, centralise the data to better >> ?> enable launcher access, and add some logic in there so it can find >> ?> any other help data (e.g. from the jdwp agent library). I feel this would >> ?> be a bigger task than adding the new return code and changing the >> ?> vm, plus it wouldn't provide for any non-help scenarios where the >> ?> vm wants to shut down without error during initialisation. >> ?> >> ?> Alan: I should mention that the silent exit solution is already in >> ?> use in the OpenJ9 VM. Not all of the exit paths have been >> ?> resolved, but many have. >> ?> >> ?> The code is open and can be found here: >> >> ?> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_eclipse_openj9&d=DwIC-g&c=jf_iaSHvJObTbx-siA1ZOg&r=P5m8KWUXJf-CeVJc0hDGD9AQ2LkcXDC0PMV9ntVw5Ho&m=c_McLiDSKtlzF_gNlWcAvBTNbDqyGHyW325GY3_3QkU&s=0QVowxRejNRAXI0Vv5whKQFWGTO36XVICmPYCG8EqIU&e= >> ?> >> ?> And though the silent exit code is disabled for the time being, it >> ?> can be re-enabled by entering this class: >> ?> >> ?> runtime/vm/jvminit.c >> ?> >> ?> and altering line 2343 ( ctrl-f for exit(1) if it's not there). >> ?> >> ?> I won't paste the full code here in case people are concerned >> ?> about contamination, but I would assert that this code (and the >> ?> associated vm files) prove that the concept is possible. >> ?> >> ?> Note that that code should not be enabled until after we've >> ?> integrated the code that can handle a silent exit. >> ?> >> ?> Best Regards >> ?> >> ?> Adam Farley >> ?> >> ?> P.S. Thank you both for your efforts on this. :) >> ?> >> ?> >> ?> >> ?> From: David Holmes >> ?> To: Alan Bateman , Adam Farley8 >> ?> , core-libs-dev at openjdk.java.net, >> ?> hotspot-runtime-dev at openjdk.java.net, thomas.stuefe at gmail.com >> ?> Date: 15/09/2017 12:03 >> ?> Subject: Re: [BUG PROPOSAL]: C++ code that calls JNI_CreateJavaVM can be >> ?> exited by java >> ?> ------------------------------------------------------------------------ >> ?> >> ?> >> ?> >> ?> >> ?> >> ?> On 15/09/2017 8:17 PM, Alan Bateman wrote: >> ?> ?> On 15/09/2017 02:47, David Holmes wrote: >> ?> ?>> Hi Adam, >> ?> ?>> >> ?> ?>> I am still very much torn over this one. I think the idea of >> ?> ?>> print-and-exit flags for a potentially hosted library like the JVM is >> ?> ?>> just wrong - we should never have done that, but we did. Fixing that >> ?> ?>> by moving the flags to the launcher is far from trivial**. Endorsing >> ?> ?>> and encouraging these sorts of flag by adding JNI support seems to be >> ?> ?>> sending the wrong message. >> ?> ?>> >> ?> ?>> ** I can envisage a "help xxx" Dcmd that can read back the info from >> ?> ?>> the VM. The launcher can send the Dcmd, print the output and >> exit. The >> ?> ?>> launcher would not need to know what the xxx values mean, but would >> ?> ?>> have to intercept the existing ones. >> ?> ?>> >> ?> ?>> Another option is just to be aware of these flags (are there more >> than >> ?> ?>> jdwp and Xlog?) and deal with them specially in your custom >> launcher - >> ?> ?>> either filter them out and ignore them, or else launch the VM in its >> ?> ?>> own process to respond to them. >> ?> ?>> >> ?> ?>> Any changes to the JNI specification need to go through the CSR >> process. >> ?> ?> Yes, it would require an update to the JNI spec, also a change to the >> ?> ?> JVM TI spec where Agent_OnLoad returning a non-0 value is specified to >> ?> ?> terminates the VM. The name and value needs discussion too, esp. >> as the >> ?> ?> JNI spec uses negative values for failure. >> ?> ?> >> ?> ?> In any case, I'm also torn over this one as it's a corner case that is >> ?> ?> only interesting for custom launchers that load agents with >> options that >> ?> ?> print usage messages. It wouldn't be hard to have the Agent_OnLoad >> ?> ?> specify a printf hook that the agent could use for output although >> there >> ?> ?> are complications with agents such as JDWP that also announce their >> ?> ?> transport end point. Beyond that there is still the issue of the >> custom >> ?> ?> launcher that would need to know to destroy the VM without >> reporting an >> ?> ?> error. >> ?> ?> >> ?> ?> So what happened to the more meaty part to this which is fixing the >> ?> ?> various cases in HotSpot that terminate the process during >> ?> ?> initialization? I would expect some progress could be made on those >> ?> ?> cases while trying to decide whether to rev the JNI and JVM TI >> specs to >> ?> ?> cover the help case. >> ?> >> ?> Trying to eliminate the vm_exit_during_initialization paths in hotspot >> ?> is a huge undertaking IMHO. >> ?> >> ?> David >> ?> >> ?> ?> >> ?> ?> -Alan >> ?> >> ?> Unless stated otherwise above: >> ?> IBM United Kingdom Limited - Registered in England and Wales with number >> ?> 741598. >> ?> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire >> PO6 3AU >> >> >> >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From kishor.kharbas at intel.com Thu Oct 26 04:27:52 2017 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Thu, 26 Oct 2017 04:27:52 +0000 Subject: RFR(M): 8171181: Supporting heap allocation on alternative memory devices In-Reply-To: <1508859525.3150.26.camel@oracle.com> References: <1508859525.3150.26.camel@oracle.com> Message-ID: Hi Sangheon, Thomas, Thanks for the review. Here is the latest webrev based on your comments and my replies to both are inline (I copied Sangheon's email here at the top above Thomas's reply) http://cr.openjdk.java.net/~kkharbas/8171181/webrev.10_to_09 http://cr.openjdk.java.net/~kkharbas/8171181/webrev.10 Please let me know your question/comments. Thanks Kishor >Hi Kishor, >On 10/19/2017 08:00 AM, Kharbas, Kishor wrote: >Hi Sangheon, >Thanks for the review and comments. >I created two webrevs: >webrev.07 : Is the rebase of webrev.06 on jdk10 (http://cr.openjdk.java.net/~kkharbas/8171181/webrev.07/) >webrev.08 : Is incremental webrev over 07. (http://cr.openjdk.java.net/~kkharbas/8171181/webrev.08/) >1. I couldn't apply webrev.07 patch cleanly. Could you test? > Just minor nit: 07 seems not only rebase of webrev.06. i.e. there seems to have some changes from my comment of alignments. [Kharbas, Kishor] My bad, I had rebased the patch on jdk10/jdk10. The further ones are on jdk10/hs >2. The description in the JEP uses 'AllocateHeapAt' while the patch uses 'HeapDir'. [Kharbas, Kishor] I changed the code. Thanks >----------------------------- >src/os/posix/vm/os_posix.cpp >159 (void)strcpy(fullname, dir); >- Please use strncpy instead. [Kharbas, Kishor] Done >160 (void)strcat(fullname, name_template); >- Please use strncat instead. [Kharbas, Kishor] Done. >180 ::free(fullname); >- os::free(). [Kharbas, Kishor] Done. >196 ::free(fullname); >- os::free(). [Kharbas, Kishor] Done. >----------------------------- >src/os/windows/vm/os_windows.cpp >2885 char *fullname = (char*)_alloca(strlen(dir) + sizeof(name_template)); >- Missing allocation failure handling. >- Please use _malloca instead. > * This function is deprecated because a more secure version is available; see _malloca. > https://msdn.microsoft.com/en-us/library/wb1s57t5.aspx [Kharbas, Kishor] Done. >2886 (void)strcpy(fullname, dir); >- Please use strncpy instead. [Kharbas, Kishor] Done. >2887 (void)strcat(fullname, name_template); >- Please use strncat instead. [Kharbas, Kishor] Done. >----------------------------- >src/share/vm/runtime/arguments.cpp >3726 } >3727 } >- My previous comment was to add explicit explanation of disabling UseNUMA and UseNUMAInterleaving. >4636 if(!FLAG_IS_DEFAULT(HeapDir)) { >- Previsouly we checked UseNUMA and UseNUMAInterleaving and then disabled those. IMHO, I think previous one seems better with more explanation that we are going to disable those options. i.e. >Similar to os_linux.cpp:4869, a warning message with disabling codes. [Kharbas, Kishor] Based on our discussion offline, I think we agree that it?s OK to disable UseNUMA and leave UseNUMAInterleaving ON since other areas of JVM such as CodeCache can use UseNUMAInterleaving. >4638 } >4639 else if (UseParallelGC || UseParallelOldGC) { >- One line please if this change is still needed. -> } else if { >----------------------------- >src/share/vm/runtime/os.cpp >1678 >- Extra line. [Kharbas, Kishor] Done. >1688 } >1689 else { >-> } else { [Kharbas, Kishor] Done. >My answers are inlined. >In webrev08 I have addressed all the comments, except for ones below: >--------------------------------------- >src/os/aix/vm/os_aix.cpp >2514 char* os::pd_attempt_reserve_memory_at(size_t bytes, char* requested_addr, bool use_SHM) { >- Question. Why os_aix has additional parameter at pd_attemp_reserve_memory_at()? Probably only AIX has shmated memory implementation? > A: Yes that?s correct. >Okay. >-------------------------------------- >137 log_debug(gc, heap, coops)("UseLargePages can't be set with HeapDir option."); >- Is it enough to print log message instead of warning message? i.e. Don't we need something at arguments.cpp:3656, similar to NUMA flags? > A: If don?t disable UseLargePages like UseNUMA because large pages can be used for other allocation such as CodeCache. >I meant to changing log_debug() to log_warning(). >If UseNUMA is enabled, we print warning at arguments.cpp:3725 while UseLargePages is enabled, we print debug message. >In addition, is this enough? Don't we need to force disabling UseLargePages? >What happens if both options are enabled? >------------------------------------ >603 ReservedHeapSpace::ReservedHeapSpace(size_t size, size_t alignment, bool large, const char* backing_fs_for_heap) >- ReservedSpace has '_backing_fd' but the constructor doesn't take it as a parameter and only ReservedHeapSpace uses it. This seems not ideal, couldn't make it better? I know actual logic is at >ReservedSpace so it is not convenient to add _backing_fs_for_heap at ReservedHeapSapce. > A: ReservedHeapSpace depends on few functions in ReservedSpace such as initialize(), release(). So instead of passing it as argument, I set it as a propert of ReservedSpace. >Okay. ----------------------------------- >1663 >- You removed os::attempt_reserve_memory_at() from os.cpp and split into os_posix.cpp and os_windows.cpp. > But I think you should remain os::attempt_reserve_memory_at() at os.cpp and implement os specific stuffs at each os_xxx.cpp files for pd_xxx. Of cource move MemTracker function call as well. > A: I do it this way to reduce the redundant code, If I implement in OS specific files in pd_xxx(), the code to replace reserved mapping with file mapping >(replace_existing_mapping_with_dax_file_mapping()) will be redundant. > Still if you feel I will do the change and see how it looks. >I would prefer to have pd_xxx() even though there's some redundant code. [Kharbas, Kishor] We also had a discussion about this offline, I will address this in the next webrev. >Thanks, >Sangheon > -----Original Message----- > From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] > Sent: Tuesday, October 24, 2017 8:39 AM > To: Kharbas, Kishor ; sangheon.kim > ; 'hotspot-gc-dev at openjdk.java.net' > > Cc: hotspot-runtime-dev at openjdk.java.net > Subject: Re: RFR(M): 8171181: Supporting heap allocation on alternative > memory devices > > Hi Kishor, > > initial review using the webrevs, and the comment thread in hotspot- > runtime-dev (readded to cc because questions were left unanswered there). > > The code also touches runtime code quite a bit, so I would appreciate > comments by the runtime team. > > On Thu, 2017-10-19 at 15:00 +0000, Kharbas, Kishor wrote: > > Hi Sangheon, > > > > Thanks for the review and comments. > > I created two webrevs: > > webrev.07 : Is the rebase of webrev.06 on jdk10??? (http://cr.openjdk > > .java.net/~kkharbas/8171181/webrev.07/) > > webrev.08 : Is incremental webrev over 07.????????????? (http://cr.op > > enjdk.java.net/~kkharbas/8171181/webrev.08/) > > > > Some first comments on the webrev follow: > > - could you please provide both a full and incremental webrev for your > changes? The former help the reviewers late to the party, the incremental > ones the people already working with you. > [Kharbas, Kishor] Yes. I will keep this in mind. > - the patch has not been rebased to jdk10/hs as Sangheon suggested. > Probably you rebased to jdk10/jdk10? > [Kharbas, Kishor] My bad, it was rebased to jdk10/jdk10. New webrev's will be based on jdk10/hs. > os_posix.cpp: > > - os::create_file_for_heap: the malloc allocates one byte too little, creating > an automatic write outside the buffer at the strcat() method. > [Kharbas, Kishor] Done. > - os::create_file_for_heap does not use the size parameter. Please remove. > [Kharbas, Kishor] Done. > - in several places: improve the output of the error messages to have > meaning for the user. Not just "malloc failed"; there were already some > suggestions in the referenced thread at ?http://mail.openjdk.java.net/p > ipermail/hotspot-runtime-dev/2017-March/022733.html . > > Same for the asserts, and particularly for them - they will implicitly terminate > the VM, so e.g. a "fchmod error" is definitely too little. At least error > no/output of os::strerror() etc. would be required for quick diagnosis. > > I read in some of the emails that you intend to let the caller print a good error > message. I fear that the caller might not have the necessary information any > more to do that in a useful way. > [Kharbas, Kishor] Done. Moved macro assert_with_errror() to top of file so I can use in my code. > - in that same thread there has also been the question about the need for > blocking the signals for the process that has gone unanswered. > [Kharbas, Kishor] Let me get back on this. The implementation at pmem.io which I use as reference for file operations does blocks signals. I will check if there is more to it than just a 'good practice'. > - in that same thread there has also been the question about why the code > sets permissions to 0600 manually; this is because of glibc 2.0.6 compatibility. > Glibc 2.06 has been released 1997-12-29. I looked a bit through glibc > requirements for the Oracle JDK, and while I could not find exact > requirements, some posts indicate the need for GLIBC_2.4 at minimum (for > jdk7) which is from 2003. I recommend removing this, it seems unnecessary > and completely outdated. > [Kharbas, Kishor] Agree. Removed it. > - os::malloc should be paired with os::free in > os::create_file_for_heap() (2x) > [Kharbas, Kishor] Done. > - I am not sure whether the methods that deal with mapping memory to a > file should have "dax" in their name. The code seems to be pretty generically > map memory into a file. > [Kharbas, Kishor] I used 'dax' in the name because to get direct mapping to DAX device you need to use 'MAP_SHARED'. That makes this map function different than generic map function. But it?s a minor point. I can change the name if it's still desired./mkstem/ > - I am not sure if the current approach to FS errors after mkstemp is very > nice. Please at least print a meaningful warning message to the log. > [Kharbas, Kishor] Added a warning() message when mkstemp could not create a file. > - please also specify the meaning of the return value for > os::create_file_for_heap() in the documentation. > [Kharbas, Kishor] Added more information in the comment before function's declaration. > - there is a missing "p" in the name of os::reserve_mmaped_memory(). > [Kharbas, Kishor] Done. > - os::util_posix_fallocate(): s/continous/continuous > [Kharbas, Kishor] Done. > - os::map_memory_to_dax_file(): according to man pages, posix_fallocate > does not set errno, but the code checking its return (or > util_posix_fallocate()) expects it to do so. > [Kharbas, Kishor] I removed printing of error string. Can the return error value of posix_fallocate() we used as argument to os::strerror(errno)? > - os::attempt_reserve_memory_at(), in the call to > pd_attempt_reserve_memory_at() for AIX, in the third parameter, please > use the exact name of the parameter in the comment. > [Kharbas, Kishor] Done. > - os::attempt_reserve_memory_at(), the second "if" needs a blank before > the bracket. > [Kharbas, Kishor] Done. > - os::attempt_reserve_memory_at(), in the comment, > s/mmemory/memory > [Kharbas, Kishor] Done. > - in that same method, the #if..#endif block should be aligned like other > similar blocks. > [Kharbas, Kishor] Done. > - in that same method, the "else" should be on the same line as the closing > bracket of the if-body. > [Kharbas, Kishor] Done. > os_windows.cpp: > > os::create_file_for_heap(): > - same bug about length of allocation the _alloca call. Not completely sure > about why use alloca (or not use alloca in os_posix.cpp). > [Kharbas, Kishor] Changed to have uniform implementation. > - this version calls os::native_path(). It seems strange to not do that as well > in the others (although it does nothing). > [Kharbas, Kishor] Added in os_posix.cpp to make it uniform. > - os::reserve_memory_aligned(): an "else" should be moved into the same > line as the closing bracket. > [Kharbas, Kishor] Done. > universe.cpp: > > - Universe::reserve_heap(): I do not think this functionality has anything to > do with compressed oops, so the log message should not have the coops tag. > The log message could/should have more information about the file location. > I also recommend using "Java heap" instead of "Heap" here, as the latter is > ambiguous. > [Kharbas, Kishor] Done. Added more info to log message. > virtualspace.cpp: > > - in failed_to_reserve_as_requested: "else" indentation. Also, the "else { if > (..." could be shortened to "} else if (..." to avoid the extra indentation level. > (2x) > [Kharbas, Kishor] Done. > - ReservedSpace::initialize(): s/upto/up to > [Kharbas, Kishor] Done. > - ReservedSpace::initialize(): remove the coops tag here as well. > Also, the info message might be more similar to the comment above where it > says that large page support is up to the file system, and the flag ignored. > [Kharbas, Kishor] Done. > - the code contains the snippet > > if (_fd_for_heap != -1) { > if (!os::unmap_memory... > } else { > if (!os::release_memory... > } > > at least twice. Maybe this code snippet could be extracted into a new > method. > [Kharbas, Kishor] Defined a new static function unmap_or_release_memory() > - (ignore if you want) ReservedHeapSpace::ReservedHeapSpace, at the end > - maybe the condition is easier to read if it looks like: > > if (_fd_for_heap != -1) { > os::close(_fd_for_heap); > } > > than it is now. [Kharbas, Kishor] I agree. Done. > > - virtualspace.hpp: in the changed constructor signature, please add a > comment about what backing_fs_for_heap actually means (and what > happens if set). > [Kharbas, Kishor] Done. > arguments.cpp: > > - Arguments::finalize_vm_init_args: maybe at the place where the change > adds the warning/info message about NUMA support being specific to the > file system; also add the same warning about UseLargePages that is located > elsewhere. > > Like "UseXXXX may not be supported in some specific file system > implementations." - or just ignore as Andrew said in the other email thread. > > Note that I am not sure that the selected place is the correct place for such > warning about incompatible flags (and maybe UseNUMA/UseLargePages > should be forcibly disabled here? But then again, it does not hurt just having > it enabled?). > > I will let the runtime team comment as a lot of that argument processing > changed in jdk9. > > Maybe Arguments::check_vm_args_consistency() is better. > > There is similar code in Arguments::adjust_after_os()... > [Kharbas, Kishor] In progress. I will get back on these comments in next reply. > os.cpp: > > - os::reserve_memory: "else" indentation > > - os::reserve_memory: I do not follow that comment - it explains the code as > written, not why, and what map_memory_to_dax_file() has to do with AIX > and SHM. It uses an outdated method name, and also s/your/our. > > Probably the whole comment can be removed. > [Kharbas, Kishor] I changed the comment. In the beginning I had called pd_reserve_memory() followed by replace_existing_mapping_with_dax_file_mapping(), but later realized that AIX can allocate from SHM, in which case it's not straight forward to replace the mapping. So I leave this comment to make sure in future I (or someone else) does not ponder over the same thing. > os.hpp: > > - indentation of the new #if defined clause > > - here I may probably be speaking wrongly on behalf of the runtime team, > but os.hpp, as an abstraction of the OS should probably not have platform > specific ifdefs in it. > [Kharbas, Kishor] Change the indentation. I will address abstraction issue in next reply. > Thanks, > Thomas > > > In webrev08 I have addressed all the comments, except for ones below: > > > > --------------------------------------- > > src/os/aix/vm/os_aix.cpp > > > > 2514 char* os::pd_attempt_reserve_memory_at(size_t bytes, char* > > requested_addr, bool use_SHM) { > > - Question. Why os_aix has additional parameter at > > pd_attemp_reserve_memory_at()? Probably only AIX has shmated > memory > > implementation? > > ???????? A: Yes that?s correct. > > > > -------------------------------------- > > 137?????? log_debug(gc, heap, coops)("UseLargePages can't be set with > > HeapDir option."); > > - Is it enough to print log message instead of warning message? i.e. > > Don't we need something at arguments.cpp:3656, similar to NUMA flags? > > ?????? A: If don?t disable UseLargePages like UseNUMA because large > > pages can be used for other allocation such as CodeCache. > > > > ------------------------------------ > > > > 603 ReservedHeapSpace::ReservedHeapSpace(size_t size, size_t > > alignment, bool large, const char* backing_fs_for_heap) > > - ReservedSpace has '_backing_fd' but the constructor doesn't take it > > as a parameter and only ReservedHeapSpace uses it. This seems not > > ideal, couldn't make it better? I know actual logic is at > > ReservedSpace so it is not convenient to add _backing_fs_for_heap at > > ReservedHeapSapce. > > ????? A: ReservedHeapSpace depends on few functions in ReservedSpace > > such as initialize(), release(). So instead of passing it as argument, > > I set it as a propert of ReservedSpace. > > > > ----------------------------------- > > 1663 > > - You removed os::attempt_reserve_memory_at() from os.cpp and split > > into os_posix.cpp and os_windows.cpp. > > ? But I think you should remain os::attempt_reserve_memory_at() at > > os.cpp and implement os specific stuffs at each os_xxx.cpp files for > > pd_xxx. Of cource move MemTracker function call as well. > > ??????? A: I do it this way to reduce the redundant code, If I > > implement in OS specific files in pd_xxx(), the code to replace > > reserved mapping with file mapping > > (replace_existing_mapping_with_dax_file_mapping()) will be redundant. > > ???????????? Still if you feel I will do the change and see how it > > looks. > > > > Regards > > Kishor > > > > > > From: sangheon.kim [mailto:sangheon.kim at oracle.com] > > Sent: Tuesday, September 26, 2017 3:18 PM > > To: Kharbas, Kishor ; 'hotspot-gc-dev at openj > > dk.java.net' > > Subject: Re: RFR(M): 8171181: Supporting heap allocation on > > alternative memory devices > > > > Hi Kishor, > > > > On 07/20/2017 06:34 PM, Kharbas, Kishor wrote: > > I have a new version of this patch at http://cr.openjdk.java.net/~kkh > > arbas/8171181/webrev.06/ > > > > This version has been tested on Windows, Linux, Solaris and Mac OS. I > > could not get access to AIX for testing. > > I used tmpfs to test the functionality. Cases that were tested were. > > 1.?????? Allocation of heap using file mapping when ?XX:HeapDir= > > option is used. > > 2.?????? Creation of nameless temporary file for Heap allocation which > > prevents access to file using its name. > > 3.?????? Correct deletion and freeing up of space allocated for file > > under different exit conditions. > > 4.?????? Error handling when path specified is not present, heap size > > is more than size of file system, etc. > > Overall seems good. > > I tried to list some missing part. > > > > 1. Please rebase with consolidated repository. (jdk10/hs) 2. Build > > failure on Solaris. > > ??? (Sorry no build error log file, as I tested a few weeks ago, it is > > deleted) 3. Have you tested with various heap reserve path using > > HeapBaseMinAddress flag? i.e. to test code path of > > ReservedHeapSpace::try_reserve_heap() and try_reserve_range(). > > 4. How about adding HeapDir allocation success message? e.g. > > gc+heap+coops=info > > 5. Adding JTREG test. Probably we would need to verify this allocation > > is succeeded via #4 added allocation success message. > > 6. CSR (Compatibility & Specification Review). At some point, you need > > to file another CR of 'CSR' type to add a new flag of 'HeapDir'. > > 7. It will be much appreciated if you provide incremental webrev. I > > think 06(this version) vs. 07(rebase version) would be hard to get. > > Probably from 08 version. > > > > Here's my comments. > > ----------------------------- > > src/os/aix/vm/os_aix.cpp > > > > 2514 char* os::pd_attempt_reserve_memory_at(size_t bytes, char* > > requested_addr, bool use_SHM) { > > - Question. Why os_aix has additional parameter at > > pd_attemp_reserve_memory_at()? Probably only AIX has shmated > memory > > implementation? > > > > ----------------------------- > > src/os/posix/vm/os_posix.cpp > > > > 148?? char *fullname = (char*)::malloc(strlen(dir) + > > sizeof(name_template)); > > - Use os::malloc() > > > > 196?? int flags; > > 197 > > 198?? flags = MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS; > > - Combining 196 and 198 seems better to me. > > > > 200???? assert((uintptr_t)requested_addr % os::Linux::page_size() == > > 0, "unaligned address"); > > - Linux dependency on posix file which makes build error on Solaris. > > Probably os::vm_page_size(). > > > > 207?? addr = (char*)::mmap(requested_addr, bytes, PROT_NONE, > > 208???? flags, -1, 0); > > - Missing some spaces? Alignment seems odd to me. > > > > 226???? if (ret == -1) > > - Probably you wanted to add handling code? If not, just return ret. > > > > 252?? if (addr == MAP_FAILED || (base != NULL && addr != base)) { > > 253???? if (addr != MAP_FAILED) { > > 254?????? if (!os::release_memory(addr, size)) { > > 255???????? warning("Could not release memory on unsuccessful file > > mapping"); > > 256?????? } > > 257???? } > > 258???? return NULL; > > 259?? } > > - Splitting MAP_FAILED case and another gives better readability to > > me. But this is your call. > > > > 269 > > - Extra line. > > > > 284?? if (result != NULL && file_desc != -1) { > > 285???? if (replace_existing_mapping_with_dax_file_mapping(result, > > bytes, file_desc) == NULL) { > > 286?????? vm_exit_during_initialization(err_msg("Error in mapping Java > > heap at the given filesystem directory")); > > 287???? } > > 288 > > > MemTracker::record_virtual_memory_reserve_and_commit((address)result > , > > bytes, CALLER_PC); > > 289???? return result; > > 290?? } > > 291?? if (result != NULL) { > > 292???? MemTracker::record_virtual_memory_reserve((address)result, > > bytes, CALLER_PC); > > 293?? } > > - Combining line 284 and 291 seems better to me. > > 284?? if (result != NULL) { > > ??????? if (file_desc != -1) { > > ????????? if (replace_existing_mapping_with_dax_file_mapping(result, > > bytes, file_desc) == NULL) { > > ??????????? vm_exit_during_initialization(err_msg("Error in mapping > > Java heap at the given filesystem directory")); > > ????????? } > > > > > MemTracker::record_virtual_memory_reserve_and_commit((address)result > , > > bytes, CALLER_PC); > > ??????? } else { > > ????????? MemTracker::record_virtual_memory_reserve((address)result, > > bytes, CALLER_PC); > > ??????? } > > ????? } > > ????? return result; > > > > ----------------------------- > > src/os/windows/vm/os_windows.cpp > > 3141 // if 'base' is not NULL, function will return NULL if it cannot > > get 'base' > > - Start with uppercase. > > > > 3142 // > > - This line seems redundant. > > > > 3151?????? vm_exit_during_initialization(err_msg("Could not allocate > > sufficient disk space for heap")); > > - heap -> Java heap (same as line 3153) > > > > 3168?? assert(base != NULL, "base cannot be NULL"); > > - 'base' -> 'Base' or 'Base address'. > > > > 3172 > > - Redundant line. > > > > 3230???? } > > 3231???? else { > > -> } else { > > > > 3278?? return reserve_memory(bytes, requested_addr, 0); > > - Is it correct to use '0' not '-1'? It would be better to explain why > > we use hard-coded value here. > > > > ----------------------------- > > src/share/vm/memory/universe.cpp > > - No comments > > > > ----------------------------- > > src/share/vm/memory/virtualspace.cpp > > - copyright update > > > > 74??????????????????????????????????????????? const size_t size, bool > > special, bool is_file_mapped= false) > > - Need space between 'is_file_mapped' and '='. > > > > 92?????????? fatal("os::release_memory failed"); > > - Typo, 'os::unmap_memory failed'. > > > > 129?? // If there is a backing file directory for this VirtualSpace > > then whether > > - This is not VirtualSpace. Probably just 'space'. > > > > 130?? // large pages are allocated is upto the filesystem the dir > > resides in. > > - 'dir'? Probably 'backing file for Java heap'. > > > > 137?????? log_debug(gc, heap, coops)("UseLargePages can't be set with > > HeapDir option."); > > - Is it enough to print log message instead of warning message? i.e. > > Don't we need something at arguments.cpp:3656, similar to NUMA flags? > > > > 191???????? // unmap_memory will do extra work esp. in Windows > > - esp. -> especially > > > > 282?????? } > > 283?????? else { > > -> } else { > > > > 346?? // If there is a backing file directory for this VirtualSpace > > then whether > > - Again this is not VirtualSpace, so just 'space'. > > > > 352???? if (UseLargePages && (!FLAG_IS_DEFAULT(UseLargePages) || > > 353?????? !FLAG_IS_DEFAULT(LargePageSizeInBytes))) { > > - Wrong alignment at line 353. Consider to make same as line 380. > > > > 603 ReservedHeapSpace::ReservedHeapSpace(size_t size, size_t > > alignment, bool large, const char* backing_fs_for_heap) > > - ReservedSpace has '_backing_fd' but the constructor doesn't take it > > as a parameter and only ReservedHeapSpace uses it. This seems not > > ideal, couldn't make it better? I know actual logic is at > > ReservedSpace so it is not convenient to add _backing_fs_for_heap at > > ReservedHeapSapce. > > > > ----------------------------- > > src/share/vm/memory/virtualspace.hpp > > 40?? int??? _backing_fd; > > - I would prefer to have better name to describe. > > ? e.g. as command-line option name is 'HeapDir', _heap_fd or > > _fd_for_heap(similar to below)? > > > > 115?? ReservedHeapSpace(size_t size, size_t forced_base_alignment, > > bool large, const char* backingFSforHeap = NULL); > > - Snake case. How about 'fs_for_heap' or 'heap_fs'? > > > > ----------------------------- > > src/share/vm/runtime/arguments.cpp > > 3655???? FLAG_SET_CMDLINE(bool, UseNUMA, false); > > - (questions) Don't need to add a warning message for > > UseLargePagesSame=true as commented virtualspace.cpp:137? > > > > ----------------------------- > > src/share/vm/runtime/globals.hpp > > - No comments > > > > ----------------------------- > > src/share/vm/runtime/os.cpp > > > > 1632 > > - Extra line. > > > > 1642?? } > > 1643?? else { > > -> } else { > > > > 1663 > > - You removed os::attempt_reserve_memory_at() from os.cpp and split > > into os_posix.cpp and os_windows.cpp. > > ? But I think you should remain os::attempt_reserve_memory_at() at > > os.cpp and implement os specific stuffs at each os_xxx.cpp files for > > pd_xxx. Of cource move MemTracker function call as well. > > > > ----------------------------- > > src/share/vm/runtime/os.hpp > > > > 349?? // replace existing reserved memory with file mapping > > - Start with uppercase letter. > > > > Thanks, > > Sangheon > > > > > > > > > > - Kishor > > > > From: Kharbas, Kishor > > Sent: Tuesday, July 11, 2017 6:40 PM > > To: 'hotspot-gc-dev at openjdk.java.net' > t> > > Cc: Kharbas, Kishor > > Subject: RFR(M): 8171181: Supporting heap allocation on alternative > > memory devices > > > > Greetings, > > > > I have an updated patch for JEP https://bugs.openjdk.java.net/browse/ > > JDK-8171181 at http://cr.openjdk.java.net/~kkharbas/8171181/webrev.05 > > This patch fixes the bugs pointed earlier and other suggestions to > > make the code less intrusive. > > > > I have also sent this to ?hotspot-runtime-dev? mailing list (included > > below). > > > > I would appreciate comments and feedback. > > > > Thanks > > Kishor > > > > From: Kharbas, Kishor > > Sent: Monday, July 10, 2017 1:53 PM > > To: hotspot-runtime-dev at openjdk.java.net > > Cc: Kharbas, Kishor > > Subject: RFR(M): 8171181: Supporting heap allocation on alternative > > memory devices > > > > Hello all! > > > > I have an updated patch for https://bugs.openjdk.java.net/browse/JDK- > > 8171181 at http://cr.openjdk.java.net/~kkharbas/8171181/webrev.05 > > I have lost the old email chain so had to start a fresh one. The > > archived conversation can be found at - http://mail.openjdk.java.net/ > > pipermail/hotspot-runtime-dev/2017-March/022733.html > > > > 1.?????? I have worked on all the comments and fixed the bugs. Mainly > > bugs fixed are related to sigprocmask() and changed the implementation > > such that ?fd? is not passed all the way down the call stack. Thus > > minimizing function signature changes. > > > > 2.?????? Patch supports all OS?es. Consolidated all Posix compliant > > OS?s implementation in os_posix.cpp. > > > > 3.?????? The patch is tested on Windows and Linux. Working on testing > > it on other OS?es. > > > > Let me know if this version looks clean and correct. > > > > Thanks > > Kishor > > From thomas.stuefe at gmail.com Thu Oct 26 06:54:47 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 26 Oct 2017 08:54:47 +0200 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <59F0EAA0.5020203@oracle.com> References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> Message-ID: Hi Calvin, this is a worthwhile addition, thank you for your work! ========================= src/hotspot/os/windows/os_windows.cpp Could you please use wchar_t instead of WCHAR? --------------- in os::stat(): This cumbersome malloc/strcpy sequence: ! size_t path_len = strlen(path) + 1; + char* pathbuf = (char*)os::malloc(path_len * sizeof(char), mtInternal); + if (pathbuf == NULL) { + errno = ENOMEM; return -1; } os::native_path(strcpy(pathbuf, path)); can be reduced to a simple strdup: char* pathbuf = os::strdup(path, mtInternal); if (pathbuf == NULL) { errno = ENOMEM; return -1; } os::native_path(pathbuf); This also would separate strcpy() from os::native_path() which is a bit unreadable. -------------------- The single-byte path (strdup, ::stat()), together with its free(), can be moved inside the (strlen(path) < MAX_PATH) condition. os::native_path will not change the path length (it better not, as it operates on the input buffer). That avoids having two allocations on the doublebyte path. ----------------------- But seeing that translation to UNC paths is something we might want more often, I would combine allocation and UNC prefix adding to one function like this, to avoid the memove and increase reusability: // Creates an unc path from a single byte path. Return buffer is allocated in C heap and needs to be freed by caller. Returns NULL on error. static whchar_t* create_unc_path(const char* s) { - does s start with \\?\ ? - yes: - os::malloc(strlen(s) + 1) and mbstowcs_s - no: - os::malloc(strlen(s) + 1 + 4), mbstowcs_s to fourth position in string, prefix with L"\\?\" } We also need error checking to mbstowcs_s. Just for cleanliness, I would then wrap deallocation into an own function. static viud destroy_unc_path(whchar_t* s) { os::free(s); } These two functions could be candidates of putting into os::win32 namespace, as this form of ANSI->UNC path translation is quite common - whoever wants to use the xxxxW() functions must do this. ----------------------- FindFirstFileW: I am pretty sure that you can achieve the same result with GetFileAttributesExW(). It also returns WIN32_FIND_DATAW. It is way more straightforward to use than FindFirstFileW, as it does not require you to write a callback hook. ------- eval_find_data(): This is more of a generic helper function, could you rename this to something clearer, e.g. make_double_word() ? Also, setup_stat(), could this be renamed more clearly to something like WIN32_FIND_DATA_to_stat? or lowercase if this bothers you :) ================================== src/hotspot/share/classfile/classLoader.cpp In ClassPathDirEntry::open_stream(), I would feel better if we were asserting _dir and name to be != NULL before feeding it to strlen. =================== Thanks! Thomas On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung wrote: > I've reworked this fix by using the Unicode version of open (wopen) to > handle path name longer than max path with the path prefixed to indicate an > extended length path as described in [1]. > > The Unicode version of stat (wstat) doesn't work well with long path [2]. > So FindFirstFileW will be used.The data in WIN32_FIND_DATA returned from > FindFirstFileW needs to be transferred to the stat structure since the > caller expects a return stat structure and other platforms return a stat > structure. > > In classLoader.cpp, calculate the size of buffer required instead of > limiting it to JVM_MAXPATHLEN. > In os_windows.cpp, dynamically allocate buffers in os::open and os::stat. > > updated webrev: http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ > > It passed hs-tier2 testing using mach5. > > thanks, > Calvin > > [1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa3 > 65247(v=vs.85).aspx#MAX_PATH > [2] https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093 > ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral > > > On 10/16/17, 3:15 PM, Calvin Cheung wrote: > >> JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 >> >> Adding a warning message if the full path or the directory length exceeds >> MAX_PATH on windows. >> >> Example warning messages. >> >> 1) The full path exceeds MAX_PATH: >> >> Java HotSpot(TM) 64-Bit Server VM warning: construct full path name >> failed: total length 270 exceeds max length of 260 >> dir T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClas >> s\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> length 259 >> name Hello.class length 11 >> >> 2) The directory path exceeds MAX_PATH: >> >> Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: path length >> 265 exceeds max length 260 >> path T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClas >> s\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >> >> webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >> >> Testing: >> JPRT (including the new test) >> >> thanks, >> Calvin >> > From per.liden at oracle.com Thu Oct 26 09:43:53 2017 From: per.liden at oracle.com (Per Liden) Date: Thu, 26 Oct 2017 11:43:53 +0200 Subject: RFR: 8189171: Move GC argument processing into GC specific classes In-Reply-To: References: <5a7fecea-1a1b-16db-e366-4dd8d0c9ed63@redhat.com> <59E9E9A9.7020306@oracle.com> <511327b5-4632-0e27-6914-2a3d076a1dc3@redhat.com> <59EA002D.1050605@oracle.com> <2b37fed8-0383-eb19-3e45-076aabd74f22@redhat.com> <59F0AD40.3020307@oracle.com> Message-ID: Hi, On 2017-10-25 18:11, Erik Osterlund wrote: [...] >> So let me just put your changes up for review (again), if you don't mind: >> >> Full webrev: >> http://cr.openjdk.java.net/~eosterlund/8189171/webrev.03/ >> I like this. Just two naming suggestions: src/hotspot/share/gc/shared/gcArguments.hpp: 39 static jint create_instance(); 40 static bool is_initialized(); 41 static GCArguments* instance(); Could we call these: 39 static jint initialize(); 40 static bool is_initialized(); 41 static GCArguments* arguments(); Reasoning: initialize() maps better to its companion is_initialized(). GCArguments::arguments() maps better to the existing patterns we have with CollectedHeap::heap(). cheers, Per From thomas.stuefe at gmail.com Thu Oct 26 12:53:05 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 26 Oct 2017 14:53:05 +0200 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> Message-ID: Hi Zhengyu, one more thing, your output does not seem to be included in the final NMT report when the VM shuts down, could this be added too? Thanks! Thomas On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe wrote: > Hi Zhengyu, > > On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu wrote: > >> Hi Thomas, >> >> On 10/24/2017 12:08 PM, Thomas St?fe wrote: >> >>> Hi Zhengyu, >>> >>> - VM_PrintMetadata still has the _unit member, but I see it nowhere used. >>> >>> - I would have split the summary between class and non-class area, that >>> may have been more useful. But this is a matter of taste, I leave it up to >>> you. >>> >>> Agree, should go little further. >> >> Summary: >> Total class loaders= 754 capacity= 67528.00KB used= 61216.88KB >> free= 9453.11KB waste= 38.73KB >> Metadata capacity= 58780.00KB used= 54053.45KB >> free= 4726.55KB waste= 36.38KB >> Class data capacity= 8748.00KB used= 7163.43KB >> free= 1584.57KB waste= 2.35KB >> >> For anonymous classes= 580 capacity= 2732.00KB used= 1065.06KB >> free= 1666.94KB waste= 16.27KB >> Metadata capacity= 2152.00KB used= 738.46KB >> free= 1413.54KB waste= 16.27KB >> Class data capacity= 580.00KB used= 326.60KB >> free= 253.40KB waste= 0.00KB >> >> >> > Nice, thanks :) > > >> Updated webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >> >> > All is well. Note that you could reduce the footprint of your change > somewhat by defining a structure like this: > > struct numbers_t { size_t cap; size_t used; size_t free; size_t waste; } > > and replacing the many members in PrintCLDMetaspaceInfoClosure with > instances of this structure: > > numbers_t total; > numbers_t total_class; > numbers_t total_anon; > numbers_t total_anon_class; > > Depending on how far you go, your code could deflate quite a bit. Give the > structure a default ctor and your large initializer list goes away; give it > a printing function and you reduce print-summary() to four function calls; > with something like an numbers_t::add(size_t cap, size_t used, size_t free, > size_t waste) you can reduce the additions in PrintCLDMetaspaceInfoClosure::print_metaspace() > to four lines. > > Just a suggestion, I leave it up to you if you do this. > > Lines 3108 and 3129 miss each a space before BytesPerWord. Change looks > fine otherwise. > > Thanks, Thomas > > > Thanks, >> >> -Zhengyu >> >> >> All else looks fine to me now. I do not need another review. >>> >>> Thanks for your work, this will come in handy! >>> >>> ..Thomas >>> >>> On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu >> zgu at redhat.com>> wrote: >>> >>> Hi Thomas, >>> >>> >>> Not sure whether we misunderstood each other. I was thinking of >>> something in the line of: >>> >>> <<<< >>> .... >>> ClassLoader: jdk/internal/reflect/DelegatingClassLoader >>> Metadata: >>> capacity: 5.00KB used: 3.32KB free: 1.68KB >>> waste: 0.00KB >>> Class data: >>> capacity: 1.00KB used: 0.54KB free: 0.46KB >>> waste: 0.00KB >>> >>> ClassLoader: for anonymous class >>> Metadata: >>> capacity: 1.00KB used: 1.00KB free: 0.00KB >>> waste: 0.00KB >>> Class data: >>> capacity: 1.00KB used: 0.64KB free: 0.36KB >>> waste: 0.00KB >>> .... >>> >>> Summary: >>> XX class loaders encountered, total capacity: xx, total waste: >>> xx. >>> >>> Peak usage by class loader: xxxx >>> >>>> >>> >>> Added summary lines: >>> >>> Total class loaders= 56 capacity= 6378.00KB used= >>> 6205.08KB free= 172.92KB waste= 1.44KB >>> For anonymous classes= 54 capacity= 212.00KB used= 96.95KB >>> free= 115.05KB waste= 0.94KB >>> >>> >>> >>> >>> These numbers would not be interpretation, just a convenience to >>> the reader to get a clear picture. >>> >>> It even may be useful to separate the output between basic and >>> detailed mode, and in basic mode omit all the single class >>> loaders and just print the summary line. >>> >>> Updated webrev: >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >>> >>> >> gu/8189688/webrev.01/index.html >>> > >>> >>> >>> >>> metaspace.hpp: >>> >>> I would have preferred to keep scale_unit file local static >>> instead of exposing it via MetaspaceAux, because it does not >>> really have anything to do with metaspace. >>> >>> Fixed >>> >>> >>> Otherwise ok. >>> >>> --- >>> >>> metaspace.cpp >>> >>> - ChunkManager::print_statistics(): thanks for reusing this >>> function! >>> -> Scale can only be ever 1, K, M, G, yes? So, could you >>> add an assert at the start of the function, and a comment at the >>> prototype or function head? >>> -> Also, now that ChunkManager::print_statistics() takes a >>> scale, could you please change the printout to use floats like >>> you did in PrintCLDMetaspaceInfoClosure::print_metaspace()? >>> >>> Done. >>> >>> >>> Chunkmanager (non-class): >>> 0 specialized (128 bytes) chunks, total 0.00KB >>> 0 small (512 bytes) chunks, total 0.00KB >>> 0 medium (8192 bytes) chunks, total 0.00KB >>> 0 humongous chunks, total 0.00KB >>> total size: 0.00KB. >>> Chunkmanager (class): >>> 0 specialized (128 bytes) chunks, total 0.00KB >>> 0 small (256 bytes) chunks, total 0.00KB >>> 0 medium (4096 bytes) chunks, total 0.00KB >>> 0 humongous chunks, total 0.00KB >>> total size: 0.00KB. >>> >>> >>> - I am concerned about races in >>> PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >>> Maybe my understanding is limited. We bail if cld->is_unloading. >>> But nothing prevents a ClassLoaderDataGraph from starting to >>> unload a ClassLoaderData and tearing down the Metaspace after we >>> started printing it in another thread, right? Also, I do not see >>> any fences. >>> >>> So, I wonder whether PrintCLDMetaspaceInfoClosure should run in >>> lock protection. And then, I wonder if it would be good to >>> separate data gathering and printing. We print to a (at least in >>> principle) unknown output stream, which may be doing slow File >>> IO or even Network IO. Doing this under lock protection seems >>> not a good idea (but I see other places where this happens too). >>> >>> For an example, see ChunkManager::get_statistic() vs >>> ChunkManager::print_statistic(). Could you do the same, have a >>> data gathering step where you collect your >>> quadrupel in a variable sized list of >>> structures in memory, and print it out in a separate step? I >>> realize though that the effort would be larger than for what we >>> did in ChunkManager::get_statistic(), where we only fill a >>> structure. >>> >>> >>> Unlike other NMT queries, this query is executed at a safepoint by >>> VMThread, so it should be okay. >>> >>> Updated webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >>> >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> ------ >>> >>> vm_operations.hpp >>> >>> - VM_PrintMetadata : do we still need _unit? >>> >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> >>> Thanks! >>> >>> Thomas >>> >>> >>> >>> On 10/23/2017 11:08 AM, Thomas St?fe wrote: >>> >>> >>> >>> On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu >>> >>> > >>> >>> >>> >>> wrote: >>> >>> >>> >>> Okay. So this is important for understanding >>> cases >>> where we have >>> a large number of miniscule class loaders, >>> each one >>> only having >>> a few numbers of chunks. In this case, would >>> it be >>> useful to >>> have a running total of "free" and "waste", >>> too? Or >>> even better, >>> also average and peak values of "free" and >>> "waste" (to tell >>> apart cases where you have one outlier vs >>> cases where >>> just all >>> metaspaces waste a lot of memory). >>> Just out of curiosity, I looked at >>> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >>> >>> >> gu/cld_metaspace/wildfly.txt >>> > >>> >> gu/cld_metaspace/wildfly.txt >>> >>> >> gu/cld_metaspace/wildfly.txt >>> >> >>> and >>> it seems that "free" is the problem, not >>> "waste"? So I >>> guess >>> what happens is that all the little >>> classloaders >>> allocate just >>> enough memory to push them over >>> "_small_chunk_limit=4", >>> so they >>> allocate the first medium chunk, from which >>> then they >>> take a >>> small bite and leave the rest laying around? >>> >>> Yes. The free space of anonymous classes should be >>> counted >>> as waste, >>> in my opinion. I am not sure if everyone agrees, >>> so I took the >>> summary line out of this patch. >>> >>> I would be more than happy to restore the summary >>> line, if >>> you find >>> it is useful :-) >>> >>> >>> I agree with you. Typically class loaders stop loading >>> at some >>> point, especially these tine ones, and often will not >>> resume >>> loading. So, effectivly, the space is wasted. It still >>> helps to >>> distinguish "free" in the current chunks from "free" in >>> the >>> other chunks to tell this situation apart from a >>> situation where >>> we have just a bug in metaspace chunk handling >>> preventing us to >>> use up our chunks. >>> >>> >>> The latter would be an important >>> addition, >>> regardless >>> if this is >>> for customers or for us. Chunks >>> idling in >>> freelists can >>> make a >>> huge impact, and unfortunately this >>> may happen >>> in perfectly >>> valid cases. See e.g. our JEP >>> " >>> https://bugs.openjdk.java.net/browse/JDK-8166690 >>> >>> >> > >>> >> /browse/JDK-8166690 >>> >>> >> >> >>> < >>> https://bugs.openjdk.java.net/browse/JDK-8166690 >>> >>> >> > >>> >> /browse/JDK-8166690 >>> >>> >> >>>". We have >>> already a printing routine for free >>> chunks, in >>> ChunkManager::print_all_chunkmanagers(). >>> Could >>> you add >>> this to >>> your output? >>> >>> >>> Make sense! Could you recommend what data >>> and >>> format you >>> would like >>> to see? >>> >>> >>> >>> Would not >>> ChunkManager::print_all_chunkmanagers() be >>> sufficient? >>> >>> Okay, I might need to tweak output format. >>> >>> >>> Fine by me. Nothing depends on it yet. >>> >>> Thanks, Thomas >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> >>> ------------- >>> >>> Other than above notes, the changes >>> seem >>> straighforward, I did >>> not see anything wrong. Minor nits: >>> >>> - PrintCLDMetaspaceInfoClosure::_out, >>> _scale >>> and _unit >>> could all >>> be constant (with _out being a >>> outputStream* >>> const). >>> - We also do not need to provide >>> scale *and* >>> unit as >>> argument, >>> one can be deduced from the other. >>> This would >>> also prevent >>> mismatches.We do need scale here, >>> since nmt >>> command >>> line can set >>> scale and we do >>> >>> have to honor that. >>> >>> However, passing unit is ugly! I don't >>> want to >>> introduce >>> inverse >>> dependence (metaspace -> nmt), which is >>> also ugly. >>> >>> >>> Yes, that would be regrettable. >>> >>> Probably should move scale mapping code >>> from >>> NMTUtil to a >>> common util? >>> >>> >>> >>> That would be possible, these functions >>> (scale_from_name etc) >>> are simple enough to be moved to a generic >>> file. >>> However, if you >>> pass scale, deducing the string that goes with >>> it is >>> trivial and >>> I would have no qualms doing this by hand in >>> metaspace.cpp. But >>> it is a matter of taste. >>> >>> >>> I did not look closely at locking >>> issues, I assume >>> MetaspaceAux::print_metadata() is >>> supposed to >>> run only in >>> Safepoints? >>> >>> >>> No. sum_xxx_xxx_in_use queries are >>> guarded by locks. >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> Thanks, Thomas >>> >>> >>> >>> Thanks & Kind Regards, Thomas >>> >>> >>> >>> >>> >>> >>> On Fri, Oct 20, 2017 at 4:00 PM, >>> Zhengyu Gu >>> >>> > >>> >>> >> >>> >> >> > >>> >>> >>> >>> >>> > >>> >>> >> >>> >>> >> >> > >>> >>> >>>>> wrote: >>> >>> Up to now, there is little >>> visibility >>> into how >>> metaspaces >>> are used, >>> and by whom. >>> >>> This proposed enhancement gives >>> NMT the >>> ability to >>> report >>> metaspace >>> usages on per-classloader level, >>> via a >>> new NMT command >>> "metadata" >>> >>> >>> For example: >>> >>> 2496: >>> Metaspaces: >>> Metadata space: reserved= >>> 8192KB >>> committed= 5888KB >>> Class space: reserved= >>> 1048576KB >>> committed= 640KB >>> >>> Per-classloader metadata: >>> >>> ClassLoader: for anonymous class >>> Metadata capacity= >>> 5.00KB >>> used= 1.16KB >>> free= 3.84KB waste= >>> 0.05KB >>> Class data capacity= >>> 1.00KB >>> used= 0.59KB >>> free= 0.41KB waste= >>> 0.00KB >>> >>> .... >>> >>> ClassLoader: >>> Metadata capacity= >>> 5640.00KB >>> used= 5607.42KB >>> free= 32.58KB waste= >>> 0.46KB >>> Class data capacity= >>> 520.00KB >>> used= 500.94KB >>> free= 19.06KB waste= >>> 0.02KB >>> >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8189688 >>> >>> >> > >>> >> /browse/JDK-8189688 >>> >>> >> >> >>> < >>> https://bugs.openjdk.java.net/browse/JDK-8189688 >>> >>> >> > >>> >> /browse/JDK-8189688 >>> >>> >> >>> >>> < >>> https://bugs.openjdk.java.net/browse/JDK-8189688 >>> >>> >> > >>> >> /browse/JDK-8189688 >>> >>> >> >> >>> < >>> https://bugs.openjdk.java.net/browse/JDK-8189688 >>> >>> >> > >>> >> /browse/JDK-8189688 >>> >>> >> >>>> >>> Webrev: >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >>> >>> >> gu/8189688/webrev.00/index.html >>> > >>> >> gu/8189688/webrev.00/index.html >>> >>> >> gu/8189688/webrev.00/index.html >>> >> >>> < >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >>> >>> >> gu/8189688/webrev.00/index.html >>> > >>> >> gu/8189688/webrev.00/index.html >>> >>> >> gu/8189688/webrev.00/index.html >>> >> >>>> >>> < >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >>> >>> >> gu/8189688/webrev.00/index.html >>> > >>> >> gu/8189688/webrev.00/index.html >>> >>> >> gu/8189688/webrev.00/index.html >>> >> >>> < >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >>> >>> >> gu/8189688/webrev.00/index.html >>> > >>> >> gu/8189688/webrev.00/index.html >>> >>> >> gu/8189688/webrev.00/index.html >>> >> >>>>> >>> >>> Test: >>> >>> hotspot_tier1_runtime, >>> fastdebug and >>> release on >>> x64 Linux. >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> >>> >>> >>> >>> >>> > From thomas.stuefe at gmail.com Thu Oct 26 13:37:54 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 26 Oct 2017 15:37:54 +0200 Subject: Register output in assert case useful? Message-ID: Hi all, when an assert happens, one does not have an ucontext_t and hence no register context - that is why hs-err files generated by an assert do not contain registers. But sometimes there are cases where it would be useful to know the content of registers at assert - e.g. to know the current stack depth, or because remnants in registers may help with investigating. At SAP we added a small feature to retrieve the current context when an assert happens and make that part of the hs-err file. Works similar to the SafeFetch logic: when an assert happens, we trigger a segfault and in the signal handler we squirrel the ucontext away, before continuing with error reporting. We do this right after the assert condition was evaluated, so this is as close to the assert point as possible (basically, just a read access which is part of the assert macro). I wonder whether there is interest in us contributing this? Kind Regards, Thomas From dean.long at oracle.com Thu Oct 26 19:00:33 2017 From: dean.long at oracle.com (dean.long at oracle.com) Date: Thu, 26 Oct 2017 12:00:33 -0700 Subject: Register output in assert case useful? In-Reply-To: References: Message-ID: <33bf3a42-7d70-d33a-868b-0e46c5dcdb90@oracle.com> On 10/26/17 6:37 AM, Thomas St?fe wrote: > Hi all, > > when an assert happens, one does not have an ucontext_t and hence no > register context - that is why hs-err files generated by an assert do not > contain registers. > > But sometimes there are cases where it would be useful to know the content > of registers at assert - e.g. to know the current stack depth, or because > remnants in registers may help with investigating. > > At SAP we added a small feature to retrieve the current context when an > assert happens and make that part of the hs-err file. Works similar to the > SafeFetch logic: when an assert happens, we trigger a segfault and in the > signal handler we squirrel the ucontext away, before continuing with error > reporting. We do this right after the assert condition was evaluated, so > this is as close to the assert point as possible (basically, just a read > access which is part of the assert macro). > > I wonder whether there is interest in us contributing this? +1 dl > Kind Regards, Thomas From vladimir.kozlov at oracle.com Thu Oct 26 19:25:41 2017 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Thu, 26 Oct 2017 12:25:41 -0700 Subject: Register output in assert case useful? In-Reply-To: References: Message-ID: Yes, please. We try to print values for failed asserts but not in all places. To have values in registers would be good. Do you also dump "Instructions:" near assert code? One thing to watch out is when debugging code (in debugger) there is ability to continue execution when assert is triggered: http://hg.openjdk.java.net/jdk10/hs/file/068d316e905e/src/hotspot/share/utilities/debug.cpp#l209 This should be preserved. Vladimir On 10/26/17 6:37 AM, Thomas St?fe wrote: > Hi all, > > when an assert happens, one does not have an ucontext_t and hence no > register context - that is why hs-err files generated by an assert do not > contain registers. > > But sometimes there are cases where it would be useful to know the content > of registers at assert - e.g. to know the current stack depth, or because > remnants in registers may help with investigating. > > At SAP we added a small feature to retrieve the current context when an > assert happens and make that part of the hs-err file. Works similar to the > SafeFetch logic: when an assert happens, we trigger a segfault and in the > signal handler we squirrel the ucontext away, before continuing with error > reporting. We do this right after the assert condition was evaluated, so > this is as close to the assert point as possible (basically, just a read > access which is part of the assert macro). > > I wonder whether there is interest in us contributing this? > > Kind Regards, Thomas > From ioi.lam at oracle.com Thu Oct 26 23:26:49 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 26 Oct 2017 16:26:49 -0700 Subject: RFR [XS] Subclasses of jdk.jfr.Event loaded from CDS breaks -XX:FlightRecorderOptions=retransform=false Message-ID: <5395e40f-e660-4e3c-5b8e-1522f0dbd78b@oracle.com> Please review the follow change: ??? https://bugs.openjdk.java.net/browse/JDK-8190191 http://cr.openjdk.java.net/~iklam/jdk10/8190191-jfr-event-retransform-false.v01.open/ Background: When -XX:FlightRecorderOptions=retransform=false is given in the command-line, subclasses of jdk.jfr.Event are instrumented at load time with information that's specific to the current JVM lifetime. As a result, we cannot perform such instrumentation at CDS dump time. A more complicated CDS solution would load these classes from the archive, and re-do the runtime instrumentation. However, there are only a very small number of these classes. The performance benefit of archiving them does not justify the extra complication. Hence, in this fix, we just identify these classes and exclude them from the CDS archive during run time. (Because JFR is still a closed feature, the test cases are in the closed repo ...) Thanks - Ioi From ioi.lam at oracle.com Thu Oct 26 23:27:57 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 26 Oct 2017 16:27:57 -0700 Subject: RFR [S] Subclasses of jdk.jfr.Event loaded from CDS breaks -XX:FlightRecorderOptions=retransform=false (closed repo) In-Reply-To: <5395e40f-e660-4e3c-5b8e-1522f0dbd78b@oracle.com> References: <5395e40f-e660-4e3c-5b8e-1522f0dbd78b@oracle.com> Message-ID: <8fb73755-ca88-4426-bd36-12d1af4f96c8@oracle.com> Hi, Here are the closed test cases: http://ioilinux.us.oracle.com/webrev/jdk10/8190191-jfr-event-retransform-false.v01.closed/ Thanks - Ioi On 10/26/17 4:26 PM, Ioi Lam wrote: > Please review the follow change: > > ??? https://bugs.openjdk.java.net/browse/JDK-8190191 > http://cr.openjdk.java.net/~iklam/jdk10/8190191-jfr-event-retransform-false.v01.open/ > > > Background: > > When -XX:FlightRecorderOptions=retransform=false is given in the > command-line, > subclasses of jdk.jfr.Event are instrumented at load time with > information that's > specific to the current JVM lifetime. As a result, we cannot perform > such instrumentation at CDS dump time. > > A more complicated CDS solution would load these classes from the > archive, > and re-do the runtime instrumentation. However, there are only a very > small number of these classes. The performance benefit of archiving them > does not justify the extra complication. > > Hence, in this fix, we just identify these classes and exclude them > from the CDS archive during run time. > > (Because JFR is still a closed feature, the test cases are in the > closed repo ...) > > Thanks > - Ioi > From calvin.cheung at oracle.com Fri Oct 27 00:03:02 2017 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 26 Oct 2017 17:03:02 -0700 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> Message-ID: <59F277B6.4010801@oracle.com> Hi Thomas, On 10/25/17, 11:54 PM, Thomas St?fe wrote: > Hi Calvin, > > this is a worthwhile addition, thank you for your work! Thanks for your review. > > ========================= > > src/hotspot/os/windows/os_windows.cpp > > Could you please use wchar_t instead of WCHAR? Fixed. > > --------------- > in os::stat(): > > This cumbersome malloc/strcpy sequence: > > ! size_t path_len = strlen(path) + 1; > + char* pathbuf = (char*)os::malloc(path_len * sizeof(char), > mtInternal); > + if (pathbuf == NULL) { > + errno = ENOMEM; > return -1; > } > os::native_path(strcpy(pathbuf, path)); > > can be reduced to a simple strdup: > > char* pathbuf = os::strdup(path, mtInternal); > if (pathbuf == NULL) { > errno = ENOMEM; > return -1; > } > os::native_path(pathbuf); > > This also would separate strcpy() from os::native_path() which is a > bit unreadable. I've made the change. > > -------------------- > > The single-byte path (strdup, ::stat()), together with its free(), can > be moved inside the (strlen(path) < MAX_PATH) condition. > os::native_path will not change the path length (it better not, as it > operates on the input buffer). That avoids having two allocations on > the doublebyte path. os::native_path() converts a path like C:\\\\somedir to C:\\somedir So I'll need the converted path in the wchar_t case too. The freeing of the pathbuf is being done at the end of the function or in the middle of the wchar_t case if there's an error. > > ----------------------- > > But seeing that translation to UNC paths is something we might want > more often, I would combine allocation and UNC prefix adding to one > function like this, to avoid the memove and increase reusability: > > // Creates an unc path from a single byte path. Return buffer is > allocated in C heap and needs to be freed by caller. Returns NULL on > error. > static whchar_t* create_unc_path(const char* s) { > - does s start with \\?\ ? > - yes: > - os::malloc(strlen(s) + 1) and mbstowcs_s > - no: > - os::malloc(strlen(s) + 1 + 4), mbstowcs_s to fourth > position in string, prefix with L"\\?\" > } I also include the case for adding L"\\\\?\\UNC\0" at the beginning to be consistent with libjava/canonicalize_md.c. > > We also need error checking to mbstowcs_s. I've added assert like the following after the call: assert(converted_chars == path_len, "sanity"); > > Just for cleanliness, I would then wrap deallocation into an own function. > > static viud destroy_unc_path(whchar_t* s) { os::free(s); } I've added the destroy function. > > These two functions could be candidates of putting into os::win32 > namespace, as this form of ANSI->UNC path translation is quite common > - whoever wants to use the xxxxW() functions must do this. I'm leaving them in os_windows.cpp since they're being used only within that file. > > ----------------------- > > FindFirstFileW: > > I am pretty sure that you can achieve the same result with > GetFileAttributesExW(). It also returns WIN32_FIND_DATAW. It actually returns WIN32_FILE_ATTRIBUTE_DATA and is very similar to WIN32_FIND_DATAW. > > It is way more straightforward to use than FindFirstFileW, as it does > not require you to write a callback hook. I've switched to using GetFileAttributesExW(). > > ------- > > eval_find_data(): This is more of a generic helper function, could you > rename this to something clearer, e.g. make_double_word() ? Ok. I've renamed it. > > Also, setup_stat(), could this be renamed more clearly to something > like WIN32_FIND_DATA_to_stat? or lowercase if this bothers you :) I'm naming the function as file_attribute_data_to_stat() to match with the data structure name. > > ================================== > src/hotspot/share/classfile/classLoader.cpp > > In ClassPathDirEntry::open_stream(), I would feel better if we were > asserting _dir and name to be != NULL before feeding it to strlen. I've added an assert statement. > > =================== Here's an updated webrev: http://cr.openjdk.java.net/~ccheung/8188122/webrev.02/ thanks, Calvin > > Thanks! > > Thomas > > > > > > > On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung > > wrote: > > I've reworked this fix by using the Unicode version of open > (wopen) to handle path name longer than max path with the path > prefixed to indicate an extended length path as described in [1]. > > The Unicode version of stat (wstat) doesn't work well with long > path [2]. So FindFirstFileW will be used.The data in > WIN32_FIND_DATA returned from FindFirstFileW needs to be > transferred to the stat structure since the caller expects a > return stat structure and other platforms return a stat structure. > > In classLoader.cpp, calculate the size of buffer required instead > of limiting it to JVM_MAXPATHLEN. > In os_windows.cpp, dynamically allocate buffers in os::open and > os::stat. > > updated webrev: > http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ > > > It passed hs-tier2 testing using mach5. > > thanks, > Calvin > > [1] > https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#MAX_PATH > > > [2] > https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral > > > > > On 10/16/17, 3:15 PM, Calvin Cheung wrote: > > JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 > > > Adding a warning message if the full path or the directory > length exceeds MAX_PATH on windows. > > Example warning messages. > > 1) The full path exceeds MAX_PATH: > > Java HotSpot(TM) 64-Bit Server VM warning: construct full path > name failed: total length 270 exceeds max length of 260 > dir > T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > length 259 > name Hello.class length 11 > > 2) The directory path exceeds MAX_PATH: > > Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: > path length 265 exceeds max length 260 > path > T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClass\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx > > webrev: > http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ > > > Testing: > JPRT (including the new test) > > thanks, > Calvin > > From jiangli.zhou at Oracle.COM Fri Oct 27 01:00:28 2017 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Thu, 26 Oct 2017 18:00:28 -0700 Subject: RFR [XS] Subclasses of jdk.jfr.Event loaded from CDS breaks -XX:FlightRecorderOptions=retransform=false In-Reply-To: <5395e40f-e660-4e3c-5b8e-1522f0dbd78b@oracle.com> References: <5395e40f-e660-4e3c-5b8e-1522f0dbd78b@oracle.com> Message-ID: <986EB008-32B1-4325-BE61-CC50CD93F746@oracle.com> Hi Ioi, SystemDictionary::reorder_dictionary_for_sharing() and Dictionary::reorder_dictionary_for_sharing() are only used for CDS code. Could you please add CDS_ONLY() to the function definitions and put the implementation under #if INCLUDE_CDS. Thanks, Jiangli > On Oct 26, 2017, at 4:26 PM, Ioi Lam wrote: > > Please review the follow change: > > https://bugs.openjdk.java.net/browse/JDK-8190191 > http://cr.openjdk.java.net/~iklam/jdk10/8190191-jfr-event-retransform-false.v01.open/ > > Background: > > When -XX:FlightRecorderOptions=retransform=false is given in the command-line, > subclasses of jdk.jfr.Event are instrumented at load time with information that's > specific to the current JVM lifetime. As a result, we cannot perform > such instrumentation at CDS dump time. > > A more complicated CDS solution would load these classes from the archive, > and re-do the runtime instrumentation. However, there are only a very > small number of these classes. The performance benefit of archiving them > does not justify the extra complication. > > Hence, in this fix, we just identify these classes and exclude them > from the CDS archive during run time. > > (Because JFR is still a closed feature, the test cases are in the closed repo ...) > > Thanks > - Ioi > From serguei.spitsyn at oracle.com Fri Oct 27 06:02:24 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 26 Oct 2017 23:02:24 -0700 Subject: RFR [XS] Subclasses of jdk.jfr.Event loaded from CDS breaks -XX:FlightRecorderOptions=retransform=false In-Reply-To: <5395e40f-e660-4e3c-5b8e-1522f0dbd78b@oracle.com> References: <5395e40f-e660-4e3c-5b8e-1522f0dbd78b@oracle.com> Message-ID: <3a190e6a-d599-a473-1447-3a714681ac29@oracle.com> Hi Ioi, It looks good modulo the comment from Jiangli. Thanks, Serguei On 10/26/17 16:26, Ioi Lam wrote: > Please review the follow change: > > ??? https://bugs.openjdk.java.net/browse/JDK-8190191 > http://cr.openjdk.java.net/~iklam/jdk10/8190191-jfr-event-retransform-false.v01.open/ > > > Background: > > When -XX:FlightRecorderOptions=retransform=false is given in the > command-line, > subclasses of jdk.jfr.Event are instrumented at load time with > information that's > specific to the current JVM lifetime. As a result, we cannot perform > such instrumentation at CDS dump time. > > A more complicated CDS solution would load these classes from the > archive, > and re-do the runtime instrumentation. However, there are only a very > small number of these classes. The performance benefit of archiving them > does not justify the extra complication. > > Hence, in this fix, we just identify these classes and exclude them > from the CDS archive during run time. > > (Because JFR is still a closed feature, the test cases are in the > closed repo ...) > > Thanks > - Ioi > From thomas.stuefe at gmail.com Fri Oct 27 06:27:32 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 27 Oct 2017 08:27:32 +0200 Subject: Register output in assert case useful? In-Reply-To: References: Message-ID: Hi Vladimir, On Thu, Oct 26, 2017 at 9:25 PM, Vladimir Kozlov wrote: > Yes, please. > > Nice, I'll do this then. One change less to maintain downstream. > We try to print values for failed asserts but not in all places. To have > values in registers would be good. Do you also dump "Instructions:" near > assert code? > > Yes. We go into the error handler with a valid ucontext, so now all reporting steps depending on a valid uncontext. The only thing missing is the signal info, because there was no crash signal. > One thing to watch out is when debugging code (in debugger) there is > ability to continue execution when assert is triggered: > > http://hg.openjdk.java.net/jdk10/hs/file/068d316e905e/src/ > hotspot/share/utilities/debug.cpp#l209 > > This should be preserved. It is. We go the same code paths to process the assert, we only take a short break before to store the ucontext. > > > Vladimir > > Cheers, Thomas > On 10/26/17 6:37 AM, Thomas St?fe wrote: > >> Hi all, >> >> when an assert happens, one does not have an ucontext_t and hence no >> register context - that is why hs-err files generated by an assert do not >> contain registers. >> >> But sometimes there are cases where it would be useful to know the content >> of registers at assert - e.g. to know the current stack depth, or because >> remnants in registers may help with investigating. >> >> At SAP we added a small feature to retrieve the current context when an >> assert happens and make that part of the hs-err file. Works similar to the >> SafeFetch logic: when an assert happens, we trigger a segfault and in the >> signal handler we squirrel the ucontext away, before continuing with error >> reporting. We do this right after the assert condition was evaluated, so >> this is as close to the assert point as possible (basically, just a read >> access which is part of the assert macro). >> >> I wonder whether there is interest in us contributing this? >> >> Kind Regards, Thomas >> >> From thomas.stuefe at gmail.com Fri Oct 27 07:55:10 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 27 Oct 2017 09:55:10 +0200 Subject: RFR(S): 8188122: Path length limits on Windows leads to obscure class loading failures In-Reply-To: <59F277B6.4010801@oracle.com> References: <59E52F99.8090903@oracle.com> <59F0EAA0.5020203@oracle.com> <59F277B6.4010801@oracle.com> Message-ID: Hi Calvin, On Fri, Oct 27, 2017 at 2:03 AM, Calvin Cheung wrote: > Hi Thomas, > > On 10/25/17, 11:54 PM, Thomas St?fe wrote: > >> Hi Calvin, >> >> this is a worthwhile addition, thank you for your work! >> > Thanks for your review. Thanks for your work :) First off, there is another issue in file_attribute_data_to_stat(). We actually had this issue before, but your work uncovered it: According to POSIX ( http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html) and every unix manpage I looked at (solaris, linux, aix..), st_ctime is not the file creation time but the last time file status was changed. Only Microsoft gets this wrong in their posix layer, its stat() function returns (https://msdn.microsoft.com/en-us/library/14h5k7ff.aspx) creation time in st_ctime. But as os::stat() is a platform independent layer, I'd say we should behave the same on all platforms, and that would be the Posix way. I did not find any code calling os::stat() and using st_ctime, so this is still save to change for windows. (Note that I found that perfMemory_xxx.cpp uses plain OS ::stat and misuses ctime as "creation time" - I opened a bug for that ( https://bugs.openjdk.java.net/browse/JDK-8190260 - but that does not affect us, as they do not call os::stat()). There is one more complication: in os::stat() we use plain ::stat() in the single byte case.: *+ if (strlen(path) < MAX_PATH) {* *+ ret = ::stat(pathbuf, sbuf);**+ } else {* But ::stat() on Windows is broken, as I wrote above. We should not use it, especially not if we change the meaing of st_ctime in the double byte path. My suggestion would be to just always call GetFileAttributes(), also for the single byte path. A very simple solution would be to just always go the double byte path with UNC translation, regardless of the path length. Or, if you are worried about performance, for paths shorter than MAX_PATH we use GetFileAttributesA(), for longer paths GetFileAttributesW with UNC translation. In both cases you get a WIN32_FILE_ATTRIBUTE_DATA which you can feed tp your file_attribute_data_to_stat() routine and have the struct stat you return be always consistent. But onward: > >> ========================= >> >> src/hotspot/os/windows/os_windows.cpp >> >> Could you please use wchar_t instead of WCHAR? >> > Fixed. > >> >> --------------- >> in os::stat(): >> >> This cumbersome malloc/strcpy sequence: >> >> ! size_t path_len = strlen(path) + 1; >> + char* pathbuf = (char*)os::malloc(path_len * sizeof(char), >> mtInternal); >> + if (pathbuf == NULL) { >> + errno = ENOMEM; >> return -1; >> } >> os::native_path(strcpy(pathbuf, path)); >> >> can be reduced to a simple strdup: >> >> char* pathbuf = os::strdup(path, mtInternal); >> if (pathbuf == NULL) { >> errno = ENOMEM; >> return -1; >> } >> os::native_path(pathbuf); >> >> This also would separate strcpy() from os::native_path() which is a bit >> unreadable. >> > I've made the change. > >> >> -------------------- >> >> The single-byte path (strdup, ::stat()), together with its free(), can be >> moved inside the (strlen(path) < MAX_PATH) condition. os::native_path will >> not change the path length (it better not, as it operates on the input >> buffer). That avoids having two allocations on the doublebyte path. >> > > os::native_path() converts a path like C:\\\\somedir to C:\\somedir > So I'll need the converted path in the wchar_t case too. The freeing of > the pathbuf is being done at the end of the function or in the middle of > the wchar_t case if there's an error. Oh, you are right,of course. I missed that it this is needed for both paths. > > >> ----------------------- >> >> But seeing that translation to UNC paths is something we might want more >> often, I would combine allocation and UNC prefix adding to one function >> like this, to avoid the memove and increase reusability: >> >> // Creates an unc path from a single byte path. Return buffer is >> allocated in C heap and needs to be freed by caller. Returns NULL on error. >> static whchar_t* create_unc_path(const char* s) { >> - does s start with \\?\ ? >> - yes: >> - os::malloc(strlen(s) + 1) and mbstowcs_s >> - no: >> - os::malloc(strlen(s) + 1 + 4), mbstowcs_s to fourth >> position in string, prefix with L"\\?\" >> } >> > > I also include the case for adding L"\\\\?\\UNC\0" at the beginning to be > consistent with libjava/canonicalize_md.c. >> We also need error checking to mbstowcs_s. >> > I've added assert like the following after the call: > > assert(converted_chars == path_len, "sanity"); create_unc_path() : - could you convert the /**/ to // comments, please? - not sure about the assert after mbstowcs_s: if we happen to encounter an invalid multibyte character, function will fail and return an error: https://msdn.microsoft.com/en-us/library/eyktyxsx.aspx "If mbstowcs_s encounters an invalid multibyte character, it puts 0 in *``pReturnValue, sets the destination buffer to an empty string, sets errno to EILSEQ, and returns EILSEQ." As this is dependent on user data, we should not assert, but handle the return code of mbstowcs_s gracefully. Same goes for libjava/canonicalize_md.c. - Here: ::mbstowcs_s(&converted_chars, &wpath[7], path_len + 7, path, path_len); third parameter is wrong, as we hand in an offset into the buffer, we must decrement the buffer size by this offset, so correct would be path_len +7 - 7 or just path_len. - Same error below: + ::mbstowcs_s(&converted_chars, &wpath[4], path_len + 4, path, path_len); > > >> Just for cleanliness, I would then wrap deallocation into an own function. >> >> static viud destroy_unc_path(whchar_t* s) { os::free(s); } >> > I've added the destroy function. > >> >> These two functions could be candidates of putting into os::win32 >> namespace, as this form of ANSI->UNC path translation is quite common - >> whoever wants to use the xxxxW() functions must do this. >> > I'm leaving them in os_windows.cpp since they're being used only within > that file. Fine by me. > > >> ----------------------- >> >> FindFirstFileW: >> >> I am pretty sure that you can achieve the same result with >> GetFileAttributesExW(). It also returns WIN32_FIND_DATAW. >> > It actually returns WIN32_FILE_ATTRIBUTE_DATA and is very similar to > WIN32_FIND_DATAW. > >> >> It is way more straightforward to use than FindFirstFileW, as it does not >> require you to write a callback hook. >> > I've switched to using GetFileAttributesExW(). Thank you, this is way more readable. Another issue: do we still need the fix for 6539723 if we switch from stat() to GetFileAttributesExW()? This fix looks important, but the comment indicates that it could break things if the original bug is not present. Btw, this is another strong argument for scrapping ::stat() altogether on all code paths, not only for long input paths, because stat() and GetFileAttributesExW() may behave differently. So it would be good to use the same API on all code paths, in order to get the best test coverage. > >> ------- >> >> eval_find_data(): This is more of a generic helper function, could you >> rename this to something clearer, e.g. make_double_word() ? >> > Ok. I've renamed it. > >> >> Also, setup_stat(), could this be renamed more clearly to something like >> WIN32_FIND_DATA_to_stat? or lowercase if this bothers you :) >> > I'm naming the function as file_attribute_data_to_stat() to match with the > data structure name. Thanks for taking my suggestions. > > >> ================================== >> src/hotspot/share/classfile/classLoader.cpp >> >> In ClassPathDirEntry::open_stream(), I would feel better if we were >> asserting _dir and name to be != NULL before feeding it to strlen. >> > I've added an assert statement. > >> >> =================== >> > Here's an updated webrev: > http://cr.openjdk.java.net/~ccheung/8188122/webrev.02/ > > Thanks! Thomas > thanks, > Calvin > > >> Thanks! >> >> Thomas >> >> >> >> >> >> >> On Wed, Oct 25, 2017 at 9:48 PM, Calvin Cheung > > wrote: >> >> I've reworked this fix by using the Unicode version of open >> (wopen) to handle path name longer than max path with the path >> prefixed to indicate an extended length path as described in [1]. >> >> The Unicode version of stat (wstat) doesn't work well with long >> path [2]. So FindFirstFileW will be used.The data in >> WIN32_FIND_DATA returned from FindFirstFileW needs to be >> transferred to the stat structure since the caller expects a >> return stat structure and other platforms return a stat structure. >> >> In classLoader.cpp, calculate the size of buffer required instead >> of limiting it to JVM_MAXPATHLEN. >> In os_windows.cpp, dynamically allocate buffers in os::open and >> os::stat. >> >> updated webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.01/ >> >> >> It passed hs-tier2 testing using mach5. >> >> thanks, >> Calvin >> >> [1] >> https://msdn.microsoft.com/en-us/library/windows/desktop/aa3 >> 65247(v=vs.85).aspx#MAX_PATH >> > 365247%28v=vs.85%29.aspx#MAX_PATH> >> >> [2] >> https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c093 >> ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral >> > 3ea9-f0aa-446d-b648-2dabe8480430/stat-and-long-names?forum=vcgeneral> >> >> >> >> On 10/16/17, 3:15 PM, Calvin Cheung wrote: >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8188122 >> >> >> Adding a warning message if the full path or the directory >> length exceeds MAX_PATH on windows. >> >> Example warning messages. >> >> 1) The full path exceeds MAX_PATH: >> >> Java HotSpot(TM) 64-Bit Server VM warning: construct full path >> name failed: total length 270 exceeds max length of 260 >> dir >> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClas >> s\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> length 259 >> name Hello.class length 11 >> >> 2) The directory path exceeds MAX_PATH: >> >> Java HotSpot(TM) 64-Bit Server VM warning: os::stat failed: >> path length 265 exceeds max length 260 >> path >> T:\\testoutput\\jtreg\\JTwork\\classes\\2\\runtime\\LoadClas >> s\\LongPath.d\\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >> xxxxxxxxxxxxxxxxxxxxxxxxxxxx\\xxxxx >> >> webrev: >> http://cr.openjdk.java.net/~ccheung/8188122/webrev.00/ >> >> >> Testing: >> JPRT (including the new test) >> >> thanks, >> Calvin >> >> >> From serguei.spitsyn at oracle.com Fri Oct 27 08:52:54 2017 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 27 Oct 2017 01:52:54 -0700 Subject: RFR: SA: JDK-8189798: SA cleanup - part 1 In-Reply-To: References: <18501902-23db-de6c-b83d-640cd33df836@oracle.com> Message-ID: <691d8166-5395-906a-4256-ef0ab2e2773a@oracle.com> Hi Jini, The fix looks good to me. Thanks, Serguei On 10/24/17 00:31, Jini George wrote: > Adding hotspot-dev too. > > Thanks, > Jini. > > On 10/24/2017 12:05 PM, Jini George wrote: >> Hello, >> >> As a part of SA next, I am working on writing a test case which >> compares the fields and the types of the fields of the SA java >> classes with the corresponding entries in the vmStructs tables. This, >> to some extent, would help in preventing errors in SA due to the >> changes in hotspot. As a precursor to this, I am in the process of >> making some cleanup related changes (mostly in SA). I plan to have >> the changes done in parts. For this webrev, most of the changes are for: >> >> 1. Avoiding having some values being redefined in SA. Instead have >> those exported through vmStructs, and read it in SA. >> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >> CompactibleFreeListSpace::IndexSetSize) >> >> Redefinition of hotspot values in SA makes SA error prone, when the >> value gets altered in hotspot and the corresponding modification gets >> missed out in SA. >> >> 2. To remove some unused code (JNIid.java). >> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >> 4. Modify variable names in SA and hotspot to match the counterpart >> names, so that the comparison of the fields become easier. Most of >> the changes belong to this group. >> >> Could I please get reviews done for these precursor changes ? >> >> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >> >> Thank you, >> Jini. >> From martin.doerr at sap.com Fri Oct 27 14:23:27 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Fri, 27 Oct 2017 14:23:27 +0000 Subject: RFR(M): 8190285: s390: Some java boolean checks are not correct Message-ID: Hi, the current s390 implementation misses a couple of small fixes which exist on other platforms. Please review: http://cr.openjdk.java.net/~mdoerr/8190285_s390_jbool/webrev.00/ Best regards, Martin From lutz.schmidt at sap.com Fri Oct 27 15:37:35 2017 From: lutz.schmidt at sap.com (Schmidt, Lutz) Date: Fri, 27 Oct 2017 15:37:35 +0000 Subject: RFR(M): 8190285: s390: Some java boolean checks are not correct In-Reply-To: References: Message-ID: Hi Martin, thank you for fixing this. Your changes look good. Best Regards, Lutz On 27.10.2017, 16:23, "Doerr, Martin" > wrote: Hi, the current s390 implementation misses a couple of small fixes which exist on other platforms. Please review: http://cr.openjdk.java.net/~mdoerr/8190285_s390_jbool/webrev.00/ Best regards, Martin From jini.george at oracle.com Fri Oct 27 15:49:45 2017 From: jini.george at oracle.com (Jini George) Date: Fri, 27 Oct 2017 21:19:45 +0530 Subject: RFR: SA: JDK-8189798: SA cleanup - part 1 In-Reply-To: <691d8166-5395-906a-4256-ef0ab2e2773a@oracle.com> References: <18501902-23db-de6c-b83d-640cd33df836@oracle.com> <691d8166-5395-906a-4256-ef0ab2e2773a@oracle.com> Message-ID: Thank you very much, Serguei. -Jini. On 10/27/2017 2:22 PM, serguei.spitsyn at oracle.com wrote: > Hi Jini, > > The fix looks good to me. > > Thanks, > Serguei > > > On 10/24/17 00:31, Jini George wrote: >> Adding hotspot-dev too. >> >> Thanks, >> Jini. >> >> On 10/24/2017 12:05 PM, Jini George wrote: >>> Hello, >>> >>> As a part of SA next, I am working on writing a test case which >>> compares the fields and the types of the fields of the SA java >>> classes with the corresponding entries in the vmStructs tables. This, >>> to some extent, would help in preventing errors in SA due to the >>> changes in hotspot. As a precursor to this, I am in the process of >>> making some cleanup related changes (mostly in SA). I plan to have >>> the changes done in parts. For this webrev, most of the changes are for: >>> >>> 1. Avoiding having some values being redefined in SA. Instead have >>> those exported through vmStructs, and read it in SA. >>> (CompactibleFreeListSpace::_min_chunk_size_in_bytes, >>> CompactibleFreeListSpace::IndexSetSize) >>> >>> Redefinition of hotspot values in SA makes SA error prone, when the >>> value gets altered in hotspot and the corresponding modification gets >>> missed out in SA. >>> >>> 2. To remove some unused code (JNIid.java). >>> 3. Add the missing "CMSBitMap::_bmStartWord" in vmStructs. >>> 4. Modify variable names in SA and hotspot to match the counterpart >>> names, so that the comparison of the fields become easier. Most of >>> the changes belong to this group. >>> >>> Could I please get reviews done for these precursor changes ? >>> >>> JBS Id: https://bugs.openjdk.java.net/browse/JDK-8189798 >>> webrev: http://cr.openjdk.java.net/~jgeorge/8189798/webrev.00/ >>> >>> Thank you, >>> Jini. >>> > From gerard.ziemski at oracle.com Fri Oct 27 18:22:10 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Fri, 27 Oct 2017 13:22:10 -0500 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <58edfce7-4fe5-edbc-2a2d-56a1dcca6ced@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <58edfce7-4fe5-edbc-2a2d-56a1dcca6ced@oracle.com> Message-ID: <7F7EA9A1-6E2F-44DE-991E-C9738A891C47@oracle.com> hi David, Thank you for your detailed review and questions - my answers below. Updated webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev2 > On Oct 11, 2017, at 12:07 AM, David Holmes wrote: > > Hi Gerard, > > On 11/10/2017 6:40 AM, Gerard Ziemski wrote: >> hi all, >> Please review this change that adds dynamic resizing of system dictionaries. > > Sounds good - but there is nothing in the bug report to justify any of the resizing choices - eg resize factor - where are the performance numbers? :) > > In the JDK libraries the policy of always doubling collections when they needed to grow was removed because as the collection gets really big, doubling its size becomes a major problem. What kind of sizes are we likely to be dealing with? This is not as much about improving performance of general apps, but more about uncommon cases for big apps (i.e. those loading 40,000 classes) > Given we don't currently resize, when will we now see resizing happening? ie what apps will be affected by this and what performance hit might they take when the resizing occurs? Resizing will take place at a safepoint when the system dictionary?s underlaying hash table load factor is too high (currently it?s 5, i.e. we will resize when we reach an average of 5 entries per a bucket) > > IIUC resizing is done lazily if we hit the right thresholds when a safepoint happens. I'm unclear what kind of margins we have to ensure we are not likely to run out of space before the next safepoint might occur? As Karen explained, we can run out of memory, not entries - if that happens, we keep system dictionaries at their old sizes. > >> The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock > > So basically anyone calling hash_to_index must now hold the SD lock when doing that call. In most places it is a simple matter of moving the call to a locked region later in the method; but in SD::resolve_instance_class_or_null you added a new locked region - I am concerned this may introduce a synchronization bottleneck. I ran various performance benchmarks (including specjvm via Aurora) and there were no regressions. > >> A few notes: >> - dynamic resizing is off when either dumping or using shared spaces >> - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) > > I don't think it makes sense to make a flag that restores the old behaviour, experimental. This should be a product flag IMHO. This flag is different to PredictedLoadedClassCount which provided an unsupported (aka experimental) means to influence the initial table size. I heard a different argument calling for no flag at all. IMHO there should either be no flag - we want resizable system dictionaries after all, and they are ON by default, but since this is a new feature I thought it would be safer to be able to turn it off, if it causes issues for the customers out in the real world. > >> - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) >> bug: https://bugs.openjdk.java.net/browse/JDK-8184765 >> webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 > > src/hotspot/share/classfile/dictionary.cpp > > 109 for (int i=n; i 110 if (i%2 != 0) { Done. > > Style nits: put spaces around all operators please (check all files) > > 123 desired_size = _next_prime((int)(_resize_factor*(double)number_of_entries())); > > You don't need the cast to double; the fact _resize_factor is a double means the calculation will be done as a double. Done. > > --- > > src/hotspot/share/utilities/hashtable.cpp > > 271 HashtableBucket* buckets_new = NEW_C_HEAP_ARRAY2(HashtableBucket, new_size, F, CURRENT_PC); > 272 if (buckets_new == NULL) { > > You're using the NEW_ macro that aborts on OOM not the one that returns NULL. > > BTW what happens when the SD does fill up? Do we abort or just stop loading classes? Done. > > --- > > src/hotspot/share/utilities/hashtable.hpp > > May be worth adding: > > assert_locked_or_safepoint(SystemDictionary_lock); > > to hash_to_index (or just a lock ownwership test if you don't expect this to be done at a safepoint) Other hash tables use it too and may not be calling it with a lock or at a safepoint (yet). > > --- > > test/runtime/LoadClass/TestResize.java > test/runtime/LoadClass/TriggerResize.java > > Shouldn't these be test/hotspot/jtreg/runtime/LoadClass/? ? Fixed, thank you for catching this. > > --- > > TestResize.java > > 24 /* > 25 * @test > 26 * @bug 8184765 > 27 * @summary Test dynamic resizing of SystemDictionary > 28 * @modules java.base/jdk.internal.misc:open > 29 * @modules java.base/java.lang:open > 30 * @library /test/lib > 31 */ > > This isn't actually a test - you never run it directly - so I don't expect to see any jtreg tags here ?? I would expect them all to be on the actual test class. Actually I'm not sure why this needs to be a separate public class in its own file ? Done. I followed the pattern, but also IMHO it?s cool to be able to run the test case on its own. > > Please add comments in load() explaining what all the byte/index stuff is doing - presumably making the class representation unique? Done. cheers From gerard.ziemski at oracle.com Fri Oct 27 18:25:27 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Fri, 27 Oct 2017 13:25:27 -0500 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> Message-ID: hi Coleen, Thank you for the review. Updated webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev2 > On Oct 11, 2017, at 9:10 AM, coleen.phillimore at oracle.com wrote: > > > Gerard, some preliminary comments. > > http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html > > *+{* > *!_MutexLocker mu(SystemDictionary_lock, THREAD_);* > *!_Klass* probe = dictionary->find(_d_hash, name, protection_domain);* > *if (probe != NULL) return probe;* > ** > *+ }* > > I don't think you need this because dictionary->find() should have the NoSafepointVerifier, so the index will not change. Done, good catch. > http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/classLoaderData.cpp.udiff.html > > I think we want a global _some_dictionary_needs_resizing to avoid walking through the CLDG. > > And have Dictionary have a field _resizing_needed to avoid the calculation during the safepoint, which is set when adding an entry. > > _resizing_needed = *(number_of_entries() > (_resize_load_trigger*table_size()); > *CLDG::_any_resizing_needed |= _resizing_needed; // or something like that. > > I can write more about the rationale of this change in the bug report, if needed. Done. > > Thank you for doing this change. > Coleen > > > On 10/10/17 4:40 PM, Gerard Ziemski wrote: >> hi all, >> >> Please review this change that adds dynamic resizing of system dictionaries. >> >> The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock >> >> A few notes: >> >> - dynamic resizing is off when either dumping or using shared spaces >> - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) >> - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8184765 >> webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 >> >> >> cheers >> > From coleen.phillimore at oracle.com Fri Oct 27 18:48:43 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 27 Oct 2017 14:48:43 -0400 Subject: RFR(M): 8190285: s390: Some java boolean checks are not correct In-Reply-To: References: Message-ID: <294b5990-fc2f-ae11-316f-b247a340945b@oracle.com> Hi This looks good, but do you have a test case for the early return case??? Many of the platforms are missing the narrow call. http://cr.openjdk.java.net/~mdoerr/8190285_s390_jbool/webrev.00/src/hotspot/cpu/s390/templateInterpreterGenerator_s390.cpp.udiff.html Thanks, Coleen On 10/27/17 10:23 AM, Doerr, Martin wrote: > Hi, > > the current s390 implementation misses a couple of small fixes which exist on other platforms. > > Please review: > http://cr.openjdk.java.net/~mdoerr/8190285_s390_jbool/webrev.00/ > > Best regards, > Martin > From zgu at redhat.com Fri Oct 27 20:34:30 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Fri, 27 Oct 2017 16:34:30 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> Message-ID: <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> On 10/26/2017 08:53 AM, Thomas St?fe wrote: > Hi Zhengyu, > > one more thing, your output does not seem to be included in the final > NMT report when the VM shuts down, could this be added too? > This turns out to be a tricky one. During VM exit, we are not supposed to request a safepoint, correct? Without a safepoint, it is unsafe to walk class loaders. Thanks, -Zhengyu > Thanks! Thomas > > On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe > wrote: > > Hi Zhengyu, > > On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu > wrote: > > Hi Thomas, > > On 10/24/2017 12:08 PM, Thomas St?fe wrote: > > Hi Zhengyu, > > - VM_PrintMetadata still has the _unit member, but I see it > nowhere used. > > - I would have split the summary between class and non-class > area, that may have been more useful. But this is a matter > of taste, I leave it up to you. > > Agree, should go little further. > > Summary: > Total class loaders= 754 capacity= 67528.00KB used= > 61216.88KB free= 9453.11KB waste= 38.73KB > Metadata capacity= 58780.00KB used= > 54053.45KB free= 4726.55KB waste= 36.38KB > Class data capacity= 8748.00KB used= > 7163.43KB free= 1584.57KB waste= 2.35KB > > For anonymous classes= 580 capacity= 2732.00KB used= > 1065.06KB free= 1666.94KB waste= 16.27KB > Metadata capacity= 2152.00KB used= > 738.46KB free= 1413.54KB waste= 16.27KB > Class data capacity= 580.00KB used= > 326.60KB free= 253.40KB waste= 0.00KB > > > > Nice, thanks :) > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ > > > > All is well. Note that you could reduce the footprint of your change > somewhat by defining a structure like this: > > struct numbers_t { size_t cap; size_t used; size_t free; size_t waste; } > > and replacing the many members in PrintCLDMetaspaceInfoClosure with > instances of this structure: > > numbers_t total; > numbers_t total_class; > numbers_t total_anon; > numbers_t total_anon_class; > Depending on how far you go, your code could deflate quite a bit. > Give the structure a default ctor and your large initializer list > goes away; give it a printing function and you reduce > print-summary() to four function calls; with something like an > numbers_t::add(size_t cap, size_t used, size_t free, size_t waste) > you can reduce the additions in > PrintCLDMetaspaceInfoClosure::print_metaspace() to four lines. > > Just a suggestion, I leave it up to you if you do this. > > Lines 3108 and 3129 miss each a space before BytesPerWord. Change > looks fine otherwise. > > Thanks, Thomas > > > Thanks, > > -Zhengyu > > > All else looks fine to me now. I do not need another review. > > Thanks for your work, this will come in handy! > > ..Thomas > > On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu >> wrote: > > Hi Thomas, > > > Not sure whether we misunderstood each other. I was > thinking of > something in the line of: > > <<<< > .... > ClassLoader: jdk/internal/reflect/DelegatingClassLoader > Metadata: > capacity: 5.00KB used: 3.32KB free: > 1.68KB > waste: 0.00KB > Class data: > capacity: 1.00KB used: 0.54KB free: > 0.46KB > waste: 0.00KB > > ClassLoader: for anonymous class > Metadata: > capacity: 1.00KB used: 1.00KB free: > 0.00KB > waste: 0.00KB > Class data: > capacity: 1.00KB used: 0.64KB free: > 0.36KB > waste: 0.00KB > .... > > Summary: > XX class loaders encountered, total capacity: xx, > total waste: xx. > > Peak usage by class loader: xxxx > >>>> > > Added summary lines: > > Total class loaders= 56 capacity= 6378.00KB > used= 6205.08KB free= 172.92KB waste= 1.44KB > For anonymous classes= 54 capacity= 212.00KB > used= 96.95KB > free= 115.05KB waste= 0.94KB > > > > > These numbers would not be interpretation, just a > convenience to > the reader to get a clear picture. > > It even may be useful to separate the output > between basic and > detailed mode, and in basic mode omit all the > single class > loaders and just print the summary line. > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html > > > > > > > > >> > > > > metaspace.hpp: > > I would have preferred to keep scale_unit file > local static > instead of exposing it via MetaspaceAux, because it > does not > really have anything to do with metaspace. > > Fixed > > > Otherwise ok. > > --- > > metaspace.cpp > > - ChunkManager::print_statistics(): thanks for > reusing this > function! > -> Scale can only be ever 1, K, M, G, yes? > So, could you > add an assert at the start of the function, and a > comment at the > prototype or function head? > -> Also, now that > ChunkManager::print_statistics() takes a > scale, could you please change the printout to use > floats like > you did in > PrintCLDMetaspaceInfoClosure::print_metaspace()? > > Done. > > > Chunkmanager (non-class): > 0 specialized (128 bytes) chunks, total 0.00KB > 0 small (512 bytes) chunks, total 0.00KB > 0 medium (8192 bytes) chunks, total 0.00KB > 0 humongous chunks, total 0.00KB > total size: 0.00KB. > Chunkmanager (class): > 0 specialized (128 bytes) chunks, total 0.00KB > 0 small (256 bytes) chunks, total 0.00KB > 0 medium (4096 bytes) chunks, total 0.00KB > 0 humongous chunks, total 0.00KB > total size: 0.00KB. > > > - I am concerned about races in > > PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). > Maybe my understanding is limited. We bail if > cld->is_unloading. > But nothing prevents a ClassLoaderDataGraph from > starting to > unload a ClassLoaderData and tearing down the > Metaspace after we > started printing it in another thread, right? Also, > I do not see > any fences. > > So, I wonder whether PrintCLDMetaspaceInfoClosure > should run in > lock protection. And then, I wonder if it would be > good to > separate data gathering and printing. We print to a > (at least in > principle) unknown output stream, which may be > doing slow File > IO or even Network IO. Doing this under lock > protection seems > not a good idea (but I see other places where this > happens too). > > For an example, see ChunkManager::get_statistic() vs > ChunkManager::print_statistic(). Could you do the > same, have a > data gathering step where you collect your > quadrupel in a variable > sized list of > structures in memory, and print it out in a > separate step? I > realize though that the effort would be larger than > for what we > did in ChunkManager::get_statistic(), where we only > fill a > structure. > > > Unlike other NMT queries, this query is executed at a > safepoint by > VMThread, so it should be okay. > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ > > > > > Thanks, > > -Zhengyu > > > ------ > > vm_operations.hpp > > - VM_PrintMetadata : do we still need _unit? > > > Thanks, > > -Zhengyu > > > > Thanks! > > Thomas > > > > On 10/23/2017 11:08 AM, Thomas St?fe wrote: > > > > On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu > > > > >> > > > > >>>> wrote: > > > > Okay. So this is important for > understanding cases > where we have > a large number of miniscule class > loaders, > each one > only having > a few numbers of chunks. In this > case, would it be > useful to > have a running total of "free" > and "waste", > too? Or > even better, > also average and peak values of > "free" and > "waste" (to tell > apart cases where you have one > outlier vs > cases where > just all > metaspaces waste a lot of memory). > Just out of curiosity, I looked at > http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt > > > > > > > > >> > > > > > > > > > >>> > and > it seems that "free" is the > problem, not > "waste"? So I > guess > what happens is that all the > little classloaders > allocate just > enough memory to push them over > "_small_chunk_limit=4", > so they > allocate the first medium chunk, > from which > then they > take a > small bite and leave the rest > laying around? > > Yes. The free space of anonymous > classes should be > counted > as waste, > in my opinion. I am not sure if > everyone agrees, > so I took the > summary line out of this patch. > > I would be more than happy to restore > the summary > line, if > you find > it is useful :-) > > > I agree with you. Typically class loaders > stop loading > at some > point, especially these tine ones, and > often will not > resume > loading. So, effectivly, the space is > wasted. It still > helps to > distinguish "free" in the current chunks > from "free" in the > other chunks to tell this situation apart > from a > situation where > we have just a bug in metaspace chunk handling > preventing us to > use up our chunks. > > > The latter would be an > important > addition, > regardless > if this is > for customers or for us. > Chunks idling in > freelists can > make a > huge impact, and > unfortunately this > may happen > in perfectly > valid cases. See e.g. > our JEP > > "https://bugs.openjdk.java.net/browse/JDK-8166690 > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>>>". We have > already a printing > routine for free > chunks, in > > ChunkManager::print_all_chunkmanagers(). Could > you add > this to > your output? > > > Make sense! Could you > recommend what data and > format you > would like > to see? > > > > Would not > ChunkManager::print_all_chunkmanagers() be > sufficient? > > Okay, I might need to tweak output > format. > > > Fine by me. Nothing depends on it yet. > > Thanks, Thomas > > Thanks, > > -Zhengyu > > > > ------------- > > Other than above notes, > the changes seem > straighforward, I did > not see anything wrong. > Minor nits: > > - > PrintCLDMetaspaceInfoClosure::_out, > _scale > and _unit > could all > be constant (with _out > being a > outputStream* > const). > - We also do not need to > provide > scale *and* > unit as > argument, > one can be deduced from > the other. > This would > also prevent > mismatches.We do need > scale here, > since nmt > command > line can set > scale and we do > > have to honor that. > > However, passing unit is > ugly! I don't > want to > introduce > inverse > dependence (metaspace -> > nmt), which is > also ugly. > > > Yes, that would be regrettable. > > Probably should move scale > mapping code from > NMTUtil to a > common util? > > > > That would be possible, these > functions > (scale_from_name etc) > are simple enough to be moved to > a generic file. > However, if you > pass scale, deducing the string > that goes with > it is > trivial and > I would have no qualms doing this > by hand in > metaspace.cpp. But > it is a matter of taste. > > > I did not look closely > at locking > issues, I assume > > MetaspaceAux::print_metadata() is > supposed to > run only in > Safepoints? > > > No. sum_xxx_xxx_in_use > queries are > guarded by locks. > > Thanks, > > -Zhengyu > > > Thanks, Thomas > > > > Thanks & Kind Regards, > Thomas > > > > > > > On Fri, Oct 20, 2017 at > 4:00 PM, > Zhengyu Gu > > > > >> > > > > >>> > > > > > >> > > > > >>>> > > > > >> > > > > >>> > > > > > > >> > > > > >>>>>> wrote: > > Up to now, there is > little > visibility > into how > metaspaces > are used, > and by whom. > > This proposed > enhancement gives > NMT the > ability to > report > metaspace > usages on > per-classloader level, > via a > new NMT command > "metadata" > > > For example: > > 2496: > Metaspaces: > Metadata space: > reserved= 8192KB > committed= 5888KB > Class space: > reserved= 1048576KB > committed= 640KB > > Per-classloader > metadata: > > ClassLoader: for > anonymous class > Metadata > capacity= 5.00KB > used= 1.16KB > free= 3.84KB > waste= 0.05KB > Class data > capacity= 1.00KB > used= 0.59KB > free= 0.41KB > waste= 0.00KB > > .... > > ClassLoader: > > Metadata > capacity= 5640.00KB > used= 5607.42KB > free= 32.58KB > waste= 0.46KB > Class data > capacity= 520.00KB > used= 500.94KB > free= 19.06KB > waste= 0.02KB > > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8189688 > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>>> > > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>>>> > Webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html > > > > > > > > >> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > >>>> > > > > > > > > > > >> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > >>>>> > > Test: > > > hotspot_tier1_runtime, > fastdebug and > release on > x64 Linux. > > Thanks, > > -Zhengyu > > > > > > > > > From thomas.stuefe at gmail.com Sat Oct 28 05:46:57 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Sat, 28 Oct 2017 07:46:57 +0200 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> Message-ID: On Fri, Oct 27, 2017 at 10:34 PM, Zhengyu Gu wrote: > > On 10/26/2017 08:53 AM, Thomas St?fe wrote: > >> Hi Zhengyu, >> >> one more thing, your output does not seem to be included in the final NMT >> report when the VM shuts down, could this be added too? >> >> This turns out to be a tricky one. > > During VM exit, we are not supposed to request a safepoint, correct? > Without a safepoint, it is unsafe to walk class loaders. > > Thanks, > > -Zhengyu > > Yes, this is not trivial. I hacked it in (because I wanted to see your output at the end of my tests) and had to disable the assert. I never ran into problems. Should java threads not all be stopped at that point? But this also can be done in a follow up issue. Thanks, Thomas > > Thanks! Thomas >> >> On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe > > wrote: >> >> Hi Zhengyu, >> >> On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu > > wrote: >> >> Hi Thomas, >> >> On 10/24/2017 12:08 PM, Thomas St?fe wrote: >> >> Hi Zhengyu, >> >> - VM_PrintMetadata still has the _unit member, but I see it >> nowhere used. >> >> - I would have split the summary between class and non-class >> area, that may have been more useful. But this is a matter >> of taste, I leave it up to you. >> >> Agree, should go little further. >> >> Summary: >> Total class loaders= 754 capacity= 67528.00KB used= >> 61216.88KB free= 9453.11KB waste= 38.73KB >> Metadata capacity= 58780.00KB used= >> 54053.45KB free= 4726.55KB waste= 36.38KB >> Class data capacity= 8748.00KB used= >> 7163.43KB free= 1584.57KB waste= 2.35KB >> >> For anonymous classes= 580 capacity= 2732.00KB used= >> 1065.06KB free= 1666.94KB waste= 16.27KB >> Metadata capacity= 2152.00KB used= >> 738.46KB free= 1413.54KB waste= 16.27KB >> Class data capacity= 580.00KB used= >> 326.60KB free= 253.40KB waste= 0.00KB >> >> >> >> Nice, thanks :) >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >> >> >> >> All is well. Note that you could reduce the footprint of your change >> somewhat by defining a structure like this: >> >> struct numbers_t { size_t cap; size_t used; size_t free; size_t >> waste; } >> >> and replacing the many members in PrintCLDMetaspaceInfoClosure with >> instances of this structure: >> >> numbers_t total; >> numbers_t total_class; >> numbers_t total_anon; >> numbers_t total_anon_class; >> Depending on how far you go, your code could deflate quite a bit. >> Give the structure a default ctor and your large initializer list >> goes away; give it a printing function and you reduce >> print-summary() to four function calls; with something like an >> numbers_t::add(size_t cap, size_t used, size_t free, size_t waste) >> you can reduce the additions in >> PrintCLDMetaspaceInfoClosure::print_metaspace() to four lines. >> >> Just a suggestion, I leave it up to you if you do this. >> >> Lines 3108 and 3129 miss each a space before BytesPerWord. Change >> looks fine otherwise. >> >> Thanks, Thomas >> >> >> Thanks, >> >> -Zhengyu >> >> >> All else looks fine to me now. I do not need another review. >> >> Thanks for your work, this will come in handy! >> >> ..Thomas >> >> On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu > > >> wrote: >> >> Hi Thomas, >> >> >> Not sure whether we misunderstood each other. I was >> thinking of >> something in the line of: >> >> <<<< >> .... >> ClassLoader: jdk/internal/reflect/Delegatin >> gClassLoader >> Metadata: >> capacity: 5.00KB used: 3.32KB free: >> 1.68KB >> waste: 0.00KB >> Class data: >> capacity: 1.00KB used: 0.54KB free: >> 0.46KB >> waste: 0.00KB >> >> ClassLoader: for anonymous class >> Metadata: >> capacity: 1.00KB used: 1.00KB free: >> 0.00KB >> waste: 0.00KB >> Class data: >> capacity: 1.00KB used: 0.64KB free: >> 0.36KB >> waste: 0.00KB >> .... >> >> Summary: >> XX class loaders encountered, total capacity: xx, >> total waste: xx. >> >> Peak usage by class loader: xxxx >> >>>> >> >> Added summary lines: >> >> Total class loaders= 56 capacity= 6378.00KB >> used= 6205.08KB free= 172.92KB waste= 1.44KB >> For anonymous classes= 54 capacity= 212.00KB >> used= 96.95KB >> free= 115.05KB waste= 0.94KB >> >> >> >> >> These numbers would not be interpretation, just a >> convenience to >> the reader to get a clear picture. >> >> It even may be useful to separate the output >> between basic and >> detailed mode, and in basic mode omit all the >> single class >> loaders and just print the summary line. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >> > > >> > gu/8189688/webrev.01/index.html > gu/8189688/webrev.01/index.html>> >> < >> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html < >> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html> >> > gu/8189688/webrev.01/index.html > gu/8189688/webrev.01/index.html>>> >> >> >> >> metaspace.hpp: >> >> I would have preferred to keep scale_unit file >> local static >> instead of exposing it via MetaspaceAux, because it >> does not >> really have anything to do with metaspace. >> >> Fixed >> >> >> Otherwise ok. >> >> --- >> >> metaspace.cpp >> >> - ChunkManager::print_statistics(): thanks for >> reusing this >> function! >> -> Scale can only be ever 1, K, M, G, yes? >> So, could you >> add an assert at the start of the function, and a >> comment at the >> prototype or function head? >> -> Also, now that >> ChunkManager::print_statistics() takes a >> scale, could you please change the printout to use >> floats like >> you did in >> PrintCLDMetaspaceInfoClosure::print_metaspace()? >> >> Done. >> >> >> Chunkmanager (non-class): >> 0 specialized (128 bytes) chunks, total 0.00KB >> 0 small (512 bytes) chunks, total 0.00KB >> 0 medium (8192 bytes) chunks, total 0.00KB >> 0 humongous chunks, total 0.00KB >> total size: 0.00KB. >> Chunkmanager (class): >> 0 specialized (128 bytes) chunks, total 0.00KB >> 0 small (256 bytes) chunks, total 0.00KB >> 0 medium (4096 bytes) chunks, total 0.00KB >> 0 humongous chunks, total 0.00KB >> total size: 0.00KB. >> >> >> - I am concerned about races in >> PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* >> cld). >> Maybe my understanding is limited. We bail if >> cld->is_unloading. >> But nothing prevents a ClassLoaderDataGraph from >> starting to >> unload a ClassLoaderData and tearing down the >> Metaspace after we >> started printing it in another thread, right? Also, >> I do not see >> any fences. >> >> So, I wonder whether PrintCLDMetaspaceInfoClosure >> should run in >> lock protection. And then, I wonder if it would be >> good to >> separate data gathering and printing. We print to a >> (at least in >> principle) unknown output stream, which may be >> doing slow File >> IO or even Network IO. Doing this under lock >> protection seems >> not a good idea (but I see other places where this >> happens too). >> >> For an example, see ChunkManager::get_statistic() vs >> ChunkManager::print_statistic(). Could you do the >> same, have a >> data gathering step where you collect your >> quadrupel in a variable >> sized list of >> structures in memory, and print it out in a >> separate step? I >> realize though that the effort would be larger than >> for what we >> did in ChunkManager::get_statistic(), where we only >> fill a >> structure. >> >> >> Unlike other NMT queries, this query is executed at a >> safepoint by >> VMThread, so it should be okay. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >> >> > > >> >> Thanks, >> >> -Zhengyu >> >> >> ------ >> >> vm_operations.hpp >> >> - VM_PrintMetadata : do we still need _unit? >> >> >> Thanks, >> >> -Zhengyu >> >> >> >> Thanks! >> >> Thomas >> >> >> >> On 10/23/2017 11:08 AM, Thomas St?fe wrote: >> >> >> >> On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu >> >> > >> > > >> >> >> > >> > > >>>> wrote: >> >> >> >> Okay. So this is important for >> understanding cases >> where we have >> a large number of miniscule class >> loaders, >> each one >> only having >> a few numbers of chunks. In this >> case, would it be >> useful to >> have a running total of "free" >> and "waste", >> too? Or >> even better, >> also average and peak values of >> "free" and >> "waste" (to tell >> apart cases where you have one >> outlier vs >> cases where >> just all >> metaspaces waste a lot of memory). >> Just out of curiosity, I looked at >> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >> >> > gu/cld_metaspace/wildfly.txt >> > >> < >> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >> >> > gu/cld_metaspace/wildfly.txt >> >> >> < >> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >> >> > gu/cld_metaspace/wildfly.txt >> > >> < >> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >> >> > gu/cld_metaspace/wildfly.txt >> >>> >> and >> it seems that "free" is the >> problem, not >> "waste"? So I >> guess >> what happens is that all the >> little classloaders >> allocate just >> enough memory to push them over >> "_small_chunk_limit=4", >> so they >> allocate the first medium chunk, >> from which >> then they >> take a >> small bite and leave the rest >> laying around? >> >> Yes. The free space of anonymous >> classes should be >> counted >> as waste, >> in my opinion. I am not sure if >> everyone agrees, >> so I took the >> summary line out of this patch. >> >> I would be more than happy to restore >> the summary >> line, if >> you find >> it is useful :-) >> >> >> I agree with you. Typically class loaders >> stop loading >> at some >> point, especially these tine ones, and >> often will not >> resume >> loading. So, effectivly, the space is >> wasted. It still >> helps to >> distinguish "free" in the current chunks >> from "free" in the >> other chunks to tell this situation apart >> from a >> situation where >> we have just a bug in metaspace chunk >> handling >> preventing us to >> use up our chunks. >> >> >> The latter would be an >> important >> addition, >> regardless >> if this is >> for customers or for us. >> Chunks idling in >> freelists can >> make a >> huge impact, and >> unfortunately this >> may happen >> in perfectly >> valid cases. See e.g. >> our JEP >> >> "https://bugs.openjdk.java.net/browse/JDK-8166690 >> >> > > >> > /browse/JDK-8166690 >> >> > >> >> < >> https://bugs.openjdk.java.net/browse/JDK-8166690 >> >> > > >> > /browse/JDK-8166690 >> >> > >>> >> >> > >> > > >> > /browse/JDK-8166690 >> >> > >> >> < >> https://bugs.openjdk.java.net/browse/JDK-8166690 >> >> > > >> > /browse/JDK-8166690 >> >> > >>>>". We >> have >> already a printing >> routine for free >> chunks, in >> >> ChunkManager::print_all_chunkmanagers(). Could >> you add >> this to >> your output? >> >> >> Make sense! Could you >> recommend what data and >> format you >> would like >> to see? >> >> >> >> Would not >> ChunkManager::print_all_chunkmanagers() be >> sufficient? >> >> Okay, I might need to tweak output >> format. >> >> >> Fine by me. Nothing depends on it yet. >> >> Thanks, Thomas >> >> Thanks, >> >> -Zhengyu >> >> >> >> ------------- >> >> Other than above notes, >> the changes seem >> straighforward, I did >> not see anything wrong. >> Minor nits: >> >> - >> PrintCLDMetaspaceInfoClosure::_out, >> _scale >> and _unit >> could all >> be constant (with _out >> being a >> outputStream* >> const). >> - We also do not need to >> provide >> scale *and* >> unit as >> argument, >> one can be deduced from >> the other. >> This would >> also prevent >> mismatches.We do need >> scale here, >> since nmt >> command >> line can set >> scale and we do >> >> have to honor that. >> >> However, passing unit is >> ugly! I don't >> want to >> introduce >> inverse >> dependence (metaspace -> >> nmt), which is >> also ugly. >> >> >> Yes, that would be regrettable. >> >> Probably should move scale >> mapping code from >> NMTUtil to a >> common util? >> >> >> >> That would be possible, these >> functions >> (scale_from_name etc) >> are simple enough to be moved to >> a generic file. >> However, if you >> pass scale, deducing the string >> that goes with >> it is >> trivial and >> I would have no qualms doing this >> by hand in >> metaspace.cpp. But >> it is a matter of taste. >> >> >> I did not look closely >> at locking >> issues, I assume >> >> MetaspaceAux::print_metadata() is >> supposed to >> run only in >> Safepoints? >> >> >> No. sum_xxx_xxx_in_use >> queries are >> guarded by locks. >> >> Thanks, >> >> -Zhengyu >> >> >> Thanks, Thomas >> >> >> >> Thanks & Kind Regards, >> Thomas >> >> >> >> >> >> >> On Fri, Oct 20, 2017 at >> 4:00 PM, >> Zhengyu Gu >> > > > >> >> >> >> > > > >> >> >>> >> > >> > >> >> >> >> > > > >> >> >>>> >> > > > >> >> >> >> > > > >> >> >>> >> >> > >> > >> >> >> >> > > > >> >> >>>>>> wrote: >> >> Up to now, there is >> little >> visibility >> into how >> metaspaces >> are used, >> and by whom. >> >> This proposed >> enhancement gives >> NMT the >> ability to >> report >> metaspace >> usages on >> per-classloader level, >> via a >> new NMT command >> "metadata" >> >> >> For example: >> >> 2496: >> Metaspaces: >> Metadata space: >> reserved= 8192KB >> committed= 5888KB >> Class space: >> reserved= 1048576KB >> committed= 640KB >> >> Per-classloader >> metadata: >> >> ClassLoader: for >> anonymous class >> Metadata >> capacity= 5.00KB >> used= 1.16KB >> free= 3.84KB >> waste= 0.05KB >> Class data >> capacity= 1.00KB >> used= 0.59KB >> free= 0.41KB >> waste= 0.00KB >> >> .... >> >> ClassLoader: >> >> Metadata >> capacity= 5640.00KB >> used= 5607.42KB >> free= 32.58KB >> waste= 0.46KB >> Class data >> capacity= 520.00KB >> used= 500.94KB >> free= 19.06KB >> waste= 0.02KB >> >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > /browse/JDK-8189688 >> >> > >> >> < >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > /browse/JDK-8189688 >> >> > >>> >> >> > >> > > >> > /browse/JDK-8189688 >> >> > >> >> < >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > /browse/JDK-8189688 >> >> > >>>> >> >> > >> > From david.holmes at oracle.com Sat Oct 28 12:28:32 2017 From: david.holmes at oracle.com (David Holmes) Date: Sat, 28 Oct 2017 22:28:32 +1000 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> Message-ID: <6710b89e-1b1a-aed2-6273-8bb2bb0fcdbe@oracle.com> On 28/10/2017 6:34 AM, Zhengyu Gu wrote: > > On 10/26/2017 08:53 AM, Thomas St?fe wrote: >> Hi Zhengyu, >> >> one more thing, your output does not seem to be included in the final >> NMT report when the VM shuts down, could this be added too? >> > This turns out to be a tricky one. > > During VM exit, we are not supposed to request a safepoint, correct? VM exit uses a safepoint - can you use it do gather the data? David > Without a safepoint, it is unsafe to walk class loaders. > > Thanks, > > -Zhengyu > > >> Thanks! Thomas >> >> On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe > > wrote: >> >> ??? Hi Zhengyu, >> >> ??? On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu > ??? > wrote: >> >> ??????? Hi Thomas, >> >> ??????? On 10/24/2017 12:08 PM, Thomas St?fe wrote: >> >> ??????????? Hi Zhengyu, >> >> ??????????? - VM_PrintMetadata still has the _unit member, but I see it >> ??????????? nowhere used. >> >> ??????????? - I would have split the summary between class and non-class >> ??????????? area, that may have been more useful. But this is a matter >> ??????????? of taste, I leave it up to you. >> >> ??????? Agree, should go little further. >> >> ??????? Summary: >> ?????????? Total class loaders=?? 754 capacity=? 67528.00KB used= >> ??????? 61216.88KB free=?? 9453.11KB waste=???? 38.73KB >> ???????????????????????????? Metadata capacity=? 58780.00KB used= >> ??????? 54053.45KB free=?? 4726.55KB waste=???? 36.38KB >> ?????????????????????????? Class data capacity=?? 8748.00KB used= >> ????????? 7163.43KB free=?? 1584.57KB waste=????? 2.35KB >> >> ??????? For anonymous classes=?? 580 capacity=?? 2732.00KB used= >> ????????? 1065.06KB free=?? 1666.94KB waste=???? 16.27KB >> ???????????????????????????? Metadata capacity=?? 2152.00KB used= >> ??????? 738.46KB free=?? 1413.54KB waste=???? 16.27KB >> ?????????????????????????? Class data capacity=??? 580.00KB used= >> ??????? 326.60KB free=??? 253.40KB waste=????? 0.00KB >> >> >> >> ??? Nice, thanks :) >> >> ??????? Updated webrev: >> ??????? http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >> ??????? >> >> >> ??? All is well. Note that you could reduce the footprint of your change >> ??? somewhat by defining a structure like this: >> >> ??? struct numbers_t { size_t cap; size_t used; size_t free; size_t >> waste; } >> >> ??? and replacing the many members in PrintCLDMetaspaceInfoClosure with >> ??? instances of this structure: >> >> ??? numbers_t total; >> ??? numbers_t total_class; >> ??? numbers_t total_anon; >> ??? numbers_t total_anon_class; >> ??? Depending on how far you go, your code could deflate quite a bit. >> ??? Give the structure a default ctor and your large initializer list >> ??? goes away; give it a printing function and you reduce >> ??? print-summary() to four function calls; with something like an >> ??? numbers_t::add(size_t cap, size_t used, size_t free, size_t waste) >> ??? you can reduce the additions in >> ??? PrintCLDMetaspaceInfoClosure::print_metaspace() to four lines. >> >> ??? Just a suggestion, I leave it up to you if you do this. >> >> ??? Lines 3108 and 3129 miss each a space before BytesPerWord. Change >> ??? looks fine otherwise. >> >> ??? Thanks, Thomas >> >> >> ??????? Thanks, >> >> ??????? -Zhengyu >> >> >> ??????????? All else looks fine to me now. I do not need another review. >> >> ??????????? Thanks for your work, this will come in handy! >> >> ??????????? ..Thomas >> >> ??????????? On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu > ??????????? > ??????????? >> wrote: >> >> ???????????????? Hi Thomas, >> >> >> ???????????????????? Not sure whether we misunderstood each other. I was >> ??????????? thinking of >> ???????????????????? something in the line of: >> >> ???????????????????? <<<< >> ???????????????????? .... >> ???????????????????? ClassLoader: >> jdk/internal/reflect/DelegatingClassLoader >> ???????????????????????? Metadata: >> ??????????????????????????? capacity:???? 5.00KB used:???? 3.32KB >> free: ??????????????? 1.68KB >> ???????????????????? waste: 0.00KB >> ???????????????????????? Class data: >> ??????????????????????????? capacity:???? 1.00KB used:???? 0.54KB >> free: ??????????????? 0.46KB >> ???????????????????? waste: 0.00KB >> >> ???????????????????? ClassLoader: for anonymous class >> ???????????????????????? Metadata: >> ??????????????????????????? capacity:???? 1.00KB used:???? 1.00KB >> free: ??????????????? 0.00KB >> ???????????????????? waste: 0.00KB >> ???????????????????????? Class data: >> ??????????????????????????? capacity:???? 1.00KB used:???? 0.64KB >> free: ??????????????? 0.36KB >> ???????????????????? waste: 0.00KB >> ???????????????????? .... >> >> ???????????????????? Summary: >> ???????????????????? XX class loaders encountered, total capacity: xx, >> ??????????? total waste: xx. >> >> ???????????????????? Peak usage by class loader: xxxx >> ?????????????????????? >>>> >> >> ???????????????? Added summary lines: >> >> ??????????????????? Total class loaders=??? 56 capacity=?? 6378.00KB >> ??????????? used=?????? 6205.08KB free=??? 172.92KB waste=????? 1.44KB >> ???????????????? For anonymous classes=??? 54 capacity=??? 212.00KB >> ??????????? used=???? 96.95KB >> ???????????????? free=??? 115.05KB waste=????? 0.94KB >> >> >> >> >> ???????????????????? These numbers would not be interpretation, just a >> ??????????? convenience to >> ???????????????????? the reader to get a clear picture. >> >> ???????????????????? It even may be useful to separate the output >> ??????????? between basic and >> ???????????????????? detailed mode, and in basic mode omit all the >> ??????????? single class >> ???????????????????? loaders and just print the summary line. >> >> ????????????????????????? Updated webrev: >> ??????????? http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >> >> >> >> > > >> >> > >> >> > >> >> >> >> >> ???????????????????? metaspace.hpp: >> >> ???????????????????? I would have preferred to keep scale_unit file >> ??????????? local static >> ???????????????????? instead of exposing it via MetaspaceAux, because it >> ??????????? does not >> ???????????????????? really have anything to do with metaspace. >> >> ???????????????? Fixed >> >> >> ???????????????????? Otherwise ok. >> >> ???????????????????? --- >> >> ???????????????????? metaspace.cpp >> >> ???????????????????? - ChunkManager::print_statistics(): thanks for >> ??????????? reusing this >> ???????????????????? function! >> ?????????????????????????? -> Scale can only be ever 1, K, M, G, yes? >> ??????????? So, could you >> ???????????????????? add an assert at the start of the function, and a >> ??????????? comment at the >> ???????????????????? prototype or function head? >> ?????????????????????????? -> Also, now that >> ??????????? ChunkManager::print_statistics() takes a >> ???????????????????? scale, could you please change the printout to use >> ??????????? floats like >> ???????????????????? you did in >> ??????????? PrintCLDMetaspaceInfoClosure::print_metaspace()? >> >> ???????????????? Done. >> >> >> ???????????????? Chunkmanager (non-class): >> ??????????????????? 0 specialized (128 bytes) chunks, total 0.00KB >> ??????????????????? 0 small (512 bytes) chunks, total 0.00KB >> ??????????????????? 0 medium (8192 bytes) chunks, total 0.00KB >> ??????????????????? 0 humongous chunks, total 0.00KB >> ??????????????????? total size: 0.00KB. >> ???????????????? Chunkmanager (class): >> ??????????????????? 0 specialized (128 bytes) chunks, total 0.00KB >> ??????????????????? 0 small (256 bytes) chunks, total 0.00KB >> ??????????????????? 0 medium (4096 bytes) chunks, total 0.00KB >> ??????????????????? 0 humongous chunks, total 0.00KB >> ??????????????????? total size: 0.00KB. >> >> >> ???????????????????? - I am concerned about races in >> ??????????? PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >> ???????????????????? Maybe my understanding is limited. We bail if >> ??????????? cld->is_unloading. >> ???????????????????? But nothing prevents a ClassLoaderDataGraph from >> ??????????? starting to >> ???????????????????? unload a ClassLoaderData and tearing down the >> ??????????? Metaspace after we >> ???????????????????? started printing it in another thread, right? Also, >> ??????????? I do not see >> ???????????????????? any fences. >> >> ???????????????????? So, I wonder whether PrintCLDMetaspaceInfoClosure >> ??????????? should run in >> ???????????????????? lock protection. And then, I wonder if it would be >> ??????????? good to >> ???????????????????? separate data gathering and printing. We print to a >> ??????????? (at least in >> ???????????????????? principle) unknown output stream, which may be >> ??????????? doing slow File >> ???????????????????? IO or even Network IO. Doing this under lock >> ??????????? protection seems >> ???????????????????? not a good idea (but I see other places where this >> ??????????? happens too). >> >> ???????????????????? For an example, see ChunkManager::get_statistic() vs >> ???????????????????? ChunkManager::print_statistic(). Could you do the >> ??????????? same, have a >> ???????????????????? data gathering step where you collect your >> ???????????????????? quadrupel in a variable >> ??????????? sized list of >> ???????????????????? structures in memory, and print it out in a >> ??????????? separate step? I >> ???????????????????? realize though that the effort would be larger than >> ??????????? for what we >> ???????????????????? did in ChunkManager::get_statistic(), where we only >> ??????????? fill a >> ???????????????????? structure. >> >> >> ???????????????? Unlike other NMT queries, this query is executed at a >> ??????????? safepoint by >> ???????????????? VMThread, so it should be okay. >> >> ???????????????? Updated webrev: >> ??????????? http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >> ??????????? >> ???????????????? > ??????????? > >> >> ???????????????? Thanks, >> >> ???????????????? -Zhengyu >> >> >> ???????????????????? ------ >> >> ???????????????????? vm_operations.hpp >> >> ???????????????????? - VM_PrintMetadata : do we still need _unit? >> >> >> ????????????????????????? Thanks, >> >> ????????????????????????? -Zhengyu >> >> >> >> ???????????????????? Thanks! >> >> ???????????????????? Thomas >> >> >> >> ????????????????????????? On 10/23/2017 11:08 AM, Thomas St?fe wrote: >> >> >> >> ????????????????????????????? On Mon, Oct 23, 2017 at 5:03 PM, Zhengyu Gu >> ???????????????????? >> ??????????? > >> ????????????????????????????? > ??????????? > ??????????? >> >> ???????????????????? >> ??????????? > >> ????????????????????????????? > ??????????? > ??????????? >>>> wrote: >> >> >> >> ?????????????????????????????????????? Okay. So this is important for >> ??????????? understanding cases >> ????????????????????????????? where we have >> ?????????????????????????????????????? a large number of miniscule class >> ??????????? loaders, >> ???????????????????? each one >> ????????????????????????????? only having >> ?????????????????????????????????????? a few numbers of chunks. In this >> ??????????? case, would it be >> ????????????????????????????? useful to >> ?????????????????????????????????????? have a running total of "free" >> ??????????? and "waste", >> ???????????????????? too? Or >> ????????????????????????????? even better, >> ?????????????????????????????????????? also average and peak values of >> ??????????? "free" and >> ???????????????????? "waste" (to tell >> ?????????????????????????????????????? apart cases where you have one >> ??????????? outlier vs >> ???????????????????? cases where >> ????????????????????????????? just all >> ?????????????????????????????????????? metaspaces waste a lot of memory). >> ?????????????????????????????????????? Just out of curiosity, I looked at >> ??????????? http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >> ??????????? >> ??????????? > ??????????? > >> ??????????? > ??????????? >> ??????????? > ??????????? >> >> ??????????? > ??????????? >> ??????????? > ??????????? > >> ??????????? > ??????????? >> ??????????? > >> >>> >> ??????????? and >> ?????????????????????????????????????? it seems that "free" is the >> ??????????? problem, not >> ???????????????????? "waste"? So I >> ????????????????????????????? guess >> ?????????????????????????????????????? what happens is that all the >> ??????????? little classloaders >> ????????????????????????????? allocate just >> ?????????????????????????????????????? enough memory to push them over >> ???????????????????? "_small_chunk_limit=4", >> ????????????????????????????? so they >> ?????????????????????????????????????? allocate the first medium chunk, >> ??????????? from which >> ???????????????????? then they >> ????????????????????????????? take a >> ?????????????????????????????????????? small bite and leave the rest >> ??????????? laying around? >> >> ?????????????????????????????????? Yes. The free space of anonymous >> ??????????? classes should be >> ???????????????????? counted >> ????????????????????????????? as waste, >> ?????????????????????????????????? in my opinion. I am not sure if >> ??????????? everyone agrees, >> ???????????????????? so I took the >> ?????????????????????????????????? summary line out of this patch. >> >> ?????????????????????????????????? I would be more than happy to restore >> ??????????? the summary >> ???????????????????? line, if >> ????????????????????????????? you find >> ?????????????????????????????????? it is useful :-) >> >> >> ????????????????????????????? I agree with you. Typically class loaders >> ??????????? stop loading >> ???????????????????? at some >> ????????????????????????????? point, especially these tine ones, and >> ??????????? often will not >> ???????????????????? resume >> ????????????????????????????? loading. So, effectivly, the space is >> ??????????? wasted. It still >> ???????????????????? helps to >> ????????????????????????????? distinguish "free" in the current chunks >> ??????????? from "free" in the >> ????????????????????????????? other chunks to tell this situation apart >> ??????????? from a >> ???????????????????? situation where >> ????????????????????????????? we have just a bug in metaspace chunk >> handling >> ???????????????????? preventing us to >> ????????????????????????????? use up our chunks. >> >> >> ??????????????????????????????????????????????? The latter would be an >> ??????????? important >> ???????????????????? addition, >> ????????????????????????????? regardless >> ?????????????????????????????????????? if this is >> ??????????????????????????????????????????????? for customers or for us. >> ??????????? Chunks idling in >> ????????????????????????????? freelists can >> ?????????????????????????????????????? make a >> ??????????????????????????????????????????????? huge impact, and >> ??????????? unfortunately this >> ???????????????????? may happen >> ????????????????????????????? in perfectly >> ??????????????????????????????????????????????? valid cases. See e.g. >> ??????????? our JEP >> ??????????? "https://bugs.openjdk.java.net/browse/JDK-8166690 >> ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >> >> ??????????? > ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >>> >> ??????????? > ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >> >> ??????????? > ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >>>>". >> We have >> ??????????????????????????????????????????????? already a printing >> ??????????? routine for free >> ???????????????????? chunks, in >> ??????????? ChunkManager::print_all_chunkmanagers(). Could >> ????????????????????????????? you add >> ?????????????????????????????????????? this to >> ??????????????????????????????????????????????? your output? >> >> >> ??????????????????????????????????????????? Make sense! Could you >> ??????????? recommend what data and >> ????????????????????????????? format you >> ?????????????????????????????????????? would like >> ??????????????????????????????????????????? to see? >> >> >> >> ?????????????????????????????????????? Would not >> ???????????????????? ChunkManager::print_all_chunkmanagers() be >> ????????????????????????????? sufficient? >> >> ?????????????????????????????????? Okay, I might need to tweak output >> ??????????? format. >> >> >> ????????????????????????????? Fine by me. Nothing depends on it yet. >> >> ????????????????????????????? Thanks, Thomas >> >> ?????????????????????????????????? Thanks, >> >> ?????????????????????????????????? -Zhengyu >> >> >> >> ??????????????????????????????????????????????? ------------- >> >> ??????????????????????????????????????????????? Other than above notes, >> ??????????? the changes seem >> ?????????????????????????????????????? straighforward, I did >> ??????????????????????????????????????????????? not see anything wrong. >> ??????????? Minor nits: >> >> ??????????????????????????????????????????????? - >> ??????????? PrintCLDMetaspaceInfoClosure::_out, >> ???????????????????? _scale >> ????????????????????????????? and _unit >> ?????????????????????????????????????? could all >> ??????????????????????????????????????????????? be constant (with _out >> ??????????? being a >> ???????????????????? outputStream* >> ????????????????????????????? const). >> ??????????????????????????????????????????????? - We also do not need to >> ??????????? provide >> ???????????????????? scale *and* >> ????????????????????????????? unit as >> ?????????????????????????????????????? argument, >> ??????????????????????????????????????????????? one can be deduced from >> ??????????? the other. >> ???????????????????? This would >> ????????????????????????????? also prevent >> ??????????????????????????????????????????????? mismatches.We do need >> ??????????? scale here, >> ???????????????????? since nmt >> ????????????????????????????? command >> ?????????????????????????????????????? line can set >> ??????????????????????????????????????????????? scale and we do >> >> ??????????????????????????????????????????? have to honor that. >> >> ??????????????????????????????????????????? However, passing unit is >> ??????????? ugly! I don't >> ???????????????????? want to >> ????????????????????????????? introduce >> ?????????????????????????????????????? inverse >> ??????????????????????????????????????????? dependence (metaspace -> >> ??????????? nmt), which is >> ???????????????????? also ugly. >> >> >> ?????????????????????????????????????? Yes, that would be regrettable. >> >> ??????????????????????????????????????????? Probably should move scale >> ??????????? mapping code from >> ????????????????????????????? NMTUtil to a >> ?????????????????????????????????????? common util? >> >> >> >> ?????????????????????????????????????? That would be possible, these >> ??????????? functions >> ????????????????????????????? (scale_from_name etc) >> ?????????????????????????????????????? are simple enough to be moved to >> ??????????? a generic file. >> ????????????????????????????? However, if you >> ?????????????????????????????????????? pass scale, deducing the string >> ??????????? that goes with >> ???????????????????? it is >> ????????????????????????????? trivial and >> ?????????????????????????????????????? I would have no qualms doing this >> ??????????? by hand in >> ????????????????????????????? metaspace.cpp. But >> ?????????????????????????????????????? it is a matter of taste. >> >> >> ??????????????????????????????????????????????? I did not look closely >> ??????????? at locking >> ???????????????????? issues, I assume >> ????????????? MetaspaceAux::print_metadata() is >> ???????????????????? supposed to >> ????????????????????????????? run only in >> ??????????????????????????????????????????????? Safepoints? >> >> >> ??????????????????????????????????????????? No. sum_xxx_xxx_in_use >> ??????????? queries are >> ???????????????????? guarded by locks. >> >> ??????????????????????????????????????????? Thanks, >> >> ??????????????????????????????????????????? -Zhengyu >> >> >> ?????????????????????????????????????? Thanks, Thomas >> >> >> >> ??????????????????????????????????????????????? Thanks & Kind Regards, >> ??????????? Thomas >> >> >> >> >> >> >> ??????????????????????????????????????????????? On Fri, Oct 20, 2017 at >> ??????????? 4:00 PM, >> ???????????????????? Zhengyu Gu >> ?????????????????????????????????????? > ??????????? > ??????????? > >> ???????????????????? >> ??????????? >> >> ????????????????????????????? > ??????????? > ??????????? > >> ???????????????????? >> ??????????? >>> >> ??????????????????????????????????????????????? > ??????????? >> ???????????????????? > >> ??????????? >> ???????????????????? >> >> ????????????????????????????? > ??????????? > ??????????? > >> ???????????????????? >> ??????????? >>>> >> ?????????????????????????????????????? > ??????????? > ??????????? > >> ???????????????????? >> ??????????? >> >> ????????????????????????????? > ??????????? > ??????????? > >> ???????????????????? >> ??????????? >>> >> >> ??????????????????????????????????????????????? > ??????????? >> ???????????????????? > >> ??????????? >> ???????????????????? >> >> ????????????????????????????? > ??????????? > ??????????? > >> ???????????????????? >> ??????????? >>>>>> wrote: >> >> ???????????????????????????????????????????????????? Up to now, there is >> ??????????? little >> ???????????????????? visibility >> ????????????????????????????? into how >> ?????????????????????????????????????? metaspaces >> ??????????????????????????????????????????????? are used, >> ???????????????????????????????????????????????????? and by whom. >> >> ???????????????????????????????????????????????????? This proposed >> ??????????? enhancement gives >> ???????????????????? NMT the >> ????????????????????????????? ability to >> ?????????????????????????????????????? report >> ??????????????????????????????????????????????? metaspace >> ???????????????????????????????????????????????????? usages on >> ??????????? per-classloader level, >> ???????????????????? via a >> ????????????????????????????? new NMT command >> ??????????????????????????????????????????????? "metadata" >> >> >> ???????????????????????????????????????????????????? For example: >> >> ???????????????????????????????????????????????????? 2496: >> ???????????????????????????????????????????????????? Metaspaces: >> ??????????????????????????????????????????????????????? Metadata space: >> ??????????? reserved=????????????? 8192KB >> ?????????????????????????????????????? committed=????? 5888KB >> ??????????????????????????????????????????????????????? Class??? space: >> ??????????? reserved=?????????? 1048576KB >> ?????????????????????????????????????? committed=?????? 640KB >> >> ???????????????????????????????????????????????????? Per-classloader >> ??????????? metadata: >> >> ???????????????????????????????????????????????????? ClassLoader: for >> ??????????? anonymous class >> ??????????????????????????????????????????????????????? Metadata >> ????????????? capacity=????? 5.00KB >> ????????????????????????????? used=????? 1.16KB >> ??????????????????????????????????????????????? free=???????? 3.84KB >> ??????????? waste=????? 0.05KB >> ??????????????????????????????????????????????????????? Class data >> ??????????? capacity=????? 1.00KB >> ????????????????????????????? used=????? 0.59KB >> ??????????????????????????????????????????????? free=???????? 0.41KB >> ??????????? waste=????? 0.00KB >> >> ???????????????????????????????????????????????????? .... >> >> ???????????????????????????????????????????????????? ClassLoader: >> ??????????? >> ??????????????????????????????????????????????????????? Metadata >> ????????????? capacity=?? 5640.00KB >> ????????????????????????????? used=?? 5607.42KB >> ??????????????????????????????????????????????? free=???????? 32.58KB >> ??????????? waste=????? 0.46KB >> ??????????????????????????????????????????????????????? Class data >> ??????????? capacity=??? 520.00KB >> ????????????????????????????? used=??? 500.94KB >> ??????????????????????????????????????????????? free=???????? 19.06KB >> ??????????? waste=????? 0.02KB >> >> >> ???????????????????????????????????????????????????? Bug: >> ??????????? https://bugs.openjdk.java.net/browse/JDK-8189688 >> ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >> >> ??????????? > ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >>> >> ??????????? > ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >> >> ??????????? > ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >>>> >> ??????????? > ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >> >> ??????????? > ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >>> >> ??????????? > ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >> >> ??????????? > ??????????? >> ???????????????????? > ??????????? > >> ????????????? > ??????????? >> ???????????????????? > ??????????? >>>>> >> ???????????????????????????????????????????????????? Webrev: >> ??????????? http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >> >> >> >> > > >> >> > >> >> > >> >> >> > >> >> > > >> >> > >> >> > >>> >> >> > >> >> > > >> >> > >> >> > >> >> >> > >> >> > > >> >> > >> >> > >>>> >> >> > >> >> > > >> >> > >> >> > >> >> >> > >> >> > > >> >> > >> >> > >>> >> >> > >> >> > > >> >> > >> >> > >> >> >> > >> >> > > >> >> > >> >> > >>>>> >> >> ???????????????????????????????????????????????????? Test: >> >> ????????????? hotspot_tier1_runtime, >> ???????????????????? fastdebug and >> ????????????????????????????? release on >> ?????????????????????????????????????? x64 Linux. >> >> ???????????????????????????????????????????????????? Thanks, >> >> ???????????????????????????????????????????????????? -Zhengyu >> >> >> >> >> >> >> >> >> From martin.doerr at sap.com Sat Oct 28 16:33:03 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Sat, 28 Oct 2017 16:33:03 +0000 Subject: RFR(M): 8190285: s390: Some java boolean checks are not correct In-Reply-To: <294b5990-fc2f-ae11-316f-b247a340945b@oracle.com> References: <294b5990-fc2f-ae11-316f-b247a340945b@oracle.com> Message-ID: <7d5e0cad6e544a8d990440d17a8a2915@sap.com> Hi Coleen, thanks for reviewing. We have only tested that the narrowing in early return doesn't break anything. It should be possible to test it by using a modified Java debugger, but I haven't tried that. Anyway, I think the narrowing should be there since ForceEarlyReturnInt can be used for any of these types and I couldn't find a check in the JVMTI implementation. Best regards, Martin -----Original Message----- From: hotspot-runtime-dev [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On Behalf Of coleen.phillimore at oracle.com Sent: Freitag, 27. Oktober 2017 20:49 To: hotspot-runtime-dev at openjdk.java.net Subject: Re: RFR(M): 8190285: s390: Some java boolean checks are not correct Hi This looks good, but do you have a test case for the early return case??? Many of the platforms are missing the narrow call. http://cr.openjdk.java.net/~mdoerr/8190285_s390_jbool/webrev.00/src/hotspot/cpu/s390/templateInterpreterGenerator_s390.cpp.udiff.html Thanks, Coleen On 10/27/17 10:23 AM, Doerr, Martin wrote: > Hi, > > the current s390 implementation misses a couple of small fixes which exist on other platforms. > > Please review: > http://cr.openjdk.java.net/~mdoerr/8190285_s390_jbool/webrev.00/ > > Best regards, > Martin > From ioi.lam at oracle.com Mon Oct 30 01:41:38 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Sun, 29 Oct 2017 18:41:38 -0700 Subject: RFR [XS] Subclasses of jdk.jfr.Event loaded from CDS breaks -XX:FlightRecorderOptions=retransform=false In-Reply-To: <986EB008-32B1-4325-BE61-CC50CD93F746@oracle.com> References: <5395e40f-e660-4e3c-5b8e-1522f0dbd78b@oracle.com> <986EB008-32B1-4325-BE61-CC50CD93F746@oracle.com> Message-ID: <11afa7a9-9621-a24a-6451-000baf847da6@oracle.com> Hi Jiangli & Sergei, Thanks for your review. I have updated the code as suggested by Jiangli and pushed. - Ioi On 10/26/17 6:00 PM, Jiangli Zhou wrote: > Hi Ioi, > > SystemDictionary::reorder_dictionary_for_sharing() and Dictionary::reorder_dictionary_for_sharing() are only used for CDS code. Could you please add CDS_ONLY() to the function definitions and put the implementation under #if INCLUDE_CDS. > > Thanks, > Jiangli > >> On Oct 26, 2017, at 4:26 PM, Ioi Lam wrote: >> >> Please review the follow change: >> >> https://bugs.openjdk.java.net/browse/JDK-8190191 >> http://cr.openjdk.java.net/~iklam/jdk10/8190191-jfr-event-retransform-false.v01.open/ >> >> Background: >> >> When -XX:FlightRecorderOptions=retransform=false is given in the command-line, >> subclasses of jdk.jfr.Event are instrumented at load time with information that's >> specific to the current JVM lifetime. As a result, we cannot perform >> such instrumentation at CDS dump time. >> >> A more complicated CDS solution would load these classes from the archive, >> and re-do the runtime instrumentation. However, there are only a very >> small number of these classes. The performance benefit of archiving them >> does not justify the extra complication. >> >> Hence, in this fix, we just identify these classes and exclude them >> from the CDS archive during run time. >> >> (Because JFR is still a closed feature, the test cases are in the closed repo ...) >> >> Thanks >> - Ioi >> From thomas.schatzl at oracle.com Mon Oct 30 08:00:41 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 30 Oct 2017 09:00:41 +0100 Subject: FW: RFR(M): 8171181: Supporting heap allocation on alternative memory devices In-Reply-To: References: <1508859525.3150.26.camel@oracle.com> Message-ID: <1509350441.3625.1.camel@oracle.com> Hi, On Thu, 2017-10-26 at 04:33 +0000, Kharbas, Kishor wrote: > I got delivery failure to the individual email addresses, so > forwarding the email. > > Regards > Kishor > > -----Original Message----- > From: Kharbas, Kishor? > Sent: Wednesday, October 25, 2017 9:28 PM > To: Thomas Schatzl ; sangheon.kim n.kim at oracle.com>; 'hotspot-gc-dev at openjdk.java.net' openjdk.java.net> > Cc: hotspot-runtime-dev at openjdk.java.net; Kharbas, Kishor rbas at intel.com> > Subject: RE: RFR(M): 8171181: Supporting heap allocation on > alternative memory devices > > Hi Sangheon, Thomas, > > Thanks for the review. Here is the latest webrev based on your > comments and my replies to both are inline (I copied Sangheon's email > here at the top above Thomas's reply) > http://cr.openjdk.java.net/~kkharbas/8171181/webrev.10_to_09 > http://cr.openjdk.java.net/~kkharbas/8171181/webrev.10 > > Please let me know your question/comments. Just minor nits for now: - virtualspace.cpp:72: s/Healper/Helper - TestAllocateHeapAt.java:2: This is a new file, so I guess copyright date should not start from 2013. Thanks for all these changes. Thanks, ? Thomas > > Thanks > Kishor > > > Hi Kishor, > > On 10/19/2017 08:00 AM, Kharbas, Kishor wrote: > > Hi Sangheon, > > ? > > Thanks for the review and comments. > > I created two webrevs: > > webrev.07 : Is the rebase of webrev.06 on jdk10????(http://cr.openj > > dk.java.net/~kkharbas/8171181/webrev.07/) > > webrev.08 : Is incremental webrev over 07.??????????????(http://cr. > > openjdk.java.net/~kkharbas/8171181/webrev.08/) > > 1. I couldn't apply webrev.07 patch cleanly. Could you test?? > > ???Just minor nit: 07 seems not only rebase of webrev.06. i.e. > > there seems to have some changes from my comment of alignments. > > [Kharbas, Kishor] My bad, I had rebased the patch on jdk10/jdk10. The > further ones are on jdk10/hs > > > 2. The description in the JEP uses 'AllocateHeapAt' while the patch > > uses 'HeapDir'. > > [Kharbas, Kishor] I changed the code. Thanks > > > ----------------------------- > > src/os/posix/vm/os_posix.cpp > > 159???(void)strcpy(fullname, dir); > > - Please use strncpy instead. > > [Kharbas, Kishor] Done > > > 160???(void)strcat(fullname, name_template); > > - Please use strncat instead. > > [Kharbas, Kishor] Done. > > > 180?????::free(fullname); > > - os::free(). > > [Kharbas, Kishor] Done. > > > 196???::free(fullname); > > - os::free(). > > [Kharbas, Kishor] Done. > > > ----------------------------- > > src/os/windows/vm/os_windows.cpp > > 2885???char *fullname = (char*)_alloca(strlen(dir) + > > sizeof(name_template)); > > - Missing allocation failure handling. > > - Please use _malloca instead. > > ?* This function is deprecated because a more secure version is > > available; see _malloca. > > ???https://msdn.microsoft.com/en-us/library/wb1s57t5.aspx > > [Kharbas, Kishor] Done. > > > 2886???(void)strcpy(fullname, dir); > > - Please use strncpy instead. > > [Kharbas, Kishor] Done. > > > 2887???(void)strcat(fullname, name_template); > > - Please use strncat instead. > > [Kharbas, Kishor] Done. > > > ----------------------------- > > src/share/vm/runtime/arguments.cpp > > 3726?????} > > 3727???} > > - My previous comment was to add explicit explanation of disabling > > UseNUMA and UseNUMAInterleaving. > > 4636?????if(!FLAG_IS_DEFAULT(HeapDir)) { > > - Previsouly we checked UseNUMA and UseNUMAInterleaving and then > > disabled those. IMHO, I think previous one seems better with more > > explanation that we are going to disable those options. i.e. > > >Similar to os_linux.cpp:4869, a warning message with disabling > > codes. > > [Kharbas, Kishor] Based on our discussion offline, I think we agree > that it?s OK to disable UseNUMA and leave UseNUMAInterleaving ON > since other areas of JVM such as CodeCache can use > UseNUMAInterleaving. > > > 4638?????} > > 4639?????else if (UseParallelGC || UseParallelOldGC) { > > - One line please if this change is still needed.? > > ? -> } else if { > > > ----------------------------- > > src/share/vm/runtime/os.cpp > > 1678 > > - Extra line. > > [Kharbas, Kishor] Done. > > > 1688???} > > 1689???else { > > -> } else { > > [Kharbas, Kishor] Done. > > > My answers are inlined. > > ? > > In webrev08 I have addressed all the comments, except for ones > > below: > > ? > > --------------------------------------- > > src/os/aix/vm/os_aix.cpp > > 2514 char* os::pd_attempt_reserve_memory_at(size_t bytes, char*? > > requested_addr, bool use_SHM) { > > - Question. Why os_aix has additional parameter at > > pd_attemp_reserve_memory_at()? Probably only AIX has shmated memory > > implementation? > > ????????A: Yes that?s correct. > > Okay. > > -------------------------------------- > > 137???????log_debug(gc, heap, coops)("UseLargePages can't be set > > with HeapDir option."); > > - Is it enough to print log message instead of warning message? > > i.e. Don't we need something at arguments.cpp:3656, similar to NUMA > > flags? > > ??????A: If don?t disable UseLargePages like UseNUMA because large > > pages can be used for other allocation such as CodeCache. > > I meant to changing log_debug() to log_warning(). > > If UseNUMA is enabled, we print warning at arguments.cpp:3725 while > > UseLargePages is enabled, we print debug message. > > In addition, is this enough? Don't we need to force disabling > > UseLargePages? > > What happens if both options are enabled? > > ------------------------------------ > > 603 ReservedHeapSpace::ReservedHeapSpace(size_t size, size_t > > alignment,? > > bool large, const char* backing_fs_for_heap) > > - ReservedSpace has '_backing_fd' but the constructor doesn't take > > it as a parameter and only ReservedHeapSpace uses it. This seems > > not ideal, couldn't make it better? I know actual logic is at > > >ReservedSpace so it is not convenient to add _backing_fs_for_heap > > at ReservedHeapSapce. > > ?????A: ReservedHeapSpace depends on few functions in ReservedSpace > > such as initialize(), release(). So instead of passing it as > > argument, I set it as a propert of ReservedSpace. > > Okay. > > ----------------------------------- > > 1663 > > - You removed os::attempt_reserve_memory_at() from os.cpp and split > > into os_posix.cpp and os_windows.cpp. > > ?But I think you should remain os::attempt_reserve_memory_at() at > > os.cpp and implement os specific stuffs at each os_xxx.cpp files > > for pd_xxx. Of cource move MemTracker function call as well. > > ???????A: I do it this way to reduce the redundant code, If I > > implement in OS specific files in pd_xxx(), the code to replace > > reserved mapping with file mapping > > >(replace_existing_mapping_with_dax_file_mapping()) will be > > redundant. > > ????????????Still if you feel I will do the change and see how it > > looks. > > I would prefer to have pd_xxx() even though there's some redundant > > code. > > [Kharbas, Kishor] We also had a discussion about this offline, I will > address this in the next webrev. > > > Thanks, > > Sangheon > > -----Original Message----- > > From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] > > Sent: Tuesday, October 24, 2017 8:39 AM > > To: Kharbas, Kishor ; sangheon.kim? > > ; 'hotspot-gc-dev at openjdk.java.net' > > > > Cc: hotspot-runtime-dev at openjdk.java.net > > Subject: Re: RFR(M): 8171181: Supporting heap allocation on? > > alternative memory devices > > > > Hi Kishor, > > > > ? initial review using the webrevs, and the comment thread in > > hotspot-? > > runtime-dev (readded to cc because questions were left unanswered > > there). > > > > The code also touches runtime code quite a bit, so I would > > appreciate? > > comments by the runtime team. > > > > On Thu, 2017-10-19 at 15:00 +0000, Kharbas, Kishor wrote: > > > Hi Sangheon, > > > > > > Thanks for the review and comments. > > > I created two webrevs: > > > webrev.07 : Is the rebase of webrev.06 on jdk10???? > > > (http://cr.openjdk > > > .java.net/~kkharbas/8171181/webrev.07/) > > > webrev.08 : Is incremental webrev over 07.?????????????? > > > (http://cr.op > > > enjdk.java.net/~kkharbas/8171181/webrev.08/) > > > > > > > Some first comments on the webrev follow: > > > > ?- could you please provide both a full and incremental webrev for? > > your changes? The former help the reviewers late to the party, the? > > incremental ones the people already working with you. > > > > [Kharbas, Kishor] Yes. I will keep this in mind. > > > ?- the patch has not been rebased to jdk10/hs as Sangheon > > suggested. > > Probably you rebased to jdk10/jdk10? > > > > [Kharbas, Kishor] My bad, it was rebased to jdk10/jdk10. New webrev's > will be based on jdk10/hs. > > > os_posix.cpp: > > > > ?- os::create_file_for_heap: the malloc allocates one byte too > > little,? > > creating an automatic write outside the buffer at the strcat() > > method. > > > > [Kharbas, Kishor] Done. > > > ?- os::create_file_for_heap does not use the size parameter. Please > > remove. > > > > [Kharbas, Kishor] Done. > > > ?- in several places: improve the output of the error messages to > > have? > > meaning for the user. Not just "malloc failed"; there were already? > > some suggestions in the referenced thread at ? > > http://mail.openjdk.java.net/p ipermail/hotspot-runtime-dev/2017- > > March/022733.html . > > > > Same for the asserts, and particularly for them - they will > > implicitly? > > terminate the VM, so e.g. a "fchmod error" is definitely too > > little.? > > At least error no/output of os::strerror() etc. would be required > > for quick diagnosis. > > > > I read in some of the emails that you intend to let the caller > > print a? > > good error message. I fear that the caller might not have the? > > necessary information any more to do that in a useful way. > > > > [Kharbas, Kishor] Done. Moved macro assert_with_errror() to top of > file so I can use in my code. > > > ?- in that same thread there has also been the question about the > > need? > > for blocking the signals for the process that has gone unanswered. > > > > [Kharbas, Kishor] Let me get back on this. The implementation at > pmem.io which I use as reference for file operations does blocks > signals. I will check if there is more to it than just a 'good > practice'. > > > ?- in that same thread there has also been the question about why > > the? > > code sets permissions to 0600 manually; this is because of glibc > > 2.0.6 compatibility. > > Glibc 2.06 has been released 1997-12-29. I looked a bit through > > glibc? > > requirements for the Oracle JDK, and while I could not find exact? > > requirements, some posts indicate the need for GLIBC_2.4 at > > minimum? > > (for > > jdk7) which is from 2003. I recommend removing this, it seems? > > unnecessary and completely outdated. > > > > [Kharbas, Kishor] Agree. Removed it. > > > ?- os::malloc should be paired with os::free in > > os::create_file_for_heap() (2x) > > > > [Kharbas, Kishor] Done. > ? > > ?- I am not sure whether the methods that deal with mapping memory > > to? > > a file should have "dax" in their name. The code seems to be > > pretty? > > generically map memory into a file. > > > > [Kharbas, Kishor] I used 'dax' in the name because to get direct > mapping to DAX device you need to use 'MAP_SHARED'. That makes this > map function different than generic map function. But it?s a minor > point. I can change the name if it's still desired./mkstem/ > > > ?- I am not sure if the current approach to FS errors after mkstemp > > is? > > very nice. Please at least print a meaningful warning message to > > the log. > > > > [Kharbas, Kishor] Added a warning() message when mkstemp could not > create a file. > > > ?- please also specify the meaning of the return value for > > os::create_file_for_heap() in the documentation. > > > > [Kharbas, Kishor] Added more information in the comment before > function's declaration. > > > ?- there is a missing "p" in the name of > > os::reserve_mmaped_memory(). > > > > [Kharbas, Kishor] Done. > > > ?- os::util_posix_fallocate(): s/continous/continuous > > > > [Kharbas, Kishor] Done. > > > ?- os::map_memory_to_dax_file(): according to man pages,? > > posix_fallocate does not set errno, but the code checking its > > return? > > (or > > util_posix_fallocate()) expects it to do so. > > > > [Kharbas, Kishor] I removed printing of error string. Can the return > error value of posix_fallocate() we used as argument to > os::strerror(errno)? > > > ?- os::attempt_reserve_memory_at(), in the call to > > pd_attempt_reserve_memory_at() for AIX, in the third parameter, > > please? > > use the exact name of the parameter in the comment. > > > > [Kharbas, Kishor] Done. > > > ?- os::attempt_reserve_memory_at(), the second "if" needs a blank? > > before the bracket. > > > > [Kharbas, Kishor] Done. > > > ?- os::attempt_reserve_memory_at(), in the comment, > > s/mmemory/memory > > > > [Kharbas, Kishor] Done. > > > ?- in that same method, the #if..#endif block should be aligned > > like? > > other similar blocks. > > > > [Kharbas, Kishor] Done. > > > ?- in that same method, the "else" should be on the same line as > > the? > > closing bracket of the if-body. > > > > [Kharbas, Kishor] Done. > > > os_windows.cpp: > > > > os::create_file_for_heap(): > > ?- same bug about length of allocation the _alloca call. Not? > > completely sure about why use alloca (or not use alloca in > > os_posix.cpp). > > > > [Kharbas, Kishor] Changed to have uniform implementation. > > > ?- this version calls os::native_path(). It seems strange to not > > do? > > that as well in the others (although it does nothing). > > > > [Kharbas, Kishor] Added in os_posix.cpp to make it uniform. > > > ?- os::reserve_memory_aligned(): an "else" should be moved into > > the? > > same line as the closing bracket. > > > > [Kharbas, Kishor] Done. > > > universe.cpp: > > > > ?- Universe::reserve_heap(): I do not think this functionality has? > > anything to do with compressed oops, so the log message should not > > have the coops tag. > > The log message could/should have more information about the file > > location. > > I also recommend using "Java heap" instead of "Heap" here, as the? > > latter is ambiguous. > > > > [Kharbas, Kishor] Done. Added more info to log message. > > > virtualspace.cpp: > > > > ?- in failed_to_reserve_as_requested: "else" indentation. Also, > > the? > > "else { if (..." could be shortened to "} else if (..." to avoid > > the extra indentation level. > > (2x) > > > > [Kharbas, Kishor] Done. > > > ?- ReservedSpace::initialize(): s/upto/up to > > > > [Kharbas, Kishor] Done. > > > ?- ReservedSpace::initialize(): remove the coops tag here as well. > > Also, the info message might be more similar to the comment above? > > where it says that large page support is up to the file system, and > > the flag ignored. > > > > [Kharbas, Kishor] Done. > > ?- the code contains the snippet > > > > if (_fd_for_heap != -1) { > > ? if (!os::unmap_memory... > > } else { > > ? if (!os::release_memory... > > } > > > > at least twice. Maybe this code snippet could be extracted into a > > new? > > method. > > > > [Kharbas, Kishor] Defined a new static function > unmap_or_release_memory() > > > ?- (ignore if you want) ReservedHeapSpace::ReservedHeapSpace, at > > the? > > end > > - maybe the condition is easier to read if it looks like: > > > > if (_fd_for_heap != -1) { > > ? os::close(_fd_for_heap); > > } > > > > than it is now. > > [Kharbas, Kishor] I agree. Done. > > > > ?- virtualspace.hpp: in the changed constructor signature, please > > add? > > a comment about what backing_fs_for_heap actually means (and what? > > happens if set). > > > > [Kharbas, Kishor] Done. > > > arguments.cpp: > > > > ?- Arguments::finalize_vm_init_args: maybe at the place where the? > > change adds the warning/info message about NUMA support being > > specific? > > to the file system; also add the same warning about UseLargePages > > that? > > is located elsewhere. > > > > Like "UseXXXX may not be supported in some specific file system? > > implementations." - or just ignore as Andrew said in the other > > email thread. > > > > Note that I am not sure that the selected place is the correct > > place? > > for such warning about incompatible flags (and maybe? > > UseNUMA/UseLargePages should be forcibly disabled here? But then? > > again, it does not hurt just having it enabled?). > > > > I will let the runtime team comment as a lot of that argument? > > processing changed in jdk9. > > > > Maybe Arguments::check_vm_args_consistency() is better. > > > > There is similar code in Arguments::adjust_after_os()... > > > > [Kharbas, Kishor] In progress. I will get back on these comments in > next reply. > > > os.cpp: > > > > ?- os::reserve_memory: "else" indentation > > > > ?- os::reserve_memory: I do not follow that comment - it explains > > the? > > code as written, not why, and what map_memory_to_dax_file() has to > > do? > > with AIX and SHM. It uses an outdated method name, and also > > s/your/our. > > > > Probably the whole comment can be removed. > > > > [Kharbas, Kishor] I changed the comment. In the beginning I had > called pd_reserve_memory() followed by > replace_existing_mapping_with_dax_file_mapping(), but later realized > that AIX can allocate from SHM, in which case it's not straight > forward to replace the mapping. So I leave this comment to make sure > in future I (or someone else) does not ponder over the same thing. > > > os.hpp: > > > > ?- indentation of the new #if defined clause > > > > ?- here I may probably be speaking wrongly on behalf of the > > runtime? > > team, but os.hpp, as an abstraction of the OS should probably not > > have? > > platform specific ifdefs in it. > > > > [Kharbas, Kishor] Change the indentation. I will address abstraction > issue in next reply. > > > Thanks, > > ? Thomas > > > > > In webrev08 I have addressed all the comments, except for ones > > > below: > > > > > > --------------------------------------- > > > src/os/aix/vm/os_aix.cpp > > > > > > 2514 char* os::pd_attempt_reserve_memory_at(size_t bytes, char*? > > > requested_addr, bool use_SHM) { > > > - Question. Why os_aix has additional parameter at? > > > pd_attemp_reserve_memory_at()? Probably only AIX has shmated > > > > memory > > > implementation? > > > ???????? A: Yes that?s correct. > > > > > > -------------------------------------- > > > 137?????? log_debug(gc, heap, coops)("UseLargePages can't be set? > > > with HeapDir option."); > > > - Is it enough to print log message instead of warning message? > > > i.e. > > > Don't we need something at arguments.cpp:3656, similar to NUMA > > > flags? > > > ?????? A: If don?t disable UseLargePages like UseNUMA because > > > large? > > > pages can be used for other allocation such as CodeCache. > > > > > > ------------------------------------ > > > > > > 603 ReservedHeapSpace::ReservedHeapSpace(size_t size, size_t? > > > alignment, bool large, const char* backing_fs_for_heap) > > > - ReservedSpace has '_backing_fd' but the constructor doesn't > > > take? > > > it as a parameter and only ReservedHeapSpace uses it. This seems > > > not? > > > ideal, couldn't make it better? I know actual logic is at? > > > ReservedSpace so it is not convenient to add _backing_fs_for_heap > > > at? > > > ReservedHeapSapce. > > > ????? A: ReservedHeapSpace depends on few functions in > > > ReservedSpace? > > > such as initialize(), release(). So instead of passing it as? > > > argument, I set it as a propert of ReservedSpace. > > > > > > ----------------------------------- > > > 1663 > > > - You removed os::attempt_reserve_memory_at() from os.cpp and > > > split? > > > into os_posix.cpp and os_windows.cpp. > > > ? But I think you should remain os::attempt_reserve_memory_at() > > > at? > > > os.cpp and implement os specific stuffs at each os_xxx.cpp files > > > for? > > > pd_xxx. Of cource move MemTracker function call as well. > > > ??????? A: I do it this way to reduce the redundant code, If I? > > > implement in OS specific files in pd_xxx(), the code to replace? > > > reserved mapping with file mapping > > > (replace_existing_mapping_with_dax_file_mapping()) will be > > > redundant. > > > ???????????? Still if you feel I will do the change and see how > > > it? > > > looks. > > > > > > Regards > > > Kishor > > > > > > > > > From: sangheon.kim [mailto:sangheon.kim at oracle.com] > > > Sent: Tuesday, September 26, 2017 3:18 PM > > > To: Kharbas, Kishor ;? > > > 'hotspot-gc-dev at openj dk.java.net' > > et> > > > Subject: Re: RFR(M): 8171181: Supporting heap allocation on? > > > alternative memory devices > > > > > > Hi Kishor, > > > > > > On 07/20/2017 06:34 PM, Kharbas, Kishor wrote: > > > I have a new version of this patch at? > > > http://cr.openjdk.java.net/~kkh arbas/8171181/webrev.06/ > > > > > > This version has been tested on Windows, Linux, Solaris and Mac > > > OS.? > > > I could not get access to AIX for testing. > > > I used tmpfs to test the functionality. Cases that were tested > > > were. > > > 1.?????? Allocation of heap using file mapping when ?XX:HeapDir=? > > > option is used. > > > 2.?????? Creation of nameless temporary file for Heap allocation? > > > which prevents access to file using its name. > > > 3.?????? Correct deletion and freeing up of space allocated for > > > file? > > > under different exit conditions. > > > 4.?????? Error handling when path specified is not present, heap? > > > size is more than size of file system, etc. > > > Overall seems good. > > > I tried to list some missing part. > > > > > > 1. Please rebase with consolidated repository. (jdk10/hs) 2. > > > Build? > > > failure on Solaris. > > > ??? (Sorry no build error log file, as I tested a few weeks ago, > > > it? > > > is > > > deleted) 3. Have you tested with various heap reserve path using? > > > HeapBaseMinAddress flag? i.e. to test code path of > > > ReservedHeapSpace::try_reserve_heap() and try_reserve_range(). > > > 4. How about adding HeapDir allocation success message? e.g. > > > gc+heap+coops=info > > > 5. Adding JTREG test. Probably we would need to verify this? > > > allocation is succeeded via #4 added allocation success message. > > > 6. CSR (Compatibility & Specification Review). At some point, > > > you? > > > need to file another CR of 'CSR' type to add a new flag of > > > 'HeapDir'. > > > 7. It will be much appreciated if you provide incremental webrev. > > > I? > > > think 06(this version) vs. 07(rebase version) would be hard to > > > get. > > > Probably from 08 version. > > > > > > Here's my comments. > > > ----------------------------- > > > src/os/aix/vm/os_aix.cpp > > > > > > 2514 char* os::pd_attempt_reserve_memory_at(size_t bytes, char*? > > > requested_addr, bool use_SHM) { > > > - Question. Why os_aix has additional parameter at? > > > pd_attemp_reserve_memory_at()? Probably only AIX has shmated > > > > memory > > > implementation? > > > > > > ----------------------------- > > > src/os/posix/vm/os_posix.cpp > > > > > > 148?? char *fullname = (char*)::malloc(strlen(dir) +? > > > sizeof(name_template)); > > > - Use os::malloc() > > > > > > 196?? int flags; > > > 197 > > > 198?? flags = MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS; > > > - Combining 196 and 198 seems better to me. > > > > > > 200???? assert((uintptr_t)requested_addr % os::Linux::page_size() > > > ==? > > > 0, "unaligned address"); > > > - Linux dependency on posix file which makes build error on > > > Solaris. > > > Probably os::vm_page_size(). > > > > > > 207?? addr = (char*)::mmap(requested_addr, bytes, PROT_NONE, > > > 208???? flags, -1, 0); > > > - Missing some spaces? Alignment seems odd to me. > > > > > > 226???? if (ret == -1) > > > - Probably you wanted to add handling code? If not, just return > > > ret. > > > > > > 252?? if (addr == MAP_FAILED || (base != NULL && addr != base)) { > > > 253???? if (addr != MAP_FAILED) { > > > 254?????? if (!os::release_memory(addr, size)) { > > > 255???????? warning("Could not release memory on unsuccessful > > > file? > > > mapping"); > > > 256?????? } > > > 257???? } > > > 258???? return NULL; > > > 259?? } > > > - Splitting MAP_FAILED case and another gives better readability > > > to? > > > me. But this is your call. > > > > > > 269 > > > - Extra line. > > > > > > 284?? if (result != NULL && file_desc != -1) { > > > 285???? if > > > (replace_existing_mapping_with_dax_file_mapping(result, > > > bytes, file_desc) == NULL) { > > > 286?????? vm_exit_during_initialization(err_msg("Error in > > > mapping? > > > Java heap at the given filesystem directory")); > > > 287???? } > > > 288 > > > > > > > MemTracker::record_virtual_memory_reserve_and_commit((address)resul > > t > > , > > > bytes, CALLER_PC); > > > 289???? return result; > > > 290?? } > > > 291?? if (result != NULL) { > > > 292???? > > > MemTracker::record_virtual_memory_reserve((address)result, > > > bytes, CALLER_PC); > > > 293?? } > > > - Combining line 284 and 291 seems better to me. > > > 284?? if (result != NULL) { > > > ??????? if (file_desc != -1) { > > > ????????? if > > > (replace_existing_mapping_with_dax_file_mapping(result, > > > bytes, file_desc) == NULL) { > > > ??????????? vm_exit_during_initialization(err_msg("Error in > > > mapping? > > > Java heap at the given filesystem directory")); > > > ????????? } > > > > > > > > > > MemTracker::record_virtual_memory_reserve_and_commit((address)resul > > t > > , > > > bytes, CALLER_PC); > > > ??????? } else { > > > ????????? > > > MemTracker::record_virtual_memory_reserve((address)result, > > > bytes, CALLER_PC); > > > ??????? } > > > ????? } > > > ????? return result; > > > > > > ----------------------------- > > > src/os/windows/vm/os_windows.cpp > > > 3141 // if 'base' is not NULL, function will return NULL if it? > > > cannot get 'base' > > > - Start with uppercase. > > > > > > 3142 // > > > - This line seems redundant. > > > > > > 3151?????? vm_exit_during_initialization(err_msg("Could not > > > allocate? > > > sufficient disk space for heap")); > > > - heap -> Java heap (same as line 3153) > > > > > > 3168?? assert(base != NULL, "base cannot be NULL"); > > > - 'base' -> 'Base' or 'Base address'. > > > > > > 3172 > > > - Redundant line. > > > > > > 3230???? } > > > 3231???? else { > > > -> } else { > > > > > > 3278?? return reserve_memory(bytes, requested_addr, 0); > > > - Is it correct to use '0' not '-1'? It would be better to > > > explain? > > > why we use hard-coded value here. > > > > > > ----------------------------- > > > src/share/vm/memory/universe.cpp > > > - No comments > > > > > > ----------------------------- > > > src/share/vm/memory/virtualspace.cpp > > > - copyright update > > > > > > 74??????????????????????????????????????????? const size_t size,? > > > bool special, bool is_file_mapped= false) > > > - Need space between 'is_file_mapped' and '='. > > > > > > 92?????????? fatal("os::release_memory failed"); > > > - Typo, 'os::unmap_memory failed'. > > > > > > 129?? // If there is a backing file directory for this > > > VirtualSpace? > > > then whether > > > - This is not VirtualSpace. Probably just 'space'. > > > > > > 130?? // large pages are allocated is upto the filesystem the > > > dir? > > > resides in. > > > - 'dir'? Probably 'backing file for Java heap'. > > > > > > 137?????? log_debug(gc, heap, coops)("UseLargePages can't be set? > > > with HeapDir option."); > > > - Is it enough to print log message instead of warning message? > > > i.e. > > > Don't we need something at arguments.cpp:3656, similar to NUMA > > > flags? > > > > > > 191???????? // unmap_memory will do extra work esp. in Windows > > > - esp. -> especially > > > > > > 282?????? } > > > 283?????? else { > > > -> } else { > > > > > > 346?? // If there is a backing file directory for this > > > VirtualSpace? > > > then whether > > > - Again this is not VirtualSpace, so just 'space'. > > > > > > 352???? if (UseLargePages && (!FLAG_IS_DEFAULT(UseLargePages) || > > > 353?????? !FLAG_IS_DEFAULT(LargePageSizeInBytes))) { > > > - Wrong alignment at line 353. Consider to make same as line 380. > > > > > > 603 ReservedHeapSpace::ReservedHeapSpace(size_t size, size_t? > > > alignment, bool large, const char* backing_fs_for_heap) > > > - ReservedSpace has '_backing_fd' but the constructor doesn't > > > take? > > > it as a parameter and only ReservedHeapSpace uses it. This seems > > > not? > > > ideal, couldn't make it better? I know actual logic is at? > > > ReservedSpace so it is not convenient to add _backing_fs_for_heap > > > at? > > > ReservedHeapSapce. > > > > > > ----------------------------- > > > src/share/vm/memory/virtualspace.hpp > > > 40?? int??? _backing_fd; > > > - I would prefer to have better name to describe. > > > ? e.g. as command-line option name is 'HeapDir', _heap_fd or? > > > _fd_for_heap(similar to below)? > > > > > > 115?? ReservedHeapSpace(size_t size, size_t > > > forced_base_alignment,? > > > bool large, const char* backingFSforHeap = NULL); > > > - Snake case. How about 'fs_for_heap' or 'heap_fs'? > > > > > > ----------------------------- > > > src/share/vm/runtime/arguments.cpp > > > 3655???? FLAG_SET_CMDLINE(bool, UseNUMA, false); > > > - (questions) Don't need to add a warning message for? > > > UseLargePagesSame=true as commented virtualspace.cpp:137? > > > > > > ----------------------------- > > > src/share/vm/runtime/globals.hpp > > > - No comments > > > > > > ----------------------------- > > > src/share/vm/runtime/os.cpp > > > > > > 1632 > > > - Extra line. > > > > > > 1642?? } > > > 1643?? else { > > > -> } else { > > > > > > 1663 > > > - You removed os::attempt_reserve_memory_at() from os.cpp and > > > split? > > > into os_posix.cpp and os_windows.cpp. > > > ? But I think you should remain os::attempt_reserve_memory_at() > > > at? > > > os.cpp and implement os specific stuffs at each os_xxx.cpp files > > > for? > > > pd_xxx. Of cource move MemTracker function call as well. > > > > > > ----------------------------- > > > src/share/vm/runtime/os.hpp > > > > > > 349?? // replace existing reserved memory with file mapping > > > - Start with uppercase letter. > > > > > > Thanks, > > > Sangheon > > > > > > > > > > > > > > > - Kishor > > > > > > From: Kharbas, Kishor > > > Sent: Tuesday, July 11, 2017 6:40 PM > > > To: 'hotspot-gc-dev at openjdk.java.net'? > > > > > t> > > > Cc: Kharbas, Kishor > > > Subject: RFR(M): 8171181: Supporting heap allocation on > > > alternative? > > > memory devices > > > > > > Greetings, > > > > > > I have an updated patch for JEP? > > > https://bugs.openjdk.java.net/browse/ > > > JDK-8171181 at? > > > http://cr.openjdk.java.net/~kkharbas/8171181/webrev.05 > > > This patch fixes the bugs pointed earlier and other suggestions > > > to? > > > make the code less intrusive. > > > > > > I have also sent this to ?hotspot-runtime-dev? mailing list? > > > (included below). > > > > > > I would appreciate comments and feedback. > > > > > > Thanks > > > Kishor > > > > > > From: Kharbas, Kishor > > > Sent: Monday, July 10, 2017 1:53 PM > > > To: hotspot-runtime-dev at openjdk.java.net > > > Cc: Kharbas, Kishor > > > Subject: RFR(M): 8171181: Supporting heap allocation on > > > alternative? > > > memory devices > > > > > > Hello all! > > > > > > I have an updated patch for? > > > https://bugs.openjdk.java.net/browse/JDK- > > > 8171181 at http://cr.openjdk.java.net/~kkharbas/8171181/webrev.05 > > > I have lost the old email chain so had to start a fresh one. The? > > > archived conversation can be found at -? > > > http://mail.openjdk.java.net/? > > > pipermail/hotspot-runtime-dev/2017-March/022733.html > > > > > > 1.?????? I have worked on all the comments and fixed the bugs.? > > > Mainly bugs fixed are related to sigprocmask() and changed the? > > > implementation such that ?fd? is not passed all the way down the? > > > call stack. Thus minimizing function signature changes. > > > > > > 2.?????? Patch supports all OS?es. Consolidated all Posix > > > compliant? > > > OS?s implementation in os_posix.cpp. > > > > > > 3.?????? The patch is tested on Windows and Linux. Working on? > > > testing it on other OS?es. > > > > > > Let me know if this version looks clean and correct. > > > > > > Thanks > > > Kishor > > > > > From dmitry.samersoff at bell-sw.com Mon Oct 30 11:04:39 2017 From: dmitry.samersoff at bell-sw.com (Dmitry Samersoff) Date: Mon, 30 Oct 2017 14:04:39 +0300 Subject: RFR(M) JDK-8188791 Move AppCDS implementation from closed repo to open repo In-Reply-To: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> Message-ID: Ioi, I'd tried to apply your patch to latest open JDK10 and the compilation fails with: /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: error: ?class SharedClassPathEntry? has no member named ?is_jrt? Did I miss something? -Dmitry On 13.10.2017 02:48, Ioi Lam wrote: > Hi, > > Please review this change set. > > http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ > ??? https://bugs.openjdk.java.net/browse/JDK-8188791 > > This is the first step of implementing the following JEP, which moves > AppCDS from > closed repos into the openjdk repo: > > ??? https://bugs.openjdk.java.net/browse/JDK-8185996 > > In JDK 9, significant portion of AppCDS code resided in the closed repo. > As part > of the open-sourcing effort of JDK 18.3, we will move the source code > into the > open repo. > > In this changeset, the code is moved verbatim as much as possible. The > intention is > only to relocate the sources, not to changing existing behaviors, and not > to do any sort of refactoring. > > Most of the "diffs" shown in this webrev are the result of copying the > closed source > files on top of files of the same name in the open repo. So in > reviewing, instead of > focusing on what's "changed", it's better to focus on the entire content > of the new > version of each file. > > The only functional change in this task is that the UseAppCDS flag is > changed from > a "commercial" flag to a regular "product" flag. This is because > "commercial" > flags are not supported by the OpenJDK build. > > Source code refactoring may be desirable, because the old open/closed > source > code structure had introduced some intermediary APIs to connect code > between > the two repos. Such API should be removed in a separate RFE. > > Also, some AppCDS tests are currently in the closed repo. These tests > will be > moved in a separate task. See JDK-8188792 for details. > > All the AppCDS tests (currently still in closed sources) passed with > both Oracle JDK > and OpenJDK. > > Thanks > - Ioi From goetz.lindenmaier at sap.com Mon Oct 30 12:49:36 2017 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Mon, 30 Oct 2017 12:49:36 +0000 Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace fragmentation In-Reply-To: References: Message-ID: <667f07ea2fb64e97b8a149310961693f@sap.com> Hi Thomas, the change looks good and I know we have made good use of this already. Can you look into the chunks? It could be useful to know that, for example, a medium chunk is used by a class loader, but still mostly empty (i.e., empty but not free). Well, there is no third variant after lower and upper case, but maybe empty parts could be indicated in the line with the dots by 'e's (obviously at the granularity of small chunks). Thanks, Goetz. > -----Original Message----- > From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- > bounces at openjdk.java.net] On Behalf Of Thomas St?fe > Sent: Wednesday, October 25, 2017 6:52 AM > To: hotspot-runtime-dev at openjdk.java.net > Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace > fragmentation > > Hi all, > > could I please have your thoughts and reviews for this enhancement: > > Issue: https://bugs.openjdk.java.net/browse/JDK-8189864 > Webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8189864- > metaspace-map/webrev.00/webrev/ > > At SAP, we added a something we call a metaspace map to the metaspace > analysis functions. This is a very simple ASCII print of the metaspace > layout. It shows the composition of the VirtualSpaceNodes on a chunk basis. > It facilitates examining fragmentation and has been proven useful when > experimenting with the metaspace allocator. > > This change adds the map printing code and invokes it if a Metaspace OOM > occurs and -Xlog:gc+metaspace+freelist=debug is active (in this case, we > already print out a bunch of things). We also may consider adding this to > NMT or as a jcmd addition, but I did not want to overload the patch. > > Example output looks like this (mind that this will only make sense with a > monospaced font). Dots indicate starts of chunks. Letters indicate chunk > type - (H)umongous, (M)edium, (S)mall (X)specialized - with lower case > letters indicating > a free chunk, upper case letters indicating a chunk in use. > > 0x0000000100000000: ...... > xxxxxxHHHHHHHHHHHHHHHHHHHHHHHH > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > HHHHHHHHH > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > 0x0000000100020000: > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > HHHHHHHHH > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > 0x0000000100040000: > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > HHHHHHHHH > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH > 0x0000000100060000: . . . . . . . . . . . . . . > SSSSSSSSSSSSMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMSSSSSSSSMMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > M > 0x0000000100080000: . . . . > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm > mmmmmmmmmmmmmmmmMMMMMM > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > M > 0x00000001000a0000: . . . . > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > M > 0x00000001000c0000: . . . . > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm > mmmmmmmmmmmmmmmmMMMMMM > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > M > 0x00000001000e0000: . . . . > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > M > 0x0000000100100000: . . . . . . . . . . . . . . . . . . . . . . . . > MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMMMMSSSSMMMMMMMMMMMMM > MMMMMMMMMMMMMMMMMMMSS > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > 0x0000000100120000: . . . . . . . . . . . . . . . . . . . . . . . . . . . > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > > > Thank you! > > Thomas From zgu at redhat.com Mon Oct 30 13:57:50 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 30 Oct 2017 09:57:50 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <6710b89e-1b1a-aed2-6273-8bb2bb0fcdbe@oracle.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <6710b89e-1b1a-aed2-6273-8bb2bb0fcdbe@oracle.com> Message-ID: Hi David, >> During VM exit, we are not supposed to request a safepoint, correct? > > VM exit uses a safepoint - can you use it do gather the data? > Apparently, not all VM exits go through safepoint. If Java is terminated due to application's main method exists (JVM is terminated via jni_DestoryJavaVM) , there is not safepoint executed. Thanks, -Zhengyu > David > >> Without a safepoint, it is unsafe to walk class loaders. >> >> Thanks, >> >> -Zhengyu >> >> >>> Thanks! Thomas >>> >>> On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe >>> > wrote: >>> >>> Hi Zhengyu, >>> >>> On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu >> > wrote: >>> >>> Hi Thomas, >>> >>> On 10/24/2017 12:08 PM, Thomas St?fe wrote: >>> >>> Hi Zhengyu, >>> >>> - VM_PrintMetadata still has the _unit member, but I see it >>> nowhere used. >>> >>> - I would have split the summary between class and non-class >>> area, that may have been more useful. But this is a matter >>> of taste, I leave it up to you. >>> >>> Agree, should go little further. >>> >>> Summary: >>> Total class loaders= 754 capacity= 67528.00KB used= >>> 61216.88KB free= 9453.11KB waste= 38.73KB >>> Metadata capacity= 58780.00KB used= >>> 54053.45KB free= 4726.55KB waste= 36.38KB >>> Class data capacity= 8748.00KB used= >>> 7163.43KB free= 1584.57KB waste= 2.35KB >>> >>> For anonymous classes= 580 capacity= 2732.00KB used= >>> 1065.06KB free= 1666.94KB waste= 16.27KB >>> Metadata capacity= 2152.00KB used= >>> 738.46KB free= 1413.54KB waste= 16.27KB >>> Class data capacity= 580.00KB used= >>> 326.60KB free= 253.40KB waste= 0.00KB >>> >>> >>> >>> Nice, thanks :) >>> >>> Updated webrev: >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >>> >>> >>> >>> All is well. Note that you could reduce the footprint of your change >>> somewhat by defining a structure like this: >>> >>> struct numbers_t { size_t cap; size_t used; size_t free; size_t >>> waste; } >>> >>> and replacing the many members in PrintCLDMetaspaceInfoClosure with >>> instances of this structure: >>> >>> numbers_t total; >>> numbers_t total_class; >>> numbers_t total_anon; >>> numbers_t total_anon_class; >>> Depending on how far you go, your code could deflate quite a bit. >>> Give the structure a default ctor and your large initializer list >>> goes away; give it a printing function and you reduce >>> print-summary() to four function calls; with something like an >>> numbers_t::add(size_t cap, size_t used, size_t free, size_t waste) >>> you can reduce the additions in >>> PrintCLDMetaspaceInfoClosure::print_metaspace() to four lines. >>> >>> Just a suggestion, I leave it up to you if you do this. >>> >>> Lines 3108 and 3129 miss each a space before BytesPerWord. Change >>> looks fine otherwise. >>> >>> Thanks, Thomas >>> >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> All else looks fine to me now. I do not need another review. >>> >>> Thanks for your work, this will come in handy! >>> >>> ..Thomas >>> >>> On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu >> >> >> wrote: >>> >>> Hi Thomas, >>> >>> >>> Not sure whether we misunderstood each other. I was >>> thinking of >>> something in the line of: >>> >>> <<<< >>> .... >>> ClassLoader: >>> jdk/internal/reflect/DelegatingClassLoader >>> Metadata: >>> capacity: 5.00KB used: 3.32KB >>> free: 1.68KB >>> waste: 0.00KB >>> Class data: >>> capacity: 1.00KB used: 0.54KB >>> free: 0.46KB >>> waste: 0.00KB >>> >>> ClassLoader: for anonymous class >>> Metadata: >>> capacity: 1.00KB used: 1.00KB >>> free: 0.00KB >>> waste: 0.00KB >>> Class data: >>> capacity: 1.00KB used: 0.64KB >>> free: 0.36KB >>> waste: 0.00KB >>> .... >>> >>> Summary: >>> XX class loaders encountered, total capacity: xx, >>> total waste: xx. >>> >>> Peak usage by class loader: xxxx >>> >>>> >>> >>> Added summary lines: >>> >>> Total class loaders= 56 capacity= 6378.00KB >>> used= 6205.08KB free= 172.92KB waste= 1.44KB >>> For anonymous classes= 54 capacity= 212.00KB >>> used= 96.95KB >>> free= 115.05KB waste= 0.94KB >>> >>> >>> >>> >>> These numbers would not be interpretation, just a >>> convenience to >>> the reader to get a clear picture. >>> >>> It even may be useful to separate the output >>> between basic and >>> detailed mode, and in basic mode omit all the >>> single class >>> loaders and just print the summary line. >>> >>> Updated webrev: >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >>> >>> >> > >>> >> >>> >> >> >>> >>> >>> >>> metaspace.hpp: >>> >>> I would have preferred to keep scale_unit file >>> local static >>> instead of exposing it via MetaspaceAux, because it >>> does not >>> really have anything to do with metaspace. >>> >>> Fixed >>> >>> >>> Otherwise ok. >>> >>> --- >>> >>> metaspace.cpp >>> >>> - ChunkManager::print_statistics(): thanks for >>> reusing this >>> function! >>> -> Scale can only be ever 1, K, M, G, yes? >>> So, could you >>> add an assert at the start of the function, and a >>> comment at the >>> prototype or function head? >>> -> Also, now that >>> ChunkManager::print_statistics() takes a >>> scale, could you please change the printout to use >>> floats like >>> you did in >>> PrintCLDMetaspaceInfoClosure::print_metaspace()? >>> >>> Done. >>> >>> >>> Chunkmanager (non-class): >>> 0 specialized (128 bytes) chunks, total 0.00KB >>> 0 small (512 bytes) chunks, total 0.00KB >>> 0 medium (8192 bytes) chunks, total 0.00KB >>> 0 humongous chunks, total 0.00KB >>> total size: 0.00KB. >>> Chunkmanager (class): >>> 0 specialized (128 bytes) chunks, total 0.00KB >>> 0 small (256 bytes) chunks, total 0.00KB >>> 0 medium (4096 bytes) chunks, total 0.00KB >>> 0 humongous chunks, total 0.00KB >>> total size: 0.00KB. >>> >>> >>> - I am concerned about races in >>> PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >>> Maybe my understanding is limited. We bail if >>> cld->is_unloading. >>> But nothing prevents a ClassLoaderDataGraph from >>> starting to >>> unload a ClassLoaderData and tearing down the >>> Metaspace after we >>> started printing it in another thread, right? Also, >>> I do not see >>> any fences. >>> >>> So, I wonder whether PrintCLDMetaspaceInfoClosure >>> should run in >>> lock protection. And then, I wonder if it would be >>> good to >>> separate data gathering and printing. We print to a >>> (at least in >>> principle) unknown output stream, which may be >>> doing slow File >>> IO or even Network IO. Doing this under lock >>> protection seems >>> not a good idea (but I see other places where this >>> happens too). >>> >>> For an example, see >>> ChunkManager::get_statistic() vs >>> ChunkManager::print_statistic(). Could you do the >>> same, have a >>> data gathering step where you collect your >>> quadrupel in a variable >>> sized list of >>> structures in memory, and print it out in a >>> separate step? I >>> realize though that the effort would be larger than >>> for what we >>> did in ChunkManager::get_statistic(), where we only >>> fill a >>> structure. >>> >>> >>> Unlike other NMT queries, this query is executed at a >>> safepoint by >>> VMThread, so it should be okay. >>> >>> Updated webrev: >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >>> >>> >> > >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> ------ >>> >>> vm_operations.hpp >>> >>> - VM_PrintMetadata : do we still need _unit? >>> >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> >>> Thanks! >>> >>> Thomas >>> >>> >>> >>> On 10/23/2017 11:08 AM, Thomas St?fe wrote: >>> >>> >>> >>> On Mon, Oct 23, 2017 at 5:03 PM, >>> Zhengyu Gu >>> >>> > >>> >> >> >> >>> >>> > >>> >> >> >>>> wrote: >>> >>> >>> >>> Okay. So this is important for >>> understanding cases >>> where we have >>> a large number of miniscule class >>> loaders, >>> each one >>> only having >>> a few numbers of chunks. In this >>> case, would it be >>> useful to >>> have a running total of "free" >>> and "waste", >>> too? Or >>> even better, >>> also average and peak values of >>> "free" and >>> "waste" (to tell >>> apart cases where you have one >>> outlier vs >>> cases where >>> just all >>> metaspaces waste a lot of >>> memory). >>> Just out of curiosity, I >>> looked at >>> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >>> >>> >> > >>> >> >>> >> >>> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> and >>> it seems that "free" is the >>> problem, not >>> "waste"? So I >>> guess >>> what happens is that all the >>> little classloaders >>> allocate just >>> enough memory to push them over >>> "_small_chunk_limit=4", >>> so they >>> allocate the first medium chunk, >>> from which >>> then they >>> take a >>> small bite and leave the rest >>> laying around? >>> >>> Yes. The free space of anonymous >>> classes should be >>> counted >>> as waste, >>> in my opinion. I am not sure if >>> everyone agrees, >>> so I took the >>> summary line out of this patch. >>> >>> I would be more than happy to restore >>> the summary >>> line, if >>> you find >>> it is useful :-) >>> >>> >>> I agree with you. Typically class loaders >>> stop loading >>> at some >>> point, especially these tine ones, and >>> often will not >>> resume >>> loading. So, effectivly, the space is >>> wasted. It still >>> helps to >>> distinguish "free" in the current chunks >>> from "free" in the >>> other chunks to tell this situation apart >>> from a >>> situation where >>> we have just a bug in metaspace chunk >>> handling >>> preventing us to >>> use up our chunks. >>> >>> >>> The latter would be an >>> important >>> addition, >>> regardless >>> if this is >>> for customers or for us. >>> Chunks idling in >>> freelists can >>> make a >>> huge impact, and >>> unfortunately this >>> may happen >>> in perfectly >>> valid cases. See e.g. >>> our JEP >>> "https://bugs.openjdk.java.net/browse/JDK-8166690 >>> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>>>". >>> We have >>> already a printing >>> routine for free >>> chunks, in >>> ChunkManager::print_all_chunkmanagers(). Could >>> you add >>> this to >>> your output? >>> >>> >>> Make sense! Could you >>> recommend what data and >>> format you >>> would like >>> to see? >>> >>> >>> >>> Would not >>> ChunkManager::print_all_chunkmanagers() be >>> sufficient? >>> >>> Okay, I might need to tweak output >>> format. >>> >>> >>> Fine by me. Nothing depends on it yet. >>> >>> Thanks, Thomas >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> >>> ------------- >>> >>> Other than above notes, >>> the changes seem >>> straighforward, I did >>> not see anything wrong. >>> Minor nits: >>> >>> - >>> PrintCLDMetaspaceInfoClosure::_out, >>> _scale >>> and _unit >>> could all >>> be constant (with _out >>> being a >>> outputStream* >>> const). >>> - We also do not need to >>> provide >>> scale *and* >>> unit as >>> argument, >>> one can be deduced from >>> the other. >>> This would >>> also prevent >>> mismatches.We do need >>> scale here, >>> since nmt >>> command >>> line can set >>> scale and we do >>> >>> have to honor that. >>> >>> However, passing unit is >>> ugly! I don't >>> want to >>> introduce >>> inverse >>> dependence (metaspace -> >>> nmt), which is >>> also ugly. >>> >>> >>> Yes, that would be regrettable. >>> >>> Probably should move scale >>> mapping code from >>> NMTUtil to a >>> common util? >>> >>> >>> >>> That would be possible, these >>> functions >>> (scale_from_name etc) >>> are simple enough to be moved to >>> a generic file. >>> However, if you >>> pass scale, deducing the string >>> that goes with >>> it is >>> trivial and >>> I would have no qualms doing this >>> by hand in >>> metaspace.cpp. But >>> it is a matter of taste. >>> >>> >>> I did not look closely >>> at locking >>> issues, I assume >>> MetaspaceAux::print_metadata() is >>> supposed to >>> run only in >>> Safepoints? >>> >>> >>> No. sum_xxx_xxx_in_use >>> queries are >>> guarded by locks. >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> Thanks, Thomas >>> >>> >>> >>> Thanks & Kind Regards, >>> Thomas >>> >>> >>> >>> >>> >>> >>> On Fri, Oct 20, 2017 at >>> 4:00 PM, >>> Zhengyu Gu >>> >> >> > >>> >>> >> >>> >> >> > >>> >>> >>> >>> >> >>> > >>> >>> >> >>> >> >> > >>> >>> >>>> >>> >> >> > >>> >>> >> >>> >> >> > >>> >>> >>> >>> >>> >> >>> > >>> >>> >> >>> >> >> > >>> >>> >>>>>> wrote: >>> >>> Up to now, there is >>> little >>> visibility >>> into how >>> metaspaces >>> are used, >>> and by whom. >>> >>> This proposed >>> enhancement gives >>> NMT the >>> ability to >>> report >>> metaspace >>> usages on >>> per-classloader level, >>> via a >>> new NMT command >>> "metadata" >>> >>> >>> For example: >>> >>> 2496: >>> Metaspaces: >>> Metadata space: >>> reserved= 8192KB >>> committed= 5888KB >>> Class space: >>> reserved= 1048576KB >>> committed= 640KB >>> >>> Per-classloader >>> metadata: >>> >>> ClassLoader: for >>> anonymous class >>> Metadata >>> capacity= 5.00KB >>> used= 1.16KB >>> free= 3.84KB >>> waste= 0.05KB >>> Class data >>> capacity= 1.00KB >>> used= 0.59KB >>> free= 0.41KB >>> waste= 0.00KB >>> >>> .... >>> >>> ClassLoader: >>> >>> Metadata >>> capacity= 5640.00KB >>> used= 5607.42KB >>> free= 32.58KB >>> waste= 0.46KB >>> Class data >>> capacity= 520.00KB >>> used= 500.94KB >>> free= 19.06KB >>> waste= 0.02KB >>> >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8189688 >>> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>>>> >>> Webrev: >>> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >>> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >> >>> >> > >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>>>> >>> >>> Test: >>> >>> hotspot_tier1_runtime, >>> fastdebug and >>> release on >>> x64 Linux. >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>> >>> >>> >>> >>> >>> >>> From coleen.phillimore at oracle.com Mon Oct 30 15:12:22 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 30 Oct 2017 11:12:22 -0400 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> Message-ID: <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> Hi Gerard, This looks great.? One small question: *+ // straight forward brute force* *+ inline static int _next_prime(int n) {* *+ int p = n;* *+ for (int i = n; i < (n * n); i++) {* *+ if ((i % 2) != 0) {* *+ p = i;* *+ break;* *+ }* *+ }* *+ return p;* *+ }* Is this how you calculate next prime?? Wouldn't you check if it can mod by 3 and 5 as well? Thanks, Coleen On 10/27/17 2:25 PM, Gerard Ziemski wrote: > hi Coleen, > > Thank you for the review. > > Updated webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev2 > >> On Oct 11, 2017, at 9:10 AM, coleen.phillimore at oracle.com wrote: >> >> >> Gerard, some preliminary comments. >> >> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html >> >> *+{* >> *!_MutexLocker mu(SystemDictionary_lock, THREAD_);* >> *!_Klass* probe = dictionary->find(_d_hash, name, protection_domain);* >> *if (probe != NULL) return probe;* >> ** >> *+ }* >> >> I don't think you need this because dictionary->find() should have the NoSafepointVerifier, so the index will not change. > Done, good catch. > > >> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/classLoaderData.cpp.udiff.html >> >> I think we want a global _some_dictionary_needs_resizing to avoid walking through the CLDG. >> >> And have Dictionary have a field _resizing_needed to avoid the calculation during the safepoint, which is set when adding an entry. >> >> _resizing_needed = *(number_of_entries() > (_resize_load_trigger*table_size()); >> *CLDG::_any_resizing_needed |= _resizing_needed; // or something like that. >> >> I can write more about the rationale of this change in the bug report, if needed. > Done. > > >> Thank you for doing this change. >> Coleen >> >> >> On 10/10/17 4:40 PM, Gerard Ziemski wrote: >>> hi all, >>> >>> Please review this change that adds dynamic resizing of system dictionaries. >>> >>> The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock >>> >>> A few notes: >>> >>> - dynamic resizing is off when either dumping or using shared spaces >>> - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) >>> - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) >>> >>> bug: https://bugs.openjdk.java.net/browse/JDK-8184765 >>> webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev1 >>> >>> >>> cheers >>> From gerard.ziemski at oracle.com Mon Oct 30 15:24:25 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Mon, 30 Oct 2017 10:24:25 -0500 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> Message-ID: hi Coleen, > On Oct 30, 2017, at 10:12 AM, coleen.phillimore at oracle.com wrote: > > > Hi Gerard, > > This looks great. One small question: > + // straight forward brute force > + inline static int _next_prime(int n) { > + int p = n; > + for (int i = n; i < (n * n); i++) { > + if ((i % 2) != 0) { > + p = i; > + break; > + } > + } > + return p; > + } > > Is this how you calculate next prime? Wouldn't you check if it can mod by 3 and 5 as well? As long as it?s not even i.e. mod 2 (and strictly speaking larger than 1). 3 and 5 are prime, so no need to check them. Thank you for the review! cheers > Thanks, > Coleen > > > On 10/27/17 2:25 PM, Gerard Ziemski wrote: >> hi Coleen, >> >> Thank you for the review. >> >> Updated webrev: >> http://cr.openjdk.java.net/~gziemski/8184765_rev2 >> >> >> >>> On Oct 11, 2017, at 9:10 AM, coleen.phillimore at oracle.com >>> wrote: >>> >>> >>> Gerard, some preliminary comments. >>> >>> >>> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html >>> >>> >>> *+{* >>> *!_MutexLocker mu(SystemDictionary_lock, THREAD_);* >>> *!_Klass* probe = dictionary->find(_d_hash, name, protection_domain);* >>> *if (probe != NULL) return probe;* >>> ** >>> *+ }* >>> >>> I don't think you need this because dictionary->find() should have the NoSafepointVerifier, so the index will not change. >>> >> Done, good catch. >> >> >> >>> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/classLoaderData.cpp.udiff.html >>> >>> >>> I think we want a global _some_dictionary_needs_resizing to avoid walking through the CLDG. >>> >>> And have Dictionary have a field _resizing_needed to avoid the calculation during the safepoint, which is set when adding an entry. >>> >>> _resizing_needed = *(number_of_entries() > (_resize_load_trigger*table_size()); >>> *CLDG::_any_resizing_needed |= _resizing_needed; // or something like that. >>> >>> I can write more about the rationale of this change in the bug report, if needed. >>> >> Done. >> >> >> >>> Thank you for doing this change. >>> Coleen >>> >>> >>> On 10/10/17 4:40 PM, Gerard Ziemski wrote: >>> >>>> hi all, >>>> >>>> Please review this change that adds dynamic resizing of system dictionaries. >>>> >>>> The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock >>>> >>>> A few notes: >>>> >>>> - dynamic resizing is off when either dumping or using shared spaces >>>> - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) >>>> - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) >>>> >>>> bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8184765 >>>> >>>> webrev: >>>> http://cr.openjdk.java.net/~gziemski/8184765_rev1 >>>> >>>> >>>> >>>> cheers >>>> >>>> > From zgu at redhat.com Mon Oct 30 15:29:11 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Mon, 30 Oct 2017 11:29:11 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> Message-ID: <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> > > > Yes, this is not trivial. I hacked it in (because I wanted to see your > output at the end of my tests) and had to disable the assert. I never > ran into problems. Should java threads not all be stopped at that point? > Yes, it looks like that all Java threads are stopped at that point. However, I am not so sure about workers. Could there be active workers while JVM exiting? > But this also can be done in a follow up issue. Yes, let's address it separately. https://bugs.openjdk.java.net/browse/JDK-8190357 Updated webrev based on your early comments: Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ Thanks, -Zhengyu > > Thanks, Thomas > > > Thanks! Thomas > > On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe > > >> wrote: > > Hi Zhengyu, > > On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu > >> wrote: > > Hi Thomas, > > On 10/24/2017 12:08 PM, Thomas St?fe wrote: > > Hi Zhengyu, > > - VM_PrintMetadata still has the _unit member, but > I see it > nowhere used. > > - I would have split the summary between class and > non-class > area, that may have been more useful. But this is a > matter > of taste, I leave it up to you. > > Agree, should go little further. > > Summary: > Total class loaders= 754 capacity= 67528.00KB > used= 61216.88KB free= 9453.11KB waste= 38.73KB > Metadata capacity= 58780.00KB > used= 54053.45KB free= 4726.55KB waste= 36.38KB > Class data capacity= 8748.00KB > used= 7163.43KB free= 1584.57KB waste= 2.35KB > > For anonymous classes= 580 capacity= 2732.00KB > used= 1065.06KB free= 1666.94KB waste= 16.27KB > Metadata capacity= 2152.00KB > used= 738.46KB free= 1413.54KB waste= 16.27KB > Class data capacity= 580.00KB > used= 326.60KB free= 253.40KB waste= 0.00KB > > > > Nice, thanks :) > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ > > > > > > All is well. Note that you could reduce the footprint of > your change > somewhat by defining a structure like this: > > struct numbers_t { size_t cap; size_t used; size_t free; > size_t waste; } > > and replacing the many members in > PrintCLDMetaspaceInfoClosure with > instances of this structure: > > numbers_t total; > numbers_t total_class; > numbers_t total_anon; > numbers_t total_anon_class; > Depending on how far you go, your code could deflate quite > a bit. > Give the structure a default ctor and your large > initializer list > goes away; give it a printing function and you reduce > print-summary() to four function calls; with something like an > numbers_t::add(size_t cap, size_t used, size_t free, size_t > waste) > you can reduce the additions in > PrintCLDMetaspaceInfoClosure::print_metaspace() to four lines. > > Just a suggestion, I leave it up to you if you do this. > > Lines 3108 and 3129 miss each a space before BytesPerWord. > Change > looks fine otherwise. > > Thanks, Thomas > > > Thanks, > > -Zhengyu > > > All else looks fine to me now. I do not need > another review. > > Thanks for your work, this will come in handy! > > ..Thomas > > On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu > > > > > >>> > wrote: > > Hi Thomas, > > > Not sure whether we misunderstood each > other. I was > thinking of > something in the line of: > > <<<< > .... > ClassLoader: > jdk/internal/reflect/DelegatingClassLoader > Metadata: > capacity: 5.00KB used: > 3.32KB free: 1.68KB > waste: 0.00KB > Class data: > capacity: 1.00KB used: > 0.54KB free: 0.46KB > waste: 0.00KB > > ClassLoader: for anonymous class > Metadata: > capacity: 1.00KB used: > 1.00KB free: 0.00KB > waste: 0.00KB > Class data: > capacity: 1.00KB used: > 0.64KB free: 0.36KB > waste: 0.00KB > .... > > Summary: > XX class loaders encountered, total > capacity: xx, > total waste: xx. > > Peak usage by class loader: xxxx > >>>> > > Added summary lines: > > Total class loaders= 56 capacity= > 6378.00KB > used= 6205.08KB free= 172.92KB waste= > 1.44KB > For anonymous classes= 54 capacity= 212.00KB > used= 96.95KB > free= 115.05KB waste= 0.94KB > > > > > These numbers would not be interpretation, > just a > convenience to > the reader to get a clear picture. > > It even may be useful to separate the output > between basic and > detailed mode, and in basic mode omit all the > single class > loaders and just print the summary line. > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html > > > > > > > >> > > > > > > > >>> > > > > metaspace.hpp: > > I would have preferred to keep scale_unit file > local static > instead of exposing it via MetaspaceAux, > because it > does not > really have anything to do with metaspace. > > Fixed > > > Otherwise ok. > > --- > > metaspace.cpp > > - ChunkManager::print_statistics(): thanks for > reusing this > function! > -> Scale can only be ever 1, K, M, > G, yes? > So, could you > add an assert at the start of the > function, and a > comment at the > prototype or function head? > -> Also, now that > ChunkManager::print_statistics() takes a > scale, could you please change the > printout to use > floats like > you did in > PrintCLDMetaspaceInfoClosure::print_metaspace()? > > Done. > > > Chunkmanager (non-class): > 0 specialized (128 bytes) chunks, total 0.00KB > 0 small (512 bytes) chunks, total 0.00KB > 0 medium (8192 bytes) chunks, total 0.00KB > 0 humongous chunks, total 0.00KB > total size: 0.00KB. > Chunkmanager (class): > 0 specialized (128 bytes) chunks, total 0.00KB > 0 small (256 bytes) chunks, total 0.00KB > 0 medium (4096 bytes) chunks, total 0.00KB > 0 humongous chunks, total 0.00KB > total size: 0.00KB. > > > - I am concerned about races in > > PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). > Maybe my understanding is limited. We bail if > cld->is_unloading. > But nothing prevents a > ClassLoaderDataGraph from > starting to > unload a ClassLoaderData and tearing down the > Metaspace after we > started printing it in another thread, > right? Also, > I do not see > any fences. > > So, I wonder whether > PrintCLDMetaspaceInfoClosure > should run in > lock protection. And then, I wonder if it > would be > good to > separate data gathering and printing. We > print to a > (at least in > principle) unknown output stream, which may be > doing slow File > IO or even Network IO. Doing this under lock > protection seems > not a good idea (but I see other places > where this > happens too). > > For an example, see > ChunkManager::get_statistic() vs > ChunkManager::print_statistic(). Could you > do the > same, have a > data gathering step where you collect your > quadrupel in a > variable > sized list of > structures in memory, and print it out in a > separate step? I > realize though that the effort would be > larger than > for what we > did in ChunkManager::get_statistic(), > where we only > fill a > structure. > > > Unlike other NMT queries, this query is > executed at a > safepoint by > VMThread, so it should be okay. > > Updated webrev: > http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ > > > > > > >> > > Thanks, > > -Zhengyu > > > ------ > > vm_operations.hpp > > - VM_PrintMetadata : do we still need _unit? > > > Thanks, > > -Zhengyu > > > > Thanks! > > Thomas > > > > On 10/23/2017 11:08 AM, Thomas St?fe > wrote: > > > > On Mon, Oct 23, 2017 at 5:03 PM, > Zhengyu Gu > > > > > >> > > > > > >>> > > > > >> > > > > > >>>>> > wrote: > > > > Okay. So this is > important for > understanding cases > where we have > a large number of > miniscule class > loaders, > each one > only having > a few numbers of chunks. > In this > case, would it be > useful to > have a running total of > "free" > and "waste", > too? Or > even better, > also average and peak > values of > "free" and > "waste" (to tell > apart cases where you > have one > outlier vs > cases where > just all > metaspaces waste a lot > of memory). > Just out of curiosity, I > looked at > http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt > > > > > > > > >> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > >>>> > and > it seems that "free" is the > problem, not > "waste"? So I > guess > what happens is that all the > little classloaders > allocate just > enough memory to push > them over > "_small_chunk_limit=4", > so they > allocate the first > medium chunk, > from which > then they > take a > small bite and leave the > rest > laying around? > > Yes. The free space of anonymous > classes should be > counted > as waste, > in my opinion. I am not sure if > everyone agrees, > so I took the > summary line out of this patch. > > I would be more than happy > to restore > the summary > line, if > you find > it is useful :-) > > > I agree with you. Typically class > loaders > stop loading > at some > point, especially these tine > ones, and > often will not > resume > loading. So, effectivly, the space is > wasted. It still > helps to > distinguish "free" in the current > chunks > from "free" in the > other chunks to tell this > situation apart > from a > situation where > we have just a bug in metaspace > chunk handling > preventing us to > use up our chunks. > > > The latter > would be an > important > addition, > regardless > if this is > for customers > or for us. > Chunks idling in > freelists can > make a > huge impact, and > unfortunately this > may happen > in perfectly > valid cases. > See e.g. > our JEP > > "https://bugs.openjdk.java.net/browse/JDK-8166690 > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>>> > > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>>>>". We have > already a printing > routine for free > chunks, in > > ChunkManager::print_all_chunkmanagers(). Could > you add > this to > your output? > > > Make sense! Could you > recommend what data and > format you > would like > to see? > > > > Would not > ChunkManager::print_all_chunkmanagers() be > sufficient? > > Okay, I might need to tweak > output > format. > > > Fine by me. Nothing depends on it > yet. > > Thanks, Thomas > > Thanks, > > -Zhengyu > > > > ------------- > > Other than > above notes, > the changes seem > straighforward, I did > not see > anything wrong. > Minor nits: > > - > PrintCLDMetaspaceInfoClosure::_out, > _scale > and _unit > could all > be constant > (with _out > being a > outputStream* > const). > - We also do > not need to > provide > scale *and* > unit as > argument, > one can be > deduced from > the other. > This would > also prevent > mismatches.We > do need > scale here, > since nmt > command > line can set > scale and we do > > have to honor that. > > However, passing > unit is > ugly! I don't > want to > introduce > inverse > dependence > (metaspace -> > nmt), which is > also ugly. > > > Yes, that would be > regrettable. > > Probably should > move scale > mapping code from > NMTUtil to a > common util? > > > > That would be possible, > these > functions > (scale_from_name etc) > are simple enough to be > moved to > a generic file. > However, if you > pass scale, deducing the > string > that goes with > it is > trivial and > I would have no qualms > doing this > by hand in > metaspace.cpp. But > it is a matter of taste. > > > I did not look > closely > at locking > issues, I assume > > MetaspaceAux::print_metadata() is > supposed to > run only in > Safepoints? > > > No. sum_xxx_xxx_in_use > queries are > guarded by locks. > > Thanks, > > -Zhengyu > > > Thanks, Thomas > > > > Thanks & Kind > Regards, > Thomas > > > > > > > On Fri, Oct 20, > 2017 at > 4:00 PM, > Zhengyu Gu > > > > > >> > > > > >>> > > > > > >> > > > > >>>> > > > > > >> > > > > >>> > > > > > >> > > > > >>>>> > > > > > >> > > > > >>> > > > > > >> > > > > >>>> > > > > > > >> > > > > >>> > > > > > >> > > > > >>>>>>> wrote: > > Up to now, > there is > little > visibility > into how > metaspaces > are used, > and by whom. > > This proposed > enhancement gives > NMT the > ability to > report > metaspace > usages on > per-classloader level, > via a > new NMT command > "metadata" > > > For example: > > 2496: > Metaspaces: > > Metadata space: > reserved= 8192KB > committed= 5888KB > Class > space: > reserved= 1048576KB > committed= 640KB > > > Per-classloader > metadata: > > > ClassLoader: for > anonymous class > > Metadata capacity= 5.00KB > used= 1.16KB > free= > 3.84KB > waste= 0.05KB > Class data > capacity= 1.00KB > used= 0.59KB > free= > 0.41KB > waste= 0.00KB > > .... > > ClassLoader: > > > Metadata capacity= 5640.00KB > used= 5607.42KB > free= > 32.58KB > waste= 0.46KB > Class data > capacity= 520.00KB > used= 500.94KB > free= > 19.06KB > waste= 0.02KB > > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8189688 > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>>> > > > > > > > >> > > > > > > > >>> > > > > > > > >> > > > > > > > >>>>> > > > > > > > > From coleen.phillimore at oracle.com Mon Oct 30 15:36:28 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 30 Oct 2017 11:36:28 -0400 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> Message-ID: <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> On 10/30/17 11:24 AM, Gerard Ziemski wrote: > hi Coleen, > > >> On Oct 30, 2017, at 10:12 AM, coleen.phillimore at oracle.com wrote: >> >> >> Hi Gerard, >> >> This looks great. One small question: >> + // straight forward brute force >> + inline static int _next_prime(int n) { >> + int p = n; >> + for (int i = n; i < (n * n); i++) { >> + if ((i % 2) != 0) { >> + p = i; >> + break; >> + } >> + } >> + return p; >> + } >> >> Is this how you calculate next prime? Wouldn't you check if it can mod by 3 and 5 as well? > As long as it?s not even i.e. mod 2 (and strictly speaking larger than 1). 3 and 5 are prime, so no need to check them. If you passed in 214 (2*107), 215 would look prime but it's not.?? I think the prime table would be better and limit resizing occurrences. Coleen > > Thank you for the review! > > > cheers > > >> Thanks, >> Coleen >> >> >> On 10/27/17 2:25 PM, Gerard Ziemski wrote: >>> hi Coleen, >>> >>> Thank you for the review. >>> >>> Updated webrev: >>> http://cr.openjdk.java.net/~gziemski/8184765_rev2 >>> >>> >>> >>>> On Oct 11, 2017, at 9:10 AM, coleen.phillimore at oracle.com >>>> wrote: >>>> >>>> >>>> Gerard, some preliminary comments. >>>> >>>> >>>> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/systemDictionary.cpp.udiff.html >>>> >>>> >>>> *+{* >>>> *!_MutexLocker mu(SystemDictionary_lock, THREAD_);* >>>> *!_Klass* probe = dictionary->find(_d_hash, name, protection_domain);* >>>> *if (probe != NULL) return probe;* >>>> ** >>>> *+ }* >>>> >>>> I don't think you need this because dictionary->find() should have the NoSafepointVerifier, so the index will not change. >>>> >>> Done, good catch. >>> >>> >>> >>>> http://cr.openjdk.java.net/~gziemski/8184765_rev1/src/hotspot/share/classfile/classLoaderData.cpp.udiff.html >>>> >>>> >>>> I think we want a global _some_dictionary_needs_resizing to avoid walking through the CLDG. >>>> >>>> And have Dictionary have a field _resizing_needed to avoid the calculation during the safepoint, which is set when adding an entry. >>>> >>>> _resizing_needed = *(number_of_entries() > (_resize_load_trigger*table_size()); >>>> *CLDG::_any_resizing_needed |= _resizing_needed; // or something like that. >>>> >>>> I can write more about the rationale of this change in the bug report, if needed. >>>> >>> Done. >>> >>> >>> >>>> Thank you for doing this change. >>>> Coleen >>>> >>>> >>>> On 10/10/17 4:40 PM, Gerard Ziemski wrote: >>>> >>>>> hi all, >>>>> >>>>> Please review this change that adds dynamic resizing of system dictionaries. >>>>> >>>>> The biggest change is refactoring of the code that protects calculating of the table entry?s index using SystemDictionary_lock >>>>> >>>>> A few notes: >>>>> >>>>> - dynamic resizing is off when either dumping or using shared spaces >>>>> - we remove the experimental runtime option ?PredictedLoadedClassCount? and add ?DynamicallyResizeSystemDictionaries? to turn the resizing off (it?s on by default) >>>>> - the jtreg test uses stream of bytes to dynamically load numbered classes from memory instead of disk (thank you Ioi) >>>>> >>>>> bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8184765 >>>>> >>>>> webrev: >>>>> http://cr.openjdk.java.net/~gziemski/8184765_rev1 >>>>> >>>>> >>>>> >>>>> cheers >>>>> >>>>> From aph at redhat.com Mon Oct 30 15:37:20 2017 From: aph at redhat.com (Andrew Haley) Date: Mon, 30 Oct 2017 15:37:20 +0000 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> Message-ID: On 30/10/17 15:24, Gerard Ziemski wrote: > As long as it?s not even i.e. mod 2 (and strictly speaking larger than 1). 3 and 5 are prime, so no need to check them. > > Thank you for the review! This function is not correctly named. For example, next_prime(8) returns 9. -- Andrew Haley Java Platform Lead Engineer Red Hat UK Ltd. EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From gerard.ziemski at oracle.com Mon Oct 30 15:52:22 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Mon, 30 Oct 2017 10:52:22 -0500 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> Message-ID: <23B218F4-7088-4EE2-B4BE-CD1443468D70@oracle.com> > On Oct 30, 2017, at 10:36 AM, coleen.phillimore at oracle.com wrote: > > > > On 10/30/17 11:24 AM, Gerard Ziemski wrote: >> hi Coleen, >> >> >>> On Oct 30, 2017, at 10:12 AM, coleen.phillimore at oracle.com wrote: >>> >>> >>> Hi Gerard, >>> >>> This looks great. One small question: >>> + // straight forward brute force >>> + inline static int _next_prime(int n) { >>> + int p = n; >>> + for (int i = n; i < (n * n); i++) { >>> + if ((i % 2) != 0) { >>> + p = i; >>> + break; >>> + } >>> + } >>> + return p; >>> + } >>> >>> Is this how you calculate next prime? Wouldn't you check if it can mod by 3 and 5 as well? >> As long as it?s not even i.e. mod 2 (and strictly speaking larger than 1). 3 and 5 are prime, so no need to check them. > > If you passed in 214 (2*107), 215 would look prime but it's not. I think the prime table would be better and limit resizing occurrences. Right, need something a bit more clever here. A precomputed prime table might be good enough, but let me see how complex (a correct) function has to be. cheers From ioi.lam at oracle.com Mon Oct 30 15:52:54 2017 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 30 Oct 2017 08:52:54 -0700 Subject: RFR(M) JDK-8188791 Move AppCDS implementation from closed repo to open repo In-Reply-To: References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> Message-ID: <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> Hi Dmitry, In the latest JDK 10 repo, is_jrt has been renamed to is_modules_image. Please change the code accordingly. I will post my latest diff soon, with some test cases as well. Thanks - Ioi On 10/30/17 4:04 AM, Dmitry Samersoff wrote: > Ioi, > > I'd tried to apply your patch to latest open JDK10 and > the compilation fails with: > > /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: > error: ?class SharedClassPathEntry? has no member named ?is_jrt? > > Did I miss something? > > -Dmitry > > On 13.10.2017 02:48, Ioi Lam wrote: >> Hi, >> >> Please review this change set. >> >> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >> ??? https://bugs.openjdk.java.net/browse/JDK-8188791 >> >> This is the first step of implementing the following JEP, which moves >> AppCDS from >> closed repos into the openjdk repo: >> >> ??? https://bugs.openjdk.java.net/browse/JDK-8185996 >> >> In JDK 9, significant portion of AppCDS code resided in the closed repo. >> As part >> of the open-sourcing effort of JDK 18.3, we will move the source code >> into the >> open repo. >> >> In this changeset, the code is moved verbatim as much as possible. The >> intention is >> only to relocate the sources, not to changing existing behaviors, and not >> to do any sort of refactoring. >> >> Most of the "diffs" shown in this webrev are the result of copying the >> closed source >> files on top of files of the same name in the open repo. So in >> reviewing, instead of >> focusing on what's "changed", it's better to focus on the entire content >> of the new >> version of each file. >> >> The only functional change in this task is that the UseAppCDS flag is >> changed from >> a "commercial" flag to a regular "product" flag. This is because >> "commercial" >> flags are not supported by the OpenJDK build. >> >> Source code refactoring may be desirable, because the old open/closed >> source >> code structure had introduced some intermediary APIs to connect code >> between >> the two repos. Such API should be removed in a separate RFE. >> >> Also, some AppCDS tests are currently in the closed repo. These tests >> will be >> moved in a separate task. See JDK-8188792 for details. >> >> All the AppCDS tests (currently still in closed sources) passed with >> both Oracle JDK >> and OpenJDK. >> >> Thanks >> - Ioi > From gerard.ziemski at oracle.com Mon Oct 30 15:53:27 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Mon, 30 Oct 2017 10:53:27 -0500 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> Message-ID: > On Oct 30, 2017, at 10:37 AM, Andrew Haley wrote: > > On 30/10/17 15:24, Gerard Ziemski wrote: >> As long as it?s not even i.e. mod 2 (and strictly speaking larger than 1). 3 and 5 are prime, so no need to check them. >> >> Thank you for the review! > > This function is not correctly named. For example, next_prime(8) > returns 9. Thanks Andrew, you?re right, the function is not correct - I?m on it. cheers From martin.doerr at sap.com Mon Oct 30 16:01:01 2017 From: martin.doerr at sap.com (Doerr, Martin) Date: Mon, 30 Oct 2017 16:01:01 +0000 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <23B218F4-7088-4EE2-B4BE-CD1443468D70@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> <23B218F4-7088-4EE2-B4BE-CD1443468D70@oracle.com> Message-ID: Hi Gerard, I guess something like the following was meant. (I assume overflow will not happen.) Best regards, Martin int next_prime(int n) { if (n < 2) return 2; int p = (n+1)|1, d=3; while (d <= p/d) { if (p%d == 0) { p+=2; d=3; } else { d+=2; } } return p; } -----Original Message----- From: hotspot-runtime-dev [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On Behalf Of Gerard Ziemski Sent: Monday, October 30, 2017 4:52 PM To: coleen.phillimore at oracle.com Cc: hotspot-runtime-dev at openjdk.java.net Subject: Re: RFR: 8184765: Dynamically resize SystemDictionary > On Oct 30, 2017, at 10:36 AM, coleen.phillimore at oracle.com wrote: > > > > On 10/30/17 11:24 AM, Gerard Ziemski wrote: >> hi Coleen, >> >> >>> On Oct 30, 2017, at 10:12 AM, coleen.phillimore at oracle.com wrote: >>> >>> >>> Hi Gerard, >>> >>> This looks great. One small question: >>> + // straight forward brute force >>> + inline static int _next_prime(int n) { >>> + int p = n; >>> + for (int i = n; i < (n * n); i++) { >>> + if ((i % 2) != 0) { >>> + p = i; >>> + break; >>> + } >>> + } >>> + return p; >>> + } >>> >>> Is this how you calculate next prime? Wouldn't you check if it can mod by 3 and 5 as well? >> As long as it?s not even i.e. mod 2 (and strictly speaking larger than 1). 3 and 5 are prime, so no need to check them. > > If you passed in 214 (2*107), 215 would look prime but it's not. I think the prime table would be better and limit resizing occurrences. Right, need something a bit more clever here. A precomputed prime table might be good enough, but let me see how complex (a correct) function has to be. cheers From gerard.ziemski at oracle.com Mon Oct 30 23:28:22 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Mon, 30 Oct 2017 18:28:22 -0500 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> Message-ID: hi Coleen, I updated the webrev to rev3 bug: https://bugs.openjdk.java.net/browse/JDK-8184765 webrev: http://cr.openjdk.java.net/~gziemski/8184765_rev3 > On Oct 30, 2017, at 10:36 AM, coleen.phillimore at oracle.com wrote: > > > > On 10/30/17 11:24 AM, Gerard Ziemski wrote: >> hi Coleen, >> >> >>> On Oct 30, 2017, at 10:12 AM, coleen.phillimore at oracle.com wrote: >>> >>> >>> Hi Gerard, >>> >>> This looks great. One small question: >>> + // straight forward brute force >>> + inline static int _next_prime(int n) { >>> + int p = n; >>> + for (int i = n; i < (n * n); i++) { >>> + if ((i % 2) != 0) { >>> + p = i; >>> + break; >>> + } >>> + } >>> + return p; >>> + } >>> >>> Is this how you calculate next prime? Wouldn't you check if it can mod by 3 and 5 as well? >> As long as it?s not even i.e. mod 2 (and strictly speaking larger than 1). 3 and 5 are prime, so no need to check them. > > If you passed in 214 (2*107), 215 would look prime but it's not. I think the prime table would be better and limit resizing occurrences. Thank you for catching the problem. I went with a simple table of prime numbers (as was used before), since Coleen pointed out to me that the resizing will occur at safe point, so we want to keep the time used to find next prime to a minimum (though resizing is probably so expensive anyways, would it really matter?) cheers From david.holmes at oracle.com Tue Oct 31 02:41:49 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 31 Oct 2017 12:41:49 +1000 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <6710b89e-1b1a-aed2-6273-8bb2bb0fcdbe@oracle.com> Message-ID: <85d1d9de-69fa-0c30-1c26-763528ffa245@oracle.com> On 30/10/2017 11:57 PM, Zhengyu Gu wrote: > Hi David, > > >>> During VM exit, we are not supposed to request a safepoint, correct? >> >> VM exit uses a safepoint - can you use it do gather the data? >> > > Apparently, not all VM exits go through safepoint. If Java is terminated > due to application's main method exists (JVM is terminated via > jni_DestoryJavaVM) , there is not safepoint executed. Yes there is. See the comments in Threads::destroy_vm and VMThread::wait_for_vm_thread_exit. JNI_DestroyJavaVM -> Threads::destroy_vm() -> VMThread::wait_for_vm_thread_exit(); _should_terminate = true; VMThread::run() this->loop(); // returns when _should_terminate is true ... // 4526887 let VM thread exit at Safepoint _no_op_reason = "Halt"; SafepointSynchronize::begin(); The VMThread always takes the VM to a safepoint on a non-aborting exit. David > Thanks, > > -Zhengyu > >> David >> >>> Without a safepoint, it is unsafe to walk class loaders. >>> >>> Thanks, >>> >>> -Zhengyu >>> >>> >>>> Thanks! Thomas >>>> >>>> On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe >>>> > wrote: >>>> >>>> ??? Hi Zhengyu, >>>> >>>> ??? On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu >>> ??? > wrote: >>>> >>>> ??????? Hi Thomas, >>>> >>>> ??????? On 10/24/2017 12:08 PM, Thomas St?fe wrote: >>>> >>>> ??????????? Hi Zhengyu, >>>> >>>> ??????????? - VM_PrintMetadata still has the _unit member, but I see it >>>> ??????????? nowhere used. >>>> >>>> ??????????? - I would have split the summary between class and >>>> non-class >>>> ??????????? area, that may have been more useful. But this is a matter >>>> ??????????? of taste, I leave it up to you. >>>> >>>> ??????? Agree, should go little further. >>>> >>>> ??????? Summary: >>>> ?????????? Total class loaders=?? 754 capacity=? 67528.00KB used= >>>> ??????? 61216.88KB free=?? 9453.11KB waste=???? 38.73KB >>>> ???????????????????????????? Metadata capacity=? 58780.00KB used= >>>> ??????? 54053.45KB free=?? 4726.55KB waste=???? 36.38KB >>>> ?????????????????????????? Class data capacity=?? 8748.00KB used= >>>> ????????? 7163.43KB free=?? 1584.57KB waste=????? 2.35KB >>>> >>>> ??????? For anonymous classes=?? 580 capacity=?? 2732.00KB used= >>>> ????????? 1065.06KB free=?? 1666.94KB waste=???? 16.27KB >>>> ???????????????????????????? Metadata capacity=?? 2152.00KB used= >>>> ??????? 738.46KB free=?? 1413.54KB waste=???? 16.27KB >>>> ?????????????????????????? Class data capacity=??? 580.00KB used= >>>> ??????? 326.60KB free=??? 253.40KB waste=????? 0.00KB >>>> >>>> >>>> >>>> ??? Nice, thanks :) >>>> >>>> ??????? Updated webrev: >>>> ??????? http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >>>> ??????? >>>> >>>> >>>> ??? All is well. Note that you could reduce the footprint of your >>>> change >>>> ??? somewhat by defining a structure like this: >>>> >>>> ??? struct numbers_t { size_t cap; size_t used; size_t free; size_t >>>> waste; } >>>> >>>> ??? and replacing the many members in PrintCLDMetaspaceInfoClosure with >>>> ??? instances of this structure: >>>> >>>> ??? numbers_t total; >>>> ??? numbers_t total_class; >>>> ??? numbers_t total_anon; >>>> ??? numbers_t total_anon_class; >>>> ??? Depending on how far you go, your code could deflate quite a bit. >>>> ??? Give the structure a default ctor and your large initializer list >>>> ??? goes away; give it a printing function and you reduce >>>> ??? print-summary() to four function calls; with something like an >>>> ??? numbers_t::add(size_t cap, size_t used, size_t free, size_t waste) >>>> ??? you can reduce the additions in >>>> ??? PrintCLDMetaspaceInfoClosure::print_metaspace() to four lines. >>>> >>>> ??? Just a suggestion, I leave it up to you if you do this. >>>> >>>> ??? Lines 3108 and 3129 miss each a space before BytesPerWord. Change >>>> ??? looks fine otherwise. >>>> >>>> ??? Thanks, Thomas >>>> >>>> >>>> ??????? Thanks, >>>> >>>> ??????? -Zhengyu >>>> >>>> >>>> ??????????? All else looks fine to me now. I do not need another >>>> review. >>>> >>>> ??????????? Thanks for your work, this will come in handy! >>>> >>>> ??????????? ..Thomas >>>> >>>> ??????????? On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu >>> ??????????? >>> ??????????? >> wrote: >>>> >>>> ???????????????? Hi Thomas, >>>> >>>> >>>> ???????????????????? Not sure whether we misunderstood each other. I >>>> was >>>> ??????????? thinking of >>>> ???????????????????? something in the line of: >>>> >>>> ???????????????????? <<<< >>>> ???????????????????? .... >>>> ???????????????????? ClassLoader: >>>> jdk/internal/reflect/DelegatingClassLoader >>>> ???????????????????????? Metadata: >>>> ??????????????????????????? capacity:???? 5.00KB used:???? 3.32KB >>>> free:???????????????? 1.68KB >>>> ???????????????????? waste: 0.00KB >>>> ???????????????????????? Class data: >>>> ??????????????????????????? capacity:???? 1.00KB used:???? 0.54KB >>>> free:???????????????? 0.46KB >>>> ???????????????????? waste: 0.00KB >>>> >>>> ???????????????????? ClassLoader: for anonymous class >>>> ???????????????????????? Metadata: >>>> ??????????????????????????? capacity:???? 1.00KB used:???? 1.00KB >>>> free:???????????????? 0.00KB >>>> ???????????????????? waste: 0.00KB >>>> ???????????????????????? Class data: >>>> ??????????????????????????? capacity:???? 1.00KB used:???? 0.64KB >>>> free:???????????????? 0.36KB >>>> ???????????????????? waste: 0.00KB >>>> ???????????????????? .... >>>> >>>> ???????????????????? Summary: >>>> ???????????????????? XX class loaders encountered, total capacity: xx, >>>> ??????????? total waste: xx. >>>> >>>> ???????????????????? Peak usage by class loader: xxxx >>>> ?????????????????????? >>>> >>>> >>>> ???????????????? Added summary lines: >>>> >>>> ??????????????????? Total class loaders=??? 56 capacity=?? 6378.00KB >>>> ??????????? used=?????? 6205.08KB free=??? 172.92KB waste=????? 1.44KB >>>> ???????????????? For anonymous classes=??? 54 capacity=??? 212.00KB >>>> ??????????? used=???? 96.95KB >>>> ???????????????? free=??? 115.05KB waste=????? 0.94KB >>>> >>>> >>>> >>>> >>>> ???????????????????? These numbers would not be interpretation, just a >>>> ??????????? convenience to >>>> ???????????????????? the reader to get a clear picture. >>>> >>>> ???????????????????? It even may be useful to separate the output >>>> ??????????? between basic and >>>> ???????????????????? detailed mode, and in basic mode omit all the >>>> ??????????? single class >>>> ???????????????????? loaders and just print the summary line. >>>> >>>> ????????????????????????? Updated webrev: >>>> >>>> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >>>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>>> >>>> >>>> ???????????????????? metaspace.hpp: >>>> >>>> ???????????????????? I would have preferred to keep scale_unit file >>>> ??????????? local static >>>> ???????????????????? instead of exposing it via MetaspaceAux, >>>> because it >>>> ??????????? does not >>>> ???????????????????? really have anything to do with metaspace. >>>> >>>> ???????????????? Fixed >>>> >>>> >>>> ???????????????????? Otherwise ok. >>>> >>>> ???????????????????? --- >>>> >>>> ???????????????????? metaspace.cpp >>>> >>>> ???????????????????? - ChunkManager::print_statistics(): thanks for >>>> ??????????? reusing this >>>> ???????????????????? function! >>>> ?????????????????????????? -> Scale can only be ever 1, K, M, G, yes? >>>> ??????????? So, could you >>>> ???????????????????? add an assert at the start of the function, and a >>>> ??????????? comment at the >>>> ???????????????????? prototype or function head? >>>> ?????????????????????????? -> Also, now that >>>> ??????????? ChunkManager::print_statistics() takes a >>>> ???????????????????? scale, could you please change the printout to use >>>> ??????????? floats like >>>> ???????????????????? you did in >>>> ??????????? PrintCLDMetaspaceInfoClosure::print_metaspace()? >>>> >>>> ???????????????? Done. >>>> >>>> >>>> ???????????????? Chunkmanager (non-class): >>>> ??????????????????? 0 specialized (128 bytes) chunks, total 0.00KB >>>> ??????????????????? 0 small (512 bytes) chunks, total 0.00KB >>>> ??????????????????? 0 medium (8192 bytes) chunks, total 0.00KB >>>> ??????????????????? 0 humongous chunks, total 0.00KB >>>> ??????????????????? total size: 0.00KB. >>>> ???????????????? Chunkmanager (class): >>>> ??????????????????? 0 specialized (128 bytes) chunks, total 0.00KB >>>> ??????????????????? 0 small (256 bytes) chunks, total 0.00KB >>>> ??????????????????? 0 medium (4096 bytes) chunks, total 0.00KB >>>> ??????????????????? 0 humongous chunks, total 0.00KB >>>> ??????????????????? total size: 0.00KB. >>>> >>>> >>>> ???????????????????? - I am concerned about races in >>>> ??????????? PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >>>> ???????????????????? Maybe my understanding is limited. We bail if >>>> ??????????? cld->is_unloading. >>>> ???????????????????? But nothing prevents a ClassLoaderDataGraph from >>>> ??????????? starting to >>>> ???????????????????? unload a ClassLoaderData and tearing down the >>>> ??????????? Metaspace after we >>>> ???????????????????? started printing it in another thread, right? >>>> Also, >>>> ??????????? I do not see >>>> ???????????????????? any fences. >>>> >>>> ???????????????????? So, I wonder whether PrintCLDMetaspaceInfoClosure >>>> ??????????? should run in >>>> ???????????????????? lock protection. And then, I wonder if it would be >>>> ??????????? good to >>>> ???????????????????? separate data gathering and printing. We print >>>> to a >>>> ??????????? (at least in >>>> ???????????????????? principle) unknown output stream, which may be >>>> ??????????? doing slow File >>>> ???????????????????? IO or even Network IO. Doing this under lock >>>> ??????????? protection seems >>>> ???????????????????? not a good idea (but I see other places where this >>>> ??????????? happens too). >>>> >>>> ???????????????????? For an example, see >>>> ChunkManager::get_statistic() vs >>>> ???????????????????? ChunkManager::print_statistic(). Could you do the >>>> ??????????? same, have a >>>> ???????????????????? data gathering step where you collect your >>>> ???????????????????? quadrupel in a variable >>>> ??????????? sized list of >>>> ???????????????????? structures in memory, and print it out in a >>>> ??????????? separate step? I >>>> ???????????????????? realize though that the effort would be larger >>>> than >>>> ??????????? for what we >>>> ???????????????????? did in ChunkManager::get_statistic(), where we >>>> only >>>> ??????????? fill a >>>> ???????????????????? structure. >>>> >>>> >>>> ???????????????? Unlike other NMT queries, this query is executed at a >>>> ??????????? safepoint by >>>> ???????????????? VMThread, so it should be okay. >>>> >>>> ???????????????? Updated webrev: >>>> ??????????? http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >>>> ??????????? >>>> ???????????????? >>> ??????????? > >>>> >>>> ???????????????? Thanks, >>>> >>>> ???????????????? -Zhengyu >>>> >>>> >>>> ???????????????????? ------ >>>> >>>> ???????????????????? vm_operations.hpp >>>> >>>> ???????????????????? - VM_PrintMetadata : do we still need _unit? >>>> >>>> >>>> ????????????????????????? Thanks, >>>> >>>> ????????????????????????? -Zhengyu >>>> >>>> >>>> >>>> ???????????????????? Thanks! >>>> >>>> ???????????????????? Thomas >>>> >>>> >>>> >>>> ????????????????????????? On 10/23/2017 11:08 AM, Thomas St?fe wrote: >>>> >>>> >>>> >>>> ????????????????????????????? On Mon, Oct 23, 2017 at 5:03 PM, >>>> Zhengyu Gu >>>> ???????????????????? >>>> ??????????? > >>>> ????????????????????????????? >>> ??????????? >>> ??????????? >> >>>> ???????????????????? >>>> ??????????? > >>>> ????????????????????????????? >>> ??????????? >>> ??????????? >>>> wrote: >>>> >>>> >>>> >>>> ?????????????????????????????????????? Okay. So this is important for >>>> ??????????? understanding cases >>>> ????????????????????????????? where we have >>>> ?????????????????????????????????????? a large number of miniscule >>>> class >>>> ??????????? loaders, >>>> ???????????????????? each one >>>> ????????????????????????????? only having >>>> ?????????????????????????????????????? a few numbers of chunks. In this >>>> ??????????? case, would it be >>>> ????????????????????????????? useful to >>>> ?????????????????????????????????????? have a running total of "free" >>>> ??????????? and "waste", >>>> ???????????????????? too? Or >>>> ????????????????????????????? even better, >>>> ?????????????????????????????????????? also average and peak values of >>>> ??????????? "free" and >>>> ???????????????????? "waste" (to tell >>>> ?????????????????????????????????????? apart cases where you have one >>>> ??????????? outlier vs >>>> ???????????????????? cases where >>>> ????????????????????????????? just all >>>> ?????????????????????????????????????? metaspaces waste a lot of >>>> memory). >>>> ?????????????????????????????????????? Just out of curiosity, I >>>> looked at >>>> ??????????? http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >>>> ??????????? >>>> ??????????? >>> >>>> > >>>> ??????????? >>> ??????????? >>>> ??????????? >>> >> >>>> ??????????? >>> ??????????? >>>> ??????????? >>> >>>> > >>>> ??????????? >>> ??????????? >>>> ??????????? >>> >>> >>>> ??????????? and >>>> ?????????????????????????????????????? it seems that "free" is the >>>> ??????????? problem, not >>>> ???????????????????? "waste"? So I >>>> ????????????????????????????? guess >>>> ?????????????????????????????????????? what happens is that all the >>>> ??????????? little classloaders >>>> ????????????????????????????? allocate just >>>> ?????????????????????????????????????? enough memory to push them over >>>> ???????????????????? "_small_chunk_limit=4", >>>> ????????????????????????????? so they >>>> ?????????????????????????????????????? allocate the first medium chunk, >>>> ??????????? from which >>>> ???????????????????? then they >>>> ????????????????????????????? take a >>>> ?????????????????????????????????????? small bite and leave the rest >>>> ??????????? laying around? >>>> >>>> ?????????????????????????????????? Yes. The free space of anonymous >>>> ??????????? classes should be >>>> ???????????????????? counted >>>> ????????????????????????????? as waste, >>>> ?????????????????????????????????? in my opinion. I am not sure if >>>> ??????????? everyone agrees, >>>> ???????????????????? so I took the >>>> ?????????????????????????????????? summary line out of this patch. >>>> >>>> ?????????????????????????????????? I would be more than happy to >>>> restore >>>> ??????????? the summary >>>> ???????????????????? line, if >>>> ????????????????????????????? you find >>>> ?????????????????????????????????? it is useful :-) >>>> >>>> >>>> ????????????????????????????? I agree with you. Typically class loaders >>>> ??????????? stop loading >>>> ???????????????????? at some >>>> ????????????????????????????? point, especially these tine ones, and >>>> ??????????? often will not >>>> ???????????????????? resume >>>> ????????????????????????????? loading. So, effectivly, the space is >>>> ??????????? wasted. It still >>>> ???????????????????? helps to >>>> ????????????????????????????? distinguish "free" in the current chunks >>>> ??????????? from "free" in the >>>> ????????????????????????????? other chunks to tell this situation apart >>>> ??????????? from a >>>> ???????????????????? situation where >>>> ????????????????????????????? we have just a bug in metaspace chunk >>>> handling >>>> ???????????????????? preventing us to >>>> ????????????????????????????? use up our chunks. >>>> >>>> >>>> ??????????????????????????????????????????????? The latter would be an >>>> ??????????? important >>>> ???????????????????? addition, >>>> ????????????????????????????? regardless >>>> ?????????????????????????????????????? if this is >>>> ??????????????????????????????????????????????? for customers or for >>>> us. >>>> ??????????? Chunks idling in >>>> ????????????????????????????? freelists can >>>> ?????????????????????????????????????? make a >>>> ??????????????????????????????????????????????? huge impact, and >>>> ??????????? unfortunately this >>>> ???????????????????? may happen >>>> ????????????????????????????? in perfectly >>>> ??????????????????????????????????????????????? valid cases. See e.g. >>>> ??????????? our JEP >>>> ??????????? "https://bugs.openjdk.java.net/browse/JDK-8166690 >>>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >> >>>> ??????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >>> >>>> ??????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >> >>>> ??????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >>>>". >>>> We have >>>> ??????????????????????????????????????????????? already a printing >>>> ??????????? routine for free >>>> ???????????????????? chunks, in >>>> ??????????? ChunkManager::print_all_chunkmanagers(). Could >>>> ????????????????????????????? you add >>>> ?????????????????????????????????????? this to >>>> ??????????????????????????????????????????????? your output? >>>> >>>> >>>> ??????????????????????????????????????????? Make sense! Could you >>>> ??????????? recommend what data and >>>> ????????????????????????????? format you >>>> ?????????????????????????????????????? would like >>>> ??????????????????????????????????????????? to see? >>>> >>>> >>>> >>>> ?????????????????????????????????????? Would not >>>> ???????????????????? ChunkManager::print_all_chunkmanagers() be >>>> ????????????????????????????? sufficient? >>>> >>>> ?????????????????????????????????? Okay, I might need to tweak output >>>> ??????????? format. >>>> >>>> >>>> ????????????????????????????? Fine by me. Nothing depends on it yet. >>>> >>>> ????????????????????????????? Thanks, Thomas >>>> >>>> ?????????????????????????????????? Thanks, >>>> >>>> ?????????????????????????????????? -Zhengyu >>>> >>>> >>>> >>>> ??????????????????????????????????????????????? ------------- >>>> >>>> ??????????????????????????????????????????????? Other than above notes, >>>> ??????????? the changes seem >>>> ?????????????????????????????????????? straighforward, I did >>>> ??????????????????????????????????????????????? not see anything wrong. >>>> ??????????? Minor nits: >>>> >>>> ??????????????????????????????????????????????? - >>>> ??????????? PrintCLDMetaspaceInfoClosure::_out, >>>> ???????????????????? _scale >>>> ????????????????????????????? and _unit >>>> ?????????????????????????????????????? could all >>>> ??????????????????????????????????????????????? be constant (with _out >>>> ??????????? being a >>>> ???????????????????? outputStream* >>>> ????????????????????????????? const). >>>> ??????????????????????????????????????????????? - We also do not >>>> need to >>>> ??????????? provide >>>> ???????????????????? scale *and* >>>> ????????????????????????????? unit as >>>> ?????????????????????????????????????? argument, >>>> ??????????????????????????????????????????????? one can be deduced from >>>> ??????????? the other. >>>> ???????????????????? This would >>>> ????????????????????????????? also prevent >>>> ??????????????????????????????????????????????? mismatches.We do need >>>> ??????????? scale here, >>>> ???????????????????? since nmt >>>> ????????????????????????????? command >>>> ?????????????????????????????????????? line can set >>>> ??????????????????????????????????????????????? scale and we do >>>> >>>> ??????????????????????????????????????????? have to honor that. >>>> >>>> ??????????????????????????????????????????? However, passing unit is >>>> ??????????? ugly! I don't >>>> ???????????????????? want to >>>> ????????????????????????????? introduce >>>> ?????????????????????????????????????? inverse >>>> ??????????????????????????????????????????? dependence (metaspace -> >>>> ??????????? nmt), which is >>>> ???????????????????? also ugly. >>>> >>>> >>>> ?????????????????????????????????????? Yes, that would be regrettable. >>>> >>>> ??????????????????????????????????????????? Probably should move scale >>>> ??????????? mapping code from >>>> ????????????????????????????? NMTUtil to a >>>> ?????????????????????????????????????? common util? >>>> >>>> >>>> >>>> ?????????????????????????????????????? That would be possible, these >>>> ??????????? functions >>>> ????????????????????????????? (scale_from_name etc) >>>> ?????????????????????????????????????? are simple enough to be moved to >>>> ??????????? a generic file. >>>> ????????????????????????????? However, if you >>>> ?????????????????????????????????????? pass scale, deducing the string >>>> ??????????? that goes with >>>> ???????????????????? it is >>>> ????????????????????????????? trivial and >>>> ?????????????????????????????????????? I would have no qualms doing >>>> this >>>> ??????????? by hand in >>>> ????????????????????????????? metaspace.cpp. But >>>> ?????????????????????????????????????? it is a matter of taste. >>>> >>>> >>>> ??????????????????????????????????????????????? I did not look closely >>>> ??????????? at locking >>>> ???????????????????? issues, I assume >>>> ????????????? MetaspaceAux::print_metadata() is >>>> ???????????????????? supposed to >>>> ????????????????????????????? run only in >>>> ??????????????????????????????????????????????? Safepoints? >>>> >>>> >>>> ??????????????????????????????????????????? No. sum_xxx_xxx_in_use >>>> ??????????? queries are >>>> ???????????????????? guarded by locks. >>>> >>>> ??????????????????????????????????????????? Thanks, >>>> >>>> ??????????????????????????????????????????? -Zhengyu >>>> >>>> >>>> ?????????????????????????????????????? Thanks, Thomas >>>> >>>> >>>> >>>> ??????????????????????????????????????????????? Thanks & Kind Regards, >>>> ??????????? Thomas >>>> >>>> >>>> >>>> >>>> >>>> >>>> ??????????????????????????????????????????????? On Fri, Oct 20, 2017 at >>>> ??????????? 4:00 PM, >>>> ???????????????????? Zhengyu Gu >>>> ?????????????????????????????????????? >>> ??????????? >>> ??????????? > >>>> ???????????????????? >>>> ??????????? >> >>>> ????????????????????????????? >>> ??????????? >>> ??????????? > >>>> ???????????????????? >>>> ??????????? >>> >>>> ??????????????????????????????????????????????? >>> ??????????? >>>> ???????????????????? > >>>> ??????????? >>>> ???????????????????? >> >>>> ????????????????????????????? >>> ??????????? >>> ??????????? > >>>> ???????????????????? >>>> ??????????? >>>> >>>> ?????????????????????????????????????? >>> ??????????? >>> ??????????? > >>>> ???????????????????? >>>> ??????????? >> >>>> ????????????????????????????? >>> ??????????? >>> ??????????? > >>>> ???????????????????? >>>> ??????????? >>> >>>> >>>> ??????????????????????????????????????????????? >>> ??????????? >>>> ???????????????????? > >>>> ??????????? >>>> ???????????????????? >> >>>> ????????????????????????????? >>> ??????????? >>> ??????????? > >>>> ???????????????????? >>>> ??????????? >>>>>> wrote: >>>> >>>> ???????????????????????????????????????????????????? Up to now, >>>> there is >>>> ??????????? little >>>> ???????????????????? visibility >>>> ????????????????????????????? into how >>>> ?????????????????????????????????????? metaspaces >>>> ??????????????????????????????????????????????? are used, >>>> ???????????????????????????????????????????????????? and by whom. >>>> >>>> ???????????????????????????????????????????????????? This proposed >>>> ??????????? enhancement gives >>>> ???????????????????? NMT the >>>> ????????????????????????????? ability to >>>> ?????????????????????????????????????? report >>>> ??????????????????????????????????????????????? metaspace >>>> ???????????????????????????????????????????????????? usages on >>>> ??????????? per-classloader level, >>>> ???????????????????? via a >>>> ????????????????????????????? new NMT command >>>> ??????????????????????????????????????????????? "metadata" >>>> >>>> >>>> ???????????????????????????????????????????????????? For example: >>>> >>>> ???????????????????????????????????????????????????? 2496: >>>> ???????????????????????????????????????????????????? Metaspaces: >>>> ??????????????????????????????????????????????????????? Metadata space: >>>> ??????????? reserved=????????????? 8192KB >>>> ?????????????????????????????????????? committed=????? 5888KB >>>> ??????????????????????????????????????????????????????? Class??? space: >>>> ??????????? reserved=?????????? 1048576KB >>>> ?????????????????????????????????????? committed=?????? 640KB >>>> >>>> ???????????????????????????????????????????????????? Per-classloader >>>> ??????????? metadata: >>>> >>>> ???????????????????????????????????????????????????? ClassLoader: for >>>> ??????????? anonymous class >>>> ??????????????????????????????????????????????????????? Metadata >>>> ????????????? capacity=????? 5.00KB >>>> ????????????????????????????? used=????? 1.16KB >>>> ??????????????????????????????????????????????? free=???????? 3.84KB >>>> ??????????? waste=????? 0.05KB >>>> ??????????????????????????????????????????????????????? Class data >>>> ??????????? capacity=????? 1.00KB >>>> ????????????????????????????? used=????? 0.59KB >>>> ??????????????????????????????????????????????? free=???????? 0.41KB >>>> ??????????? waste=????? 0.00KB >>>> >>>> ???????????????????????????????????????????????????? .... >>>> >>>> ???????????????????????????????????????????????????? ClassLoader: >>>> ??????????? >>>> ??????????????????????????????????????????????????????? Metadata >>>> ????????????? capacity=?? 5640.00KB >>>> ????????????????????????????? used=?? 5607.42KB >>>> ??????????????????????????????????????????????? free=???????? 32.58KB >>>> ??????????? waste=????? 0.46KB >>>> ??????????????????????????????????????????????????????? Class data >>>> ??????????? capacity=??? 520.00KB >>>> ????????????????????????????? used=??? 500.94KB >>>> ??????????????????????????????????????????????? free=???????? 19.06KB >>>> ??????????? waste=????? 0.02KB >>>> >>>> >>>> ???????????????????????????????????????????????????? Bug: >>>> ??????????? https://bugs.openjdk.java.net/browse/JDK-8189688 >>>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >> >>>> ??????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >>> >>>> ??????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >> >>>> ??????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >>>> >>>> ??????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >> >>>> ??????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >>> >>>> ??????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >> >>>> ??????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? > >>>> ????????????? >>> ??????????? >>>> ???????????????????? >>> ??????????? >>>>> >>>> ???????????????????????????????????????????????????? Webrev: >>>> >>>> http://cr.openjdk.java.net/~zgu/8189688/webrev.00/index.html >>>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >> >>>> >>> >>>> >>> > >>>> >>> >>>> >>> >>>>> >>>> >>>> ???????????????????????????????????????????????????? Test: >>>> >>>> ????????????? hotspot_tier1_runtime, >>>> ???????????????????? fastdebug and >>>> ????????????????????????????? release on >>>> ?????????????????????????????????????? x64 Linux. >>>> >>>> ???????????????????????????????????????????????????? Thanks, >>>> >>>> ???????????????????????????????????????????????????? -Zhengyu >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> From john.r.rose at oracle.com Tue Oct 31 03:23:41 2017 From: john.r.rose at oracle.com (John Rose) Date: Mon, 30 Oct 2017 20:23:41 -0700 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <23B218F4-7088-4EE2-B4BE-CD1443468D70@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> <23B218F4-7088-4EE2-B4BE-CD1443468D70@oracle.com> Message-ID: On Oct 30, 2017, at 8:52 AM, Gerard Ziemski wrote: > > Right, need something a bit more clever here. A precomputed prime table might be good enough, but let me see how complex (a correct) function has to be. I suggest a precomputed prime table and an inefficient-but-simple validation that runs only in the debug build, via assert(is_prime(tab[i])). From david.holmes at oracle.com Tue Oct 31 03:50:05 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 31 Oct 2017 13:50:05 +1000 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> Message-ID: <9dde915c-90ed-be0b-fec1-5b7a1f80338a@oracle.com> Hi Gerard, The flag should not be experimental - enabling resizable tables is not an experiment that the user is controlling, it is the default behaviour that we intend to occur. The flag is there to turn it off if for some reason this doesn't work - the flag should be a product flag. The flag also requires a CSR request. Thanks, David On 31/10/2017 9:28 AM, Gerard Ziemski wrote: > hi Coleen, > > I updated the webrev to rev3 > > bug: > https://bugs.openjdk.java.net/browse/JDK-8184765 > > webrev: > http://cr.openjdk.java.net/~gziemski/8184765_rev3 > > >> On Oct 30, 2017, at 10:36 AM, coleen.phillimore at oracle.com wrote: >> >> >> >> On 10/30/17 11:24 AM, Gerard Ziemski wrote: >>> hi Coleen, >>> >>> >>>> On Oct 30, 2017, at 10:12 AM, coleen.phillimore at oracle.com wrote: >>>> >>>> >>>> Hi Gerard, >>>> >>>> This looks great. One small question: >>>> + // straight forward brute force >>>> + inline static int _next_prime(int n) { >>>> + int p = n; >>>> + for (int i = n; i < (n * n); i++) { >>>> + if ((i % 2) != 0) { >>>> + p = i; >>>> + break; >>>> + } >>>> + } >>>> + return p; >>>> + } >>>> >>>> Is this how you calculate next prime? Wouldn't you check if it can mod by 3 and 5 as well? >>> As long as it?s not even i.e. mod 2 (and strictly speaking larger than 1). 3 and 5 are prime, so no need to check them. >> >> If you passed in 214 (2*107), 215 would look prime but it's not. I think the prime table would be better and limit resizing occurrences. > > Thank you for catching the problem. > > I went with a simple table of prime numbers (as was used before), since Coleen pointed out to me that the resizing will occur at safe point, so we want to keep the time used to find next prime to a minimum (though resizing is probably so expensive anyways, would it really matter?) > > > cheers > > From dmitry.samersoff at bell-sw.com Tue Oct 31 07:37:10 2017 From: dmitry.samersoff at bell-sw.com (Dmitry Samersoff) Date: Tue, 31 Oct 2017 10:37:10 +0300 Subject: JDK-8190336 Make sure that AppCDS works on aarch64 platform In-Reply-To: <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> References: <7b13162f-0240-1695-15b5-c7b6baad8555@oracle.com> <97e0d61f-30ad-b60d-fb8f-c83c839213f4@oracle.com> Message-ID: Ioi, I can confirm that current patchset works on aarch64 as expected. It passes all tests in runtime/SharedArchiveFile. I'll re-check it when other version of fix/more tests become available. -Dmitry On 30.10.2017 18:52, Ioi Lam wrote: > Hi Dmitry, > > In the latest JDK 10 repo, is_jrt has been renamed to is_modules_image. > Please change the code accordingly. > > I will post my latest diff soon, with some test cases as well. > > Thanks > > - Ioi > > > On 10/30/17 4:04 AM, Dmitry Samersoff wrote: >> Ioi, >> >> I'd tried to apply your patch to latest open JDK10 and >> the compilation fails with: >> >> /root/dsamersoff/ESC/appcds/hs/src/hotspot/share/classfile/systemDictionaryShared.cpp:400:16: >> >> error: ?class SharedClassPathEntry? has no member named ?is_jrt? >> >> Did I miss something? >> >> -Dmitry >> >> On 13.10.2017 02:48, Ioi Lam wrote: >>> Hi, >>> >>> Please review this change set. >>> >>> http://cr.openjdk.java.net/~iklam/jdk10/8188791-open-appcds-impl.v01/ >>> ???? https://bugs.openjdk.java.net/browse/JDK-8188791 >>> >>> This is the first step of implementing the following JEP, which moves >>> AppCDS from >>> closed repos into the openjdk repo: >>> >>> ???? https://bugs.openjdk.java.net/browse/JDK-8185996 >>> >>> In JDK 9, significant portion of AppCDS code resided in the closed repo. >>> As part >>> of the open-sourcing effort of JDK 18.3, we will move the source code >>> into the >>> open repo. >>> >>> In this changeset, the code is moved verbatim as much as possible. The >>> intention is >>> only to relocate the sources, not to changing existing behaviors, and >>> not >>> to do any sort of refactoring. >>> >>> Most of the "diffs" shown in this webrev are the result of copying the >>> closed source >>> files on top of files of the same name in the open repo. So in >>> reviewing, instead of >>> focusing on what's "changed", it's better to focus on the entire content >>> of the new >>> version of each file. >>> >>> The only functional change in this task is that the UseAppCDS flag is >>> changed from >>> a "commercial" flag to a regular "product" flag. This is because >>> "commercial" >>> flags are not supported by the OpenJDK build. >>> >>> Source code refactoring may be desirable, because the old open/closed >>> source >>> code structure had introduced some intermediary APIs to connect code >>> between >>> the two repos. Such API should be removed in a separate RFE. >>> >>> Also, some AppCDS tests are currently in the closed repo. These tests >>> will be >>> moved in a separate task. See JDK-8188792 for details. >>> >>> All the AppCDS tests (currently still in closed sources) passed with >>> both Oracle JDK >>> and OpenJDK. >>> >>> Thanks >>> - Ioi >> > From zgu at redhat.com Tue Oct 31 13:50:36 2017 From: zgu at redhat.com (Zhengyu Gu) Date: Tue, 31 Oct 2017 09:50:36 -0400 Subject: RFR(S) 8189688: NMT: Report per-class load metadata information In-Reply-To: <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> References: <26fafebd-559e-a738-dcda-3142ba01190e@redhat.com> <779089d0-6a43-b7f3-7d4b-e06569063b89@redhat.com> <6701243d-ce77-6790-8ea7-e712764fc154@redhat.com> <11eb7336-12d3-c9ea-9c98-b76271470400@redhat.com> <87a84db5-2dc0-2192-939c-6e1fe4d0e308@redhat.com> Message-ID: <40c26fee-968e-c7c0-ea5e-2ba6df42fdb1@redhat.com> On 10/30/2017 11:29 AM, Zhengyu Gu wrote: >> >> >> Yes, this is not trivial. I hacked it in (because I wanted to see your >> output at the end of my tests) and had to disable the assert. I never >> ran into problems. Should java threads not all be stopped at that point? >> > Yes, it looks like that all Java threads are stopped at that point. > However, I am not so sure about workers. Could there be active workers > while JVM exiting? Never mind. Actually, there could be live Java threads. e.g. when you exit JVM via CTRL+BREAK (JVM_HALT). Thanks, -Zhengyu > > >> But this also can be done in a follow up issue. > > Yes, let's address it separately. > https://bugs.openjdk.java.net/browse/JDK-8190357 > > > Updated webrev based on your early comments: > > Webrev: http://cr.openjdk.java.net/~zgu/8189688/webrev.04/ > > Thanks, > > -Zhengyu > >> >> Thanks, Thomas >> >> >> Thanks! Thomas >> >> On Wed, Oct 25, 2017 at 7:00 AM, Thomas St?fe >> >> > >> wrote: >> >> Hi Zhengyu, >> >> On Tue, Oct 24, 2017 at 8:04 PM, Zhengyu Gu > >> >> wrote: >> >> Hi Thomas, >> >> On 10/24/2017 12:08 PM, Thomas St?fe wrote: >> >> Hi Zhengyu, >> >> - VM_PrintMetadata still has the _unit member, but >> I see it >> nowhere used. >> >> - I would have split the summary between class and >> non-class >> area, that may have been more useful. But this is a >> matter >> of taste, I leave it up to you. >> >> Agree, should go little further. >> >> Summary: >> Total class loaders= 754 capacity= 67528.00KB >> used= 61216.88KB free= 9453.11KB waste= 38.73KB >> Metadata capacity= 58780.00KB >> used= 54053.45KB free= 4726.55KB waste= 36.38KB >> Class data capacity= 8748.00KB >> used= 7163.43KB free= 1584.57KB waste= 2.35KB >> >> For anonymous classes= 580 capacity= 2732.00KB >> used= 1065.06KB free= 1666.94KB waste= 16.27KB >> Metadata capacity= 2152.00KB >> used= 738.46KB free= 1413.54KB waste= 16.27KB >> Class data capacity= 580.00KB >> used= 326.60KB free= 253.40KB waste= 0.00KB >> >> >> >> Nice, thanks :) >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.03/ >> >> > > >> >> >> All is well. Note that you could reduce the footprint of >> your change >> somewhat by defining a structure like this: >> >> struct numbers_t { size_t cap; size_t used; size_t free; >> size_t waste; } >> >> and replacing the many members in >> PrintCLDMetaspaceInfoClosure with >> instances of this structure: >> >> numbers_t total; >> numbers_t total_class; >> numbers_t total_anon; >> numbers_t total_anon_class; >> Depending on how far you go, your code could deflate quite >> a bit. >> Give the structure a default ctor and your large >> initializer list >> goes away; give it a printing function and you reduce >> print-summary() to four function calls; with something >> like an >> numbers_t::add(size_t cap, size_t used, size_t free, size_t >> waste) >> you can reduce the additions in >> PrintCLDMetaspaceInfoClosure::print_metaspace() to four >> lines. >> >> Just a suggestion, I leave it up to you if you do this. >> >> Lines 3108 and 3129 miss each a space before BytesPerWord. >> Change >> looks fine otherwise. >> >> Thanks, Thomas >> >> >> Thanks, >> >> -Zhengyu >> >> >> All else looks fine to me now. I do not need >> another review. >> >> Thanks for your work, this will come in handy! >> >> ..Thomas >> >> On Tue, Oct 24, 2017 at 5:08 PM, Zhengyu Gu >> >> > >> >> >>> >> wrote: >> >> Hi Thomas, >> >> >> Not sure whether we misunderstood each >> other. I was >> thinking of >> something in the line of: >> >> <<<< >> .... >> ClassLoader: >> jdk/internal/reflect/DelegatingClassLoader >> Metadata: >> capacity: 5.00KB used: >> 3.32KB free: 1.68KB >> waste: 0.00KB >> Class data: >> capacity: 1.00KB used: >> 0.54KB free: 0.46KB >> waste: 0.00KB >> >> ClassLoader: for anonymous class >> Metadata: >> capacity: 1.00KB used: >> 1.00KB free: 0.00KB >> waste: 0.00KB >> Class data: >> capacity: 1.00KB used: >> 0.64KB free: 0.36KB >> waste: 0.00KB >> .... >> >> Summary: >> XX class loaders encountered, total >> capacity: xx, >> total waste: xx. >> >> Peak usage by class loader: xxxx >> >>>> >> >> Added summary lines: >> >> Total class loaders= 56 capacity= >> 6378.00KB >> used= 6205.08KB free= 172.92KB waste= >> 1.44KB >> For anonymous classes= 54 capacity= >> 212.00KB >> used= 96.95KB >> free= 115.05KB waste= 0.94KB >> >> >> >> >> These numbers would not be interpretation, >> just a >> convenience to >> the reader to get a clear picture. >> >> It even may be useful to separate the >> output >> between basic and >> detailed mode, and in basic mode omit >> all the >> single class >> loaders and just print the summary line. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.01/index.html >> >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> >> >> >> metaspace.hpp: >> >> I would have preferred to keep >> scale_unit file >> local static >> instead of exposing it via MetaspaceAux, >> because it >> does not >> really have anything to do with metaspace. >> >> Fixed >> >> >> Otherwise ok. >> >> --- >> >> metaspace.cpp >> >> - ChunkManager::print_statistics(): >> thanks for >> reusing this >> function! >> -> Scale can only be ever 1, K, M, >> G, yes? >> So, could you >> add an assert at the start of the >> function, and a >> comment at the >> prototype or function head? >> -> Also, now that >> ChunkManager::print_statistics() takes a >> scale, could you please change the >> printout to use >> floats like >> you did in >> PrintCLDMetaspaceInfoClosure::print_metaspace()? >> >> Done. >> >> >> Chunkmanager (non-class): >> 0 specialized (128 bytes) chunks, total >> 0.00KB >> 0 small (512 bytes) chunks, total 0.00KB >> 0 medium (8192 bytes) chunks, total 0.00KB >> 0 humongous chunks, total 0.00KB >> total size: 0.00KB. >> Chunkmanager (class): >> 0 specialized (128 bytes) chunks, total >> 0.00KB >> 0 small (256 bytes) chunks, total 0.00KB >> 0 medium (4096 bytes) chunks, total 0.00KB >> 0 humongous chunks, total 0.00KB >> total size: 0.00KB. >> >> >> - I am concerned about races in >> PrintCLDMetaspaceInfoClosure::do_cld(ClassLoaderData* cld). >> Maybe my understanding is limited. We >> bail if >> cld->is_unloading. >> But nothing prevents a >> ClassLoaderDataGraph from >> starting to >> unload a ClassLoaderData and tearing >> down the >> Metaspace after we >> started printing it in another thread, >> right? Also, >> I do not see >> any fences. >> >> So, I wonder whether >> PrintCLDMetaspaceInfoClosure >> should run in >> lock protection. And then, I wonder if it >> would be >> good to >> separate data gathering and printing. We >> print to a >> (at least in >> principle) unknown output stream, which >> may be >> doing slow File >> IO or even Network IO. Doing this under >> lock >> protection seems >> not a good idea (but I see other places >> where this >> happens too). >> >> For an example, see >> ChunkManager::get_statistic() vs >> ChunkManager::print_statistic(). Could you >> do the >> same, have a >> data gathering step where you collect your >> quadrupel in a >> variable >> sized list of >> structures in memory, and print it out in a >> separate step? I >> realize though that the effort would be >> larger than >> for what we >> did in ChunkManager::get_statistic(), >> where we only >> fill a >> structure. >> >> >> Unlike other NMT queries, this query is >> executed at a >> safepoint by >> VMThread, so it should be okay. >> >> Updated webrev: >> http://cr.openjdk.java.net/~zgu/8189688/webrev.02/ >> >> > > >> > >> > >> >> >> Thanks, >> >> -Zhengyu >> >> >> ------ >> >> vm_operations.hpp >> >> - VM_PrintMetadata : do we still need >> _unit? >> >> >> Thanks, >> >> -Zhengyu >> >> >> >> Thanks! >> >> Thomas >> >> >> >> On 10/23/2017 11:08 AM, Thomas St?fe >> wrote: >> >> >> >> On Mon, Oct 23, 2017 at 5:03 PM, >> Zhengyu Gu >> >> > >> >> >> >> > >> > >> >> >>> >> > > > >> >> >> >> > >> > >> >> >>>>> >> wrote: >> >> >> >> Okay. So this is >> important for >> understanding cases >> where we have >> a large number of >> miniscule class >> loaders, >> each one >> only having >> a few numbers of chunks. >> In this >> case, would it be >> useful to >> have a running total of >> "free" >> and "waste", >> too? Or >> even better, >> also average and peak >> values of >> "free" and >> "waste" (to tell >> apart cases where you >> have one >> outlier vs >> cases where >> just all >> metaspaces waste a lot >> of memory). >> Just out of curiosity, I >> looked at >> http://cr.openjdk.java.net/~zgu/cld_metaspace/wildfly.txt >> >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >> >>>> >> and >> it seems that "free" >> is the >> problem, not >> "waste"? So I >> guess >> what happens is that >> all the >> little classloaders >> allocate just >> enough memory to push >> them over >> "_small_chunk_limit=4", >> so they >> allocate the first >> medium chunk, >> from which >> then they >> take a >> small bite and leave the >> rest >> laying around? >> >> Yes. The free space of >> anonymous >> classes should be >> counted >> as waste, >> in my opinion. I am not >> sure if >> everyone agrees, >> so I took the >> summary line out of this >> patch. >> >> I would be more than happy >> to restore >> the summary >> line, if >> you find >> it is useful :-) >> >> >> I agree with you. Typically class >> loaders >> stop loading >> at some >> point, especially these tine >> ones, and >> often will not >> resume >> loading. So, effectivly, the >> space is >> wasted. It still >> helps to >> distinguish "free" in the current >> chunks >> from "free" in the >> other chunks to tell this >> situation apart >> from a >> situation where >> we have just a bug in metaspace >> chunk handling >> preventing us to >> use up our chunks. >> >> >> The latter >> would be an >> important >> addition, >> regardless >> if this is >> for customers >> or for us. >> Chunks idling in >> freelists can >> make a >> huge impact, and >> unfortunately this >> may happen >> in perfectly >> valid cases. >> See e.g. >> our JEP >> "https://bugs.openjdk.java.net/browse/JDK-8166690 >> >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>>>>". We have >> already a >> printing >> routine for free >> chunks, in >> ChunkManager::print_all_chunkmanagers(). Could >> you add >> this to >> your output? >> >> >> Make sense! Could >> you >> recommend what data and >> format you >> would like >> to see? >> >> >> >> Would not >> ChunkManager::print_all_chunkmanagers() be >> sufficient? >> >> Okay, I might need to tweak >> output >> format. >> >> >> Fine by me. Nothing depends on it >> yet. >> >> Thanks, Thomas >> >> Thanks, >> >> -Zhengyu >> >> >> >> ------------- >> >> Other than >> above notes, >> the changes seem >> straighforward, I did >> not see >> anything wrong. >> Minor nits: >> >> - >> PrintCLDMetaspaceInfoClosure::_out, >> _scale >> and _unit >> could all >> be constant >> (with _out >> being a >> outputStream* >> const). >> - We also do >> not need to >> provide >> scale *and* >> unit as >> argument, >> one can be >> deduced from >> the other. >> This would >> also prevent >> mismatches.We >> do need >> scale here, >> since nmt >> command >> line can set >> scale and we do >> >> have to honor that. >> >> However, passing >> unit is >> ugly! I don't >> want to >> introduce >> inverse >> dependence >> (metaspace -> >> nmt), which is >> also ugly. >> >> >> Yes, that would be >> regrettable. >> >> Probably should >> move scale >> mapping code from >> NMTUtil to a >> common util? >> >> >> >> That would be possible, >> these >> functions >> (scale_from_name etc) >> are simple enough to be >> moved to >> a generic file. >> However, if you >> pass scale, deducing the >> string >> that goes with >> it is >> trivial and >> I would have no qualms >> doing this >> by hand in >> metaspace.cpp. But >> it is a matter of taste. >> >> >> I did not look >> closely >> at locking >> issues, I assume >> MetaspaceAux::print_metadata() is >> supposed to >> run only in >> Safepoints? >> >> >> No. >> sum_xxx_xxx_in_use >> queries are >> guarded by locks. >> >> Thanks, >> >> -Zhengyu >> >> >> Thanks, Thomas >> >> >> >> Thanks & Kind >> Regards, >> Thomas >> >> >> >> >> >> >> On Fri, Oct 20, >> 2017 at >> 4:00 PM, >> Zhengyu Gu >> > >> > >> >> >> >> > > > >> >> >>> >> > >> > >> >> >> >> > > > >> >> >>>> >> >> > >> > > >> >> >> > >> > > >>> >> > >> > >> >> >> >> > > > >> >> >>>>> >> > >> > >> >> >> >> > > > >> >> >>> >> > >> > >> >> >> >> > > > >> >> >>>> >> >> >> > >> > > >> >> >> > >> > > >>> >> > >> > >> >> >> >> > > > >> >> >>>>>>> wrote: >> >> Up to now, >> there is >> little >> visibility >> into how >> metaspaces >> are used, >> and by >> whom. >> >> This >> proposed >> enhancement gives >> NMT the >> ability to >> report >> metaspace >> usages on >> per-classloader level, >> via a >> new NMT command >> "metadata" >> >> >> For >> example: >> >> 2496: >> Metaspaces: >> Metadata space: >> reserved= 8192KB >> committed= 5888KB >> Class >> space: >> reserved= 1048576KB >> committed= 640KB >> >> Per-classloader >> metadata: >> >> ClassLoader: for >> anonymous class >> Metadata capacity= 5.00KB >> used= 1.16KB >> free= >> 3.84KB >> waste= 0.05KB >> Class >> data >> capacity= 1.00KB >> used= 0.59KB >> free= >> 0.41KB >> waste= 0.00KB >> >> .... >> >> >> ClassLoader: >> >> Metadata capacity= 5640.00KB >> used= 5607.42KB >> free= >> 32.58KB >> waste= 0.46KB >> Class >> data >> capacity= 520.00KB >> used= 500.94KB >> free= >> 19.06KB >> waste= 0.02KB >> >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8189688 >> >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>> >> > >> > > >> > >> > >> >> > >> > > >> > >> > >>>>> >> > >> > > >> > >> >> From gerard.ziemski at oracle.com Tue Oct 31 14:20:37 2017 From: gerard.ziemski at oracle.com (Gerard Ziemski) Date: Tue, 31 Oct 2017 09:20:37 -0500 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <9dde915c-90ed-be0b-fec1-5b7a1f80338a@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> <9dde915c-90ed-be0b-fec1-5b7a1f80338a@oracle.com> Message-ID: <93023F3B-7B03-4B27-A536-C89BA1F285C3@oracle.com> hi David, I humbly disagree. The flag was only meant to be there temporarily, and the fact that the feature is turned ON by default here means that turning it OFF would be considered ?experimental?, since we don?t want such option to be considered a valid product configuration. Does that argument convince you? If not, I?d rather remove the flag altogether. cheers > On Oct 30, 2017, at 10:50 PM, David Holmes wrote: > > Hi Gerard, > > The flag should not be experimental - enabling resizable tables is not an experiment that the user is controlling, it is the default behaviour that we intend to occur. The flag is there to turn it off if for some reason this doesn't work - the flag should be a product flag. > > The flag also requires a CSR request. > > Thanks, > David > > On 31/10/2017 9:28 AM, Gerard Ziemski wrote: >> hi Coleen, >> I updated the webrev to rev3 >> bug: >> https://bugs.openjdk.java.net/browse/JDK-8184765 >> webrev: >> http://cr.openjdk.java.net/~gziemski/8184765_rev3 >>> On Oct 30, 2017, at 10:36 AM, coleen.phillimore at oracle.com wrote: >>> >>> >>> >>> On 10/30/17 11:24 AM, Gerard Ziemski wrote: >>>> hi Coleen, >>>> >>>> >>>>> On Oct 30, 2017, at 10:12 AM, coleen.phillimore at oracle.com wrote: >>>>> >>>>> >>>>> Hi Gerard, >>>>> >>>>> This looks great. One small question: >>>>> + // straight forward brute force >>>>> + inline static int _next_prime(int n) { >>>>> + int p = n; >>>>> + for (int i = n; i < (n * n); i++) { >>>>> + if ((i % 2) != 0) { >>>>> + p = i; >>>>> + break; >>>>> + } >>>>> + } >>>>> + return p; >>>>> + } >>>>> >>>>> Is this how you calculate next prime? Wouldn't you check if it can mod by 3 and 5 as well? >>>> As long as it?s not even i.e. mod 2 (and strictly speaking larger than 1). 3 and 5 are prime, so no need to check them. >>> >>> If you passed in 214 (2*107), 215 would look prime but it's not. I think the prime table would be better and limit resizing occurrences. >> Thank you for catching the problem. >> I went with a simple table of prime numbers (as was used before), since Coleen pointed out to me that the resizing will occur at safe point, so we want to keep the time used to find next prime to a minimum (though resizing is probably so expensive anyways, would it really matter?) >> cheers From john.r.rose at oracle.com Tue Oct 31 16:39:36 2017 From: john.r.rose at oracle.com (John Rose) Date: Tue, 31 Oct 2017 09:39:36 -0700 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: <9dde915c-90ed-be0b-fec1-5b7a1f80338a@oracle.com> References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> <9dde915c-90ed-be0b-fec1-5b7a1f80338a@oracle.com> Message-ID: On Oct 30, 2017, at 8:50 PM, David Holmes wrote: > > The flag should not be experimental - enabling resizable tables is not an experiment that the user is controlling, it is the default behaviour that we intend to occur. The flag is there to turn it off if for some reason this doesn't work This describes a diagnostic flag, not a product flag. We mask those flags to avoid polluting the list of true product flags with diagnostic hooks. > - the flag should be a product flag. Hence, it should be a diagnostic flag, if we think there is a long-term diagnostic benefit to turning it off. Otherwise it should be removed as Gerard suggests. Diagnosis could be for either bugs or performance. If there is a non-diagnostic reason, where we expect a user to flip it more or less permanently for their use case, it could be product. Experimental flags should be short-lived. ? John From coleen.phillimore at oracle.com Tue Oct 31 18:39:45 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 31 Oct 2017 14:39:45 -0400 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> <9dde915c-90ed-be0b-fec1-5b7a1f80338a@oracle.com> Message-ID: On 10/31/17 12:39 PM, John Rose wrote: > On Oct 30, 2017, at 8:50 PM, David Holmes > wrote: >> >> The flag should not be experimental - enabling resizable tables is >> not an experiment that the user is controlling, it is the default >> behaviour that we intend to occur. The flag is there to turn it off >> if for some reason this doesn't work > > This describes a diagnostic flag, not a product flag. ?We mask those > flags to avoid polluting the list of true product flags with > diagnostic hooks. > >> - the flag should be a product flag. > > Hence, it should be a diagnostic flag, if we think there is a long-term > diagnostic benefit to turning it off. ?Otherwise it should be removed as > Gerard suggests. ?Diagnosis could be for either bugs or performance. > > If there is a non-diagnostic reason, where we expect a user to flip it > more or less permanently for their use case, it could be product. I agree this should be a diagnostic flag.?? Technically we shouldn't need it because this feature has been well developed and tested, but we've been erring on the side of caution lately. Do we still need a CSR for the diagnostic flag? > > Experimental flags should be short-lived. > Hopefully the diagnostic flag should be short-lived as well. Thanks, Coleen > ? John From coleen.phillimore at oracle.com Tue Oct 31 19:10:18 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 31 Oct 2017 15:10:18 -0400 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> Message-ID: This looks good now.? The table would also limit the number of resizing events, until you hit the max. thanks, Coleen On 10/30/17 7:28 PM, Gerard Ziemski wrote: > hi Coleen, > > I updated the webrev to rev3 > > bug: > https://bugs.openjdk.java.net/browse/JDK-8184765 > > webrev: > http://cr.openjdk.java.net/~gziemski/8184765_rev3 > > >> On Oct 30, 2017, at 10:36 AM, coleen.phillimore at oracle.com wrote: >> >> >> >> On 10/30/17 11:24 AM, Gerard Ziemski wrote: >>> hi Coleen, >>> >>> >>>> On Oct 30, 2017, at 10:12 AM, coleen.phillimore at oracle.com wrote: >>>> >>>> >>>> Hi Gerard, >>>> >>>> This looks great. One small question: >>>> + // straight forward brute force >>>> + inline static int _next_prime(int n) { >>>> + int p = n; >>>> + for (int i = n; i < (n * n); i++) { >>>> + if ((i % 2) != 0) { >>>> + p = i; >>>> + break; >>>> + } >>>> + } >>>> + return p; >>>> + } >>>> >>>> Is this how you calculate next prime? Wouldn't you check if it can mod by 3 and 5 as well? >>> As long as it?s not even i.e. mod 2 (and strictly speaking larger than 1). 3 and 5 are prime, so no need to check them. >> If you passed in 214 (2*107), 215 would look prime but it's not. I think the prime table would be better and limit resizing occurrences. > Thank you for catching the problem. > > I went with a simple table of prime numbers (as was used before), since Coleen pointed out to me that the resizing will occur at safe point, so we want to keep the time used to find next prime to a minimum (though resizing is probably so expensive anyways, would it really matter?) > > > cheers > > From coleen.phillimore at oracle.com Tue Oct 31 19:12:46 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 31 Oct 2017 15:12:46 -0400 Subject: RFR: 8184765: Dynamically resize SystemDictionary In-Reply-To: References: <845C746C-0013-4592-85E6-67F82FAAD590@oracle.com> <9f3083f1-a837-dd2b-58bc-bd80c1a9364f@oracle.com> <8d18d4d8-e659-1d9e-75e0-1a066f774c98@oracle.com> Message-ID: On 10/31/17 3:10 PM, coleen.phillimore at oracle.com wrote: > > This looks good now.? The table would also limit the number of > resizing events, until you hit the max. Sorry, modulo making the flag a diagnostic flag. thanks, Coleen > > thanks, > Coleen > > On 10/30/17 7:28 PM, Gerard Ziemski wrote: >> hi Coleen, >> >> I updated the webrev to rev3 >> >> bug: >> https://bugs.openjdk.java.net/browse/JDK-8184765 >> >> webrev: >> http://cr.openjdk.java.net/~gziemski/8184765_rev3 >> >> >>> On Oct 30, 2017, at 10:36 AM, coleen.phillimore at oracle.com wrote: >>> >>> >>> >>> On 10/30/17 11:24 AM, Gerard Ziemski wrote: >>>> hi Coleen, >>>> >>>> >>>>> On Oct 30, 2017, at 10:12 AM, coleen.phillimore at oracle.com wrote: >>>>> >>>>> >>>>> Hi Gerard, >>>>> >>>>> This looks great.? One small question: >>>>> + // straight forward brute force >>>>> + inline static int _next_prime(int n) { >>>>> +?? int p = n; >>>>> +?? for (int i = n; i < (n * n); i++) { >>>>> +???? if ((i % 2) != 0) { >>>>> +?????? p = i; >>>>> +?????? break; >>>>> +???? } >>>>> +?? } >>>>> +?? return p; >>>>> + } >>>>> >>>>> Is this how you calculate next prime?? Wouldn't you check if it >>>>> can mod by 3 and 5 as well? >>>> As long as it?s not even i.e. mod 2 (and strictly speaking larger >>>> than 1). 3 and 5 are prime, so no need to check them. >>> If you passed in 214 (2*107), 215 would look prime but it's not.?? I >>> think the prime table would be better and limit resizing occurrences. >> Thank you for catching the problem. >> >> I went with a simple table of prime numbers (as was used before), >> since Coleen pointed out to me that the resizing will occur at safe >> point, so we want to keep the time used to find next prime to a >> minimum (though resizing is probably so expensive anyways, would it >> really matter?) >> >> >> cheers >> >> > From coleen.phillimore at oracle.com Tue Oct 31 19:18:49 2017 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 31 Oct 2017 15:18:49 -0400 Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace fragmentation In-Reply-To: <667f07ea2fb64e97b8a149310961693f@sap.com> References: <667f07ea2fb64e97b8a149310961693f@sap.com> Message-ID: This change looks good. Coleen On 10/30/17 8:49 AM, Lindenmaier, Goetz wrote: > Hi Thomas, > > the change looks good and I know we have made good use of this > already. > > Can you look into the chunks? It could be useful to know that, > for example, a medium chunk is used by a class loader, but > still mostly empty (i.e., empty but not free). Well, there is no > third variant after lower and upper case, but maybe empty > parts could be indicated in the line with the dots by 'e's (obviously > at the granularity of small chunks). > > Thanks, > Goetz. > >> -----Original Message----- >> From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- >> bounces at openjdk.java.net] On Behalf Of Thomas St?fe >> Sent: Wednesday, October 25, 2017 6:52 AM >> To: hotspot-runtime-dev at openjdk.java.net >> Subject: RFR(s): 8189864: Provide an ascii map to visualize metaspace >> fragmentation >> >> Hi all, >> >> could I please have your thoughts and reviews for this enhancement: >> >> Issue: https://bugs.openjdk.java.net/browse/JDK-8189864 >> Webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8189864- >> metaspace-map/webrev.00/webrev/ >> >> At SAP, we added a something we call a metaspace map to the metaspace >> analysis functions. This is a very simple ASCII print of the metaspace >> layout. It shows the composition of the VirtualSpaceNodes on a chunk basis. >> It facilitates examining fragmentation and has been proven useful when >> experimenting with the metaspace allocator. >> >> This change adds the map printing code and invokes it if a Metaspace OOM >> occurs and -Xlog:gc+metaspace+freelist=debug is active (in this case, we >> already print out a bunch of things). We also may consider adding this to >> NMT or as a jcmd addition, but I did not want to overload the patch. >> >> Example output looks like this (mind that this will only make sense with a >> monospaced font). Dots indicate starts of chunks. Letters indicate chunk >> type - (H)umongous, (M)edium, (S)mall (X)specialized - with lower case >> letters indicating >> a free chunk, upper case letters indicating a chunk in use. >> >> 0x0000000100000000: ...... >> xxxxxxHHHHHHHHHHHHHHHHHHHHHHHH >> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> HHHHHHHHH >> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> 0x0000000100020000: >> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> HHHHHHHHH >> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> 0x0000000100040000: >> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> HHHHHHHHH >> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH >> 0x0000000100060000: . . . . . . . . . . . . . . >> SSSSSSSSSSSSMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMSSSSSSSSMMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> M >> 0x0000000100080000: . . . . >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm >> mmmmmmmmmmmmmmmmMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> M >> 0x00000001000a0000: . . . . >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> M >> 0x00000001000c0000: . . . . >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMmmmmmmmmmmmmmmmm >> mmmmmmmmmmmmmmmmMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> M >> 0x00000001000e0000: . . . . >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> M >> 0x0000000100100000: . . . . . . . . . . . . . . . . . . . . . . . . >> MMMMMMMMMMMMMMMMMMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMMMMSSSSMMMMMMMMMMMMM >> MMMMMMMMMMMMMMMMMMMSS >> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >> 0x0000000100120000: . . . . . . . . . . . . . . . . . . . . . . . . . . . >> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . >> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS >> >> >> Thank you! >> >> Thomas From kishor.kharbas at intel.com Tue Oct 31 23:53:12 2017 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Tue, 31 Oct 2017 23:53:12 +0000 Subject: RFR(M): 8190308: Supporting heap allocation on alternative memory devices and CSR review Message-ID: Greetings, I am continuing the earlier discussion [1] with a revised issue number representing the implementation of the JEP [2]. I have addressed the remaining unaddressed comments (copied below) in these webrevs - http://cr.openjdk.java.net/~kkharbas/8190308/webrev.11/ http://cr.openjdk.java.net/~kkharbas/8190308/webrev.11_to_10/ Also, I would appreciate a review of the CSR for the new flag at https://bugs.openjdk.java.net/browse/JDK-8190309. > > - in that same thread there has also been the question about the > > need > > for blocking the signals for the process that has gone unanswered. > > Removed the blocking of signals. > > - Arguments::finalize_vm_init_args: maybe at the place where the > > change adds the warning/info message about NUMA support being > > specific > > to the file system; also add the same warning about UseLargePages > > that > > is located elsewhere. > > > > Like "UseXXXX may not be supported in some specific file system > > implementations." - or just ignore as Andrew said in the other > > email thread. > > > > Note that I am not sure that the selected place is the correct > > place > > for such warning about incompatible flags (and maybe > > UseNUMA/UseLargePages should be forcibly disabled here? But then > > again, it does not hurt just having it enabled?). > > > > I will let the runtime team comment as a lot of that argument > > processing changed in jdk9. > > > > Maybe Arguments::check_vm_args_consistency() is better. > > > > There is similar code in Arguments::adjust_after_os()... > > 1. Moved the check from finalize_vm_init_args() to check_vm_args_consistency() 2. When UseNUMA flag is true, adaptive logical group chunk resizing is used for Numa aware collectors such as ParallelOld. Just like UseNUMA is disabled in os::inti_2() in Linux when UseLargePages is set, it will be a good idea to disable UseNUMA when the new option is used. 3. I think its ok to leave UseNUMAInterleaving ON as it can be used for other memory areas. 4. For the same reason I also do not disable UseLargePages. 5. About handling UseLargePages, I thought of handling it the same way as when reserve_memory_special() fails to allocate large pages, i.e. generating a log_debug message. // failed; try to reserve regular memory below if (UseLargePages && (!FLAG_IS_DEFAULT(UseLargePages) || !FLAG_IS_DEFAULT(LargePageSizeInBytes))) { log_debug(gc, heap, coops)("Reserve regular memory without large pages"); > > - here I may probably be speaking wrongly on behalf of the > > runtime > > team, but os.hpp, as an abstraction of the OS should probably not > > have > > platform specific ifdefs in it. > > and > > > - You removed os::attempt_reserve_memory_at() from os.cpp and > > > split > > > into os_posix.cpp and os_windows.cpp. > > > But I think you should remain os::attempt_reserve_memory_at() > > > at > > > os.cpp and implement os specific stuffs at each os_xxx.cpp files > > > for > > > pd_xxx. Of cource move MemTracker function call as well. In the updated webrev, I move the implementation from os_posix.cpp and os_window.cpp to respective pd_xxx functions. No AIX specific ifdef is required now. Thank you and looking forward for your feedback. - Kishor [1] http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2017-October/020682.html [2] https://bugs.openjdk.java.net/browse/JDK-8171181