From david.holmes at oracle.com Tue Nov 1 01:45:21 2016 From: david.holmes at oracle.com (David Holmes) Date: Tue, 1 Nov 2016 11:45:21 +1000 Subject: How to use @ReservedStackAccess? In-Reply-To: <8850ae23-fda8-2481-261e-42b53131eb72@codefx.org> References: <8850ae23-fda8-2481-261e-42b53131eb72@codefx.org> Message-ID: <30ffcdb9-d2be-2c6e-4fd6-328282408eb3@oracle.com> Hi Nicolai, On 1/11/2016 1:39 AM, Nicolai Parlog wrote: > Hi! > > I've been experimenting with @ReservedStackAccess but couldn't get it > to work. Any help would be highly appreciated. > > ## SETUP > > I'm artificially creating a stack overflow by recursing indefinitely. > I then want to benefit from @ReservedStackAccess by executing some > code outside of an exception handler. > > Here's my code: > > public static void main(String[] args) { > try { > recurseThenGreet(); > } catch (StackOverflowError err) { > // to not have the console spammed with output > System.out.println("Error"); > } > } > > @ReservedStackAccess > private static void recurseThenGreet() { > recurse(); > System.out.println("Hi!"); > } > > private static void recurse() { > recurse(); > } > > I'm using build 9-ea+141-jigsaw-nightly-h5650-20161026. I compile with > > --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED > > to make the annotation available and launch with > > -XX:-RestrictReservedStack > > to activate the reserved stack for user land code. > > ## OBSERVED > > This is the output I get: > > Java HotSpot(TM) 64-Bit Server VM warning: Potentially dangerous stack > overflow in ReservedStackAccess annotated method > org.codefx.demo.java9.internal.stack.ReservingStackFrames_Simple.recurseThenGreet()V[1] > Error > > ## EXPECTED > > I expected "Hi!" to show up somewhere there. ReservedStackAccess doesn't give you unlimited stack, it gives you a little extra to try and complete the method that was annotated. In your case you will continue to recurse and exhaust the additional stack area as well. David ----- > My best guess is that I put the annotation in the wrong place but > experimenting didn't help. Any help would be greatly appreciated! > > Thanks! > Nicolai > > > From nipa at codefx.org Tue Nov 1 07:48:23 2016 From: nipa at codefx.org (Nicolai Parlog) Date: Tue, 1 Nov 2016 08:48:23 +0100 Subject: How to use @ReservedStackAccess? In-Reply-To: <30ffcdb9-d2be-2c6e-4fd6-328282408eb3@oracle.com> References: <8850ae23-fda8-2481-261e-42b53131eb72@codefx.org> <30ffcdb9-d2be-2c6e-4fd6-328282408eb3@oracle.com> Message-ID: <03117312-c921-d18d-13f2-932fb5f16f68@codefx.org> Hi David, oh, now I get it! I knew that I only get 4k of reserved space but I didn't quite get where computation would resume once stack space ran out. I got lost in the weeds and thought the JVM would fall back to the annotated frame and somehow continue in there... But it's much simpler: I just get a little more rope to hang myself with. :) Thanks! Nicolai On 01.11.2016 02:45, David Holmes wrote: > Hi Nicolai, > > On 1/11/2016 1:39 AM, Nicolai Parlog wrote: >> Hi! >> >> I've been experimenting with @ReservedStackAccess but couldn't >> get it to work. Any help would be highly appreciated. >> >> ## SETUP >> >> I'm artificially creating a stack overflow by recursing >> indefinitely. I then want to benefit from @ReservedStackAccess by >> executing some code outside of an exception handler. >> >> Here's my code: >> >> public static void main(String[] args) { try { >> recurseThenGreet(); } catch (StackOverflowError err) { // to not >> have the console spammed with output >> System.out.println("Error"); } } >> >> @ReservedStackAccess private static void recurseThenGreet() { >> recurse(); System.out.println("Hi!"); } >> >> private static void recurse() { recurse(); } >> >> I'm using build 9-ea+141-jigsaw-nightly-h5650-20161026. I compile >> with >> >> --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED >> >> to make the annotation available and launch with >> >> -XX:-RestrictReservedStack >> >> to activate the reserved stack for user land code. >> >> ## OBSERVED >> >> This is the output I get: >> >> Java HotSpot(TM) 64-Bit Server VM warning: Potentially dangerous >> stack overflow in ReservedStackAccess annotated method >> org.codefx.demo.java9.internal.stack.ReservingStackFrames_Simple.recu rseThenGreet()V[1] >> >> >> Error >> >> ## EXPECTED >> >> I expected "Hi!" to show up somewhere there. > > ReservedStackAccess doesn't give you unlimited stack, it gives you > a little extra to try and complete the method that was annotated. > In your case you will continue to recurse and exhaust the > additional stack area as well. > > David ----- > >> My best guess is that I put the annotation in the wrong place >> but experimenting didn't help. Any help would be greatly >> appreciated! >> >> Thanks! Nicolai >> >> >> > -- PGP Key: http://keys.gnupg.net/pks/lookup?op=vindex&search=0xCA3BAD2E9CCCD509 Web: http://codefx.org a blog about software development https://www.sitepoint.com/java high-quality Java/JVM content http://do-foss.de Free and Open Source Software for the City of Dortmund Twitter: https://twitter.com/nipafx From aph at redhat.com Tue Nov 1 09:44:15 2016 From: aph at redhat.com (Andrew Haley) Date: Tue, 1 Nov 2016 09:44:15 +0000 Subject: RFR(M): 8154736: enhancement of cmpxchg and copy_to_survivor for ppc64 In-Reply-To: References: <201604221228.u3MCSXCL020021@d19av07.sagamino.japan.ibm.com> <1475236951.6301.72.camel@oracle.com> <6ee4f1c6-f638-c5b9-7475-8fb6aeabf20b@oracle.com> <14c2eff4-4f90-caa0-17a7-835e6f1f1167@oracle.com> <1cbb094f-b29b-c6b3-1e50-bed21b140fcb@oracle.com> Message-ID: On 31/10/16 21:30, David Holmes wrote: > > > On 31/10/2016 7:32 PM, Andrew Haley wrote: >> On 30/10/16 21:26, David Holmes wrote: >>> On 31/10/2016 4:36 AM, Andrew Haley wrote: >>>> >>>> And, while we're on the subject, is memory_order_conservative actually >>>> defined anywhere? >>> >>> No. It was chosen to represent the current status quo that the Atomic:: >>> ops should all be (by default) full bi-directional fences. >> >> Does that mean that a CAS is actually stronger than a load acquire >> followed by a store release? And that a CAS is a release fence even >> when it fails and no store happens? > > Yes. Yes. > > // All of the atomic operations that imply a read-modify-write > // action guarantee a two-way memory barrier across that > // operation. Historically these semantics reflect the strength > // of atomic operations that are provided on SPARC/X86. We assume > // that strength is necessary unless we can prove that a weaker > // form is sufficiently safe. Mmmm, but that doesn't say anything about a CAS that fails. But fair enough, I accept your interpretation. > But there is some contention as to whether the actual implementations > obey this completely. Linux/AArch64 uses GCC's __sync_val_compare_and_swap, which is specified as a "full barrier". That is, no memory operand is moved across the operation, either forward or backward. Further, instructions are issued as necessary to prevent the processor from speculating loads across the operation and from queuing stores after the operation. ... which reads the same as the language you quoted above, but looking at the assembly code I'm sure that it's really no stronger than a seq cst load followed by a seq cst store. I guess maybe I could give up fighting this and implement all AArch64 CAS sequences as CAS(seq_cst); full fence or, even more extremely, full fence; CAS(relaxed); full fence but it all seems unreasonably heavyweight. >> And that a conservative load is a *store* barrier? > > Not sure what you mean. Atomic::load is not a r-m-w action so not > expected to be a two-way memory barrier. OK. Thanks, Andrew. From serguei.spitsyn at oracle.com Tue Nov 1 12:58:56 2016 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 1 Nov 2016 08:58:56 -0400 Subject: RFR (XS): 8160024 jdb returns invalid argument count if first parameter to Arrays.asList is null In-Reply-To: References: Message-ID: <64c20710-5257-e4d6-c613-06ff0c66cae4@oracle.com> Please, review this small fix. Bug: https://bugs.openjdk.java.net/browse/JDK-8160024 JDK webrev: http://cr.openjdk.java.net/~sspitsyn/webrevs/2016/hotspot/8160024-JDI-eval.jdk1/ The bug is confidential, so the RFR is sent to the confidential mailing lists. Summary: The JDI incorrectly evaluates the expression: Arrays.asList(null, "a"). The root cause is a special case in the method MethodImpl.handleVarArgs() that has been corrected in order to fix the problem. New unit test is included: jdk/test/com/sun/jdi/EvalArraysAsList.sh Testing: Ran new unit test, Jtreg com/sun/jdi and nsk.jdi.testlist. All tests are passed. Thanks, Serguei From serguei.spitsyn at oracle.com Tue Nov 1 13:03:16 2016 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 1 Nov 2016 09:03:16 -0400 Subject: RFR (XS): 8160024 jdb returns invalid argument count if first parameter to Arrays.asList is null In-Reply-To: <64c20710-5257-e4d6-c613-06ff0c66cae4@oracle.com> References: <64c20710-5257-e4d6-c613-06ff0c66cae4@oracle.com> Message-ID: On 11/1/16 08:58, serguei.spitsyn at oracle.com wrote: > Please, review this small fix. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8160024 > > JDK webrev: > http://cr.openjdk.java.net/~sspitsyn/webrevs/2016/hotspot/8160024-JDI-eval.jdk1/ > > > The bug is confidential, so the RFR is sent to the confidential mailing > lists. Sorry, I sent it in the open, but bug is confidential. Thanks, Serguei > > > Summary: > > The JDI incorrectly evaluates the expression: Arrays.asList(null, "a"). > The root cause is a special case in the method > MethodImpl.handleVarArgs() > that has been corrected in order to fix the problem. > > New unit test is included: jdk/test/com/sun/jdi/EvalArraysAsList.sh > > Testing: > Ran new unit test, Jtreg com/sun/jdi and nsk.jdi.testlist. > All tests are passed. > > Thanks, > Serguei > From kirill.zhaldybin at oracle.com Tue Nov 1 13:45:46 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Tue, 1 Nov 2016 16:45:46 +0300 Subject: RFR(S): 8168493: Convert TestBitMap_test to GTest Message-ID: Dear all, Could you please review this fix for 8168493? WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168493/webrev.00/ CR: https://bugs.openjdk.java.net/browse/JDK-8168493 Changes: 1. Usual gtest conversion - asserts, file location etc 2. Simplified the test a bit - deleted unused template parameter, refactoring 3. Deleted friend class from BitMap class since we actually do not need it Thank you. Regards, Kirill From daniel.fuchs at oracle.com Tue Nov 1 14:04:25 2016 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 1 Nov 2016 14:04:25 +0000 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> Message-ID: <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> Hi Mandy, I am sure I must be missing something: 322 if (s == null) { 323 // all elements will be included 324 s = ""; 325 if (classLoaderName != null && !classLoaderName.isEmpty()) { 326 s += classLoaderName + "/"; 327 } 328 if (moduleName != null && !moduleName.isEmpty()) { 329 s += moduleName; 330 } 331 if (moduleVersion != null && !moduleVersion.isEmpty()) { 332 s += "@" + moduleVersion; 333 } 334 s += declaringClass; 335 } but should line 334 instead be s = (s.isEmpty() ? declaringClass : s + "/" + declaringClass; Otherwise "/" will be missing after module at version.. Also should there be some asserts somewhere verifying that moduleVersion is null or empty when moduleName is null or empty? At the moment the constructor will blindly accept a version for an unnamed module, and I am assuming this is wrong - am I right? toLoaderModuleClassName will call ClassLoader.getName(). If a class loader overrides getName() then that might be a different name than what StackTraceElement.getClassLoaderName() returns. Is that an issue? Also I wonder whether you should be considering including a fix for https://bugs.openjdk.java.net/browse/JDK-8167099 as part of this change (though arguably the bug has been there since new fields were added to StackTraceElement in 9). best regards, -- daniel On 28/10/16 21:44, Mandy Chung wrote: > >> On Oct 28, 2016, at 11:11 AM, Brent Christian wrote: >> >> Should something be done for STEs returned from StackFrameInfo.toStackTraceElement() ? > > Good catch - I missed it. I added package-private static methods in StackTraceElement class for both Throwable and StackFrameInfo to get StackTraceElement(s). > > http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.02/ > > Mandy > From kirill.zhaldybin at oracle.com Tue Nov 1 14:10:57 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Tue, 1 Nov 2016 17:10:57 +0300 Subject: RFR(S): 8168492: Convert CollectedHeap_test to GTest Message-ID: <489f77cb-f7be-51f5-b602-9e2eab3ee807@oracle.com> Dear all, Could you please review this fix for 8168492? WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168492/webrev.00/ CR: https://bugs.openjdk.java.net/browse/JDK-8168492 Changes: 1. Changed _reserved to reserved_region() since _reserved is private but we already had an accessor. 2. Added more detailed messages for sanity checks 3. Otherwise pretty straight-forward conversion to gtest. Thank you. Regards, Kirill From kirill.zhaldybin at oracle.com Tue Nov 1 17:30:09 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Tue, 1 Nov 2016 20:30:09 +0300 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part Message-ID: Dear all, Could you please review this fix for 8169003? I changed parsing of time string so now it is not depend on LC_NUMERIC locale so the test does not fail if locale where "floating point" is actually a comma is set. WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ CR: https://bugs.openjdk.java.net/browse/JDK-8169003 Thank you. Regards, Kirill From daniel.daugherty at oracle.com Tue Nov 1 22:03:33 2016 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 1 Nov 2016 16:03:33 -0600 Subject: RFR (XS): 8160024 jdb returns invalid argument count if first parameter to Arrays.asList is null In-Reply-To: <64c20710-5257-e4d6-c613-06ff0c66cae4@oracle.com> References: <64c20710-5257-e4d6-c613-06ff0c66cae4@oracle.com> Message-ID: On 11/1/16 6:58 AM, serguei.spitsyn at oracle.com wrote: > Please, review this small fix. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8160024 > > JDK webrev: > http://cr.openjdk.java.net/~sspitsyn/webrevs/2016/hotspot/8160024-JDI-eval.jdk1/ > src/jdk.jdi/share/classes/com/sun/tools/jdi/MethodImpl.java L318: if (nthArgValue == null && argCount == paramCount) { L319: // The only one varargs parameter is null Perhaps this comment rewrite: // We have one varargs parameter and it is null // so we don't have to do anything. L322: Type nthArgType = (nthArgValue == null) ? null : nthArgValue.type(); Perhaps add this comment above L322: // If the first varargs parameter is null, then don't // access its type since it can't be an array. L347: argArray.setValues(0, arguments, paramCount - 1, count); Looks like setValues() will happily copy any null values that happen to be in the arguments List to the argArray. Good. test/com/sun/jdi/EvalArraysAsList.sh No comments. Thumbs up, but I thought we weren't suppose to add any more new shell script tests... :-) Dan > > The bug is confidential, so the RFR is sent to the confidential mailing > lists. > > > Summary: > > The JDI incorrectly evaluates the expression: Arrays.asList(null, "a"). > The root cause is a special case in the method > MethodImpl.handleVarArgs() > that has been corrected in order to fix the problem. > > New unit test is included: jdk/test/com/sun/jdi/EvalArraysAsList.sh > > Testing: > Ran new unit test, Jtreg com/sun/jdi and nsk.jdi.testlist. > All tests are passed. > > Thanks, > Serguei > From mandy.chung at oracle.com Tue Nov 1 23:42:18 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 1 Nov 2016 16:42:18 -0700 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> Message-ID: <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> Hi Daniel, Here is the updated webrev incorporating your feedback: http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.03 > On Nov 1, 2016, at 7:04 AM, Daniel Fuchs wrote: > > : > 334 s += declaringClass; > > > but should line 334 instead be > > s = (s.isEmpty() ? declaringClass : s + "/" + declaringClass; > > Otherwise "/" will be missing after module at version.. > Good catch. I added a test for this. > Also should there be some asserts somewhere verifying that moduleVersion > is null or empty when moduleName is null or empty? At the moment the > constructor will blindly accept a version for an unnamed module, > and I am assuming this is wrong - am I right? > Good point. The constructor should throw IAE if module name/version and class loader name is an empty string. > toLoaderModuleClassName will call ClassLoader.getName(). > If a class loader overrides getName() then that might be a different > name than what StackTraceElement.getClassLoaderName() returns. > > Is that an issue? > I added a test to verify StackTraceElement that uses the ClassLoader's name field. I also added @apiNote in ClassLoader::getName. > Also I wonder whether you should be considering including a fix > for https://bugs.openjdk.java.net/browse/JDK-8167099 as part > of this change (though arguably the bug has been there since > new fields were added to StackTraceElement in 9). > It is related but it?s better to separate it from this issue. Do you have cycle to help write a test case? I can help fix JDK-8167099 later. Mandy From kim.barrett at oracle.com Wed Nov 2 01:39:28 2016 From: kim.barrett at oracle.com (Kim Barrett) Date: Tue, 1 Nov 2016 21:39:28 -0400 Subject: RFR(S): 8168493: Convert TestBitMap_test to GTest In-Reply-To: References: Message-ID: <726590BD-746B-45E6-92B3-52A199058B20@oracle.com> > On Nov 1, 2016, at 9:45 AM, Kirill Zhaldybin wrote: > > Dear all, > > Could you please review this fix for 8168493? > > WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168493/webrev.00/ > CR: https://bugs.openjdk.java.net/browse/JDK-8168493 > > Changes: > 1. Usual gtest conversion - asserts, file location etc > 2. Simplified the test a bit - deleted unused template parameter, refactoring > 3. Deleted friend class from BitMap class since we actually do not need it > > Thank you. > > Regards, Kirill Conversion looks good. We ought to have more extensive tests of this class, but that?s out of scope for this conversion. I have a test for the search and iteration operations that is waiting on JDK 10: JDK-8169039. From serguei.spitsyn at oracle.com Wed Nov 2 02:53:35 2016 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Tue, 1 Nov 2016 22:53:35 -0400 Subject: RFR (XS): 8160024 jdb returns invalid argument count if first parameter to Arrays.asList is null In-Reply-To: References: <64c20710-5257-e4d6-c613-06ff0c66cae4@oracle.com> Message-ID: Dan, Thank you for the review and the suggestions! I'll add the suggested comments. The test is ridiculously simple so that I did not want to make it more complex just for the sake of no shell scripting. :) Thanks! Serguei On 11/1/16 18:03, Daniel D. Daugherty wrote: > On 11/1/16 6:58 AM, serguei.spitsyn at oracle.com wrote: >> Please, review this small fix. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8160024 >> >> JDK webrev: >> http://cr.openjdk.java.net/~sspitsyn/webrevs/2016/hotspot/8160024-JDI-eval.jdk1/ >> > > src/jdk.jdi/share/classes/com/sun/tools/jdi/MethodImpl.java > L318: if (nthArgValue == null && argCount == paramCount) { > L319: // The only one varargs parameter is null > Perhaps this comment rewrite: > // We have one varargs parameter and it is null > // so we don't have to do anything. > > L322: Type nthArgType = (nthArgValue == null) ? null : > nthArgValue.type(); > Perhaps add this comment above L322: > // If the first varargs parameter is null, then don't > // access its type since it can't be an array. > > L347: argArray.setValues(0, arguments, paramCount - 1, > count); > Looks like setValues() will happily copy any null values that > happen to be in the arguments List to the argArray. Good. > > test/com/sun/jdi/EvalArraysAsList.sh > No comments. > > Thumbs up, but I thought we weren't suppose to add any more > new shell script tests... :-) > > Dan > > >> >> The bug is confidential, so the RFR is sent to the confidential mailing >> lists. >> >> >> Summary: >> >> The JDI incorrectly evaluates the expression: Arrays.asList(null, >> "a"). >> The root cause is a special case in the method >> MethodImpl.handleVarArgs() >> that has been corrected in order to fix the problem. >> >> New unit test is included: jdk/test/com/sun/jdi/EvalArraysAsList.sh >> >> Testing: >> Ran new unit test, Jtreg com/sun/jdi and nsk.jdi.testlist. >> All tests are passed. >> >> Thanks, >> Serguei >> > From daniel.fuchs at oracle.com Wed Nov 2 11:29:30 2016 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Wed, 2 Nov 2016 11:29:30 +0000 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> Message-ID: <2392ca27-b110-ce1c-0cc7-38447e575756@oracle.com> Hi Mandy, On 01/11/16 23:42, Mandy Chung wrote: > Hi Daniel, > > Here is the updated webrev incorporating your feedback: > http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.03 ClassLoader.java: 345 if (name != null && name.isEmpty()) { 346 throw new IllegalArgumentException("name must be non-empty or null"); 347 } I'd suggest passing 'name' to 'checkCreateClassLoader()' and do this check in checkCreateClassLoader instead - in order to do the checks before 'this' is created. [...] >> Also should there be some asserts somewhere verifying that moduleVersion >> is null or empty when moduleName is null or empty? At the moment the >> constructor will blindly accept a version for an unnamed module, >> and I am assuming this is wrong - am I right? >> > > Good point. The constructor should throw IAE if module name/version and class loader name is an empty string. That's not exactly what I had in mind. I don't have any particular feeling for whether "" should be allowed or not. Maybe it would be simpler to just allow "" to mean the same thing as 'null'. What I meant was: Can moduleVersion be non null (and non empty) if moduleName is null (or empty)? In other words can an unnamed module have a version? Also if you add some validation to the constructor then you will need to add a readObject to duplicate the validation checks. And if you disallow "" then checking for isEmpty() in toString() is not needed. But I have the feeling that simply allowing "" and null to mean the same thing would be more robust. It's just the question of having a moduleVersion for an unnamed module that bothers me. >> toLoaderModuleClassName will call ClassLoader.getName(). >> If a class loader overrides getName() then that might be a different >> name than what StackTraceElement.getClassLoaderName() returns. >> >> Is that an issue? >> > > I added a test to verify StackTraceElement that uses the ClassLoader's name field. I also added @apiNote in ClassLoader::getName. Ok - the @apiNote should cover that. >> Also I wonder whether you should be considering including a fix >> for https://bugs.openjdk.java.net/browse/JDK-8167099 as part >> of this change (though arguably the bug has been there since >> new fields were added to StackTraceElement in 9). >> > > It is related but it?s better to separate it from this issue. Do you have cycle to help write a test case? I can help fix JDK-8167099 later. Right - no problem with that. I haven't given any thought to a test case for JDK-8167099. I believe it will prevent to connect a JDK 9 jvisualvm to a JDK 8 process if not fixed though. Maybe serializing a ThreadInfo composite data containing some MonitorInfo in JDK 8, and then deserializing that in JDK 9 and trying to convert it back to ThreadInfo in 9 would show the issue. best regards, -- daniel > > Mandy > From kirill.zhaldybin at oracle.com Wed Nov 2 11:48:50 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldbybin) Date: Wed, 2 Nov 2016 14:48:50 +0300 Subject: RFR(S): 8168493: Convert TestBitMap_test to GTest In-Reply-To: <726590BD-746B-45E6-92B3-52A199058B20@oracle.com> References: <726590BD-746B-45E6-92B3-52A199058B20@oracle.com> Message-ID: <5819D2A2.3030109@oracle.com> Kim, Thank you for review! Regards, Kirill On 11/02/2016 04:39 AM, Kim Barrett wrote: >> On Nov 1, 2016, at 9:45 AM, Kirill Zhaldybin wrote: >> >> Dear all, >> >> Could you please review this fix for 8168493? >> >> WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168493/webrev.00/ >> CR: https://bugs.openjdk.java.net/browse/JDK-8168493 >> >> Changes: >> 1. Usual gtest conversion - asserts, file location etc >> 2. Simplified the test a bit - deleted unused template parameter, refactoring >> 3. Deleted friend class from BitMap class since we actually do not need it >> >> Thank you. >> >> Regards, Kirill > Conversion looks good. > > We ought to have more extensive tests of this class, but that?s out of scope for this conversion. > I have a test for the search and iteration operations that is waiting on JDK 10: JDK-8169039. > > From mandy.chung at oracle.com Wed Nov 2 19:11:19 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 2 Nov 2016 12:11:19 -0700 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <2392ca27-b110-ce1c-0cc7-38447e575756@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <2392ca27-b110-ce1c-0cc7-38447e575756@oracle.com> Message-ID: > On Nov 2, 2016, at 4:29 AM, Daniel Fuchs wrote: > > I'd suggest passing 'name' to 'checkCreateClassLoader()' and do > this check in checkCreateClassLoader instead - in order to do > the checks before 'this' is created. > Fixed. > That's not exactly what I had in mind. I don't have > any particular feeling for whether "" should be allowed > or not. Maybe it would be simpler to just allow "" to mean the > same thing as 'null'. What I meant was: > Can moduleVersion be non null (and non empty) if moduleName is null > (or empty)? In other words can an unnamed module have a version? > > Also if you add some validation to the constructor then > you will need to add a readObject to duplicate the validation > checks. And if you disallow "" then checking for isEmpty() in > toString() is not needed. But I have the feeling that simply > allowing "" and null to mean the same thing would be more > robust. It's just the question of having a moduleVersion > for an unnamed module that bothers me. > Having a second thought - it can?t do much validation with these strings e.g. whether the class loader name is the one loading the class or the module where this class is in. I think it?s better to keep it simple and allow empty string and StackTraceElement::toString should not print module version for unnamed module (I fixed a bug there. It was intended to do that anyway). Would that address your concern? > > Right - no problem with that. > I haven't given any thought to a test case for JDK-8167099. > I believe it will prevent to connect a JDK 9 jvisualvm to a > JDK 8 process if not fixed though. Maybe serializing a ThreadInfo > composite data containing some MonitorInfo in JDK 8, and > then deserializing that in JDK 9 and trying to convert it > back to ThreadInfo in 9 would show the issue. We should fix that. Updated version: http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.04/ Mandy From daniel.fuchs at oracle.com Wed Nov 2 19:57:49 2016 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Wed, 2 Nov 2016 19:57:49 +0000 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <2392ca27-b110-ce1c-0cc7-38447e575756@oracle.com> Message-ID: On 02/11/16 19:11, Mandy Chung wrote: > Updated version: > http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.04/ +1. I like that toString() is now consitent with toLoaderModuleClassName() in the way both methods handle module name and version. best regards, -- daniel From coleen.phillimore at oracle.com Thu Nov 3 00:40:34 2016 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Wed, 2 Nov 2016 20:40:34 -0400 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> Message-ID: Hi, I missed some of the iterations of this. To avoid making a large change larger, I think you should keep the original names of ToStackTraceElement and GetStackTraceElements. I don't think the new names are better and it leads to confusion of where the names are. It breaks the (hopefully ubiquitous) convention of naming Java functions similarly to the JVM functions that implement them. +JNIEXPORT void JNICALL Java_java_lang_StackTraceElement_fillFromStackFrameInfo + (JNIEnv *env, jobject element, jobject stackframeinfo) { + JVM_ToStackTraceElement(env, stackframeinfo, element); +} + +JNIEXPORT void JNICALL Java_java_lang_StackTraceElement_fillStackTraceFromThrowable + (JNIEnv *env, jobject dummy, jobjectArray elements, jobject throwable) +{ + JVM_GetStackTraceElements(env, throwable, elements); +} I had already filed a compatibility request using these names to replace the old ones. If you want to change them again, you should file a new compatibility request to change the old jdk8 name (which I don't remember right now). http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.03/jdk/src/java.base/share/classes/java/lang/StackFrameInfo.java.udiff.html Maybe you can use the name "fill" for this rather than "of". If you don't use an IDE, finding the function name "of" is really challenging. Also, I think you should check this into the hs repository when done so that this bug can be unblocked sooner: https://bugs.openjdk.java.net/browse/JDK-8165550. Thanks, Coleen On 11/1/16 7:42 PM, Mandy Chung wrote: > Hi Daniel, > > Here is the updated webrev incorporating your feedback: > http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.03 > >> On Nov 1, 2016, at 7:04 AM, Daniel Fuchs wrote: >> >> : >> 334 s += declaringClass; >> >> >> but should line 334 instead be >> >> s = (s.isEmpty() ? declaringClass : s + "/" + declaringClass; >> >> Otherwise "/" will be missing after module at version.. >> > Good catch. I added a test for this. > >> Also should there be some asserts somewhere verifying that moduleVersion >> is null or empty when moduleName is null or empty? At the moment the >> constructor will blindly accept a version for an unnamed module, >> and I am assuming this is wrong - am I right? >> > Good point. The constructor should throw IAE if module name/version and class loader name is an empty string. > >> toLoaderModuleClassName will call ClassLoader.getName(). >> If a class loader overrides getName() then that might be a different >> name than what StackTraceElement.getClassLoaderName() returns. >> >> Is that an issue? >> > I added a test to verify StackTraceElement that uses the ClassLoader's name field. I also added @apiNote in ClassLoader::getName. > >> Also I wonder whether you should be considering including a fix >> for https://bugs.openjdk.java.net/browse/JDK-8167099 as part >> of this change (though arguably the bug has been there since >> new fields were added to StackTraceElement in 9). >> > It is related but it?s better to separate it from this issue. Do you have cycle to help write a test case? I can help fix JDK-8167099 later. > > Mandy > From mandy.chung at oracle.com Thu Nov 3 00:47:29 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 2 Nov 2016 17:47:29 -0700 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> Message-ID: <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> > On Nov 2, 2016, at 5:40 PM, Coleen Phillimore wrote: > > > Hi, > > I missed some of the iterations of this. To avoid making a large change larger, I think you should keep the original names of ToStackTraceElement and GetStackTraceElements. There is no change to the name of these JVM entry points. This patch changes the private native methods in the library. Is that what concerns you? Mandy > I don't think the new names are better and it leads to confusion of where the names are. It breaks the (hopefully ubiquitous) convention of naming Java functions similarly to the JVM functions that implement them. > > +JNIEXPORT void JNICALL Java_java_lang_StackTraceElement_fillFromStackFrameInfo > + (JNIEnv *env, jobject element, jobject stackframeinfo) { > + JVM_ToStackTraceElement(env, stackframeinfo, element); > +} > + > +JNIEXPORT void JNICALL Java_java_lang_StackTraceElement_fillStackTraceFromThrowable > + (JNIEnv *env, jobject dummy, jobjectArray elements, jobject throwable) > +{ > + JVM_GetStackTraceElements(env, throwable, elements); > +} > > I had already filed a compatibility request using these names to replace the old ones. If you want to change them again, you should file a new compatibility request to change the old jdk8 name (which I don't remember right now). > > http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.03/jdk/src/java.base/share/classes/java/lang/StackFrameInfo.java.udiff.html > > Maybe you can use the name "fill" for this rather than "of". If you don't use an IDE, finding the function name "of" is really challenging. > > Also, I think you should check this into the hs repository when done so that this bug can be unblocked sooner: https://bugs.openjdk.java.net/browse/JDK-8165550. > > Thanks, > Coleen > > > On 11/1/16 7:42 PM, Mandy Chung wrote: >> Hi Daniel, >> >> Here is the updated webrev incorporating your feedback: >> http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.03 >> >>> On Nov 1, 2016, at 7:04 AM, Daniel Fuchs wrote: >>> >>> : >>> 334 s += declaringClass; >>> >>> >>> but should line 334 instead be >>> >>> s = (s.isEmpty() ? declaringClass : s + "/" + declaringClass; >>> >>> Otherwise "/" will be missing after module at version.. >>> >> Good catch. I added a test for this. >> >>> Also should there be some asserts somewhere verifying that moduleVersion >>> is null or empty when moduleName is null or empty? At the moment the >>> constructor will blindly accept a version for an unnamed module, >>> and I am assuming this is wrong - am I right? >>> >> Good point. The constructor should throw IAE if module name/version and class loader name is an empty string. >> >>> toLoaderModuleClassName will call ClassLoader.getName(). >>> If a class loader overrides getName() then that might be a different >>> name than what StackTraceElement.getClassLoaderName() returns. >>> >>> Is that an issue? >>> >> I added a test to verify StackTraceElement that uses the ClassLoader's name field. I also added @apiNote in ClassLoader::getName. >> >>> Also I wonder whether you should be considering including a fix >>> for https://bugs.openjdk.java.net/browse/JDK-8167099 as part >>> of this change (though arguably the bug has been there since >>> new fields were added to StackTraceElement in 9). >>> >> It is related but it?s better to separate it from this issue. Do you have cycle to help write a test case? I can help fix JDK-8167099 later. >> >> Mandy >> > From coleen.phillimore at oracle.com Thu Nov 3 00:48:36 2016 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Wed, 2 Nov 2016 20:48:36 -0400 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> Message-ID: On 11/2/16 8:47 PM, Mandy Chung wrote: >> On Nov 2, 2016, at 5:40 PM, Coleen Phillimore wrote: >> >> >> Hi, >> >> I missed some of the iterations of this. To avoid making a large change larger, I think you should keep the original names of ToStackTraceElement and GetStackTraceElements. > > There is no change to the name of these JVM entry points. > > This patch changes the private native methods in the library. Is that what concerns you? Yes, the names of the library methods don't match the JVM names anymore. I think changing the names are unnecessary and makes it not obvious the mapping. Thanks, Coleen > > Mandy > >> I don't think the new names are better and it leads to confusion of where the names are. It breaks the (hopefully ubiquitous) convention of naming Java functions similarly to the JVM functions that implement them. >> >> +JNIEXPORT void JNICALL Java_java_lang_StackTraceElement_fillFromStackFrameInfo >> + (JNIEnv *env, jobject element, jobject stackframeinfo) { >> + JVM_ToStackTraceElement(env, stackframeinfo, element); >> +} >> + >> +JNIEXPORT void JNICALL Java_java_lang_StackTraceElement_fillStackTraceFromThrowable >> + (JNIEnv *env, jobject dummy, jobjectArray elements, jobject throwable) >> +{ >> + JVM_GetStackTraceElements(env, throwable, elements); >> +} >> >> I had already filed a compatibility request using these names to replace the old ones. If you want to change them again, you should file a new compatibility request to change the old jdk8 name (which I don't remember right now). >> >> http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.03/jdk/src/java.base/share/classes/java/lang/StackFrameInfo.java.udiff.html >> >> Maybe you can use the name "fill" for this rather than "of". If you don't use an IDE, finding the function name "of" is really challenging. >> >> Also, I think you should check this into the hs repository when done so that this bug can be unblocked sooner: https://bugs.openjdk.java.net/browse/JDK-8165550. >> >> Thanks, >> Coleen >> >> >> On 11/1/16 7:42 PM, Mandy Chung wrote: >>> Hi Daniel, >>> >>> Here is the updated webrev incorporating your feedback: >>> http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.03 >>> >>>> On Nov 1, 2016, at 7:04 AM, Daniel Fuchs wrote: >>>> >>>> : >>>> 334 s += declaringClass; >>>> >>>> >>>> but should line 334 instead be >>>> >>>> s = (s.isEmpty() ? declaringClass : s + "/" + declaringClass; >>>> >>>> Otherwise "/" will be missing after module at version.. >>>> >>> Good catch. I added a test for this. >>> >>>> Also should there be some asserts somewhere verifying that moduleVersion >>>> is null or empty when moduleName is null or empty? At the moment the >>>> constructor will blindly accept a version for an unnamed module, >>>> and I am assuming this is wrong - am I right? >>>> >>> Good point. The constructor should throw IAE if module name/version and class loader name is an empty string. >>> >>>> toLoaderModuleClassName will call ClassLoader.getName(). >>>> If a class loader overrides getName() then that might be a different >>>> name than what StackTraceElement.getClassLoaderName() returns. >>>> >>>> Is that an issue? >>>> >>> I added a test to verify StackTraceElement that uses the ClassLoader's name field. I also added @apiNote in ClassLoader::getName. >>> >>>> Also I wonder whether you should be considering including a fix >>>> for https://bugs.openjdk.java.net/browse/JDK-8167099 as part >>>> of this change (though arguably the bug has been there since >>>> new fields were added to StackTraceElement in 9). >>>> >>> It is related but it?s better to separate it from this issue. Do you have cycle to help write a test case? I can help fix JDK-8167099 later. >>> >>> Mandy >>> From mandy.chung at oracle.com Thu Nov 3 00:53:40 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 2 Nov 2016 17:53:40 -0700 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> Message-ID: > On Nov 2, 2016, at 5:48 PM, Coleen Phillimore wrote: > > > > On 11/2/16 8:47 PM, Mandy Chung wrote: >>> On Nov 2, 2016, at 5:40 PM, Coleen Phillimore wrote: >>> >>> >>> Hi, >>> >>> I missed some of the iterations of this. To avoid making a large change larger, I think you should keep the original names of ToStackTraceElement and GetStackTraceElements. >> >> There is no change to the name of these JVM entry points. >> >> This patch changes the private native methods in the library. Is that what concerns you? > > Yes, the names of the library methods don't match the JVM names anymore. I think changing the names are unnecessary and makes it not obvious the mapping. Is there such convention? I wasn?t aware of that. In addition, it would be impractical to synchronize the native methods to match JVM function names. Mandy From coleen.phillimore at oracle.com Thu Nov 3 01:10:39 2016 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Wed, 2 Nov 2016 21:10:39 -0400 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> Message-ID: On 11/2/16 8:53 PM, Mandy Chung wrote: >> On Nov 2, 2016, at 5:48 PM, Coleen Phillimore wrote: >> >> >> >> On 11/2/16 8:47 PM, Mandy Chung wrote: >>>> On Nov 2, 2016, at 5:40 PM, Coleen Phillimore wrote: >>>> >>>> >>>> Hi, >>>> >>>> I missed some of the iterations of this. To avoid making a large change larger, I think you should keep the original names of ToStackTraceElement and GetStackTraceElements. >>> There is no change to the name of these JVM entry points. >>> >>> This patch changes the private native methods in the library. Is that what concerns you? >> Yes, the names of the library methods don't match the JVM names anymore. I think changing the names are unnecessary and makes it not obvious the mapping. > Is there such convention? I wasn?t aware of that. In addition, it would be impractical to synchronize the native methods to match JVM function names. Yes, there's a convention of having very similar names. It's not 100% but having completely different names here is really not helpful for maintaining this code. This is from looking at the names in src/java.base/share/native/libjava. Some of the names have additional namespace-like strings like "Class" but basically they're close to the same. I think it's a good convention to have. In this case, I think the names toStackTraceElement and getStackTraceElements are pretty well descriptive and don't see the reason to change them. I'll read the thread. We've had a runtime offsite all week and haven't been able to read this thread, and thought you'd checked in what I'd reviewed before. thanks, Coleen > > Mandy From mandy.chung at oracle.com Thu Nov 3 02:00:06 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 2 Nov 2016 19:00:06 -0700 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> Message-ID: <6E3E1E6E-EB63-4DF2-856E-E27D3DE0DD58@oracle.com> > On Nov 2, 2016, at 6:10 PM, Coleen Phillimore wrote: > > > > On 11/2/16 8:53 PM, Mandy Chung wrote: >>> On Nov 2, 2016, at 5:48 PM, Coleen Phillimore wrote: >>> >>> Yes, the names of the library methods don't match the JVM names anymore. I think changing the names are unnecessary and makes it not obvious the mapping. >> Is there such convention? I wasn?t aware of that. In addition, it would be impractical to synchronize the native methods to match JVM function names. > > Yes, there's a convention of having very similar names. It's not 100% but having completely different names here is really not helpful for maintaining this code. This is from looking at the names in src/java.base/share/native/libjava. Some of the names have additional namespace-like strings like "Class" but basically they're close to the same. I think it's a good convention to have. In this case, I think the names toStackTraceElement and getStackTraceElements are pretty well descriptive and don't see the reason to change them. > This change was motivated due to two different places filling StackTraceElement (Throwable and StackFrameInfo). So I added the factory methods to allocate and fill in StackTraceElement object(s) in StackTraceElement and removed the package-private no-arg constructor added by JDK-8150778 [1]. Any change to create such factory methods will be in one single place. I like toStackTraceElement while getStackTraceElements does not quite represent what it does. ?fillStackTraceFromThrowable? is a more appropriate name. The original method Throwable::getStackTraceElements fills in the StackTraceElement array with this Throwable. I think that method can be renamed to ?fillStackTraceFromThrowable? which is clearer. I suggest to rename the JVM function: JVM_GetStackTraceElements => JVM_FillStackTraceElementsFromThrowable JVM_ToStackTraceElement => JVM_FillStackTraceElement Are you okay with this? > I'll read the thread. We've had a runtime offsite all week and haven't been able to read this thread, and thought you'd checked in what I'd reviewed before. > There is no change in the hotspot part. Mandy [1] https://bugs.openjdk.java.net/browse/JDK-8150778 From coleen.phillimore at oracle.com Thu Nov 3 15:52:55 2016 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Thu, 3 Nov 2016 11:52:55 -0400 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <6E3E1E6E-EB63-4DF2-856E-E27D3DE0DD58@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> <6E3E1E6E-EB63-4DF2-856E-E27D3DE0DD58@oracle.com> Message-ID: On 11/2/16 10:00 PM, Mandy Chung wrote: >> On Nov 2, 2016, at 6:10 PM, Coleen Phillimore wrote: >> >> >> >> On 11/2/16 8:53 PM, Mandy Chung wrote: >>>> On Nov 2, 2016, at 5:48 PM, Coleen Phillimore wrote: >>>> >>>> Yes, the names of the library methods don't match the JVM names anymore. I think changing the names are unnecessary and makes it not obvious the mapping. >>> Is there such convention? I wasn?t aware of that. In addition, it would be impractical to synchronize the native methods to match JVM function names. >> Yes, there's a convention of having very similar names. It's not 100% but having completely different names here is really not helpful for maintaining this code. This is from looking at the names in src/java.base/share/native/libjava. Some of the names have additional namespace-like strings like "Class" but basically they're close to the same. I think it's a good convention to have. In this case, I think the names toStackTraceElement and getStackTraceElements are pretty well descriptive and don't see the reason to change them. >> > > This change was motivated due to two different places filling StackTraceElement (Throwable and StackFrameInfo). So I added the factory methods to allocate and fill in StackTraceElement object(s) in StackTraceElement and removed the package-private no-arg constructor added by JDK-8150778 [1]. Any change to create such factory methods will be in one single place. This is cool, and nicer encapsulation. > > I like toStackTraceElement while getStackTraceElements does not quite represent what it does. ?fillStackTraceFromThrowable? is a more appropriate name. toStackTraceElement seems good in the context where this is called now. I agree that getStackTraceElements is less so. > > The original method Throwable::getStackTraceElements fills in the StackTraceElement array with this Throwable. I think that method can be renamed to ?fillStackTraceFromThrowable? which is clearer. > > I suggest to rename the JVM function: > JVM_GetStackTraceElements => JVM_FillStackTraceElementsFromThrowable > JVM_ToStackTraceElement => JVM_FillStackTraceElement > > Are you okay with this? My problem with fillStackTraceFromThrowable, besides it being a really long name, is that it's too close to fillInStackTrace which is a known API and should not be changed. Why not keep JVM_ToStackTraceElement, and change JVM_GetStackTraceElements if you really need to to JVM_ToStackTraceElementArray(). Or even JVM_ToStackTraceElements() which is very close to the function that it should conceptually be an overloaded instance. Throwable is part of the API, so putting it in the name is redundant and too long. Sorry for caring maybe overly about the names. That's how I find things. thanks, Coleen > >> I'll read the thread. We've had a runtime offsite all week and haven't been able to read this thread, and thought you'd checked in what I'd reviewed before. >> > There is no change in the hotspot part. > > Mandy > [1] https://bugs.openjdk.java.net/browse/JDK-8150778 > > From mandy.chung at oracle.com Thu Nov 3 16:29:10 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 3 Nov 2016 09:29:10 -0700 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> <6E3E1E6E-EB63-4DF2-856E-E27D3DE0DD58@oracle.com> Message-ID: <1C640D4B-6EE5-497B-96CB-E1E3CDF350EF@oracle.com> > On Nov 3, 2016, at 8:52 AM, Coleen Phillimore wrote: > > My problem with fillStackTraceFromThrowable, besides it being a really long name, is that it's too close to fillInStackTrace which is a known API and should not be changed. > > Why not keep JVM_ToStackTraceElement, and change JVM_GetStackTraceElements if you really need to to JVM_ToStackTraceElementArray(). > > Or even JVM_ToStackTraceElements() which is very close to the function that it should conceptually be an overloaded instance. > > Throwable is part of the API, so putting it in the name is redundant and too long. Sorry for caring maybe overly about the names. That's how I find things. These two methods are part of StackTraceElement APIs not Throwable, in case it?s not clear to you. What about SetStackTraceElement* or another alternative InitStackTraceElement* JNIEXPORT void JNICALL JVM_SetStackTraceElementArray(JNIEnv *env, jobjectArray elements, jobject throwable); JNIEXPORT void JNICALL JVM_SetStackTraceElement(JNIEnv* env, jobject element, jobject stackFrameInfo); BTW - do you know any known issue as JPRT hotspot test jobs are failing on jdk9/hs and jdk9/dev? Mandy From coleen.phillimore at oracle.com Thu Nov 3 17:12:36 2016 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Thu, 3 Nov 2016 13:12:36 -0400 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <1C640D4B-6EE5-497B-96CB-E1E3CDF350EF@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> <6E3E1E6E-EB63-4DF2-856E-E27D3DE0DD58@oracle.com> <1C640D4B-6EE5-497B-96CB-E1E3CDF350EF@oracle.com> Message-ID: <7ed5924c-5b1b-a21a-a24c-095f2cb7aa24@oracle.com> On 11/3/16 12:29 PM, Mandy Chung wrote: >> On Nov 3, 2016, at 8:52 AM, Coleen Phillimore wrote: >> >> My problem with fillStackTraceFromThrowable, besides it being a really long name, is that it's too close to fillInStackTrace which is a known API and should not be changed. >> >> Why not keep JVM_ToStackTraceElement, and change JVM_GetStackTraceElements if you really need to to JVM_ToStackTraceElementArray(). >> >> Or even JVM_ToStackTraceElements() which is very close to the function that it should conceptually be an overloaded instance. >> >> Throwable is part of the API, so putting it in the name is redundant and too long. Sorry for caring maybe overly about the names. That's how I find things. > These two methods are part of StackTraceElement APIs not Throwable, in case it?s not clear to you. Yes, I know but the parameter to the one is Throwable, which is what I meant. > > What about SetStackTraceElement* or another alternative InitStackTraceElement* > > JNIEXPORT void JNICALL > JVM_SetStackTraceElementArray(JNIEnv *env, jobjectArray elements, jobject throwable); > > JNIEXPORT void JNICALL > JVM_SetStackTraceElement(JNIEnv* env, jobject element, jobject stackFrameInfo); Ok. These names are better. And they'd match the java method name, right? > BTW - do you know any known issue as JPRT hotspot test jobs are failing on jdk9/hs and jdk9/dev? No, they shouldn't fail unless the machines are down, which they could be. All hotspot changes are tested in hs with each repository push. Can you rebase this for hs repository (in case that request got lost in the discussion)? Thanks so much, Coleen > Mandy From mandy.chung at oracle.com Thu Nov 3 17:56:07 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 3 Nov 2016 10:56:07 -0700 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <7ed5924c-5b1b-a21a-a24c-095f2cb7aa24@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> <6E3E1E6E-EB63-4DF2-856E-E27D3DE0DD58@oracle.com> <1C640D4B-6EE5-497B-96CB-E1E3CDF350EF@oracle.com> <7ed5924c-5b1b-a21a-a24c-095f2cb7aa24@oracle.com> Message-ID: <691D53ED-BB2D-4D04-B49C-E40B7A152EE9@oracle.com> > On Nov 3, 2016, at 10:12 AM, Coleen Phillimore wrote: >> What about SetStackTraceElement* or another alternative InitStackTraceElement* >> : > > Ok. These names are better. The methods are now named: JVM_InitStackTraceElement* StackTraceElement::initStackTraceElement* Can you please review: http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.05/ > > No, they shouldn't fail unless the machines are down, which they could be. All hotspot changes are tested in hs with each repository push. Can you rebase this for hs repository (in case that request got lost in the discussion)? I rebased to jdk9/dev as you have agreed earlier. When will the next jdk9/hs be integrated to jdk9/dev? I could push it to jdk9/hs but I need to know which build this would go into jdk9 to evaluate the implication to jake. This patch is in jake forest and we want this to go into jdk9 first before jake goes to jdk9. Mandy From coleen.phillimore at oracle.com Thu Nov 3 22:26:36 2016 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Thu, 3 Nov 2016 18:26:36 -0400 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <691D53ED-BB2D-4D04-B49C-E40B7A152EE9@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <237186f1-13b4-5a94-099d-cf1f07e7b120@oracle.com> <82595580-6100-4F6C-905C-3061D3AE43C6@oracle.com> <30A2DED2-D27E-4ECF-A09E-8775FAA54F63@oracle.com> <6E3E1E6E-EB63-4DF2-856E-E27D3DE0DD58@oracle.com> <1C640D4B-6EE5-497B-96CB-E1E3CDF350EF@oracle.com> <7ed5924c-5b1b-a21a-a24c-095f2cb7aa24@oracle.com> <691D53ED-BB2D-4D04-B49C-E40B7A152EE9@oracle.com> Message-ID: <7c063977-490a-576a-283f-aa504af6ee8f@oracle.com> On 11/3/16 1:56 PM, Mandy Chung wrote: >> On Nov 3, 2016, at 10:12 AM, Coleen Phillimore wrote: >>> What about SetStackTraceElement* or another alternative InitStackTraceElement* >>> : >> Ok. These names are better. > The methods are now named: > JVM_InitStackTraceElement* > StackTraceElement::initStackTraceElement* > > Can you please review: > http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.05/ Sorry for the delay, yes, this looks great. Thanks for making this change. > >> No, they shouldn't fail unless the machines are down, which they could be. All hotspot changes are tested in hs with each repository push. Can you rebase this for hs repository (in case that request got lost in the discussion)? > I rebased to jdk9/dev as you have agreed earlier. > > When will the next jdk9/hs be integrated to jdk9/dev? I could push it to jdk9/hs but I need to know which build this would go into jdk9 to evaluate the implication to jake. This patch is in jake forest and we want this to go into jdk9 first before jake goes to jdk9. I don't know, I'll agree with what Karen and Lois say in your mail to them. Thanks, Coleen > Mandy From dmitry.samersoff at oracle.com Fri Nov 4 12:31:19 2016 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Fri, 4 Nov 2016 15:31:19 +0300 Subject: RFR (XS): 8160024 jdb returns invalid argument count if first parameter to Arrays.asList is null In-Reply-To: <64c20710-5257-e4d6-c613-06ff0c66cae4@oracle.com> References: <64c20710-5257-e4d6-c613-06ff0c66cae4@oracle.com> Message-ID: <23c267c0-6f36-69a6-9c46-bb97c04e267e@oracle.com> Serguei, Looks good to me. PS: It might be better to refactor this code sometimes to don't check the same things multiple time. -Dmitry On 2016-11-01 15:58, serguei.spitsyn at oracle.com wrote: > Please, review this small fix. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8160024 > > JDK webrev: > http://cr.openjdk.java.net/~sspitsyn/webrevs/2016/hotspot/8160024-JDI-eval.jdk1/ > > > The bug is confidential, so the RFR is sent to the confidential mailing > lists. > > > Summary: > > The JDI incorrectly evaluates the expression: Arrays.asList(null, "a"). > The root cause is a special case in the method MethodImpl.handleVarArgs() > that has been corrected in order to fix the problem. > > New unit test is included: jdk/test/com/sun/jdi/EvalArraysAsList.sh > > Testing: > Ran new unit test, Jtreg com/sun/jdi and nsk.jdi.testlist. > All tests are passed. > > Thanks, > Serguei > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From serguei.spitsyn at oracle.com Fri Nov 4 12:35:00 2016 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Fri, 4 Nov 2016 08:35:00 -0400 Subject: RFR (XS): 8160024 jdb returns invalid argument count if first parameter to Arrays.asList is null In-Reply-To: <23c267c0-6f36-69a6-9c46-bb97c04e267e@oracle.com> References: <64c20710-5257-e4d6-c613-06ff0c66cae4@oracle.com> <23c267c0-6f36-69a6-9c46-bb97c04e267e@oracle.com> Message-ID: Dmitry, Thank you for the review! Serguei On 11/4/16 08:31, Dmitry Samersoff wrote: > Serguei, > > Looks good to me. > > PS: It might be better to refactor this code sometimes to don't check > the same things multiple time. > > -Dmitry > > On 2016-11-01 15:58, serguei.spitsyn at oracle.com wrote: >> Please, review this small fix. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8160024 >> >> JDK webrev: >> http://cr.openjdk.java.net/~sspitsyn/webrevs/2016/hotspot/8160024-JDI-eval.jdk1/ >> >> >> The bug is confidential, so the RFR is sent to the confidential mailing >> lists. >> >> >> Summary: >> >> The JDI incorrectly evaluates the expression: Arrays.asList(null, "a"). >> The root cause is a special case in the method MethodImpl.handleVarArgs() >> that has been corrected in order to fix the problem. >> >> New unit test is included: jdk/test/com/sun/jdi/EvalArraysAsList.sh >> >> Testing: >> Ran new unit test, Jtreg com/sun/jdi and nsk.jdi.testlist. >> All tests are passed. >> >> Thanks, >> Serguei >> > From marcus.larsson at oracle.com Fri Nov 4 15:16:34 2016 From: marcus.larsson at oracle.com (Marcus Larsson) Date: Fri, 4 Nov 2016 16:16:34 +0100 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part In-Reply-To: References: Message-ID: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> Hi, Thanks for fixing this. On 2016-11-01 18:30, Kirill Zhaldybin wrote: > Dear all, > > Could you please review this fix for 8169003? > > I changed parsing of time string so now it is not depend on LC_NUMERIC > locale so the test does not fail if locale where "floating point" is > actually a comma is set. > > WebRev: > http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ ISO8601 says the decimal point can be either '.' or ',' so the test should accept either. You could let sscanf read out the decimal point as a character and just verify that it is one of the two. In the UTC test you changed ASSERT_GT to ASSERT_EQ, which means that we won't accept "Z" suffixed strings. Please revert that. Thanks, Marcus > CR: https://bugs.openjdk.java.net/browse/JDK-8169003 > > Thank you. > > Regards, Kirill From david.holmes at oracle.com Sat Nov 5 17:48:29 2016 From: david.holmes at oracle.com (David Holmes) Date: Sun, 6 Nov 2016 03:48:29 +1000 Subject: RFR(M): 8154736: enhancement of cmpxchg and copy_to_survivor for ppc64 In-Reply-To: References: <201604221228.u3MCSXCL020021@d19av07.sagamino.japan.ibm.com> <1475236951.6301.72.camel@oracle.com> <6ee4f1c6-f638-c5b9-7475-8fb6aeabf20b@oracle.com> <14c2eff4-4f90-caa0-17a7-835e6f1f1167@oracle.com> <1cbb094f-b29b-c6b3-1e50-bed21b140fcb@oracle.com> Message-ID: Hi Andrew, Sorry for the delayed response. I think I should start a new thread to discuss Atomic::r-m-w memory semantics. David On 1/11/2016 7:44 PM, Andrew Haley wrote: > On 31/10/16 21:30, David Holmes wrote: >> >> >> On 31/10/2016 7:32 PM, Andrew Haley wrote: >>> On 30/10/16 21:26, David Holmes wrote: >>>> On 31/10/2016 4:36 AM, Andrew Haley wrote: >>>>> >>>>> And, while we're on the subject, is memory_order_conservative actually >>>>> defined anywhere? >>>> >>>> No. It was chosen to represent the current status quo that the Atomic:: >>>> ops should all be (by default) full bi-directional fences. >>> >>> Does that mean that a CAS is actually stronger than a load acquire >>> followed by a store release? And that a CAS is a release fence even >>> when it fails and no store happens? >> >> Yes. Yes. >> >> // All of the atomic operations that imply a read-modify-write >> // action guarantee a two-way memory barrier across that >> // operation. Historically these semantics reflect the strength >> // of atomic operations that are provided on SPARC/X86. We assume >> // that strength is necessary unless we can prove that a weaker >> // form is sufficiently safe. > > Mmmm, but that doesn't say anything about a CAS that fails. But fair > enough, I accept your interpretation. > >> But there is some contention as to whether the actual implementations >> obey this completely. > > Linux/AArch64 uses GCC's __sync_val_compare_and_swap, which is specified > as a > > "full barrier". That is, no memory operand is moved across the > operation, either forward or backward. Further, instructions are > issued as necessary to prevent the processor from speculating loads > across the operation and from queuing stores after the operation. > > ... which reads the same as the language you quoted above, but looking > at the assembly code I'm sure that it's really no stronger than a seq > cst load followed by a seq cst store. > > I guess maybe I could give up fighting this and implement all AArch64 > CAS sequences as > > CAS(seq_cst); full fence > > or, even more extremely, > > full fence; CAS(relaxed); full fence > > but it all seems unreasonably heavyweight. > >>> And that a conservative load is a *store* barrier? >> >> Not sure what you mean. Atomic::load is not a r-m-w action so not >> expected to be a two-way memory barrier. > > OK. > > Thanks, > > Andrew. > From thomas.stuefe at gmail.com Wed Nov 9 15:00:20 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 9 Nov 2016 16:00:20 +0100 Subject: Question about VirtualSpace and MPSS support Message-ID: Hi all, please help me understand some details about the VirtualSpace class. Background: I am experimenting with putting humongous metaspace chunks at the end of the VirtualSpace (part of https://bugs.openjdk.java.net/browse/JDK-8166690). So I would like to have a VirtualSpace which allows me to allocate from both bottom and top. Therefore I am checking how difficult it would be to extend the VirtualSpace class to allow me to grow from both sides. Now I try to understand the three-region-logic implemented to support multiple page sizes: VirtualSpace has three regions with three different granularities. I think the intent is to support large pages for address ranges which are not necessarily large page aligned, so this leaves small-paged regions at top and/or bottom. Looking at VirtualSpace::expand_by() I think the idea is that there is always one continuous committed area spanning those three regions, right? (instead of, say, each of the regions being committed independently). So, just one high water mark, which is either in the low, middle or upper region. If it is in the middle region, low region should be completely committed, if it is in the upper region, low and middle regions should be completely committed? I think it must be this way, because from the outside I only should have one continuous committed address range, otherwise it would be difficult to use this VirtualSpace. But then, I have a function like VirtualSpace::actual_committed_size() - in addition to VirtualSpace::committed_size() - which counts the committed area for all three regions separately. Is there ever a case where comitted size and actual comitted size are different? Can it be that lower regions are partly comitted even if higher regions are already used? Thank you! Kind Regards, Thomas From goetz.lindenmaier at sap.com Thu Nov 10 15:58:55 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Thu, 10 Nov 2016 15:58:55 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. Message-ID: Hi, Please review this change. I please need a sponsor: http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.01/ In the Linux NPTL pthread implementation the guard size mechanism is not implemented properly. The posix standard requires to add the size of the guard pages to the stack size, instead Linux takes the space out of 'stacksize'. The Posix standard http://pubs.opengroup.org/onlinepubs/9699919799/ says "the implementation allocates extra memory at the overflow end of the stack". The linux man page https://linux.die.net/man/3/pthread_attr_setguardsize says "As at glibc 2.8, the NPTL threading implementation includes the guard area within the stack size allocation, rather than allocating extra space at the end of the stack, as POSIX.1 requires". I encounter this problem in runtime/Thread/TooSmallStackSize.java on ppc with 64K pages. _compiler_thread_min_stack_allowed is 128K on ppc, and ppc specifies two OS guard pages. The VM crashes in pthread creation because there is no usable space in the thread stack after allocating the guard pages. But TooSmallStackSize.java requires that the VM comes up with the stack size mentioned in the error message. This fix adapts the requested stack size on Linux by the size of the guard pages to mimick proper behaviour, see change to os_linux.cpp. The change also streamlines usage of stack_guard_page on linuxppc, linuxppcle, aixppc and linuxs390. To reproduce the error on linux_x86_64, apply below patch and call the VM with -XX:CompilerThreadStackSize=64. I'm still exploring why I had to choose such big compiler stacks on ppc to get -version passing, but I wanted to send the RFR now as people obviously looked at the bug I opened (Thanks David!). Best regards, Goetz. diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 12:37:28 2016 +0100 +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 16:52:17 2016 +0100 @@ -701,7 +701,7 @@ size_t os::Linux::default_guard_size(os::ThreadType thr_type) { // Creating guard page is very expensive. Java thread has HotSpot // guard page, only enable glibc guard page for non-Java threads. - return (thr_type == java_thread ? 0 : page_size()); + return (thr_type == java_thread ? 0 : 64*K); } // Java thread: From thomas.stuefe at gmail.com Thu Nov 10 16:49:12 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 10 Nov 2016 17:49:12 +0100 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: Message-ID: Hi Goetz, have not looked at everything, but code may be actually incur unnecessary costs for AIX if native stack page size was 4K but we decided to lie about it and set os::Aix::_page_size - and hence os::vm_page_size() - to 64K to get 64k based heap. See os_aix.cpp, multipage support initialization in os::init(). My proposal would be to just set guard page size to 4k and in case we have real native 64k pages on the stack, the OS will round the size up to the page size. Kind Regards, Thomas On Thu, Nov 10, 2016 at 4:58 PM, Lindenmaier, Goetz < goetz.lindenmaier at sap.com> wrote: > Hi, > > Please review this change. I please need a sponsor: > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.01/ > > In the Linux NPTL pthread implementation the guard size mechanism is not > implemented properly. The posix standard requires to add the size of the > guard pages to the stack size, instead Linux takes the space out of > 'stacksize'. > > The Posix standard http://pubs.opengroup.org/onlinepubs/9699919799/ says > "the implementation allocates extra memory at the overflow end of the > stack". The linux man page https://linux.die.net/man/3/ > pthread_attr_setguardsize says "As at glibc 2.8, the NPTL threading > implementation includes the guard area within the stack size allocation, > rather than allocating extra space at the end of the stack, as POSIX.1 > requires". > > I encounter this problem in runtime/Thread/TooSmallStackSize.java on ppc > with 64K pages. _compiler_thread_min_stack_allowed is 128K on ppc, and > ppc specifies two OS guard pages. The VM crashes in pthread creation > because there is no usable space in the thread stack after allocating the > guard pages. > But TooSmallStackSize.java requires that the VM comes up with the stack > size mentioned in the error message. > > This fix adapts the requested stack size on Linux by the size of the guard > pages to mimick proper behaviour, see change to os_linux.cpp. > > The change also streamlines usage of stack_guard_page on linuxppc, > linuxppcle, aixppc and linuxs390. > > To reproduce the error on linux_x86_64, apply below patch and call the VM > with -XX:CompilerThreadStackSize=64. > > I'm still exploring why I had to choose such big compiler stacks on ppc to > get -version passing, but I wanted to send the RFR now as people obviously > looked at the bug I opened (Thanks David!). > > Best regards, > Goetz. > > > > diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp > --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 12:37:28 2016 > +0100 > +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 16:52:17 2016 > +0100 > @@ -701,7 +701,7 @@ > size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > // Creating guard page is very expensive. Java thread has HotSpot > // guard page, only enable glibc guard page for non-Java threads. > - return (thr_type == java_thread ? 0 : page_size()); > + return (thr_type == java_thread ? 0 : 64*K); > } > > // Java thread: > From calvin.cheung at oracle.com Thu Nov 10 19:33:24 2016 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 10 Nov 2016 11:33:24 -0800 Subject: RFR(xs): 8168796: relax vm options checking during CDS dump time Message-ID: <5824CB84.8040309@oracle.com> bug: https://bugs.openjdk.java.net/browse/JDK-8168796 webrev: http://cr.openjdk.java.net/~ccheung/8168796/webrev.00/ Before this change, if any one of the following vm options ("-module-path", "-add-modules", "--upgrade-module-path") is specified during CDS dump time, the vm exits with an error as follows: bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump Error occurred during initialization of VM Cannot use the following option when dumping the shared archive: --upgrade-module-path This change will print a warning message and continues as follows: bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump Warning: the --upgrade-module-path option is ignored when dumping the shared archive Allocated shared space: 50577408 bytes at 0x0000000800000000 Loading classes to share ... Test case is in the hotspot closed repo. Testing: JPRT all tests under hotspot/test/runtime thanks, Calvin From harold.seigel at oracle.com Thu Nov 10 19:42:37 2016 From: harold.seigel at oracle.com (harold seigel) Date: Thu, 10 Nov 2016 14:42:37 -0500 Subject: RFR(xs): 8168796: relax vm options checking during CDS dump time In-Reply-To: <5824CB84.8040309@oracle.com> References: <5824CB84.8040309@oracle.com> Message-ID: Hi Calvin, The changes look good. Could you add a comment to the declaration of warning_idx explaining what it means to be above and below the that index? Also, instead of calling tty->print_cr(), can you call warning(), instead? Thanks, Harold On 11/10/2016 2:33 PM, Calvin Cheung wrote: > bug: https://bugs.openjdk.java.net/browse/JDK-8168796 > > webrev: http://cr.openjdk.java.net/~ccheung/8168796/webrev.00/ > > Before this change, if any one of the following vm options > ("-module-path", "-add-modules", "--upgrade-module-path") is specified > during CDS dump time, the vm exits with an error as follows: > > bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump > Error occurred during initialization of VM > Cannot use the following option when dumping the shared archive: > --upgrade-module-path > > This change will print a warning message and continues as follows: > > bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump > Warning: the --upgrade-module-path option is ignored when dumping the > shared archive > Allocated shared space: 50577408 bytes at 0x0000000800000000 > Loading classes to share ... > > Test case is in the hotspot closed repo. > > Testing: > JPRT > all tests under hotspot/test/runtime > > thanks, > Calvin > > > From jiangli.zhou at oracle.com Thu Nov 10 19:52:07 2016 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Thu, 10 Nov 2016 11:52:07 -0800 Subject: RFR(xs): 8168796: relax vm options checking during CDS dump time In-Reply-To: References: <5824CB84.8040309@oracle.com> Message-ID: <5BBF7048-407D-4890-9EE9-AC86EAF8FF6E@oracle.com> +1 Thanks, Jiangli > On Nov 10, 2016, at 11:42 AM, harold seigel wrote: > > Hi Calvin, > > The changes look good. Could you add a comment to the declaration of warning_idx explaining what it means to be above and below the that index? > > Also, instead of calling tty->print_cr(), can you call warning(), instead? > > Thanks, Harold > > > On 11/10/2016 2:33 PM, Calvin Cheung wrote: >> bug: https://bugs.openjdk.java.net/browse/JDK-8168796 >> >> webrev: http://cr.openjdk.java.net/~ccheung/8168796/webrev.00/ >> >> Before this change, if any one of the following vm options ("-module-path", "-add-modules", "--upgrade-module-path") is specified during CDS dump time, the vm exits with an error as follows: >> >> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >> Error occurred during initialization of VM >> Cannot use the following option when dumping the shared archive: --upgrade-module-path >> >> This change will print a warning message and continues as follows: >> >> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >> Warning: the --upgrade-module-path option is ignored when dumping the shared archive >> Allocated shared space: 50577408 bytes at 0x0000000800000000 >> Loading classes to share ... >> >> Test case is in the hotspot closed repo. >> >> Testing: >> JPRT >> all tests under hotspot/test/runtime >> >> thanks, >> Calvin >> >> >> > From david.holmes at oracle.com Thu Nov 10 21:02:27 2016 From: david.holmes at oracle.com (David Holmes) Date: Fri, 11 Nov 2016 07:02:27 +1000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: Message-ID: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> Hi Goetz, As per the bug report, this issue was already known (6675312) and we chose not to try and address it due to no reported issues at the time. While I see that you have encountered an issue (is it real or fabricated?) I think this change is too intrusive to be applied at this stage of the JDK 9 release cycle, as it will change the stack requirements of every application running on Linux. Thanks, David On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > Hi, > > > > Please review this change. I please need a sponsor: > > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.01/ > > > > In the Linux NPTL pthread implementation the guard size mechanism is not > implemented properly. The posix standard requires to add the size of the > guard pages to the stack size, instead Linux takes the space out of > 'stacksize'. > > The Posix standard http://pubs.opengroup.org/onlinepubs/9699919799/ > says "the implementation allocates extra memory at the overflow end of > the stack". The linux man page > https://linux.die.net/man/3/pthread_attr_setguardsize says "As at glibc > 2.8, the NPTL threading implementation includes the guard area within > the stack size allocation, rather than allocating extra space at the end > of the stack, as POSIX.1 requires". > > I encounter this problem in runtime/Thread/TooSmallStackSize.java on ppc > with 64K pages. _compiler_thread_min_stack_allowed is 128K on ppc, and > ppc specifies two OS guard pages. The VM crashes in pthread creation > because there is no usable space in the thread stack after allocating > the guard pages. > > But TooSmallStackSize.java requires that the VM comes up with the stack > size mentioned in the error message. > > This fix adapts the requested stack size on Linux by the size of the > guard pages to mimick proper behaviour, see change to os_linux.cpp. > > > > The change also streamlines usage of stack_guard_page on linuxppc, > linuxppcle, aixppc and linuxs390. > > > > To reproduce the error on linux_x86_64, apply below patch and call the > VM with -XX:CompilerThreadStackSize=64. > > > > I?m still exploring why I had to choose such big compiler stacks on ppc > to get ?version passing, but I wanted to send the RFR now as people > obviously looked at the bug I opened (Thanks David!). > > > > Best regards, > > Goetz. > > > > > > > > diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp > > --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 12:37:28 2016 > +0100 > > +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 16:52:17 2016 > +0100 > > @@ -701,7 +701,7 @@ > > size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > > // Creating guard page is very expensive. Java thread has HotSpot > > // guard page, only enable glibc guard page for non-Java threads. > > - return (thr_type == java_thread ? 0 : page_size()); > > + return (thr_type == java_thread ? 0 : 64*K); > > } > > > > // Java thread: > From rachel.protacio at oracle.com Thu Nov 10 21:38:10 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Thu, 10 Nov 2016 16:38:10 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message Message-ID: Hi, Please review this Jigsaw enhancement which adds the new format of class loader names to VM ClassCastException messages. Includes an updated/expanded jtreg test. Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ Passes JPRT and jck vm and lang tests. Thanks! Rachel From goetz.lindenmaier at sap.com Thu Nov 10 22:00:49 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Thu, 10 Nov 2016 22:00:49 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> Message-ID: Hi David, This issue is different to 6675312, see also my comment in the bug. It appears running jtreg test runtime/Thread/TooSmallStackSize.java, with my patch below you can reproduce it on linuxx86_64. You can not do that with 6675312. Also, I would assume there are systems out there on x86 that uses 64-K pages, did you run the tests on these? I would assume you get hard crashes with stack overflows in the first C2 compilation if there is only 64K usable CompilerThreadStack. My fix does not affect Java threads, which are the largest amount of threads used by the VM. It affects only the non-Java threads. It adds one page to these threads. The page does not require memory, as it's protected. The stack will only require more space if the thread ran into a stack overflow before the fix as else the pages are not mapped. This are stack overflows that cause hard crashes, at least on ppc the VM does not properly catch these stack overflows, so any setup working now will not run into the additional space. Altogether there should be no effect on running systems besides requiring one more entry in the page table per non-Java thread. The problem is caused by a rather recent change (8140520: segfault on solaris-amd64 with "-XX:VMThreadStackSize=1" option) which was pushed after feature-close. As this was a rather recent change, it must be possible to fix this follow up issue. What else is this period in the project good for if not fixing issues? So I really think this issue should be fixed. Best regards, Goetz. > -----Original Message----- > From: David Holmes [mailto:david.holmes at oracle.com] > Sent: Thursday, November 10, 2016 10:02 PM > To: Lindenmaier, Goetz ; hotspot-runtime- > dev at openjdk.java.net > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > Hi Goetz, > > As per the bug report, this issue was already known (6675312) and we > chose not to try and address it due to no reported issues at the time. > While I see that you have encountered an issue (is it real or > fabricated?) I think this change is too intrusive to be applied at this > stage of the JDK 9 release cycle, as it will change the stack > requirements of every application running on Linux. > > Thanks, > David > > On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > > Hi, > > > > > > > > Please review this change. I please need a sponsor: > > > > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.01/ > > > > > > > > In the Linux NPTL pthread implementation the guard size mechanism is not > > implemented properly. The posix standard requires to add the size of the > > guard pages to the stack size, instead Linux takes the space out of > > 'stacksize'. > > > > The Posix standard http://pubs.opengroup.org/onlinepubs/9699919799/ > > says "the implementation allocates extra memory at the overflow end of > > the stack". The linux man page > > https://linux.die.net/man/3/pthread_attr_setguardsize says "As at glibc > > 2.8, the NPTL threading implementation includes the guard area within > > the stack size allocation, rather than allocating extra space at the end > > of the stack, as POSIX.1 requires". > > > > I encounter this problem in runtime/Thread/TooSmallStackSize.java on ppc > > with 64K pages. _compiler_thread_min_stack_allowed is 128K on ppc, and > > ppc specifies two OS guard pages. The VM crashes in pthread creation > > because there is no usable space in the thread stack after allocating > > the guard pages. > > > > But TooSmallStackSize.java requires that the VM comes up with the stack > > size mentioned in the error message. > > > > This fix adapts the requested stack size on Linux by the size of the > > guard pages to mimick proper behaviour, see change to os_linux.cpp. > > > > > > > > The change also streamlines usage of stack_guard_page on linuxppc, > > linuxppcle, aixppc and linuxs390. > > > > > > > > To reproduce the error on linux_x86_64, apply below patch and call the > > VM with -XX:CompilerThreadStackSize=64. > > > > > > > > I'm still exploring why I had to choose such big compiler stacks on ppc > > to get -version passing, but I wanted to send the RFR now as people > > obviously looked at the bug I opened (Thanks David!). > > > > > > > > Best regards, > > > > Goetz. > > > > > > > > > > > > > > > > diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp > > > > --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 12:37:28 > 2016 > > +0100 > > > > +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 16:52:17 > 2016 > > +0100 > > > > @@ -701,7 +701,7 @@ > > > > size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > > > > // Creating guard page is very expensive. Java thread has HotSpot > > > > // guard page, only enable glibc guard page for non-Java threads. > > > > - return (thr_type == java_thread ? 0 : page_size()); > > > > + return (thr_type == java_thread ? 0 : 64*K); > > > > } > > > > > > > > // Java thread: > > From david.holmes at oracle.com Thu Nov 10 22:02:49 2016 From: david.holmes at oracle.com (David Holmes) Date: Fri, 11 Nov 2016 08:02:49 +1000 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: Message-ID: Hi Rachel, Just wondering, in the most complex case, exactly what will this new message look like? We seem to be including module names as well as classloader names, and of course class names. Arguably we should only add the additional info when they differ between the types ie if both in same classloader and same module then only report class name; if same loader but different module report module:class; if different loaders then report everything loader:module:class. Also not sure module version is needed at all in this message. Thanks, David On 11/11/2016 7:38 AM, Rachel Protacio wrote: > Hi, > > Please review this Jigsaw enhancement which adds the new format of class > loader names to VM ClassCastException messages. Includes an > updated/expanded jtreg test. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 > Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ > > Passes JPRT and jck vm and lang tests. > > Thanks! > Rachel From david.holmes at oracle.com Thu Nov 10 22:20:21 2016 From: david.holmes at oracle.com (David Holmes) Date: Fri, 11 Nov 2016 08:20:21 +1000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> Message-ID: <96bd3531-130e-1fb8-923c-1bb5612588d3@oracle.com> Hi Goetz, On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > Hi David, > > This issue is different to 6675312, see also my comment in the bug. > > It appears running jtreg test runtime/Thread/TooSmallStackSize.java, > with my patch below you can reproduce it on linuxx86_64. You can not > do that with 6675312. Also, I would assume there are systems out there > on x86 that uses 64-K pages, did you run the tests on these? I would > assume you get hard crashes with stack overflows in the first C2 > compilation if there is only 64K usable CompilerThreadStack. > > My fix does not affect Java threads, which are the largest amount > of threads used by the VM. It affects only the non-Java threads. > It adds one page to these threads. The page does not require memory, > as it's protected. The stack will only require more space if the thread > ran into a stack overflow before the fix as else the pages are not mapped. > This are stack overflows that cause hard crashes, at least on ppc the VM > does not properly catch these stack overflows, so any setup working now > will not run into the additional space. Altogether there should be no > effect on running systems besides requiring one more entry in the > page table per non-Java thread. > > The problem is caused by a rather recent change (8140520: segfault on solaris-amd64 > with "-XX:VMThreadStackSize=1" option) which was pushed after > feature-close. As this was a rather recent change, it must be possible to > fix this follow up issue. What else is this period in the project good > for if not fixing issues? > > So I really think this issue should be fixed. I only gave it a cursory look initially and had not noticed that the effective change was only being applied to PPC and S390 code. If this truly relates to the 8140520 fix and is not specifically trying to address the NPTL bug then we should be able to do something. But at the moment I'm confused by exactly what 8140520 changed that broke this, or why a change is only needed on PPC and only for non-Java threads. The special casing here actually makes things worse at the moment - but I will examine this in more detail. Thanks, David > Best regards, > Goetz. > > >> -----Original Message----- >> From: David Holmes [mailto:david.holmes at oracle.com] >> Sent: Thursday, November 10, 2016 10:02 PM >> To: Lindenmaier, Goetz ; hotspot-runtime- >> dev at openjdk.java.net >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >> >> Hi Goetz, >> >> As per the bug report, this issue was already known (6675312) and we >> chose not to try and address it due to no reported issues at the time. >> While I see that you have encountered an issue (is it real or >> fabricated?) I think this change is too intrusive to be applied at this >> stage of the JDK 9 release cycle, as it will change the stack >> requirements of every application running on Linux. >> >> Thanks, >> David >> >> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: >>> Hi, >>> >>> >>> >>> Please review this change. I please need a sponsor: >>> >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.01/ >>> >>> >>> >>> In the Linux NPTL pthread implementation the guard size mechanism is not >>> implemented properly. The posix standard requires to add the size of the >>> guard pages to the stack size, instead Linux takes the space out of >>> 'stacksize'. >>> >>> The Posix standard http://pubs.opengroup.org/onlinepubs/9699919799/ >>> says "the implementation allocates extra memory at the overflow end of >>> the stack". The linux man page >>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As at glibc >>> 2.8, the NPTL threading implementation includes the guard area within >>> the stack size allocation, rather than allocating extra space at the end >>> of the stack, as POSIX.1 requires". >>> >>> I encounter this problem in runtime/Thread/TooSmallStackSize.java on ppc >>> with 64K pages. _compiler_thread_min_stack_allowed is 128K on ppc, and >>> ppc specifies two OS guard pages. The VM crashes in pthread creation >>> because there is no usable space in the thread stack after allocating >>> the guard pages. >>> >>> But TooSmallStackSize.java requires that the VM comes up with the stack >>> size mentioned in the error message. >>> >>> This fix adapts the requested stack size on Linux by the size of the >>> guard pages to mimick proper behaviour, see change to os_linux.cpp. >>> >>> >>> >>> The change also streamlines usage of stack_guard_page on linuxppc, >>> linuxppcle, aixppc and linuxs390. >>> >>> >>> >>> To reproduce the error on linux_x86_64, apply below patch and call the >>> VM with -XX:CompilerThreadStackSize=64. >>> >>> >>> >>> I'm still exploring why I had to choose such big compiler stacks on ppc >>> to get -version passing, but I wanted to send the RFR now as people >>> obviously looked at the bug I opened (Thanks David!). >>> >>> >>> >>> Best regards, >>> >>> Goetz. >>> >>> >>> >>> >>> >>> >>> >>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>> >>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 12:37:28 >> 2016 >>> +0100 >>> >>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 16:52:17 >> 2016 >>> +0100 >>> >>> @@ -701,7 +701,7 @@ >>> >>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>> >>> // Creating guard page is very expensive. Java thread has HotSpot >>> >>> // guard page, only enable glibc guard page for non-Java threads. >>> >>> - return (thr_type == java_thread ? 0 : page_size()); >>> >>> + return (thr_type == java_thread ? 0 : 64*K); >>> >>> } >>> >>> >>> >>> // Java thread: >>> From rachel.protacio at oracle.com Thu Nov 10 22:31:32 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Thu, 10 Nov 2016 17:31:32 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: Message-ID: Ah yes, my apologies - I should have provided more context. This is the VM corollary of Mandy's change http://hg.openjdk.java.net/jdk9/dev/hotspot/rev/48fce55afe71 which implements the new classloader name scheme and uses it in Stack Traces. In the fullest instance, it could look like MyClassLoader/module1 at 9.0/package1.class1 The class loader is not listed if it is built-in, the module is not listed if it is unnamed, the version is not listed if it is a "non-jdk" module. I appreciate your point about shortening this specific CCE message to not repeat information, but I think there's a stronger case for maintaining a VM-wide format for all ClassLoader-module-fqn strings. This format will be used for more messages as well, e.g. unified logging for class loading. (The follow-up bug: https://bugs.openjdk.java.net/browse/JDK-8169559.) And most importantly, because there are the specific instances listed above for leaving out the CL or module, it could potentially be more confusing to use shorthand. Would the user know that the module isn't listed the second time because it's the same module or because it's the unnamed module? Rachel On 11/10/2016 5:02 PM, David Holmes wrote: > Hi Rachel, > > Just wondering, in the most complex case, exactly what will this new > message look like? We seem to be including module names as well as > classloader names, and of course class names. > > Arguably we should only add the additional info when they differ > between the types ie if both in same classloader and same module then > only report class name; if same loader but different module report > module:class; if different loaders then report everything > loader:module:class. Also not sure module version is needed at all in > this message. > > Thanks, > David > > On 11/11/2016 7:38 AM, Rachel Protacio wrote: >> Hi, >> >> Please review this Jigsaw enhancement which adds the new format of class >> loader names to VM ClassCastException messages. Includes an >> updated/expanded jtreg test. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >> >> Passes JPRT and jck vm and lang tests. >> >> Thanks! >> Rachel From david.holmes at oracle.com Thu Nov 10 23:24:07 2016 From: david.holmes at oracle.com (David Holmes) Date: Fri, 11 Nov 2016 09:24:07 +1000 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: Message-ID: Hi Rachel, On 11/11/2016 8:31 AM, Rachel Protacio wrote: > Ah yes, my apologies - I should have provided more context. This is the > VM corollary of Mandy's change > http://hg.openjdk.java.net/jdk9/dev/hotspot/rev/48fce55afe71 which > implements the new classloader name scheme and uses it in Stack Traces. > In the fullest instance, it could look like > MyClassLoader/module1 at 9.0/package1.class1 > The class loader is not listed if it is built-in, the module is not > listed if it is unnamed, the version is not listed if it is a "non-jdk" > module. Ah I see. Though not at all clear why we would care about showing the JDK module version ?? > I appreciate your point about shortening this specific CCE message to > not repeat information, but I think there's a stronger case for > maintaining a VM-wide format for all ClassLoader-module-fqn strings. > This format will be used for more messages as well, e.g. unified logging > for class loading. (The follow-up bug: > https://bugs.openjdk.java.net/browse/JDK-8169559.) And most importantly, > because there are the specific instances listed above for leaving out > the CL or module, it could potentially be more confusing to use > shorthand. Would the user know that the module isn't listed the second > time because it's the same module or because it's the unnamed module? Yes good point. Okay on with the review: src/share/vm/classfile/moduleEntry.cpp + const char* loc = ""; + loc = location()->as_C_string(); can just be: + const char* loc = location()->as_C_string(); --- src/share/vm/runtime/sharedRuntime.cpp 1941 const char* class_loader_and_module_name(Klass* klass, Klass* base_klass) { I'm unclear in what way "base_klass" is a "base" class ?? And why, if it is not an instance class must be it in java.base ?? Nit: I don't think these need to be #defines nor have such verbose names: 1936 #define CLASS_LOADER_NAME_DELIMITER "/" 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH strlen(CLASS_LOADER_NAME_DELIMITER) why not just const char* delim = "\"; int delim_len = strlen(delim); Plus you are inconsistent with the handling of the "/" delimiter versus the "@" delimiter. --- test/runtime/modules/CCE_module_msg.java Nit: in the catch block from line #95 can you store getMessage() into a local so it isn't called 5 times. Thanks. :) Thanks, David > Rachel > > On 11/10/2016 5:02 PM, David Holmes wrote: >> Hi Rachel, >> >> Just wondering, in the most complex case, exactly what will this new >> message look like? We seem to be including module names as well as >> classloader names, and of course class names. >> >> Arguably we should only add the additional info when they differ >> between the types ie if both in same classloader and same module then >> only report class name; if same loader but different module report >> module:class; if different loaders then report everything >> loader:module:class. Also not sure module version is needed at all in >> this message. >> >> Thanks, >> David >> >> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>> Hi, >>> >>> Please review this Jigsaw enhancement which adds the new format of class >>> loader names to VM ClassCastException messages. Includes an >>> updated/expanded jtreg test. >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>> >>> Passes JPRT and jck vm and lang tests. >>> >>> Thanks! >>> Rachel > From calvin.cheung at oracle.com Thu Nov 10 23:42:08 2016 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 10 Nov 2016 15:42:08 -0800 Subject: RFR(xs): 8168796: relax vm options checking during CDS dump time In-Reply-To: <5BBF7048-407D-4890-9EE9-AC86EAF8FF6E@oracle.com> References: <5824CB84.8040309@oracle.com> <5BBF7048-407D-4890-9EE9-AC86EAF8FF6E@oracle.com> Message-ID: <582505D0.6040907@oracle.com> Hi Harold, Jiangli, Thanks for your review. I've updated the webrev based on your review comments: http://cr.openjdk.java.net/~ccheung/8168796/webrev.01/ thanks, Calvin On 11/10/16, 11:52 AM, Jiangli Zhou wrote: > +1 > > Thanks, > Jiangli > >> On Nov 10, 2016, at 11:42 AM, harold seigel wrote: >> >> Hi Calvin, >> >> The changes look good. Could you add a comment to the declaration of warning_idx explaining what it means to be above and below the that index? >> >> Also, instead of calling tty->print_cr(), can you call warning(), instead? >> >> Thanks, Harold >> >> >> On 11/10/2016 2:33 PM, Calvin Cheung wrote: >>> bug: https://bugs.openjdk.java.net/browse/JDK-8168796 >>> >>> webrev: http://cr.openjdk.java.net/~ccheung/8168796/webrev.00/ >>> >>> Before this change, if any one of the following vm options ("-module-path", "-add-modules", "--upgrade-module-path") is specified during CDS dump time, the vm exits with an error as follows: >>> >>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>> Error occurred during initialization of VM >>> Cannot use the following option when dumping the shared archive: --upgrade-module-path >>> >>> This change will print a warning message and continues as follows: >>> >>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>> Warning: the --upgrade-module-path option is ignored when dumping the shared archive >>> Allocated shared space: 50577408 bytes at 0x0000000800000000 >>> Loading classes to share ... >>> >>> Test case is in the hotspot closed repo. >>> >>> Testing: >>> JPRT >>> all tests under hotspot/test/runtime >>> >>> thanks, >>> Calvin >>> >>> >>> From jiangli.zhou at Oracle.COM Fri Nov 11 00:00:53 2016 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Thu, 10 Nov 2016 16:00:53 -0800 Subject: RFR(s): 8168790: CDS: assert(max_delta <= (0x7FFFFFFF)) failed: range check Message-ID: <3B0B08DC-9FBA-4390-A372-AB99A30BE974@oracle.com> Hi, Please review the following fix for JDK-8168790 . http://cr.openjdk.java.net/~jiangli/8168790/webrev.00/ The assert(max_delta <= MAX_SHARED_DELTA, "range check?) in CompactSymbolTableWriter::add(unsigned in hash, Symbol *symbol) was not necessary and over restrictive. Currently only shared symbols use encoding with delta to the shared space base. The shared symbols are allocated in the RO space, which is first space in the shared archive. The RO space cannot be larger than MAX_SHARED_READ_ONLY_SIZE (the check is done as part of the VM argument checks). MAX_SHARED_READ_ONLY_SIZE is (MAX_SHARED_DELTA - RW_size - MC_size - MD_size). I changed the assertion to make sure the currently ?delta' is encodable (less than MAX_SHARED_DELTA). Tested with JPRT, RBT runtime and AppCDS tests. Thanks, Jiangli From serguei.spitsyn at oracle.com Fri Nov 11 00:43:40 2016 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Thu, 10 Nov 2016 16:43:40 -0800 Subject: RFR(xs): 8168796: relax vm options checking during CDS dump time In-Reply-To: <582505D0.6040907@oracle.com> References: <5824CB84.8040309@oracle.com> <5BBF7048-407D-4890-9EE9-AC86EAF8FF6E@oracle.com> <582505D0.6040907@oracle.com> Message-ID: <0753d192-cb00-eb7d-93b8-eb889a51ab2e@oracle.com> Hi Calvin, Looks good. Thanks, Serguei On 11/10/16 15:42, Calvin Cheung wrote: > Hi Harold, Jiangli, > > Thanks for your review. > > I've updated the webrev based on your review comments: > http://cr.openjdk.java.net/~ccheung/8168796/webrev.01/ > > thanks, > Calvin > > On 11/10/16, 11:52 AM, Jiangli Zhou wrote: >> +1 >> >> Thanks, >> Jiangli >> >>> On Nov 10, 2016, at 11:42 AM, harold >>> seigel wrote: >>> >>> Hi Calvin, >>> >>> The changes look good. Could you add a comment to the declaration >>> of warning_idx explaining what it means to be above and below the >>> that index? >>> >>> Also, instead of calling tty->print_cr(), can you call warning(), >>> instead? >>> >>> Thanks, Harold >>> >>> >>> On 11/10/2016 2:33 PM, Calvin Cheung wrote: >>>> bug: https://bugs.openjdk.java.net/browse/JDK-8168796 >>>> >>>> webrev: http://cr.openjdk.java.net/~ccheung/8168796/webrev.00/ >>>> >>>> Before this change, if any one of the following vm options >>>> ("-module-path", "-add-modules", "--upgrade-module-path") is >>>> specified during CDS dump time, the vm exits with an error as follows: >>>> >>>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>>> Error occurred during initialization of VM >>>> Cannot use the following option when dumping the shared archive: >>>> --upgrade-module-path >>>> >>>> This change will print a warning message and continues as follows: >>>> >>>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>>> Warning: the --upgrade-module-path option is ignored when dumping >>>> the shared archive >>>> Allocated shared space: 50577408 bytes at 0x0000000800000000 >>>> Loading classes to share ... >>>> >>>> Test case is in the hotspot closed repo. >>>> >>>> Testing: >>>> JPRT >>>> all tests under hotspot/test/runtime >>>> >>>> thanks, >>>> Calvin >>>> >>>> >>>> From calvin.cheung at oracle.com Fri Nov 11 01:05:56 2016 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Thu, 10 Nov 2016 17:05:56 -0800 Subject: RFR(xs): 8168796: relax vm options checking during CDS dump time In-Reply-To: <0753d192-cb00-eb7d-93b8-eb889a51ab2e@oracle.com> References: <5824CB84.8040309@oracle.com> <5BBF7048-407D-4890-9EE9-AC86EAF8FF6E@oracle.com> <582505D0.6040907@oracle.com> <0753d192-cb00-eb7d-93b8-eb889a51ab2e@oracle.com> Message-ID: <58251974.8070006@oracle.com> Thank you, Serguei. Calvin On 11/10/16, 4:43 PM, serguei.spitsyn at oracle.com wrote: > Hi Calvin, > > Looks good. > > Thanks, > Serguei > > > On 11/10/16 15:42, Calvin Cheung wrote: >> Hi Harold, Jiangli, >> >> Thanks for your review. >> >> I've updated the webrev based on your review comments: >> http://cr.openjdk.java.net/~ccheung/8168796/webrev.01/ >> >> thanks, >> Calvin >> >> On 11/10/16, 11:52 AM, Jiangli Zhou wrote: >>> +1 >>> >>> Thanks, >>> Jiangli >>> >>>> On Nov 10, 2016, at 11:42 AM, harold >>>> seigel wrote: >>>> >>>> Hi Calvin, >>>> >>>> The changes look good. Could you add a comment to the declaration >>>> of warning_idx explaining what it means to be above and below the >>>> that index? >>>> >>>> Also, instead of calling tty->print_cr(), can you call warning(), >>>> instead? >>>> >>>> Thanks, Harold >>>> >>>> >>>> On 11/10/2016 2:33 PM, Calvin Cheung wrote: >>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8168796 >>>>> >>>>> webrev: http://cr.openjdk.java.net/~ccheung/8168796/webrev.00/ >>>>> >>>>> Before this change, if any one of the following vm options >>>>> ("-module-path", "-add-modules", "--upgrade-module-path") is >>>>> specified during CDS dump time, the vm exits with an error as >>>>> follows: >>>>> >>>>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>>>> Error occurred during initialization of VM >>>>> Cannot use the following option when dumping the shared archive: >>>>> --upgrade-module-path >>>>> >>>>> This change will print a warning message and continues as follows: >>>>> >>>>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>>>> Warning: the --upgrade-module-path option is ignored when dumping >>>>> the shared archive >>>>> Allocated shared space: 50577408 bytes at 0x0000000800000000 >>>>> Loading classes to share ... >>>>> >>>>> Test case is in the hotspot closed repo. >>>>> >>>>> Testing: >>>>> JPRT >>>>> all tests under hotspot/test/runtime >>>>> >>>>> thanks, >>>>> Calvin >>>>> >>>>> >>>>> > From jiangli.zhou at oracle.com Fri Nov 11 02:14:30 2016 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Thu, 10 Nov 2016 18:14:30 -0800 Subject: RFR(xs): 8168796: relax vm options checking during CDS dump time In-Reply-To: <582505D0.6040907@oracle.com> References: <5824CB84.8040309@oracle.com> <5BBF7048-407D-4890-9EE9-AC86EAF8FF6E@oracle.com> <582505D0.6040907@oracle.com> Message-ID: <98772B15-D51E-4473-8D58-42FBE3E66BF6@oracle.com> Looks good. Thanks, Jiangli > On Nov 10, 2016, at 3:42 PM, Calvin Cheung wrote: > > Hi Harold, Jiangli, > > Thanks for your review. > > I've updated the webrev based on your review comments: > http://cr.openjdk.java.net/~ccheung/8168796/webrev.01/ > > thanks, > Calvin > > On 11/10/16, 11:52 AM, Jiangli Zhou wrote: >> +1 >> >> Thanks, >> Jiangli >> >>> On Nov 10, 2016, at 11:42 AM, harold seigel wrote: >>> >>> Hi Calvin, >>> >>> The changes look good. Could you add a comment to the declaration of warning_idx explaining what it means to be above and below the that index? >>> >>> Also, instead of calling tty->print_cr(), can you call warning(), instead? >>> >>> Thanks, Harold >>> >>> >>> On 11/10/2016 2:33 PM, Calvin Cheung wrote: >>>> bug: https://bugs.openjdk.java.net/browse/JDK-8168796 >>>> >>>> webrev: http://cr.openjdk.java.net/~ccheung/8168796/webrev.00/ >>>> >>>> Before this change, if any one of the following vm options ("-module-path", "-add-modules", "--upgrade-module-path") is specified during CDS dump time, the vm exits with an error as follows: >>>> >>>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>>> Error occurred during initialization of VM >>>> Cannot use the following option when dumping the shared archive: --upgrade-module-path >>>> >>>> This change will print a warning message and continues as follows: >>>> >>>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>>> Warning: the --upgrade-module-path option is ignored when dumping the shared archive >>>> Allocated shared space: 50577408 bytes at 0x0000000800000000 >>>> Loading classes to share ... >>>> >>>> Test case is in the hotspot closed repo. >>>> >>>> Testing: >>>> JPRT >>>> all tests under hotspot/test/runtime >>>> >>>> thanks, >>>> Calvin >>>> >>>> >>>> From david.holmes at oracle.com Fri Nov 11 03:40:43 2016 From: david.holmes at oracle.com (David Holmes) Date: Fri, 11 Nov 2016 13:40:43 +1000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> Message-ID: Hi Goetz, On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > Hi David, > > This issue is different to 6675312, see also my comment in the bug. > > It appears running jtreg test runtime/Thread/TooSmallStackSize.java, > with my patch below you can reproduce it on linuxx86_64. You can not > do that with 6675312. Also, I would assume there are systems out there > on x86 that uses 64-K pages, did you run the tests on these? I would > assume you get hard crashes with stack overflows in the first C2 > compilation if there is only 64K usable CompilerThreadStack. > > My fix does not affect Java threads, which are the largest amount > of threads used by the VM. It affects only the non-Java threads. > It adds one page to these threads. The page does not require memory, > as it's protected. The stack will only require more space if the thread > ran into a stack overflow before the fix as else the pages are not mapped. > This are stack overflows that cause hard crashes, at least on ppc the VM > does not properly catch these stack overflows, so any setup working now > will not run into the additional space. Altogether there should be no > effect on running systems besides requiring one more entry in the > page table per non-Java thread. > > The problem is caused by a rather recent change (8140520: segfault on solaris-amd64 > with "-XX:VMThreadStackSize=1" option) which was pushed after > feature-close. As this was a rather recent change, it must be possible to > fix this follow up issue. What else is this period in the project good > for if not fixing issues? So I am seeing a number of factors here. First, 8140520, set: size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; Second on linux PPC it is hardwired to use 2 guard pages: return 2 * page_size(); Third, you had a pagesize of 64K. Fourth, NPTL takes the guard space from the stack space - hence with 2 x 64K guard, and a 128K stack it was all consumed. --- In the proposed changes you now only use page_size() for the guard, so that alone would have fixed the observed problem. But in addition you want to address the NPTL problem by adding back the guard space to the stack size requested. That alone would also have fixed the observed problem. :) But in addition you have increased the minimum stack size: ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; which again, on its own would have fixed the original problem. :) Did you really intend to increase the real minimum stack from 128K to 256K ? (on a 64K page system) --- Focusing simply on the shared code change to adjust the requested stacksize by the amount of guard space (if any), this does not seem unreasonable. As you note it is restricted to non-JavaThreads and only adds a page to reserved stack space. My only query now is whether the minimum stacksize detection logic will correctly report the real minimum stack size (taking into account the need for the guard page) ? Thanks, David > So I really think this issue should be fixed. > > Best regards, > Goetz. > > >> -----Original Message----- >> From: David Holmes [mailto:david.holmes at oracle.com] >> Sent: Thursday, November 10, 2016 10:02 PM >> To: Lindenmaier, Goetz ; hotspot-runtime- >> dev at openjdk.java.net >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >> >> Hi Goetz, >> >> As per the bug report, this issue was already known (6675312) and we >> chose not to try and address it due to no reported issues at the time. >> While I see that you have encountered an issue (is it real or >> fabricated?) I think this change is too intrusive to be applied at this >> stage of the JDK 9 release cycle, as it will change the stack >> requirements of every application running on Linux. >> >> Thanks, >> David >> >> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: >>> Hi, >>> >>> >>> >>> Please review this change. I please need a sponsor: >>> >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.01/ >>> >>> >>> >>> In the Linux NPTL pthread implementation the guard size mechanism is not >>> implemented properly. The posix standard requires to add the size of the >>> guard pages to the stack size, instead Linux takes the space out of >>> 'stacksize'. >>> >>> The Posix standard http://pubs.opengroup.org/onlinepubs/9699919799/ >>> says "the implementation allocates extra memory at the overflow end of >>> the stack". The linux man page >>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As at glibc >>> 2.8, the NPTL threading implementation includes the guard area within >>> the stack size allocation, rather than allocating extra space at the end >>> of the stack, as POSIX.1 requires". >>> >>> I encounter this problem in runtime/Thread/TooSmallStackSize.java on ppc >>> with 64K pages. _compiler_thread_min_stack_allowed is 128K on ppc, and >>> ppc specifies two OS guard pages. The VM crashes in pthread creation >>> because there is no usable space in the thread stack after allocating >>> the guard pages. >>> >>> But TooSmallStackSize.java requires that the VM comes up with the stack >>> size mentioned in the error message. >>> >>> This fix adapts the requested stack size on Linux by the size of the >>> guard pages to mimick proper behaviour, see change to os_linux.cpp. >>> >>> >>> >>> The change also streamlines usage of stack_guard_page on linuxppc, >>> linuxppcle, aixppc and linuxs390. >>> >>> >>> >>> To reproduce the error on linux_x86_64, apply below patch and call the >>> VM with -XX:CompilerThreadStackSize=64. >>> >>> >>> >>> I'm still exploring why I had to choose such big compiler stacks on ppc >>> to get -version passing, but I wanted to send the RFR now as people >>> obviously looked at the bug I opened (Thanks David!). >>> >>> >>> >>> Best regards, >>> >>> Goetz. >>> >>> >>> >>> >>> >>> >>> >>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>> >>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 12:37:28 >> 2016 >>> +0100 >>> >>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 16:52:17 >> 2016 >>> +0100 >>> >>> @@ -701,7 +701,7 @@ >>> >>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>> >>> // Creating guard page is very expensive. Java thread has HotSpot >>> >>> // guard page, only enable glibc guard page for non-Java threads. >>> >>> - return (thr_type == java_thread ? 0 : page_size()); >>> >>> + return (thr_type == java_thread ? 0 : 64*K); >>> >>> } >>> >>> >>> >>> // Java thread: >>> From goetz.lindenmaier at sap.com Fri Nov 11 08:12:55 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Fri, 11 Nov 2016 08:12:55 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> Message-ID: <56741cc6200846b28c9e1741d83c1d5f@derote13de14.global.corp.sap> Hi David, thanks for having a closer look! > -----Original Message----- > From: David Holmes [mailto:david.holmes at oracle.com] > Sent: Freitag, 11. November 2016 04:41 > To: Lindenmaier, Goetz ; hotspot-runtime- > dev at openjdk.java.net > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > Hi Goetz, > > On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > > Hi David, > > > > This issue is different to 6675312, see also my comment in the bug. > > > > It appears running jtreg test runtime/Thread/TooSmallStackSize.java, > > with my patch below you can reproduce it on linuxx86_64. You can not > > do that with 6675312. Also, I would assume there are systems out there > > on x86 that uses 64-K pages, did you run the tests on these? I would > > assume you get hard crashes with stack overflows in the first C2 > > compilation if there is only 64K usable CompilerThreadStack. > > > > My fix does not affect Java threads, which are the largest amount > > of threads used by the VM. It affects only the non-Java threads. > > It adds one page to these threads. The page does not require memory, > > as it's protected. The stack will only require more space if the thread > > ran into a stack overflow before the fix as else the pages are not mapped. > > This are stack overflows that cause hard crashes, at least on ppc the VM > > does not properly catch these stack overflows, so any setup working now > > will not run into the additional space. Altogether there should be no > > effect on running systems besides requiring one more entry in the > > page table per non-Java thread. > > > > The problem is caused by a rather recent change (8140520: segfault on > solaris-amd64 > > with "-XX:VMThreadStackSize=1" option) which was pushed after > > feature-close. As this was a rather recent change, it must be possible to > > fix this follow up issue. What else is this period in the project good > > for if not fixing issues? > > So I am seeing a number of factors here. > > First, 8140520, set: > > size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > > Second on linux PPC it is hardwired to use 2 guard pages: > > return 2 * page_size(); > > Third, you had a pagesize of 64K. > > Fourth, NPTL takes the guard space from the stack space - hence with 2 x > 64K guard, and a 128K stack it was all consumed. Yes. > > --- > > In the proposed changes you now only use page_size() for the guard, so > that alone would have fixed the observed problem. 2*page_size() is a setting we inherited from our ia64 port, from where we branched the original ppc port somewhere around 2006/7. I took the chance to adapt this. > But in addition you want to address the NPTL problem by adding back the > guard space to the stack size requested. That alone would also have > fixed the observed problem. :) Yes, this is the real problem. _compiler_thread_min_stack_allowed & friends are minimal values required to start up the VM. If you reduce them slightly, the VM will run into stack overflow. I would assume they are tuned on linuxx86_64 for 4K page size. As the value there is 64K, the VM probably utilizes around 50K in a simple -version run. Thus, if you start the VM on a system with 32K page size, there is only 32K usable space on the stack, and -version will crash (I verified this by setting _compiler_thread_min_stack_allowed = 32K on linuxx86_64). The lower bound is pointless if it is that much dependent on the system. It should reflect the size of the frames of the binary, which are basically the same independent of page size. > But in addition you have increased the minimum stack size: > > ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; I had to do so because I ran into stack overflow of the compiler thread starting with 128K after fixing the other issues. I'm looking for the cause of this. I thought we have more aggressive compiler flag settings, but these flags look quite reasonable so far. > which again, on its own would have fixed the original problem. :) Well, I would have needed even more ... > > Did you really intend to increase the real minimum stack from 128K to > 256K ? (on a 64K page system) > > --- > > Focusing simply on the shared code change to adjust the requested > stacksize by the amount of guard space (if any), this does not seem > unreasonable. As you note it is restricted to non-JavaThreads and only > adds a page to reserved stack space. > > My only query now is whether the minimum stacksize detection logic will > correctly report the real minimum stack size (taking into account the > need for the guard page) ? It will report the proper value that the user must set on the command line (i.e., the test mentioned above passes). This value does not include the size required for the pthread guard page. > Thanks, > David > > > So I really think this issue should be fixed. > > > > Best regards, > > Goetz. > > > > > >> -----Original Message----- > >> From: David Holmes [mailto:david.holmes at oracle.com] > >> Sent: Thursday, November 10, 2016 10:02 PM > >> To: Lindenmaier, Goetz ; hotspot-runtime- > >> dev at openjdk.java.net > >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > >> > >> Hi Goetz, > >> > >> As per the bug report, this issue was already known (6675312) and we > >> chose not to try and address it due to no reported issues at the time. > >> While I see that you have encountered an issue (is it real or > >> fabricated?) I think this change is too intrusive to be applied at this > >> stage of the JDK 9 release cycle, as it will change the stack > >> requirements of every application running on Linux. > >> > >> Thanks, > >> David > >> > >> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > >>> Hi, > >>> > >>> > >>> > >>> Please review this change. I please need a sponsor: > >>> > >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > stackFix/webrev.01/ > >>> > >>> > >>> > >>> In the Linux NPTL pthread implementation the guard size mechanism is > not > >>> implemented properly. The posix standard requires to add the size of > the > >>> guard pages to the stack size, instead Linux takes the space out of > >>> 'stacksize'. > >>> > >>> The Posix standard http://pubs.opengroup.org/onlinepubs/9699919799/ > >>> says "the implementation allocates extra memory at the overflow end > of > >>> the stack". The linux man page > >>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As at glibc > >>> 2.8, the NPTL threading implementation includes the guard area within > >>> the stack size allocation, rather than allocating extra space at the end > >>> of the stack, as POSIX.1 requires". > >>> > >>> I encounter this problem in runtime/Thread/TooSmallStackSize.java on > ppc > >>> with 64K pages. _compiler_thread_min_stack_allowed is 128K on ppc, > and > >>> ppc specifies two OS guard pages. The VM crashes in pthread creation > >>> because there is no usable space in the thread stack after allocating > >>> the guard pages. > >>> > >>> But TooSmallStackSize.java requires that the VM comes up with the > stack > >>> size mentioned in the error message. > >>> > >>> This fix adapts the requested stack size on Linux by the size of the > >>> guard pages to mimick proper behaviour, see change to os_linux.cpp. > >>> > >>> > >>> > >>> The change also streamlines usage of stack_guard_page on linuxppc, > >>> linuxppcle, aixppc and linuxs390. > >>> > >>> > >>> > >>> To reproduce the error on linux_x86_64, apply below patch and call the > >>> VM with -XX:CompilerThreadStackSize=64. > >>> > >>> > >>> > >>> I'm still exploring why I had to choose such big compiler stacks on ppc > >>> to get -version passing, but I wanted to send the RFR now as people > >>> obviously looked at the bug I opened (Thanks David!). > >>> > >>> > >>> > >>> Best regards, > >>> > >>> Goetz. > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp > >>> > >>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 12:37:28 > >> 2016 > >>> +0100 > >>> > >>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 16:52:17 > >> 2016 > >>> +0100 > >>> > >>> @@ -701,7 +701,7 @@ > >>> > >>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > >>> > >>> // Creating guard page is very expensive. Java thread has HotSpot > >>> > >>> // guard page, only enable glibc guard page for non-Java threads. > >>> > >>> - return (thr_type == java_thread ? 0 : page_size()); > >>> > >>> + return (thr_type == java_thread ? 0 : 64*K); > >>> > >>> } > >>> > >>> > >>> > >>> // Java thread: > >>> From daniel.fuchs at oracle.com Fri Nov 11 11:13:38 2016 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Fri, 11 Nov 2016 11:13:38 +0000 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: Message-ID: <2418683b-b574-7646-cdef-eb21038c64eb@oracle.com> Hi, On 10/11/16 23:24, David Holmes wrote: > Hi Rachel, > > On 11/11/2016 8:31 AM, Rachel Protacio wrote: >> Ah yes, my apologies - I should have provided more context. This is the >> VM corollary of Mandy's change >> http://hg.openjdk.java.net/jdk9/dev/hotspot/rev/48fce55afe71 which >> implements the new classloader name scheme and uses it in Stack Traces. >> In the fullest instance, it could look like >> MyClassLoader/module1 at 9.0/package1.class1 >> The class loader is not listed if it is built-in, the module is not >> listed if it is unnamed, the version is not listed if it is a "non-jdk" >> module. > > Ah I see. Though not at all clear why we would care about showing the > JDK module version ?? Mandy will correct me if I'm wrong, but AFAIU, for JDK modules, the version is displayed only if the module is an upgradeable module. The relevant algorithm [1] that does that can be seen in StackTraceElement::toLoaderModuleClassName(Class): [1] http://hg.openjdk.java.net/jdk9/dev/jdk/file/5b6b8e24a20b/src/java.base/share/classes/java/lang/StackTraceElement.java#l414 best regards, -- daniel From david.holmes at oracle.com Fri Nov 11 22:49:48 2016 From: david.holmes at oracle.com (David Holmes) Date: Sat, 12 Nov 2016 08:49:48 +1000 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <2418683b-b574-7646-cdef-eb21038c64eb@oracle.com> References: <2418683b-b574-7646-cdef-eb21038c64eb@oracle.com> Message-ID: <6ad872cd-9684-e184-e1bf-74ebd561d3f4@oracle.com> On 11/11/2016 9:13 PM, Daniel Fuchs wrote: > Hi, > > On 10/11/16 23:24, David Holmes wrote: >> Hi Rachel, >> >> On 11/11/2016 8:31 AM, Rachel Protacio wrote: >>> Ah yes, my apologies - I should have provided more context. This is the >>> VM corollary of Mandy's change >>> http://hg.openjdk.java.net/jdk9/dev/hotspot/rev/48fce55afe71 which >>> implements the new classloader name scheme and uses it in Stack Traces. >>> In the fullest instance, it could look like >>> MyClassLoader/module1 at 9.0/package1.class1 >>> The class loader is not listed if it is built-in, the module is not >>> listed if it is unnamed, the version is not listed if it is a "non-jdk" >>> module. >> >> Ah I see. Though not at all clear why we would care about showing the >> JDK module version ?? > > Mandy will correct me if I'm wrong, but AFAIU, for JDK modules, > the version is displayed only if the module is an upgradeable > module. The relevant algorithm [1] that does that can be seen in > StackTraceElement::toLoaderModuleClassName(Class): That seems to be almost the complete opposite of what we have at the moment. But it makes sense to me to omit stable JDK version info. Thanks, David > > [1] > http://hg.openjdk.java.net/jdk9/dev/jdk/file/5b6b8e24a20b/src/java.base/share/classes/java/lang/StackTraceElement.java#l414 > > > best regards, > > -- daniel > > From mandy.chung at oracle.com Fri Nov 11 22:58:22 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Fri, 11 Nov 2016 14:58:22 -0800 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <2418683b-b574-7646-cdef-eb21038c64eb@oracle.com> References: <2418683b-b574-7646-cdef-eb21038c64eb@oracle.com> Message-ID: > On Nov 11, 2016, at 3:13 AM, Daniel Fuchs wrote: > > Hi, > > On 10/11/16 23:24, David Holmes wrote: >> Hi Rachel, >> >> On 11/11/2016 8:31 AM, Rachel Protacio wrote: >>> Ah yes, my apologies - I should have provided more context. This is the >>> VM corollary of Mandy's change >>> http://hg.openjdk.java.net/jdk9/dev/hotspot/rev/48fce55afe71 which >>> implements the new classloader name scheme and uses it in Stack Traces. >>> In the fullest instance, it could look like >>> MyClassLoader/module1 at 9.0/package1.class1 >>> The class loader is not listed if it is built-in, the module is not >>> listed if it is unnamed, the version is not listed if it is a "non-jdk" >>> module. >> >> Ah I see. Though not at all clear why we would care about showing the >> JDK module version ?? > > Mandy will correct me if I'm wrong, but AFAIU, for JDK modules, > the version is displayed only if the module is an upgradeable > module. The relevant algorithm [1] that does that can be seen in > StackTraceElement::toLoaderModuleClassName(Class): > That is correct and that?s the format we implemented for the stack trace. The VM does not have an easy way to determine if a JDK module is upgradeable or not. There are a couple options that we considered for example include such information when defining the module. The reason to include a version in an upgradeable module so that from the stack trace it can tell if the class is not the version in the JDK. So we came up with this option such that the version is omitted if it?s a JDK module coming from the image that includes upgradeable modules that are not upgraded. If these upgradeable modules are loaded from image, they are the same version as the runtime. If the module is upgraded, the version will be included. We agree to not to change the JVM_DefineModule interface since jake changes are in flight. The exception message will have a small difference but really minor and still provides the useful information for troubleshooting. Mandy > > [1] http://hg.openjdk.java.net/jdk9/dev/jdk/file/5b6b8e24a20b/src/java.base/share/classes/java/lang/StackTraceElement.java#l414 > > best regards, > > -- daniel > > From david.holmes at oracle.com Fri Nov 11 23:05:51 2016 From: david.holmes at oracle.com (David Holmes) Date: Sat, 12 Nov 2016 09:05:51 +1000 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: <2418683b-b574-7646-cdef-eb21038c64eb@oracle.com> Message-ID: On 12/11/2016 8:58 AM, Mandy Chung wrote: > >> On Nov 11, 2016, at 3:13 AM, Daniel Fuchs wrote: >> >> Hi, >> >> On 10/11/16 23:24, David Holmes wrote: >>> Hi Rachel, >>> >>> On 11/11/2016 8:31 AM, Rachel Protacio wrote: >>>> Ah yes, my apologies - I should have provided more context. This is the >>>> VM corollary of Mandy's change >>>> http://hg.openjdk.java.net/jdk9/dev/hotspot/rev/48fce55afe71 which >>>> implements the new classloader name scheme and uses it in Stack Traces. >>>> In the fullest instance, it could look like >>>> MyClassLoader/module1 at 9.0/package1.class1 >>>> The class loader is not listed if it is built-in, the module is not >>>> listed if it is unnamed, the version is not listed if it is a "non-jdk" >>>> module. >>> >>> Ah I see. Though not at all clear why we would care about showing the >>> JDK module version ?? >> >> Mandy will correct me if I'm wrong, but AFAIU, for JDK modules, >> the version is displayed only if the module is an upgradeable >> module. The relevant algorithm [1] that does that can be seen in >> StackTraceElement::toLoaderModuleClassName(Class): >> > > That is correct and that?s the format we implemented for the stack trace. > > The VM does not have an easy way to determine if a JDK module is upgradeable or not. There are a couple options that we considered for example include such information when defining the module. The reason to include a version in an upgradeable module so that from the stack trace it can tell if the class is not the version in the JDK. So we came up with this option such that the version is omitted if it?s a JDK module coming from the image that includes upgradeable modules that are not upgraded. If these upgradeable modules are loaded from image, they are the same version as the runtime. If the module is upgraded, the version will be included. > > We agree to not to change the JVM_DefineModule interface since jake changes are in flight. The exception message will have a small difference but really minor and still provides the useful information for troubleshooting. But what is being proposed for the ClassCastException message is almost the complete opposite of what was done for the stacktrace. This needs to be reconciled IMO, even if the VM can't deal with the upgradeable part. Thanks, David > Mandy > >> >> [1] http://hg.openjdk.java.net/jdk9/dev/jdk/file/5b6b8e24a20b/src/java.base/share/classes/java/lang/StackTraceElement.java#l414 >> >> best regards, >> >> -- daniel >> >> > From david.holmes at oracle.com Sat Nov 12 01:36:43 2016 From: david.holmes at oracle.com (David Holmes) Date: Sat, 12 Nov 2016 11:36:43 +1000 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: <2418683b-b574-7646-cdef-eb21038c64eb@oracle.com> Message-ID: <267d1530-d128-4e6a-af8d-02fbd6c938fb@oracle.com> On 12/11/2016 9:05 AM, David Holmes wrote: > On 12/11/2016 8:58 AM, Mandy Chung wrote: >> >>> On Nov 11, 2016, at 3:13 AM, Daniel Fuchs >>> wrote: >>> >>> Hi, >>> >>> On 10/11/16 23:24, David Holmes wrote: >>>> Hi Rachel, >>>> >>>> On 11/11/2016 8:31 AM, Rachel Protacio wrote: >>>>> Ah yes, my apologies - I should have provided more context. This is >>>>> the >>>>> VM corollary of Mandy's change >>>>> http://hg.openjdk.java.net/jdk9/dev/hotspot/rev/48fce55afe71 which >>>>> implements the new classloader name scheme and uses it in Stack >>>>> Traces. >>>>> In the fullest instance, it could look like >>>>> MyClassLoader/module1 at 9.0/package1.class1 >>>>> The class loader is not listed if it is built-in, the module is not >>>>> listed if it is unnamed, the version is not listed if it is a >>>>> "non-jdk" >>>>> module. >>>> >>>> Ah I see. Though not at all clear why we would care about showing the >>>> JDK module version ?? >>> >>> Mandy will correct me if I'm wrong, but AFAIU, for JDK modules, >>> the version is displayed only if the module is an upgradeable >>> module. The relevant algorithm [1] that does that can be seen in >>> StackTraceElement::toLoaderModuleClassName(Class): >>> >> >> That is correct and that?s the format we implemented for the stack trace. >> >> The VM does not have an easy way to determine if a JDK module is >> upgradeable or not. There are a couple options that we considered for >> example include such information when defining the module. The reason >> to include a version in an upgradeable module so that from the stack >> trace it can tell if the class is not the version in the JDK. So we >> came up with this option such that the version is omitted if it?s a >> JDK module coming from the image that includes upgradeable modules >> that are not upgraded. If these upgradeable modules are loaded from >> image, they are the same version as the runtime. If the module is >> upgraded, the version will be included. >> >> We agree to not to change the JVM_DefineModule interface since jake >> changes are in flight. The exception message will have a small >> difference but really minor and still provides the useful information >> for troubleshooting. > > But what is being proposed for the ClassCastException message is almost > the complete opposite of what was done for the stacktrace. This needs to > be reconciled IMO, even if the VM can't deal with the upgradeable part. Okay there is a difference between the code and how Rachel described it. The code: 1977 // Use version if exists and is not a jdk module 1978 if (module->is_non_jdk_module() && module->version() != NULL) { What Rachel said: "the version is not listed if it is a "non-jdk" module." So I'll take the "not" as a typo and all is good. :) Cheers, David > Thanks, > David > >> Mandy >> >>> >>> [1] >>> http://hg.openjdk.java.net/jdk9/dev/jdk/file/5b6b8e24a20b/src/java.base/share/classes/java/lang/StackTraceElement.java#l414 >>> >>> >>> best regards, >>> >>> -- daniel >>> >>> >> From Alan.Bateman at oracle.com Sat Nov 12 08:05:09 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 12 Nov 2016 08:05:09 +0000 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: <2418683b-b574-7646-cdef-eb21038c64eb@oracle.com> Message-ID: On 11/11/2016 22:58, Mandy Chung wrote: > : > > We agree to not to change the JVM_DefineModule interface since jake changes are in flight. Right, several reasons. One is that the notion of upgradeable module is specific to the boot layer and so overburdens this function. Additionally, it would be highly disruptive to be passing this type of information to the VM at this point during startup. So it would be good to keep that out for now. -Alan From ioi.lam at oracle.com Mon Nov 14 04:26:19 2016 From: ioi.lam at oracle.com (Ioi Lam) Date: Sun, 13 Nov 2016 20:26:19 -0800 Subject: RFR(s): 8168790: CDS: assert(max_delta <= (0x7FFFFFFF)) failed: range check In-Reply-To: <3B0B08DC-9FBA-4390-A372-AB99A30BE974@oracle.com> References: <3B0B08DC-9FBA-4390-A372-AB99A30BE974@oracle.com> Message-ID: <58293CEB.4080408@oracle.com> Hi Jiangli, The code change looks good to me. For the test case, have you tested on 32-bit? Reservation of 1GB of address space may fail on 32-bit platforms. 41 ProcessBuilder pb = ProcessTools.createJavaProcessBuilder( 42 "-XX:SharedMiscCodeSize=1066924031", "-Xshare:dump"); I think it's better to limit the test case to run on only 64-bit platforms. Thanks - Ioi On 11/10/16 4:00 PM, Jiangli Zhou wrote: > Hi, > > Please review the following fix for JDK-8168790 . > > http://cr.openjdk.java.net/~jiangli/8168790/webrev.00/ > > The assert(max_delta <= MAX_SHARED_DELTA, "range check?) in CompactSymbolTableWriter::add(unsigned in hash, Symbol *symbol) was not necessary and over restrictive. > > Currently only shared symbols use encoding with delta to the shared space base. The shared symbols are allocated in the RO space, which is first space in the shared archive. The RO space cannot be larger than MAX_SHARED_READ_ONLY_SIZE (the check is done as part of the VM argument checks). MAX_SHARED_READ_ONLY_SIZE is (MAX_SHARED_DELTA - RW_size - MC_size - MD_size). I changed the assertion to make sure the currently ?delta' is encodable (less than MAX_SHARED_DELTA). > > Tested with JPRT, RBT runtime and AppCDS tests. > > Thanks, > Jiangli From harold.seigel at oracle.com Mon Nov 14 14:02:25 2016 From: harold.seigel at oracle.com (harold seigel) Date: Mon, 14 Nov 2016 09:02:25 -0500 Subject: RFR(XS) 8169551: Segfaults in error traces when module version is null Message-ID: Hi, Please review this small fix for JDK-8169551. Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8169551/ JBS bug: https://bugs.openjdk.java.net/browse/JDK-8169551 The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, java/io, java/lang, and java/util tests, the nsk co-located and the non-co-located tests, and the RBT tier2 tests. Thanks, Harold From harold.seigel at oracle.com Mon Nov 14 14:10:56 2016 From: harold.seigel at oracle.com (harold seigel) Date: Mon, 14 Nov 2016 09:10:56 -0500 Subject: RFR(xs): 8168796: relax vm options checking during CDS dump time In-Reply-To: <582505D0.6040907@oracle.com> References: <5824CB84.8040309@oracle.com> <5BBF7048-407D-4890-9EE9-AC86EAF8FF6E@oracle.com> <582505D0.6040907@oracle.com> Message-ID: <0378f0d6-09ef-9ab2-13b2-cb4392ae7177@oracle.com> Hi Calvin, It looks good! Thanks, Harold On 11/10/2016 6:42 PM, Calvin Cheung wrote: > Hi Harold, Jiangli, > > Thanks for your review. > > I've updated the webrev based on your review comments: > http://cr.openjdk.java.net/~ccheung/8168796/webrev.01/ > > thanks, > Calvin > > On 11/10/16, 11:52 AM, Jiangli Zhou wrote: >> +1 >> >> Thanks, >> Jiangli >> >>> On Nov 10, 2016, at 11:42 AM, harold >>> seigel wrote: >>> >>> Hi Calvin, >>> >>> The changes look good. Could you add a comment to the declaration >>> of warning_idx explaining what it means to be above and below the >>> that index? >>> >>> Also, instead of calling tty->print_cr(), can you call warning(), >>> instead? >>> >>> Thanks, Harold >>> >>> >>> On 11/10/2016 2:33 PM, Calvin Cheung wrote: >>>> bug: https://bugs.openjdk.java.net/browse/JDK-8168796 >>>> >>>> webrev: http://cr.openjdk.java.net/~ccheung/8168796/webrev.00/ >>>> >>>> Before this change, if any one of the following vm options >>>> ("-module-path", "-add-modules", "--upgrade-module-path") is >>>> specified during CDS dump time, the vm exits with an error as follows: >>>> >>>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>>> Error occurred during initialization of VM >>>> Cannot use the following option when dumping the shared archive: >>>> --upgrade-module-path >>>> >>>> This change will print a warning message and continues as follows: >>>> >>>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>>> Warning: the --upgrade-module-path option is ignored when dumping >>>> the shared archive >>>> Allocated shared space: 50577408 bytes at 0x0000000800000000 >>>> Loading classes to share ... >>>> >>>> Test case is in the hotspot closed repo. >>>> >>>> Testing: >>>> JPRT >>>> all tests under hotspot/test/runtime >>>> >>>> thanks, >>>> Calvin >>>> >>>> >>>> From george.triantafillou at oracle.com Mon Nov 14 14:12:06 2016 From: george.triantafillou at oracle.com (George Triantafillou) Date: Mon, 14 Nov 2016 09:12:06 -0500 Subject: RFR(XS) 8169551: Segfaults in error traces when module version is null In-Reply-To: References: Message-ID: Hi Harold, Looks good! -George On 11/14/2016 9:02 AM, harold seigel wrote: > Hi, > > Please review this small fix for JDK-8169551. > > Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8169551/ > > JBS bug: https://bugs.openjdk.java.net/browse/JDK-8169551 > > The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, > java/io, java/lang, and java/util tests, the nsk co-located and the > non-co-located tests, and the RBT tier2 tests. > > Thanks, Harold > From harold.seigel at oracle.com Mon Nov 14 14:16:12 2016 From: harold.seigel at oracle.com (harold seigel) Date: Mon, 14 Nov 2016 09:16:12 -0500 Subject: RFR(XS) 8169551: Segfaults in error traces when module version is null In-Reply-To: References: Message-ID: <8ad98db9-699d-7c0a-14d5-eea799e64f5e@oracle.com> Thanks George! Harold On 11/14/2016 9:12 AM, George Triantafillou wrote: > Hi Harold, > > Looks good! > > -George > > On 11/14/2016 9:02 AM, harold seigel wrote: >> Hi, >> >> Please review this small fix for JDK-8169551. >> >> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8169551/ >> >> JBS bug: https://bugs.openjdk.java.net/browse/JDK-8169551 >> >> The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, >> java/io, java/lang, and java/util tests, the nsk co-located and the >> non-co-located tests, and the RBT tier2 tests. >> >> Thanks, Harold >> > From dmitry.dmitriev at oracle.com Mon Nov 14 14:20:03 2016 From: dmitry.dmitriev at oracle.com (Dmitry Dmitriev) Date: Mon, 14 Nov 2016 17:20:03 +0300 Subject: RFR(s): 8168790: CDS: assert(max_delta <= (0x7FFFFFFF)) failed: range check In-Reply-To: <3B0B08DC-9FBA-4390-A372-AB99A30BE974@oracle.com> References: <3B0B08DC-9FBA-4390-A372-AB99A30BE974@oracle.com> Message-ID: <10c82a3d-e5aa-23b0-ac48-4feba0f2ee8e@oracle.com> Hello Jiangli, Can comment only new test: I think it will be better to dump classes to the non-default archive, i.e. add following options to the test, e.g.: "-XX:+UnlockDiagnosticVMOptions -XX:SharedArchiveFile=LargeSharedSpace.jsa" Thanks, Dmitry On 11.11.2016 3:00, Jiangli Zhou wrote: > Hi, > > Please review the following fix for JDK-8168790 . > > http://cr.openjdk.java.net/~jiangli/8168790/webrev.00/ > > The assert(max_delta <= MAX_SHARED_DELTA, "range check?) in CompactSymbolTableWriter::add(unsigned in hash, Symbol *symbol) was not necessary and over restrictive. > > Currently only shared symbols use encoding with delta to the shared space base. The shared symbols are allocated in the RO space, which is first space in the shared archive. The RO space cannot be larger than MAX_SHARED_READ_ONLY_SIZE (the check is done as part of the VM argument checks). MAX_SHARED_READ_ONLY_SIZE is (MAX_SHARED_DELTA - RW_size - MC_size - MD_size). I changed the assertion to make sure the currently ?delta' is encodable (less than MAX_SHARED_DELTA). > > Tested with JPRT, RBT runtime and AppCDS tests. > > Thanks, > Jiangli From lois.foltan at oracle.com Mon Nov 14 14:22:56 2016 From: lois.foltan at oracle.com (Lois Foltan) Date: Mon, 14 Nov 2016 09:22:56 -0500 Subject: RFR(XS) 8169551: Segfaults in error traces when module version is null In-Reply-To: References: Message-ID: <5829C8C0.4090105@oracle.com> Looks good. Lois On 11/14/2016 9:02 AM, harold seigel wrote: > Hi, > > Please review this small fix for JDK-8169551. > > Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8169551/ > > JBS bug: https://bugs.openjdk.java.net/browse/JDK-8169551 > > The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, > java/io, java/lang, and java/util tests, the nsk co-located and the > non-co-located tests, and the RBT tier2 tests. > > Thanks, Harold > From harold.seigel at oracle.com Mon Nov 14 14:24:27 2016 From: harold.seigel at oracle.com (harold seigel) Date: Mon, 14 Nov 2016 09:24:27 -0500 Subject: RFR(XS) 8169551: Segfaults in error traces when module version is null In-Reply-To: <5829C8C0.4090105@oracle.com> References: <5829C8C0.4090105@oracle.com> Message-ID: Thanks! Harold On 11/14/2016 9:22 AM, Lois Foltan wrote: > Looks good. > Lois > > On 11/14/2016 9:02 AM, harold seigel wrote: >> Hi, >> >> Please review this small fix for JDK-8169551. >> >> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8169551/ >> >> JBS bug: https://bugs.openjdk.java.net/browse/JDK-8169551 >> >> The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, >> java/io, java/lang, and java/util tests, the nsk co-located and the >> non-co-located tests, and the RBT tier2 tests. >> >> Thanks, Harold >> > From rachel.protacio at oracle.com Mon Nov 14 16:48:22 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Mon, 14 Nov 2016 11:48:22 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <267d1530-d128-4e6a-af8d-02fbd6c938fb@oracle.com> References: <2418683b-b574-7646-cdef-eb21038c64eb@oracle.com> <267d1530-d128-4e6a-af8d-02fbd6c938fb@oracle.com> Message-ID: <03f2a9cf-aa5d-423b-5677-84574b19fbc0@oracle.com> On 11/11/2016 8:36 PM, David Holmes wrote: > On 12/11/2016 9:05 AM, David Holmes wrote: >> On 12/11/2016 8:58 AM, Mandy Chung wrote: >>> >>>> On Nov 11, 2016, at 3:13 AM, Daniel Fuchs >>>> wrote: >>>> >>>> Hi, >>>> >>>> On 10/11/16 23:24, David Holmes wrote: >>>>> Hi Rachel, >>>>> >>>>> On 11/11/2016 8:31 AM, Rachel Protacio wrote: >>>>>> Ah yes, my apologies - I should have provided more context. This is >>>>>> the >>>>>> VM corollary of Mandy's change >>>>>> http://hg.openjdk.java.net/jdk9/dev/hotspot/rev/48fce55afe71 which >>>>>> implements the new classloader name scheme and uses it in Stack >>>>>> Traces. >>>>>> In the fullest instance, it could look like >>>>>> MyClassLoader/module1 at 9.0/package1.class1 >>>>>> The class loader is not listed if it is built-in, the module is not >>>>>> listed if it is unnamed, the version is not listed if it is a >>>>>> "non-jdk" >>>>>> module. >>>>> >>>>> Ah I see. Though not at all clear why we would care about showing the >>>>> JDK module version ?? >>>> >>>> Mandy will correct me if I'm wrong, but AFAIU, for JDK modules, >>>> the version is displayed only if the module is an upgradeable >>>> module. The relevant algorithm [1] that does that can be seen in >>>> StackTraceElement::toLoaderModuleClassName(Class): >>>> >>> >>> That is correct and that?s the format we implemented for the stack >>> trace. >>> >>> The VM does not have an easy way to determine if a JDK module is >>> upgradeable or not. There are a couple options that we considered for >>> example include such information when defining the module. The reason >>> to include a version in an upgradeable module so that from the stack >>> trace it can tell if the class is not the version in the JDK. So we >>> came up with this option such that the version is omitted if it?s a >>> JDK module coming from the image that includes upgradeable modules >>> that are not upgraded. If these upgradeable modules are loaded from >>> image, they are the same version as the runtime. If the module is >>> upgraded, the version will be included. >>> >>> We agree to not to change the JVM_DefineModule interface since jake >>> changes are in flight. The exception message will have a small >>> difference but really minor and still provides the useful information >>> for troubleshooting. >> >> But what is being proposed for the ClassCastException message is almost >> the complete opposite of what was done for the stacktrace. This needs to >> be reconciled IMO, even if the VM can't deal with the upgradeable part. > > Okay there is a difference between the code and how Rachel described > it. The code: > > 1977 // Use version if exists and is not a jdk module > 1978 if (module->is_non_jdk_module() && module->version() != > NULL) { > > What Rachel said: "the version is not listed if it is a "non-jdk" > module." > > So I'll take the "not" as a typo and all is good. :) Yes, I got caught up by the double-negation with the function name. The version is not listed if it is a "jdk" module. Rachel > > Cheers, > David > >> Thanks, >> David >> >>> Mandy >>> >>>> >>>> [1] >>>> http://hg.openjdk.java.net/jdk9/dev/jdk/file/5b6b8e24a20b/src/java.base/share/classes/java/lang/StackTraceElement.java#l414 >>>> >>>> >>>> >>>> best regards, >>>> >>>> -- daniel >>>> >>>> >>> From coleen.phillimore at oracle.com Mon Nov 14 16:49:07 2016 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Mon, 14 Nov 2016 11:49:07 -0500 Subject: RFR(XS) 8169551: Segfaults in error traces when module version is null In-Reply-To: References: Message-ID: This looks okay. Coleen On 11/14/16 9:02 AM, harold seigel wrote: > Hi, > > Please review this small fix for JDK-8169551. > > Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8169551/ > > JBS bug: https://bugs.openjdk.java.net/browse/JDK-8169551 > > The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, > java/io, java/lang, and java/util tests, the nsk co-located and the > non-co-located tests, and the RBT tier2 tests. > > Thanks, Harold > From harold.seigel at oracle.com Mon Nov 14 16:57:37 2016 From: harold.seigel at oracle.com (harold seigel) Date: Mon, 14 Nov 2016 11:57:37 -0500 Subject: RFR(XS) 8169551: Segfaults in error traces when module version is null In-Reply-To: References: Message-ID: <28139c27-c4bd-feb7-f9f8-cd0500706330@oracle.com> Thanks Coleen! Harold On 11/14/2016 11:49 AM, Coleen Phillimore wrote: > This looks okay. > Coleen > > On 11/14/16 9:02 AM, harold seigel wrote: >> Hi, >> >> Please review this small fix for JDK-8169551. >> >> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8169551/ >> >> JBS bug: https://bugs.openjdk.java.net/browse/JDK-8169551 >> >> The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, >> java/io, java/lang, and java/util tests, the nsk co-located and the >> non-co-located tests, and the RBT tier2 tests. >> >> Thanks, Harold >> > From goetz.lindenmaier at sap.com Mon Nov 14 17:12:48 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Mon, 14 Nov 2016 17:12:48 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> Message-ID: <5e9b0cf6243e471287da233b16fe1f63@derote13de14.global.corp.sap> Hi David, > Did you really intend to increase the real minimum stack from 128K to > 256K ? (on a 64K page system) I finally figured why I have to set such high minimum values. A CompilerThread is a JavaThread. It's getting yellow, red etc pages at the top. As these are 64K on 64K systems, the compiler thread stack has to be this big. Thus you actually need similar values once you run a linuxx86_64 VM on a system with 64K pages. Does this exist? But, this means that there is no use of the OS guard pages configured in default_guard_size(). This is called with 'compiler_thread' for CompilerThreads. We should probably also return '0' in that case. What do you think? Best regards, Goetz. size_t os::Linux::default_guard_size(os::ThreadType thr_type) { // Creating guard page is very expensive. Java thread has HotSpot // guard page, only enable glibc guard page for non-Java threads. return (thr_type == java_thread ? 0 : page_size()); } From calvin.cheung at oracle.com Mon Nov 14 17:35:51 2016 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Mon, 14 Nov 2016 09:35:51 -0800 Subject: RFR(xs): 8168796: relax vm options checking during CDS dump time In-Reply-To: <0378f0d6-09ef-9ab2-13b2-cb4392ae7177@oracle.com> References: <5824CB84.8040309@oracle.com> <5BBF7048-407D-4890-9EE9-AC86EAF8FF6E@oracle.com> <582505D0.6040907@oracle.com> <0378f0d6-09ef-9ab2-13b2-cb4392ae7177@oracle.com> Message-ID: <5829F5F7.2030506@oracle.com> Thanks - Harold, Jiangli. Calvin On 11/14/16, 6:10 AM, harold seigel wrote: > Hi Calvin, > > It looks good! > > Thanks, Harold > > On 11/10/2016 6:42 PM, Calvin Cheung wrote: >> Hi Harold, Jiangli, >> >> Thanks for your review. >> >> I've updated the webrev based on your review comments: >> http://cr.openjdk.java.net/~ccheung/8168796/webrev.01/ >> >> thanks, >> Calvin >> >> On 11/10/16, 11:52 AM, Jiangli Zhou wrote: >>> +1 >>> >>> Thanks, >>> Jiangli >>> >>>> On Nov 10, 2016, at 11:42 AM, harold >>>> seigel wrote: >>>> >>>> Hi Calvin, >>>> >>>> The changes look good. Could you add a comment to the declaration >>>> of warning_idx explaining what it means to be above and below the >>>> that index? >>>> >>>> Also, instead of calling tty->print_cr(), can you call warning(), >>>> instead? >>>> >>>> Thanks, Harold >>>> >>>> >>>> On 11/10/2016 2:33 PM, Calvin Cheung wrote: >>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8168796 >>>>> >>>>> webrev: http://cr.openjdk.java.net/~ccheung/8168796/webrev.00/ >>>>> >>>>> Before this change, if any one of the following vm options >>>>> ("-module-path", "-add-modules", "--upgrade-module-path") is >>>>> specified during CDS dump time, the vm exits with an error as >>>>> follows: >>>>> >>>>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>>>> Error occurred during initialization of VM >>>>> Cannot use the following option when dumping the shared archive: >>>>> --upgrade-module-path >>>>> >>>>> This change will print a warning message and continues as follows: >>>>> >>>>> bash-4.1$ java --upgrade-module-path=/tmp -Xshare:dump >>>>> Warning: the --upgrade-module-path option is ignored when dumping >>>>> the shared archive >>>>> Allocated shared space: 50577408 bytes at 0x0000000800000000 >>>>> Loading classes to share ... >>>>> >>>>> Test case is in the hotspot closed repo. >>>>> >>>>> Testing: >>>>> JPRT >>>>> all tests under hotspot/test/runtime >>>>> >>>>> thanks, >>>>> Calvin >>>>> >>>>> >>>>> > From jiangli.zhou at oracle.com Mon Nov 14 18:36:24 2016 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 14 Nov 2016 10:36:24 -0800 Subject: RFR(s): 8168790: CDS: assert(max_delta <= (0x7FFFFFFF)) failed: range check In-Reply-To: <58293CEB.4080408@oracle.com> References: <3B0B08DC-9FBA-4390-A372-AB99A30BE974@oracle.com> <58293CEB.4080408@oracle.com> Message-ID: Hi Ioi and Dmitry, Thank you both for the review and comments! Those are good suggestions. Here is the updated webrev with updated test. No change to compactHashtable.cpp. http://cr.openjdk.java.net/~jiangli/8168790/webrev.01/ Thanks! Jiangli From rachel.protacio at oracle.com Mon Nov 14 21:00:07 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Mon, 14 Nov 2016 16:00:07 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: Message-ID: New webrev: http://cr.openjdk.java.net/~rprotacio/8165550.01/ Corrected according to David's review. Comments below. On 11/10/2016 6:24 PM, David Holmes wrote: > Okay on with the review: > > src/share/vm/classfile/moduleEntry.cpp > > + const char* loc = ""; > + loc = location()->as_C_string(); > > can just be: > > + const char* loc = location()->as_C_string(); > Fixed. > --- > > src/share/vm/runtime/sharedRuntime.cpp > > 1941 const char* class_loader_and_module_name(Klass* klass, Klass* > base_klass) { > > I'm unclear in what way "base_klass" is a "base" class ?? And why, if > it is not an instance class must be it in java.base ?? I really meant "bottom" class, following the logic used to parse it out. I've moved that internal to the function now. So if the klass is an instance klass or instance klass array, we look for further information. Otherwise (if it's a primitive array), it's from java.base. > > Nit: I don't think these need to be #defines nor have such verbose names: > > 1936 #define CLASS_LOADER_NAME_DELIMITER "/" > 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH > strlen(CLASS_LOADER_NAME_DELIMITER) > > why not just > > const char* delim = "\"; > int delim_len = strlen(delim); > > Plus you are inconsistent with the handling of the "/" delimiter > versus the "@" delimiter. Fixed. > > --- > > test/runtime/modules/CCE_module_msg.java > > Nit: in the catch block from line #95 can you store getMessage() into > a local so it isn't called 5 times. Thanks. :) Fixed too. Thanks! Rachel > > Thanks, > David > >> Rachel >> >> On 11/10/2016 5:02 PM, David Holmes wrote: >>> Hi Rachel, >>> >>> Just wondering, in the most complex case, exactly what will this new >>> message look like? We seem to be including module names as well as >>> classloader names, and of course class names. >>> >>> Arguably we should only add the additional info when they differ >>> between the types ie if both in same classloader and same module then >>> only report class name; if same loader but different module report >>> module:class; if different loaders then report everything >>> loader:module:class. Also not sure module version is needed at all in >>> this message. >>> >>> Thanks, >>> David >>> >>> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>>> Hi, >>>> >>>> Please review this Jigsaw enhancement which adds the new format of >>>> class >>>> loader names to VM ClassCastException messages. Includes an >>>> updated/expanded jtreg test. >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>>> >>>> Passes JPRT and jck vm and lang tests. >>>> >>>> Thanks! >>>> Rachel >> From ioi.lam at oracle.com Mon Nov 14 21:19:22 2016 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 14 Nov 2016 13:19:22 -0800 Subject: RFR(s): 8168790: CDS: assert(max_delta <= (0x7FFFFFFF)) failed: range check In-Reply-To: References: <3B0B08DC-9FBA-4390-A372-AB99A30BE974@oracle.com> <58293CEB.4080408@oracle.com> Message-ID: <582A2A5A.7070704@oracle.com> Looks good. Thanks Jiangli. - Ioi On 11/14/16 10:36 AM, Jiangli Zhou wrote: > Hi Ioi and Dmitry, > > Thank you both for the review and comments! Those are good > suggestions. Here is the updated webrev with updated test. No change > to compactHashtable.cpp. > > http://cr.openjdk.java.net/~jiangli/8168790/webrev.01/ > > > Thanks! > Jiangli > > From jiangli.zhou at oracle.com Mon Nov 14 21:36:29 2016 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 14 Nov 2016 13:36:29 -0800 Subject: RFR(s): 8168790: CDS: assert(max_delta <= (0x7FFFFFFF)) failed: range check In-Reply-To: <582A2A5A.7070704@oracle.com> References: <3B0B08DC-9FBA-4390-A372-AB99A30BE974@oracle.com> <58293CEB.4080408@oracle.com> <582A2A5A.7070704@oracle.com> Message-ID: <0BB43D0D-FE2F-4896-B473-DBB6876893A2@oracle.com> Thanks. Jiangli > On Nov 14, 2016, at 1:19 PM, Ioi Lam wrote: > > Looks good. Thanks Jiangli. > > - Ioi > > On 11/14/16 10:36 AM, Jiangli Zhou wrote: >> Hi Ioi and Dmitry, >> >> Thank you both for the review and comments! Those are good suggestions. Here is the updated webrev with updated test. No change to compactHashtable.cpp. >> >> http://cr.openjdk.java.net/~jiangli/8168790/webrev.01/ >> >> Thanks! >> Jiangli >> >> > From david.holmes at oracle.com Mon Nov 14 23:43:15 2016 From: david.holmes at oracle.com (David Holmes) Date: Tue, 15 Nov 2016 09:43:15 +1000 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: Message-ID: <7f6a01ef-e86f-8e75-b237-2830b2d08bba@oracle.com> Hi Rachel, On 15/11/2016 7:00 AM, Rachel Protacio wrote: > New webrev: http://cr.openjdk.java.net/~rprotacio/8165550.01/ > > Corrected according to David's review. Comments below. Thanks. Only one follow up that I missed the first time (and which is a pre-existing bug) in sharedRuntime.cpp: 1996 char* message = NEW_RESOURCE_ARRAY(char, msglen); 1997 1998 // Just return the FQN if error in allocating string 1999 if (message == NULL) { 2000 return fqn; 2001 } If NEW_RESOURCE_ARRAY can't allocate it will not return but cause vm_exit_out_of_memory to be called. We should be using NEW_RESOURCE_ARRAY_RETURN_NULL here. Thanks, David ----- > > On 11/10/2016 6:24 PM, David Holmes wrote: >> Okay on with the review: >> >> src/share/vm/classfile/moduleEntry.cpp >> >> + const char* loc = ""; >> + loc = location()->as_C_string(); >> >> can just be: >> >> + const char* loc = location()->as_C_string(); >> > Fixed. >> --- >> >> src/share/vm/runtime/sharedRuntime.cpp >> >> 1941 const char* class_loader_and_module_name(Klass* klass, Klass* >> base_klass) { >> >> I'm unclear in what way "base_klass" is a "base" class ?? And why, if >> it is not an instance class must be it in java.base ?? > I really meant "bottom" class, following the logic used to parse it out. > I've moved that internal to the function now. So if the klass is an > instance klass or instance klass array, we look for further information. > Otherwise (if it's a primitive array), it's from java.base. >> >> Nit: I don't think these need to be #defines nor have such verbose names: >> >> 1936 #define CLASS_LOADER_NAME_DELIMITER "/" >> 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH >> strlen(CLASS_LOADER_NAME_DELIMITER) >> >> why not just >> >> const char* delim = "\"; >> int delim_len = strlen(delim); >> >> Plus you are inconsistent with the handling of the "/" delimiter >> versus the "@" delimiter. > Fixed. >> >> --- >> >> test/runtime/modules/CCE_module_msg.java >> >> Nit: in the catch block from line #95 can you store getMessage() into >> a local so it isn't called 5 times. Thanks. :) > Fixed too. Thanks! > Rachel >> >> Thanks, >> David >> >>> Rachel >>> >>> On 11/10/2016 5:02 PM, David Holmes wrote: >>>> Hi Rachel, >>>> >>>> Just wondering, in the most complex case, exactly what will this new >>>> message look like? We seem to be including module names as well as >>>> classloader names, and of course class names. >>>> >>>> Arguably we should only add the additional info when they differ >>>> between the types ie if both in same classloader and same module then >>>> only report class name; if same loader but different module report >>>> module:class; if different loaders then report everything >>>> loader:module:class. Also not sure module version is needed at all in >>>> this message. >>>> >>>> Thanks, >>>> David >>>> >>>> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>>>> Hi, >>>>> >>>>> Please review this Jigsaw enhancement which adds the new format of >>>>> class >>>>> loader names to VM ClassCastException messages. Includes an >>>>> updated/expanded jtreg test. >>>>> >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>>>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>>>> >>>>> Passes JPRT and jck vm and lang tests. >>>>> >>>>> Thanks! >>>>> Rachel >>> > From david.holmes at oracle.com Mon Nov 14 23:54:27 2016 From: david.holmes at oracle.com (David Holmes) Date: Tue, 15 Nov 2016 09:54:27 +1000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <5e9b0cf6243e471287da233b16fe1f63@derote13de14.global.corp.sap> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <5e9b0cf6243e471287da233b16fe1f63@derote13de14.global.corp.sap> Message-ID: Hi Goetz, On 15/11/2016 3:12 AM, Lindenmaier, Goetz wrote: > Hi David, > >> Did you really intend to increase the real minimum stack from 128K to >> 256K ? (on a 64K page system) > I finally figured why I have to set such high minimum values. > > A CompilerThread is a JavaThread. It's getting yellow, red etc > pages at the top. As these are 64K on 64K systems, the compiler > thread stack has to be this big. Thus you actually need similar > values once you run a linuxx86_64 VM on a system with 64K pages. > Does this exist? No idea, sorry. > But, this means that there is no use of the OS guard pages > configured in default_guard_size(). This is called with 'compiler_thread' > for CompilerThreads. > > We should probably also return '0' in that case. > > What do you think? I think I agree. As you note a CompilerThread is-a JavaThread even though the ThreadType enum treats them as completely distinct: enum ThreadType { vm_thread, cgc_thread, // Concurrent GC thread pgc_thread, // Parallel GC thread java_thread, compiler_thread, watcher_thread, os_thread }; so anything with VM supplied guard pages doesn't need the glibc ones as well. I'm struggling to understand why os::Linux::default_guard_size is defined per CPU type? I would not expect this to need to vary based on CPU at all. ?? Thanks, David > Best regards, > Goetz. > > size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > // Creating guard page is very expensive. Java thread has HotSpot > // guard page, only enable glibc guard page for non-Java threads. > return (thr_type == java_thread ? 0 : page_size()); > } > From lois.foltan at oracle.com Tue Nov 15 12:30:15 2016 From: lois.foltan at oracle.com (Lois Foltan) Date: Tue, 15 Nov 2016 07:30:15 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <7f6a01ef-e86f-8e75-b237-2830b2d08bba@oracle.com> References: <7f6a01ef-e86f-8e75-b237-2830b2d08bba@oracle.com> Message-ID: <582AFFD7.6060104@oracle.com> On 11/14/2016 6:43 PM, David Holmes wrote: > Hi Rachel, > > On 15/11/2016 7:00 AM, Rachel Protacio wrote: >> New webrev: http://cr.openjdk.java.net/~rprotacio/8165550.01/ >> >> Corrected according to David's review. Comments below. > > Thanks. Only one follow up that I missed the first time (and which is > a pre-existing bug) in sharedRuntime.cpp: > > 1996 char* message = NEW_RESOURCE_ARRAY(char, msglen); > 1997 > 1998 // Just return the FQN if error in allocating string > 1999 if (message == NULL) { > 2000 return fqn; > 2001 } > > If NEW_RESOURCE_ARRAY can't allocate it will not return but cause > vm_exit_out_of_memory to be called. We should be using > NEW_RESOURCE_ARRAY_RETURN_NULL here. I think line #2023 in SharedRuntime::generate_class_cast_message() should be changed in the same manner as well? Lois > > Thanks, > David > ----- > >> >> On 11/10/2016 6:24 PM, David Holmes wrote: >>> Okay on with the review: >>> >>> src/share/vm/classfile/moduleEntry.cpp >>> >>> + const char* loc = ""; >>> + loc = location()->as_C_string(); >>> >>> can just be: >>> >>> + const char* loc = location()->as_C_string(); >>> >> Fixed. >>> --- >>> >>> src/share/vm/runtime/sharedRuntime.cpp >>> >>> 1941 const char* class_loader_and_module_name(Klass* klass, Klass* >>> base_klass) { >>> >>> I'm unclear in what way "base_klass" is a "base" class ?? And why, if >>> it is not an instance class must be it in java.base ?? >> I really meant "bottom" class, following the logic used to parse it out. >> I've moved that internal to the function now. So if the klass is an >> instance klass or instance klass array, we look for further information. >> Otherwise (if it's a primitive array), it's from java.base. >>> >>> Nit: I don't think these need to be #defines nor have such verbose >>> names: >>> >>> 1936 #define CLASS_LOADER_NAME_DELIMITER "/" >>> 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH >>> strlen(CLASS_LOADER_NAME_DELIMITER) >>> >>> why not just >>> >>> const char* delim = "\"; >>> int delim_len = strlen(delim); >>> >>> Plus you are inconsistent with the handling of the "/" delimiter >>> versus the "@" delimiter. >> Fixed. >>> >>> --- >>> >>> test/runtime/modules/CCE_module_msg.java >>> >>> Nit: in the catch block from line #95 can you store getMessage() into >>> a local so it isn't called 5 times. Thanks. :) >> Fixed too. Thanks! >> Rachel >>> >>> Thanks, >>> David >>> >>>> Rachel >>>> >>>> On 11/10/2016 5:02 PM, David Holmes wrote: >>>>> Hi Rachel, >>>>> >>>>> Just wondering, in the most complex case, exactly what will this new >>>>> message look like? We seem to be including module names as well as >>>>> classloader names, and of course class names. >>>>> >>>>> Arguably we should only add the additional info when they differ >>>>> between the types ie if both in same classloader and same module then >>>>> only report class name; if same loader but different module report >>>>> module:class; if different loaders then report everything >>>>> loader:module:class. Also not sure module version is needed at all in >>>>> this message. >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>>>>> Hi, >>>>>> >>>>>> Please review this Jigsaw enhancement which adds the new format of >>>>>> class >>>>>> loader names to VM ClassCastException messages. Includes an >>>>>> updated/expanded jtreg test. >>>>>> >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>>>>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>>>>> >>>>>> Passes JPRT and jck vm and lang tests. >>>>>> >>>>>> Thanks! >>>>>> Rachel >>>> >> From lois.foltan at oracle.com Tue Nov 15 12:36:03 2016 From: lois.foltan at oracle.com (Lois Foltan) Date: Tue, 15 Nov 2016 07:36:03 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: Message-ID: <582B0133.1010001@oracle.com> On 11/14/2016 4:00 PM, Rachel Protacio wrote: > New webrev: http://cr.openjdk.java.net/~rprotacio/8165550.01/ Hi Rachel, Looks good. One comment about hard coding the string "java.base" on line #1989. Please consider obtaining the string for the java.base module from vmSymbols::java_base()->as_C_string(). Thanks, Lois > > Corrected according to David's review. Comments below. > > > On 11/10/2016 6:24 PM, David Holmes wrote: >> Okay on with the review: >> >> src/share/vm/classfile/moduleEntry.cpp >> >> + const char* loc = ""; >> + loc = location()->as_C_string(); >> >> can just be: >> >> + const char* loc = location()->as_C_string(); >> > Fixed. >> --- >> >> src/share/vm/runtime/sharedRuntime.cpp >> >> 1941 const char* class_loader_and_module_name(Klass* klass, Klass* >> base_klass) { >> >> I'm unclear in what way "base_klass" is a "base" class ?? And why, if >> it is not an instance class must be it in java.base ?? > I really meant "bottom" class, following the logic used to parse it > out. I've moved that internal to the function now. So if the klass is > an instance klass or instance klass array, we look for further > information. Otherwise (if it's a primitive array), it's from java.base. >> >> Nit: I don't think these need to be #defines nor have such verbose >> names: >> >> 1936 #define CLASS_LOADER_NAME_DELIMITER "/" >> 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH >> strlen(CLASS_LOADER_NAME_DELIMITER) >> >> why not just >> >> const char* delim = "\"; >> int delim_len = strlen(delim); >> >> Plus you are inconsistent with the handling of the "/" delimiter >> versus the "@" delimiter. > Fixed. >> >> --- >> >> test/runtime/modules/CCE_module_msg.java >> >> Nit: in the catch block from line #95 can you store getMessage() into >> a local so it isn't called 5 times. Thanks. :) > Fixed too. Thanks! > Rachel >> >> Thanks, >> David >> >>> Rachel >>> >>> On 11/10/2016 5:02 PM, David Holmes wrote: >>>> Hi Rachel, >>>> >>>> Just wondering, in the most complex case, exactly what will this new >>>> message look like? We seem to be including module names as well as >>>> classloader names, and of course class names. >>>> >>>> Arguably we should only add the additional info when they differ >>>> between the types ie if both in same classloader and same module then >>>> only report class name; if same loader but different module report >>>> module:class; if different loaders then report everything >>>> loader:module:class. Also not sure module version is needed at all in >>>> this message. >>>> >>>> Thanks, >>>> David >>>> >>>> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>>>> Hi, >>>>> >>>>> Please review this Jigsaw enhancement which adds the new format of >>>>> class >>>>> loader names to VM ClassCastException messages. Includes an >>>>> updated/expanded jtreg test. >>>>> >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>>>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>>>> >>>>> Passes JPRT and jck vm and lang tests. >>>>> >>>>> Thanks! >>>>> Rachel >>> > From harold.seigel at oracle.com Tue Nov 15 14:09:31 2016 From: harold.seigel at oracle.com (harold seigel) Date: Tue, 15 Nov 2016 09:09:31 -0500 Subject: RFR(S) 8166304: Skipping access check for classes generated by core reflection Message-ID: Hi, Please review this fix for bug JDK-8166304. The fix throws an IllegalAccessError exception when attempting to create a class that is a sub-type of a jdk.internal.reflect class, unless the class loader is either the boot loader or the special reflection class loader. This prevents user classes from being able to extend jdk.internal.reflect classes in order to bypass Reflection.getCallerClass. Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8166304/ JBS bug: https://bugs.openjdk.java.net/browse/JDK-8166304 The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, java/io, java/lang, and java/util tests, the nsk co-located and the non-co-located tests, the RBT tier2 tests, and the test in the webrev. A sample error message looks like this: java.lang.IllegalAccessError: class FakeMethodAccessor loaded by jdk/internal/loader/ClassLoaders$AppClassLoader cannot access jdk/internal/reflect superclass jdk.internal.reflect.MethodAccessorImpl Thanks, Harold From harold.seigel at oracle.com Tue Nov 15 14:38:05 2016 From: harold.seigel at oracle.com (harold seigel) Date: Tue, 15 Nov 2016 09:38:05 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <582B0133.1010001@oracle.com> References: <582B0133.1010001@oracle.com> Message-ID: Please note that "java.base" is hard coded in multiple places in the JVM. For example: src/share/vm/runtime/arguments.cpp: if (strcmp(module_name, "java.base") == 0) { src/share/vm/classfile/classLoader.cpp: location = (*JImageFindResource)(_jimage, "java.base", get_jimage_version_s src/share/vm/classfile/modules.cpp: "Bad package name for module: java.base"); src/share/vm/oops/instanceKlass.cpp: assert(ModuleEntryTable::javabase_moduleEntry() != NULL, "java.base module is NULL"); Should we enter a new bug for this or just ignore it? Thanks, Harold On 11/15/2016 7:36 AM, Lois Foltan wrote: > > On 11/14/2016 4:00 PM, Rachel Protacio wrote: >> New webrev: http://cr.openjdk.java.net/~rprotacio/8165550.01/ > > Hi Rachel, > > Looks good. One comment about hard coding the string "java.base" on > line #1989. Please consider obtaining the string for the java.base > module from vmSymbols::java_base()->as_C_string(). > > Thanks, > Lois > >> >> Corrected according to David's review. Comments below. >> >> >> On 11/10/2016 6:24 PM, David Holmes wrote: >>> Okay on with the review: >>> >>> src/share/vm/classfile/moduleEntry.cpp >>> >>> + const char* loc = ""; >>> + loc = location()->as_C_string(); >>> >>> can just be: >>> >>> + const char* loc = location()->as_C_string(); >>> >> Fixed. >>> --- >>> >>> src/share/vm/runtime/sharedRuntime.cpp >>> >>> 1941 const char* class_loader_and_module_name(Klass* klass, Klass* >>> base_klass) { >>> >>> I'm unclear in what way "base_klass" is a "base" class ?? And why, >>> if it is not an instance class must be it in java.base ?? >> I really meant "bottom" class, following the logic used to parse it >> out. I've moved that internal to the function now. So if the klass is >> an instance klass or instance klass array, we look for further >> information. Otherwise (if it's a primitive array), it's from java.base. >>> >>> Nit: I don't think these need to be #defines nor have such verbose >>> names: >>> >>> 1936 #define CLASS_LOADER_NAME_DELIMITER "/" >>> 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH >>> strlen(CLASS_LOADER_NAME_DELIMITER) >>> >>> why not just >>> >>> const char* delim = "\"; >>> int delim_len = strlen(delim); >>> >>> Plus you are inconsistent with the handling of the "/" delimiter >>> versus the "@" delimiter. >> Fixed. >>> >>> --- >>> >>> test/runtime/modules/CCE_module_msg.java >>> >>> Nit: in the catch block from line #95 can you store getMessage() >>> into a local so it isn't called 5 times. Thanks. :) >> Fixed too. Thanks! >> Rachel >>> >>> Thanks, >>> David >>> >>>> Rachel >>>> >>>> On 11/10/2016 5:02 PM, David Holmes wrote: >>>>> Hi Rachel, >>>>> >>>>> Just wondering, in the most complex case, exactly what will this new >>>>> message look like? We seem to be including module names as well as >>>>> classloader names, and of course class names. >>>>> >>>>> Arguably we should only add the additional info when they differ >>>>> between the types ie if both in same classloader and same module then >>>>> only report class name; if same loader but different module report >>>>> module:class; if different loaders then report everything >>>>> loader:module:class. Also not sure module version is needed at all in >>>>> this message. >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>>>>> Hi, >>>>>> >>>>>> Please review this Jigsaw enhancement which adds the new format >>>>>> of class >>>>>> loader names to VM ClassCastException messages. Includes an >>>>>> updated/expanded jtreg test. >>>>>> >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>>>>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>>>>> >>>>>> Passes JPRT and jck vm and lang tests. >>>>>> >>>>>> Thanks! >>>>>> Rachel >>>> >> > From lois.foltan at oracle.com Tue Nov 15 14:45:05 2016 From: lois.foltan at oracle.com (Lois Foltan) Date: Tue, 15 Nov 2016 09:45:05 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: <582B0133.1010001@oracle.com> Message-ID: <582B1F71.2080401@oracle.com> On 11/15/2016 9:38 AM, harold seigel wrote: > Please note that "java.base" is hard coded in multiple places in the > JVM. For example: > > src/share/vm/runtime/arguments.cpp: if (strcmp(module_name, > "java.base") == 0) { > > src/share/vm/classfile/classLoader.cpp: location = > (*JImageFindResource)(_jimage, "java.base", get_jimage_version_s > > src/share/vm/classfile/modules.cpp: "Bad package name > for module: java.base"); > > src/share/vm/oops/instanceKlass.cpp: > assert(ModuleEntryTable::javabase_moduleEntry() != NULL, "java.base > module is NULL"); > > Should we enter a new bug for this or just ignore it? Sure, enter a bug to clean those up as well. Lois > > Thanks, Harold > > > On 11/15/2016 7:36 AM, Lois Foltan wrote: >> >> On 11/14/2016 4:00 PM, Rachel Protacio wrote: >>> New webrev: http://cr.openjdk.java.net/~rprotacio/8165550.01/ >> >> Hi Rachel, >> >> Looks good. One comment about hard coding the string "java.base" on >> line #1989. Please consider obtaining the string for the java.base >> module from vmSymbols::java_base()->as_C_string(). >> >> Thanks, >> Lois >> >>> >>> Corrected according to David's review. Comments below. >>> >>> >>> On 11/10/2016 6:24 PM, David Holmes wrote: >>>> Okay on with the review: >>>> >>>> src/share/vm/classfile/moduleEntry.cpp >>>> >>>> + const char* loc = ""; >>>> + loc = location()->as_C_string(); >>>> >>>> can just be: >>>> >>>> + const char* loc = location()->as_C_string(); >>>> >>> Fixed. >>>> --- >>>> >>>> src/share/vm/runtime/sharedRuntime.cpp >>>> >>>> 1941 const char* class_loader_and_module_name(Klass* klass, Klass* >>>> base_klass) { >>>> >>>> I'm unclear in what way "base_klass" is a "base" class ?? And why, >>>> if it is not an instance class must be it in java.base ?? >>> I really meant "bottom" class, following the logic used to parse it >>> out. I've moved that internal to the function now. So if the klass >>> is an instance klass or instance klass array, we look for further >>> information. Otherwise (if it's a primitive array), it's from >>> java.base. >>>> >>>> Nit: I don't think these need to be #defines nor have such verbose >>>> names: >>>> >>>> 1936 #define CLASS_LOADER_NAME_DELIMITER "/" >>>> 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH >>>> strlen(CLASS_LOADER_NAME_DELIMITER) >>>> >>>> why not just >>>> >>>> const char* delim = "\"; >>>> int delim_len = strlen(delim); >>>> >>>> Plus you are inconsistent with the handling of the "/" delimiter >>>> versus the "@" delimiter. >>> Fixed. >>>> >>>> --- >>>> >>>> test/runtime/modules/CCE_module_msg.java >>>> >>>> Nit: in the catch block from line #95 can you store getMessage() >>>> into a local so it isn't called 5 times. Thanks. :) >>> Fixed too. Thanks! >>> Rachel >>>> >>>> Thanks, >>>> David >>>> >>>>> Rachel >>>>> >>>>> On 11/10/2016 5:02 PM, David Holmes wrote: >>>>>> Hi Rachel, >>>>>> >>>>>> Just wondering, in the most complex case, exactly what will this new >>>>>> message look like? We seem to be including module names as well as >>>>>> classloader names, and of course class names. >>>>>> >>>>>> Arguably we should only add the additional info when they differ >>>>>> between the types ie if both in same classloader and same module >>>>>> then >>>>>> only report class name; if same loader but different module report >>>>>> module:class; if different loaders then report everything >>>>>> loader:module:class. Also not sure module version is needed at >>>>>> all in >>>>>> this message. >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Please review this Jigsaw enhancement which adds the new format >>>>>>> of class >>>>>>> loader names to VM ClassCastException messages. Includes an >>>>>>> updated/expanded jtreg test. >>>>>>> >>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>>>>>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>>>>>> >>>>>>> Passes JPRT and jck vm and lang tests. >>>>>>> >>>>>>> Thanks! >>>>>>> Rachel >>>>> >>> >> > From igor.ignatyev at oracle.com Tue Nov 15 14:45:21 2016 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Tue, 15 Nov 2016 17:45:21 +0300 Subject: RFR(S): 8168492: Convert CollectedHeap_test to GTest In-Reply-To: <489f77cb-f7be-51f5-b602-9e2eab3ee807@oracle.com> References: <489f77cb-f7be-51f5-b602-9e2eab3ee807@oracle.com> Message-ID: <721A3D9B-36D2-41F1-B965-E46CED8F8FA8@oracle.com> Hi Kirill, looks good to me. Thanks, ? Igor > On Nov 1, 2016, at 5:10 PM, Kirill Zhaldybin wrote: > > Dear all, > > Could you please review this fix for 8168492? > > WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168492/webrev.00/ > CR: https://bugs.openjdk.java.net/browse/JDK-8168492 > > Changes: > 1. Changed _reserved to reserved_region() since _reserved is private but we already had an accessor. > 2. Added more detailed messages for sanity checks > 3. Otherwise pretty straight-forward conversion to gtest. > > Thank you. > > Regards, Kirill From kirill.zhaldybin at oracle.com Tue Nov 15 14:50:25 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Tue, 15 Nov 2016 17:50:25 +0300 Subject: RFR(S): 8168492: Convert CollectedHeap_test to GTest In-Reply-To: <721A3D9B-36D2-41F1-B965-E46CED8F8FA8@oracle.com> References: <489f77cb-f7be-51f5-b602-9e2eab3ee807@oracle.com> <721A3D9B-36D2-41F1-B965-E46CED8F8FA8@oracle.com> Message-ID: Igor, Thank you for review! Regards, Kirill On 15.11.2016 17:45, Igor Ignatyev wrote: > Hi Kirill, > > looks good to me. > > Thanks, > ? Igor >> On Nov 1, 2016, at 5:10 PM, Kirill Zhaldybin wrote: >> >> Dear all, >> >> Could you please review this fix for 8168492? >> >> WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168492/webrev.00/ >> CR: https://bugs.openjdk.java.net/browse/JDK-8168492 >> >> Changes: >> 1. Changed _reserved to reserved_region() since _reserved is private but we already had an accessor. >> 2. Added more detailed messages for sanity checks >> 3. Otherwise pretty straight-forward conversion to gtest. >> >> Thank you. >> >> Regards, Kirill From dmitry.samersoff at oracle.com Tue Nov 15 17:04:49 2016 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 15 Nov 2016 20:04:49 +0300 Subject: RFR: JDK-8160997 Solaris: deprecated and interfaces should be replaced In-Reply-To: <57839D8A.9050108@oracle.com> References: <57839D8A.9050108@oracle.com> Message-ID: <7f74a2da-e78f-4174-198c-889fb816c059@oracle.com> Alan, Solaris.c:73 if (getpwuid_r(getuid(), &pwd, pwd_buf, sizeof(pwd_buf), &p) != NULL getpwuid_r returns int and we have to check that p is not null. Otherwise looks good to me. -Dmitry On 2016-07-11 16:22, Alan Burlison wrote: > Hi, I'd like to ask for reviews for: > > JDK-8160997 Solaris: deprecated and interfaces should be > replaced > > This is to remove code that was a workaround that was needed for Solaris > 2.4. In 2.4 we implemented interfaces from the draft of POSIX.1c Draft 6 > which then changed in the final POSIX release, and we were then stuck > with the warts. > > For Solaris 11, defining the macro _POSIX_PTHREAD_SEMANTICS enables the > 'official' POSIX information and means that the code which works on > Linux etc also works the same on Solaris, so the Solaris-specific > workarounds can be removed. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8160997 > > Webrev: > http://cr.openjdk.java.net/~dcubed/for_alanbur/8160997-webrev/0-jdk9-hs-all/ > > > JPRT + the hotspot testset was clean. > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From rachel.protacio at oracle.com Tue Nov 15 18:57:32 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Tue, 15 Nov 2016 13:57:32 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <582AFFD7.6060104@oracle.com> References: <7f6a01ef-e86f-8e75-b237-2830b2d08bba@oracle.com> <582AFFD7.6060104@oracle.com> Message-ID: <8fe1432a-4b43-c16b-9097-c5978141eb03@oracle.com> Thanks, David and Lois - I'll fix both of these. Rachel On 11/15/2016 7:30 AM, Lois Foltan wrote: > > On 11/14/2016 6:43 PM, David Holmes wrote: >> Hi Rachel, >> >> On 15/11/2016 7:00 AM, Rachel Protacio wrote: >>> New webrev: http://cr.openjdk.java.net/~rprotacio/8165550.01/ >>> >>> Corrected according to David's review. Comments below. >> >> Thanks. Only one follow up that I missed the first time (and which is >> a pre-existing bug) in sharedRuntime.cpp: >> >> 1996 char* message = NEW_RESOURCE_ARRAY(char, msglen); >> 1997 >> 1998 // Just return the FQN if error in allocating string >> 1999 if (message == NULL) { >> 2000 return fqn; >> 2001 } >> >> If NEW_RESOURCE_ARRAY can't allocate it will not return but cause >> vm_exit_out_of_memory to be called. We should be using >> NEW_RESOURCE_ARRAY_RETURN_NULL here. > > I think line #2023 in SharedRuntime::generate_class_cast_message() > should be changed in the same manner as well? > Lois > >> >> Thanks, >> David >> ----- >> >>> >>> On 11/10/2016 6:24 PM, David Holmes wrote: >>>> Okay on with the review: >>>> >>>> src/share/vm/classfile/moduleEntry.cpp >>>> >>>> + const char* loc = ""; >>>> + loc = location()->as_C_string(); >>>> >>>> can just be: >>>> >>>> + const char* loc = location()->as_C_string(); >>>> >>> Fixed. >>>> --- >>>> >>>> src/share/vm/runtime/sharedRuntime.cpp >>>> >>>> 1941 const char* class_loader_and_module_name(Klass* klass, Klass* >>>> base_klass) { >>>> >>>> I'm unclear in what way "base_klass" is a "base" class ?? And why, if >>>> it is not an instance class must be it in java.base ?? >>> I really meant "bottom" class, following the logic used to parse it >>> out. >>> I've moved that internal to the function now. So if the klass is an >>> instance klass or instance klass array, we look for further >>> information. >>> Otherwise (if it's a primitive array), it's from java.base. >>>> >>>> Nit: I don't think these need to be #defines nor have such verbose >>>> names: >>>> >>>> 1936 #define CLASS_LOADER_NAME_DELIMITER "/" >>>> 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH >>>> strlen(CLASS_LOADER_NAME_DELIMITER) >>>> >>>> why not just >>>> >>>> const char* delim = "\"; >>>> int delim_len = strlen(delim); >>>> >>>> Plus you are inconsistent with the handling of the "/" delimiter >>>> versus the "@" delimiter. >>> Fixed. >>>> >>>> --- >>>> >>>> test/runtime/modules/CCE_module_msg.java >>>> >>>> Nit: in the catch block from line #95 can you store getMessage() into >>>> a local so it isn't called 5 times. Thanks. :) >>> Fixed too. Thanks! >>> Rachel >>>> >>>> Thanks, >>>> David >>>> >>>>> Rachel >>>>> >>>>> On 11/10/2016 5:02 PM, David Holmes wrote: >>>>>> Hi Rachel, >>>>>> >>>>>> Just wondering, in the most complex case, exactly what will this new >>>>>> message look like? We seem to be including module names as well as >>>>>> classloader names, and of course class names. >>>>>> >>>>>> Arguably we should only add the additional info when they differ >>>>>> between the types ie if both in same classloader and same module >>>>>> then >>>>>> only report class name; if same loader but different module report >>>>>> module:class; if different loaders then report everything >>>>>> loader:module:class. Also not sure module version is needed at >>>>>> all in >>>>>> this message. >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Please review this Jigsaw enhancement which adds the new format of >>>>>>> class >>>>>>> loader names to VM ClassCastException messages. Includes an >>>>>>> updated/expanded jtreg test. >>>>>>> >>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>>>>>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>>>>>> >>>>>>> Passes JPRT and jck vm and lang tests. >>>>>>> >>>>>>> Thanks! >>>>>>> Rachel >>>>> >>> > From lois.foltan at oracle.com Tue Nov 15 20:40:50 2016 From: lois.foltan at oracle.com (Lois Foltan) Date: Tue, 15 Nov 2016 15:40:50 -0500 Subject: RFR(S) 8166304: Skipping access check for classes generated by core reflection In-Reply-To: References: Message-ID: <582B72D2.6030506@oracle.com> On 11/15/2016 9:09 AM, harold seigel wrote: > Hi, > > Please review this fix for bug JDK-8166304. The fix throws an > IllegalAccessError exception when attempting to create a class that is > a sub-type of a jdk.internal.reflect class, unless the class loader is > either the boot loader or the special reflection class loader. This > prevents user classes from being able to extend jdk.internal.reflect > classes in order to bypass Reflection.getCallerClass. > > Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8166304/ Hi Harold, A couple of comments. src/share/vm/classfile/classFileParser.cpp line #4360 - I think you should reconfigure the if conditional. Make the check for the boot class loader first so a fast exit from the conditional exists if this is the boot class loader. In other words don't compare the symbol names. Also, use a Symbol::fast_compare instead of "==" to compare the name of the packages. line #4362 - I would like to understand better the following check 4362 this_klass->class_loader() == 4363 java_lang_ClassLoader::non_reflection_class_loader(this_klass->class_loader()) When this_klass's class loader data's _class_loader field is set the non_reflection_class_loader is not called, so I am having difficulty understanding how this conditional could ever be true. I am curious why the check isn't instead, "this_klass->class_loader()->is_a(SystemDictionary::reflect_DelegatingClassLoader_klass())"? Thanks, Lois > > JBS bug: https://bugs.openjdk.java.net/browse/JDK-8166304 > > The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, > java/io, java/lang, and java/util tests, the nsk co-located and the > non-co-located tests, the RBT tier2 tests, and the test in the webrev. > > A sample error message looks like this: > > java.lang.IllegalAccessError: class FakeMethodAccessor loaded by > jdk/internal/loader/ClassLoaders$AppClassLoader cannot access > jdk/internal/reflect superclass jdk.internal.reflect.MethodAccessorImpl > > Thanks, Harold > From rachel.protacio at oracle.com Tue Nov 15 20:49:26 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Tue, 15 Nov 2016 15:49:26 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <582B1F71.2080401@oracle.com> References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> Message-ID: <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> Updated webrev: http://cr.openjdk.java.net/~rprotacio/8165550.02/ - added macro in ModuleEntry.hpp for "java.base" to address Lois' comment (opened enhancement https://bugs.openjdk.java.net/browse/JDK-8169734 for other existing cases) - added an assert instead of a NULL check for the class loader data - fixed NEW_RESOURCE_ARRAY_RETURN_NULL calls Thanks! Rachel On 11/15/2016 9:45 AM, Lois Foltan wrote: > > On 11/15/2016 9:38 AM, harold seigel wrote: >> Please note that "java.base" is hard coded in multiple places in the >> JVM. For example: >> >> src/share/vm/runtime/arguments.cpp: if (strcmp(module_name, >> "java.base") == 0) { >> >> src/share/vm/classfile/classLoader.cpp: location = >> (*JImageFindResource)(_jimage, "java.base", get_jimage_version_s >> >> src/share/vm/classfile/modules.cpp: "Bad package name >> for module: java.base"); >> >> src/share/vm/oops/instanceKlass.cpp: >> assert(ModuleEntryTable::javabase_moduleEntry() != NULL, "java.base >> module is NULL"); >> >> Should we enter a new bug for this or just ignore it? > > Sure, enter a bug to clean those up as well. > Lois > >> >> Thanks, Harold >> >> >> On 11/15/2016 7:36 AM, Lois Foltan wrote: >>> >>> On 11/14/2016 4:00 PM, Rachel Protacio wrote: >>>> New webrev: http://cr.openjdk.java.net/~rprotacio/8165550.01/ >>> >>> Hi Rachel, >>> >>> Looks good. One comment about hard coding the string "java.base" on >>> line #1989. Please consider obtaining the string for the java.base >>> module from vmSymbols::java_base()->as_C_string(). >>> >>> Thanks, >>> Lois >>> >>>> >>>> Corrected according to David's review. Comments below. >>>> >>>> >>>> On 11/10/2016 6:24 PM, David Holmes wrote: >>>>> Okay on with the review: >>>>> >>>>> src/share/vm/classfile/moduleEntry.cpp >>>>> >>>>> + const char* loc = ""; >>>>> + loc = location()->as_C_string(); >>>>> >>>>> can just be: >>>>> >>>>> + const char* loc = location()->as_C_string(); >>>>> >>>> Fixed. >>>>> --- >>>>> >>>>> src/share/vm/runtime/sharedRuntime.cpp >>>>> >>>>> 1941 const char* class_loader_and_module_name(Klass* klass, Klass* >>>>> base_klass) { >>>>> >>>>> I'm unclear in what way "base_klass" is a "base" class ?? And why, >>>>> if it is not an instance class must be it in java.base ?? >>>> I really meant "bottom" class, following the logic used to parse it >>>> out. I've moved that internal to the function now. So if the klass >>>> is an instance klass or instance klass array, we look for further >>>> information. Otherwise (if it's a primitive array), it's from >>>> java.base. >>>>> >>>>> Nit: I don't think these need to be #defines nor have such verbose >>>>> names: >>>>> >>>>> 1936 #define CLASS_LOADER_NAME_DELIMITER "/" >>>>> 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH >>>>> strlen(CLASS_LOADER_NAME_DELIMITER) >>>>> >>>>> why not just >>>>> >>>>> const char* delim = "\"; >>>>> int delim_len = strlen(delim); >>>>> >>>>> Plus you are inconsistent with the handling of the "/" delimiter >>>>> versus the "@" delimiter. >>>> Fixed. >>>>> >>>>> --- >>>>> >>>>> test/runtime/modules/CCE_module_msg.java >>>>> >>>>> Nit: in the catch block from line #95 can you store getMessage() >>>>> into a local so it isn't called 5 times. Thanks. :) >>>> Fixed too. Thanks! >>>> Rachel >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> Rachel >>>>>> >>>>>> On 11/10/2016 5:02 PM, David Holmes wrote: >>>>>>> Hi Rachel, >>>>>>> >>>>>>> Just wondering, in the most complex case, exactly what will this >>>>>>> new >>>>>>> message look like? We seem to be including module names as well as >>>>>>> classloader names, and of course class names. >>>>>>> >>>>>>> Arguably we should only add the additional info when they differ >>>>>>> between the types ie if both in same classloader and same module >>>>>>> then >>>>>>> only report class name; if same loader but different module report >>>>>>> module:class; if different loaders then report everything >>>>>>> loader:module:class. Also not sure module version is needed at >>>>>>> all in >>>>>>> this message. >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> Please review this Jigsaw enhancement which adds the new format >>>>>>>> of class >>>>>>>> loader names to VM ClassCastException messages. Includes an >>>>>>>> updated/expanded jtreg test. >>>>>>>> >>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>>>>>>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>>>>>>> >>>>>>>> Passes JPRT and jck vm and lang tests. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> Rachel >>>>>> >>>> >>> >> > From mandy.chung at oracle.com Tue Nov 15 21:03:13 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 15 Nov 2016 13:03:13 -0800 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> Message-ID: > On Nov 15, 2016, at 12:49 PM, Rachel Protacio wrote: > > Updated webrev: http://cr.openjdk.java.net/~rprotacio/8165550.02/ > - added macro in ModuleEntry.hpp for "java.base" to address Lois' comment (opened enhancement https://bugs.openjdk.java.net/browse/JDK-8169734 for other existing cases) > - added an assert instead of a NULL check for the class loader data > - fixed NEW_RESOURCE_ARRAY_RETURN_NULL calls Looks okay to me. It?d be good if you can add a test case testing a custom named class loader with unnamed module. Mandy From david.holmes at oracle.com Wed Nov 16 05:18:04 2016 From: david.holmes at oracle.com (David Holmes) Date: Wed, 16 Nov 2016 15:18:04 +1000 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> Message-ID: Looks good Rachel! No further comments from me. Thanks, David On 16/11/2016 6:49 AM, Rachel Protacio wrote: > Updated webrev: http://cr.openjdk.java.net/~rprotacio/8165550.02/ > - added macro in ModuleEntry.hpp for "java.base" to address Lois' > comment (opened enhancement > https://bugs.openjdk.java.net/browse/JDK-8169734 for other existing cases) > - added an assert instead of a NULL check for the class loader data > - fixed NEW_RESOURCE_ARRAY_RETURN_NULL calls > > Thanks! > Rachel > > On 11/15/2016 9:45 AM, Lois Foltan wrote: >> >> On 11/15/2016 9:38 AM, harold seigel wrote: >>> Please note that "java.base" is hard coded in multiple places in the >>> JVM. For example: >>> >>> src/share/vm/runtime/arguments.cpp: if (strcmp(module_name, >>> "java.base") == 0) { >>> >>> src/share/vm/classfile/classLoader.cpp: location = >>> (*JImageFindResource)(_jimage, "java.base", get_jimage_version_s >>> >>> src/share/vm/classfile/modules.cpp: "Bad package name >>> for module: java.base"); >>> >>> src/share/vm/oops/instanceKlass.cpp: >>> assert(ModuleEntryTable::javabase_moduleEntry() != NULL, "java.base >>> module is NULL"); >>> >>> Should we enter a new bug for this or just ignore it? >> >> Sure, enter a bug to clean those up as well. >> Lois >> >>> >>> Thanks, Harold >>> >>> >>> On 11/15/2016 7:36 AM, Lois Foltan wrote: >>>> >>>> On 11/14/2016 4:00 PM, Rachel Protacio wrote: >>>>> New webrev: http://cr.openjdk.java.net/~rprotacio/8165550.01/ >>>> >>>> Hi Rachel, >>>> >>>> Looks good. One comment about hard coding the string "java.base" on >>>> line #1989. Please consider obtaining the string for the java.base >>>> module from vmSymbols::java_base()->as_C_string(). >>>> >>>> Thanks, >>>> Lois >>>> >>>>> >>>>> Corrected according to David's review. Comments below. >>>>> >>>>> >>>>> On 11/10/2016 6:24 PM, David Holmes wrote: >>>>>> Okay on with the review: >>>>>> >>>>>> src/share/vm/classfile/moduleEntry.cpp >>>>>> >>>>>> + const char* loc = ""; >>>>>> + loc = location()->as_C_string(); >>>>>> >>>>>> can just be: >>>>>> >>>>>> + const char* loc = location()->as_C_string(); >>>>>> >>>>> Fixed. >>>>>> --- >>>>>> >>>>>> src/share/vm/runtime/sharedRuntime.cpp >>>>>> >>>>>> 1941 const char* class_loader_and_module_name(Klass* klass, Klass* >>>>>> base_klass) { >>>>>> >>>>>> I'm unclear in what way "base_klass" is a "base" class ?? And why, >>>>>> if it is not an instance class must be it in java.base ?? >>>>> I really meant "bottom" class, following the logic used to parse it >>>>> out. I've moved that internal to the function now. So if the klass >>>>> is an instance klass or instance klass array, we look for further >>>>> information. Otherwise (if it's a primitive array), it's from >>>>> java.base. >>>>>> >>>>>> Nit: I don't think these need to be #defines nor have such verbose >>>>>> names: >>>>>> >>>>>> 1936 #define CLASS_LOADER_NAME_DELIMITER "/" >>>>>> 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH >>>>>> strlen(CLASS_LOADER_NAME_DELIMITER) >>>>>> >>>>>> why not just >>>>>> >>>>>> const char* delim = "\"; >>>>>> int delim_len = strlen(delim); >>>>>> >>>>>> Plus you are inconsistent with the handling of the "/" delimiter >>>>>> versus the "@" delimiter. >>>>> Fixed. >>>>>> >>>>>> --- >>>>>> >>>>>> test/runtime/modules/CCE_module_msg.java >>>>>> >>>>>> Nit: in the catch block from line #95 can you store getMessage() >>>>>> into a local so it isn't called 5 times. Thanks. :) >>>>> Fixed too. Thanks! >>>>> Rachel >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>>> Rachel >>>>>>> >>>>>>> On 11/10/2016 5:02 PM, David Holmes wrote: >>>>>>>> Hi Rachel, >>>>>>>> >>>>>>>> Just wondering, in the most complex case, exactly what will this >>>>>>>> new >>>>>>>> message look like? We seem to be including module names as well as >>>>>>>> classloader names, and of course class names. >>>>>>>> >>>>>>>> Arguably we should only add the additional info when they differ >>>>>>>> between the types ie if both in same classloader and same module >>>>>>>> then >>>>>>>> only report class name; if same loader but different module report >>>>>>>> module:class; if different loaders then report everything >>>>>>>> loader:module:class. Also not sure module version is needed at >>>>>>>> all in >>>>>>>> this message. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> David >>>>>>>> >>>>>>>> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Please review this Jigsaw enhancement which adds the new format >>>>>>>>> of class >>>>>>>>> loader names to VM ClassCastException messages. Includes an >>>>>>>>> updated/expanded jtreg test. >>>>>>>>> >>>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>>>>>>>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>>>>>>>> >>>>>>>>> Passes JPRT and jck vm and lang tests. >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> Rachel >>>>>>> >>>>> >>>> >>> >> > From david.holmes at oracle.com Wed Nov 16 05:28:29 2016 From: david.holmes at oracle.com (David Holmes) Date: Wed, 16 Nov 2016 15:28:29 +1000 Subject: RFR: JDK-8160997 Solaris: deprecated and interfaces should be replaced In-Reply-To: <7f74a2da-e78f-4174-198c-889fb816c059@oracle.com> References: <57839D8A.9050108@oracle.com> <7f74a2da-e78f-4174-198c-889fb816c059@oracle.com> Message-ID: <8b8cbb78-feab-88be-0856-cd41a28125b1@oracle.com> Hi Dmitry, This changeset was pushed at the end of July! David On 16/11/2016 3:04 AM, Dmitry Samersoff wrote: > Alan, > > Solaris.c:73 > > if (getpwuid_r(getuid(), &pwd, pwd_buf, sizeof(pwd_buf), &p) != NULL > > getpwuid_r returns int and we have to check that p is not null. > > > Otherwise looks good to me. > > -Dmitry > > > > On 2016-07-11 16:22, Alan Burlison wrote: >> Hi, I'd like to ask for reviews for: >> >> JDK-8160997 Solaris: deprecated and interfaces should be >> replaced >> >> This is to remove code that was a workaround that was needed for Solaris >> 2.4. In 2.4 we implemented interfaces from the draft of POSIX.1c Draft 6 >> which then changed in the final POSIX release, and we were then stuck >> with the warts. >> >> For Solaris 11, defining the macro _POSIX_PTHREAD_SEMANTICS enables the >> 'official' POSIX information and means that the code which works on >> Linux etc also works the same on Solaris, so the Solaris-specific >> workarounds can be removed. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8160997 >> >> Webrev: >> http://cr.openjdk.java.net/~dcubed/for_alanbur/8160997-webrev/0-jdk9-hs-all/ >> >> >> JPRT + the hotspot testset was clean. >> > > From robbin.ehn at oracle.com Wed Nov 16 09:49:56 2016 From: robbin.ehn at oracle.com (Robbin Ehn) Date: Wed, 16 Nov 2016 10:49:56 +0100 Subject: RFR(S) 8166304: Skipping access check for classes generated by core reflection In-Reply-To: References: Message-ID: <4fce17aa-87e4-7a36-167b-cc737d5c0809@oracle.com> Hi Harlod, Is there a reason for not doing a return after the exception is thrown? + Exceptions::fthrow( + THREAD_AND_LOCATION, + vmSymbols::java_lang_IllegalAccessError(), + "class %s loaded by %s cannot access jdk/internal/reflect superclass %s", + this_klass->external_name(), this_cld->loader_name(), super->external_name()); ----> return; ???? + } + } /Robbin On 11/15/2016 03:09 PM, harold seigel wrote: > Hi, > > Please review this fix for bug JDK-8166304. The fix throws an IllegalAccessError exception when attempting to create a class that is a sub-type of a jdk.internal.reflect > class, unless the class loader is either the boot loader or the special reflection class loader. This prevents user classes from being able to extend jdk.internal.reflect > classes in order to bypass Reflection.getCallerClass. > > Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8166304/ > > JBS bug: https://bugs.openjdk.java.net/browse/JDK-8166304 > > The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, java/io, java/lang, and java/util tests, the nsk co-located and the non-co-located tests, the RBT > tier2 tests, and the test in the webrev. > > A sample error message looks like this: > > java.lang.IllegalAccessError: class FakeMethodAccessor loaded by jdk/internal/loader/ClassLoaders$AppClassLoader cannot access jdk/internal/reflect superclass > jdk.internal.reflect.MethodAccessorImpl > > Thanks, Harold > From dmitry.samersoff at oracle.com Wed Nov 16 10:28:20 2016 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Wed, 16 Nov 2016 13:28:20 +0300 Subject: RFR: JDK-8160997 Solaris: deprecated and interfaces should be replaced In-Reply-To: <8b8cbb78-feab-88be-0856-cd41a28125b1@oracle.com> References: <57839D8A.9050108@oracle.com> <7f74a2da-e78f-4174-198c-889fb816c059@oracle.com> <8b8cbb78-feab-88be-0856-cd41a28125b1@oracle.com> Message-ID: David, OK. Sorry! -Dmitry On 2016-11-16 08:28, David Holmes wrote: > Hi Dmitry, > > This changeset was pushed at the end of July! > > David > > On 16/11/2016 3:04 AM, Dmitry Samersoff wrote: >> Alan, >> >> Solaris.c:73 >> >> if (getpwuid_r(getuid(), &pwd, pwd_buf, sizeof(pwd_buf), &p) != NULL >> >> getpwuid_r returns int and we have to check that p is not null. >> >> >> Otherwise looks good to me. >> >> -Dmitry >> >> >> >> On 2016-07-11 16:22, Alan Burlison wrote: >>> Hi, I'd like to ask for reviews for: >>> >>> JDK-8160997 Solaris: deprecated and interfaces should be >>> replaced >>> >>> This is to remove code that was a workaround that was needed for Solaris >>> 2.4. In 2.4 we implemented interfaces from the draft of POSIX.1c Draft 6 >>> which then changed in the final POSIX release, and we were then stuck >>> with the warts. >>> >>> For Solaris 11, defining the macro _POSIX_PTHREAD_SEMANTICS enables the >>> 'official' POSIX information and means that the code which works on >>> Linux etc also works the same on Solaris, so the Solaris-specific >>> workarounds can be removed. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8160997 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~dcubed/for_alanbur/8160997-webrev/0-jdk9-hs-all/ >>> >>> >>> >>> JPRT + the hotspot testset was clean. >>> >> >> -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From harold.seigel at oracle.com Wed Nov 16 13:33:13 2016 From: harold.seigel at oracle.com (harold seigel) Date: Wed, 16 Nov 2016 08:33:13 -0500 Subject: RFR(S) 8166304: Skipping access check for classes generated by core reflection In-Reply-To: <4fce17aa-87e4-7a36-167b-cc737d5c0809@oracle.com> References: <4fce17aa-87e4-7a36-167b-cc737d5c0809@oracle.com> Message-ID: <4321d250-cdb6-09f1-1352-00317449f7b0@oracle.com> Thanks Robbin! I'll add the missing 'return;'. Harold On 11/16/2016 4:49 AM, Robbin Ehn wrote: > Hi Harlod, > > Is there a reason for not doing a return after the exception is thrown? > + Exceptions::fthrow( > + THREAD_AND_LOCATION, > + vmSymbols::java_lang_IllegalAccessError(), > + "class %s loaded by %s cannot access jdk/internal/reflect > superclass %s", > + this_klass->external_name(), this_cld->loader_name(), > super->external_name()); > ----> return; ???? > + } > + } > > /Robbin > > On 11/15/2016 03:09 PM, harold seigel wrote: >> Hi, >> >> Please review this fix for bug JDK-8166304. The fix throws an >> IllegalAccessError exception when attempting to create a class that >> is a sub-type of a jdk.internal.reflect >> class, unless the class loader is either the boot loader or the >> special reflection class loader. This prevents user classes from >> being able to extend jdk.internal.reflect >> classes in order to bypass Reflection.getCallerClass. >> >> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8166304/ >> >> JBS bug: https://bugs.openjdk.java.net/browse/JDK-8166304 >> >> The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, >> java/io, java/lang, and java/util tests, the nsk co-located and the >> non-co-located tests, the RBT >> tier2 tests, and the test in the webrev. >> >> A sample error message looks like this: >> >> java.lang.IllegalAccessError: class FakeMethodAccessor loaded by >> jdk/internal/loader/ClassLoaders$AppClassLoader cannot access >> jdk/internal/reflect superclass >> jdk.internal.reflect.MethodAccessorImpl >> >> Thanks, Harold >> From karen.kinnear at oracle.com Wed Nov 16 14:31:04 2016 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Wed, 16 Nov 2016 09:31:04 -0500 Subject: RFR(S) 8166304: Skipping access check for classes generated by core reflection In-Reply-To: <582B72D2.6030506@oracle.com> References: <582B72D2.6030506@oracle.com> Message-ID: Harold, Creative solution to this problem! I have to agree with Lois? comments. I struggled with the non_reflection_class_loader call which is maybe too subtle since you were piggybacking on an existing call. The way I understand it, when you call non_reflection_class_loader, IF it is a reflection class loader, then it returns the delegating parent class, and therefore the result you come back with is different than the class loader you called it with. It would be cleaner to just check if the class loader is not null, check against reflect_DelegatingClassLoader_klass() as Lois suggested, which is the core check you were extracting from non_reflection_class_loader anyway. thanks, Karen > On Nov 15, 2016, at 3:40 PM, Lois Foltan wrote: > > > On 11/15/2016 9:09 AM, harold seigel wrote: >> Hi, >> >> Please review this fix for bug JDK-8166304. The fix throws an IllegalAccessError exception when attempting to create a class that is a sub-type of a jdk.internal.reflect class, unless the class loader is either the boot loader or the special reflection class loader. This prevents user classes from being able to extend jdk.internal.reflect classes in order to bypass Reflection.getCallerClass. >> >> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8166304/ > > Hi Harold, > > A couple of comments. > > src/share/vm/classfile/classFileParser.cpp > line #4360 - I think you should reconfigure the if conditional. Make the check for the boot class loader first so a fast exit from the conditional exists if this is the boot class loader. In other words don't compare the symbol names. Also, use a Symbol::fast_compare instead of "==" to compare the name of the packages. > > line #4362 - I would like to understand better the following check > > 4362 this_klass->class_loader() == > 4363 java_lang_ClassLoader::non_reflection_class_loader(this_klass->class_loader()) > > When this_klass's class loader data's _class_loader field is set the non_reflection_class_loader is not called, so I am having difficulty understanding how this conditional could ever be true. I am curious why the check isn't instead, "this_klass->class_loader()->is_a(SystemDictionary::reflect_DelegatingClassLoader_klass())"? > > Thanks, > Lois > >> >> JBS bug: https://bugs.openjdk.java.net/browse/JDK-8166304 >> >> The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, java/io, java/lang, and java/util tests, the nsk co-located and the non-co-located tests, the RBT tier2 tests, and the test in the webrev. >> >> A sample error message looks like this: >> >> java.lang.IllegalAccessError: class FakeMethodAccessor loaded by jdk/internal/loader/ClassLoaders$AppClassLoader cannot access jdk/internal/reflect superclass jdk.internal.reflect.MethodAccessorImpl >> >> Thanks, Harold >> > From goetz.lindenmaier at sap.com Wed Nov 16 15:52:46 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Wed, 16 Nov 2016 15:52:46 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <5e9b0cf6243e471287da233b16fe1f63@derote13de14.global.corp.sap> Message-ID: <59f839950a21446fbe26aedfeb8151b8@derote13de14.global.corp.sap> Hi, I addressed the additional topics, but in an extra change. First, I rounded up the NPTL bug workaround: http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.02 I fixed os::Posix::describe_pthread_attr() and current_stack_region() In the os_cpu files. Interestingly, in zero, current_stack_region() is already fixed. Then I edited an extra webrev to fix - unnecessary OS guard page on Compiler threads - unpredictable minimal stack sizes because of varying (guard) page sizes: http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.minStSz/ Compiler threads now don't get OS guard pages. Saves one page per compiler thread. Also, the constants in the os__.cpp files now list minimum space required for frames. I basically reduced them by the size of the guard zones. Where minimum stack sizes are computed, the page size is known and the minimal value is extended by the required space for the guard pages. As compiler threads don't bang, no space for the shadow zone is needed. Should I add this additional change to 8169373? As this only deals with minimal stack sizes, and reduces these sizes, I don't see a relevant risk. Only on systems with large pages the minimal sizes now might be bigger, but then the old sizes would have led to immediate stack overflows. Best regards, Goetz. > -----Original Message----- > From: David Holmes [mailto:david.holmes at oracle.com] > Sent: Dienstag, 15. November 2016 00:54 > To: Lindenmaier, Goetz ; hotspot-runtime- > dev at openjdk.java.net > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > Hi Goetz, > > On 15/11/2016 3:12 AM, Lindenmaier, Goetz wrote: > > Hi David, > > > >> Did you really intend to increase the real minimum stack from 128K to > >> 256K ? (on a 64K page system) > > I finally figured why I have to set such high minimum values. > > > > A CompilerThread is a JavaThread. It's getting yellow, red etc > > pages at the top. As these are 64K on 64K systems, the compiler > > thread stack has to be this big. Thus you actually need similar > > values once you run a linuxx86_64 VM on a system with 64K pages. > > Does this exist? > > No idea, sorry. > > > But, this means that there is no use of the OS guard pages > > configured in default_guard_size(). This is called with 'compiler_thread' > > for CompilerThreads. > > > > We should probably also return '0' in that case. > > > > What do you think? > > I think I agree. As you note a CompilerThread is-a JavaThread even > though the ThreadType enum treats them as completely distinct: > > enum ThreadType { > vm_thread, > cgc_thread, // Concurrent GC thread > pgc_thread, // Parallel GC thread > java_thread, > compiler_thread, > watcher_thread, > os_thread > }; > > so anything with VM supplied guard pages doesn't need the glibc ones as > well. > > I'm struggling to understand why os::Linux::default_guard_size is > defined per CPU type? I would not expect this to need to vary based on > CPU at all. ?? > > Thanks, > David > > > Best regards, > > Goetz. > > > > size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > > // Creating guard page is very expensive. Java thread has HotSpot > > // guard page, only enable glibc guard page for non-Java threads. > > return (thr_type == java_thread ? 0 : page_size()); > > } > > From erik.helin at oracle.com Thu Nov 17 10:33:17 2016 From: erik.helin at oracle.com (Erik Helin) Date: Thu, 17 Nov 2016 11:33:17 +0100 Subject: Question about VirtualSpace and MPSS support In-Reply-To: References: Message-ID: On 11/09/2016 04:00 PM, Thomas St?fe wrote: > Hi all, Hey Thomas, sorry my somewhat late reply. > please help me understand some details about the VirtualSpace class. > > Background: I am experimenting with putting humongous metaspace chunks at > the end of the VirtualSpace (part of > https://bugs.openjdk.java.net/browse/JDK-8166690). So I would like to have > a VirtualSpace which allows me to allocate from both bottom and top. > Therefore I am checking how difficult it would be to extend the > VirtualSpace class to allow me to grow from both sides. > > Now I try to understand the three-region-logic implemented to support > multiple page sizes: > > VirtualSpace has three regions with three different granularities. I think > the intent is to support large pages for address ranges which are not > necessarily large page aligned, so this leaves small-paged regions at top > and/or bottom. > > Looking at VirtualSpace::expand_by() I think the idea is that there is > always one continuous committed area spanning those three regions, right? > (instead of, say, each of the regions being committed independently). So, > just one high water mark, which is either in the low, middle or upper > region. If it is in the middle region, low region should be completely > committed, if it is in the upper region, low and middle regions should be > completely committed? I think it must be this way, because from the outside > I only should have one continuous committed address range, otherwise it > would be difficult to use this VirtualSpace. > > But then, I have a function like VirtualSpace::actual_committed_size() - in > addition to VirtualSpace::committed_size() - which counts the committed > area for all three regions separately. Is there ever a case where comitted > size and actual comitted size are different? Yes, consider the case when you have small pages for the "lower" part of the Virtualspace. Due to how paging works in operating systems, you can't commit less memory than the size of a small page. So even if a user of a VirtualSpace instance wants to expand by a half small page size, the OS will still commit a whole small page (the OS can't work on a smaller granularity). If that has happened, then commited_size() will return half a small page size as its result and actual_commited_size will return one small page size as its result. Do you follow? > Can it be that lower regions are partly comitted even if higher regions are already used? No, that should never happen (if it does, then that is a bug). The following assert should trigger in that case (for fastdebug or debug builds): http://hg.openjdk.java.net/jdk9/hs/hotspot/file/4d1cc698c0b6/src/share/vm/memory/virtualspace.cpp#l734 Thanks, Erik > Thank you! > > Kind Regards, Thomas > From thomas.stuefe at gmail.com Thu Nov 17 11:27:46 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 17 Nov 2016 12:27:46 +0100 Subject: Question about VirtualSpace and MPSS support In-Reply-To: References: Message-ID: Hi Erik, On Thu, Nov 17, 2016 at 11:33 AM, Erik Helin wrote: > On 11/09/2016 04:00 PM, Thomas St?fe wrote: > >> Hi all, >> > > Hey Thomas, sorry my somewhat late reply. > > no problem, thanks for thinking this thru. > > please help me understand some details about the VirtualSpace class. >> >> Background: I am experimenting with putting humongous metaspace chunks at >> the end of the VirtualSpace (part of >> https://bugs.openjdk.java.net/browse/JDK-8166690). So I would like to >> have >> a VirtualSpace which allows me to allocate from both bottom and top. >> Therefore I am checking how difficult it would be to extend the >> VirtualSpace class to allow me to grow from both sides. >> >> Now I try to understand the three-region-logic implemented to support >> multiple page sizes: >> >> VirtualSpace has three regions with three different granularities. I think >> the intent is to support large pages for address ranges which are not >> necessarily large page aligned, so this leaves small-paged regions at top >> and/or bottom. >> >> Looking at VirtualSpace::expand_by() I think the idea is that there is >> always one continuous committed area spanning those three regions, right? >> (instead of, say, each of the regions being committed independently). So, >> just one high water mark, which is either in the low, middle or upper >> region. If it is in the middle region, low region should be completely >> committed, if it is in the upper region, low and middle regions should be >> completely committed? I think it must be this way, because from the >> outside >> I only should have one continuous committed address range, otherwise it >> would be difficult to use this VirtualSpace. >> >> But then, I have a function like VirtualSpace::actual_committed_size() - >> in >> addition to VirtualSpace::committed_size() - which counts the committed >> area for all three regions separately. Is there ever a case where comitted >> size and actual comitted size are different? >> > > Yes, consider the case when you have small pages for the "lower" part of > the Virtualspace. Due to how paging works in operating systems, you can't > commit less memory than the size of a small page. So even if a user of a > VirtualSpace instance wants to expand by a half small page size, the OS > will still commit a whole small page (the OS can't work on a smaller > granularity). If that has happened, then commited_size() will return half a > small page size as its result and actual_commited_size will return one > small page size as its result. Do you follow? > > Sure! This slipped me. I was assuming VirtualSpace::_high to be always page aligned, but now I see that this is not the case. Instead, we seem to have _high, which is the unaligned high water mark the user set with expand_by(), and then _lower_high, _middle_high, _upper_high, which seem to be the page-aligned highs for the various regions. I was expecting the caller to always expand by page size units. But of course this is wrong because the caller should not have to deal with the page granularity of the space, especially if it was stitched together from various page sizes. Can it be that lower regions are partly comitted even if higher regions are >> already used? >> > > No, that should never happen (if it does, then that is a bug). The > following assert should trigger in that case (for fastdebug or debug > builds): > http://hg.openjdk.java.net/jdk9/hs/hotspot/file/4d1cc698c0b6 > /src/share/vm/memory/virtualspace.cpp#l734 > > How did I miss this :) Thanks for pointing this out! I wonder why VirtualSpace keeps the unaligned high water mark around instead of, say, just rounding _high up to the next page boundary. A simpler approach would have been to advance _high to the next page boundary and just forget the exact increments the caller requested. Then, caller needs to deal with the fact that an VirtualSpace may expand by more than the requested size, and for future size calculations always needs to refer to VirtualSpace::high(). This could be a cleanup for jdk10. Thanks, Thomas > Thanks, > Erik > > > Thank you! >> >> Kind Regards, Thomas >> >> From harold.seigel at oracle.com Thu Nov 17 15:41:00 2016 From: harold.seigel at oracle.com (harold seigel) Date: Thu, 17 Nov 2016 10:41:00 -0500 Subject: RFR(S) 8166304: Skipping access check for classes generated by core reflection In-Reply-To: References: <582B72D2.6030506@oracle.com> Message-ID: <105708ff-96eb-84d8-9816-3e7d3d172c8e@oracle.com> Hi Karen, Lois, Thanks for your reviews! Please see this new webrev that incorporates your comments: http://cr.openjdk.java.net/~hseigel/bug_8166304.2/ Thanks, Harold On 11/16/2016 9:31 AM, Karen Kinnear wrote: > Harold, > > Creative solution to this problem! > > I have to agree with Lois? comments. I struggled with the non_reflection_class_loader call which is maybe too subtle > since you were piggybacking on an existing call. The way I understand it, > when you call non_reflection_class_loader, IF it is a reflection class loader, then it returns the delegating parent class, > and therefore the result you come back with is different than the class loader you called it with. > > It would be cleaner to just check if the class loader is not null, check against reflect_DelegatingClassLoader_klass() as > Lois suggested, which is the core check you were extracting from non_reflection_class_loader anyway. > > thanks, > Karen > >> On Nov 15, 2016, at 3:40 PM, Lois Foltan wrote: >> >> >> On 11/15/2016 9:09 AM, harold seigel wrote: >>> Hi, >>> >>> Please review this fix for bug JDK-8166304. The fix throws an IllegalAccessError exception when attempting to create a class that is a sub-type of a jdk.internal.reflect class, unless the class loader is either the boot loader or the special reflection class loader. This prevents user classes from being able to extend jdk.internal.reflect classes in order to bypass Reflection.getCallerClass. >>> >>> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8166304/ >> Hi Harold, >> >> A couple of comments. >> >> src/share/vm/classfile/classFileParser.cpp >> line #4360 - I think you should reconfigure the if conditional. Make the check for the boot class loader first so a fast exit from the conditional exists if this is the boot class loader. In other words don't compare the symbol names. Also, use a Symbol::fast_compare instead of "==" to compare the name of the packages. >> >> line #4362 - I would like to understand better the following check >> >> 4362 this_klass->class_loader() == >> 4363 java_lang_ClassLoader::non_reflection_class_loader(this_klass->class_loader()) >> >> When this_klass's class loader data's _class_loader field is set the non_reflection_class_loader is not called, so I am having difficulty understanding how this conditional could ever be true. I am curious why the check isn't instead, "this_klass->class_loader()->is_a(SystemDictionary::reflect_DelegatingClassLoader_klass())"? >> >> Thanks, >> Lois >> >>> JBS bug: https://bugs.openjdk.java.net/browse/JDK-8166304 >>> >>> The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, java/io, java/lang, and java/util tests, the nsk co-located and the non-co-located tests, the RBT tier2 tests, and the test in the webrev. >>> >>> A sample error message looks like this: >>> >>> java.lang.IllegalAccessError: class FakeMethodAccessor loaded by jdk/internal/loader/ClassLoaders$AppClassLoader cannot access jdk/internal/reflect superclass jdk.internal.reflect.MethodAccessorImpl >>> >>> Thanks, Harold >>> From rachel.protacio at oracle.com Thu Nov 17 16:12:52 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Thu, 17 Nov 2016 11:12:52 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> Message-ID: <043037b6-f474-1552-fbe9-3a9adf1db7ee@oracle.com> Thanks for the review, David! Rachel On 11/16/2016 12:18 AM, David Holmes wrote: > Looks good Rachel! > > No further comments from me. > > Thanks, > David > > On 16/11/2016 6:49 AM, Rachel Protacio wrote: >> Updated webrev: http://cr.openjdk.java.net/~rprotacio/8165550.02/ >> - added macro in ModuleEntry.hpp for "java.base" to address Lois' >> comment (opened enhancement >> https://bugs.openjdk.java.net/browse/JDK-8169734 for other existing >> cases) >> - added an assert instead of a NULL check for the class loader data >> - fixed NEW_RESOURCE_ARRAY_RETURN_NULL calls >> >> Thanks! >> Rachel >> >> On 11/15/2016 9:45 AM, Lois Foltan wrote: >>> >>> On 11/15/2016 9:38 AM, harold seigel wrote: >>>> Please note that "java.base" is hard coded in multiple places in the >>>> JVM. For example: >>>> >>>> src/share/vm/runtime/arguments.cpp: if (strcmp(module_name, >>>> "java.base") == 0) { >>>> >>>> src/share/vm/classfile/classLoader.cpp: location = >>>> (*JImageFindResource)(_jimage, "java.base", get_jimage_version_s >>>> >>>> src/share/vm/classfile/modules.cpp: "Bad package >>>> name >>>> for module: java.base"); >>>> >>>> src/share/vm/oops/instanceKlass.cpp: >>>> assert(ModuleEntryTable::javabase_moduleEntry() != NULL, "java.base >>>> module is NULL"); >>>> >>>> Should we enter a new bug for this or just ignore it? >>> >>> Sure, enter a bug to clean those up as well. >>> Lois >>> >>>> >>>> Thanks, Harold >>>> >>>> >>>> On 11/15/2016 7:36 AM, Lois Foltan wrote: >>>>> >>>>> On 11/14/2016 4:00 PM, Rachel Protacio wrote: >>>>>> New webrev: http://cr.openjdk.java.net/~rprotacio/8165550.01/ >>>>> >>>>> Hi Rachel, >>>>> >>>>> Looks good. One comment about hard coding the string "java.base" on >>>>> line #1989. Please consider obtaining the string for the java.base >>>>> module from vmSymbols::java_base()->as_C_string(). >>>>> >>>>> Thanks, >>>>> Lois >>>>> >>>>>> >>>>>> Corrected according to David's review. Comments below. >>>>>> >>>>>> >>>>>> On 11/10/2016 6:24 PM, David Holmes wrote: >>>>>>> Okay on with the review: >>>>>>> >>>>>>> src/share/vm/classfile/moduleEntry.cpp >>>>>>> >>>>>>> + const char* loc = ""; >>>>>>> + loc = location()->as_C_string(); >>>>>>> >>>>>>> can just be: >>>>>>> >>>>>>> + const char* loc = location()->as_C_string(); >>>>>>> >>>>>> Fixed. >>>>>>> --- >>>>>>> >>>>>>> src/share/vm/runtime/sharedRuntime.cpp >>>>>>> >>>>>>> 1941 const char* class_loader_and_module_name(Klass* klass, Klass* >>>>>>> base_klass) { >>>>>>> >>>>>>> I'm unclear in what way "base_klass" is a "base" class ?? And why, >>>>>>> if it is not an instance class must be it in java.base ?? >>>>>> I really meant "bottom" class, following the logic used to parse it >>>>>> out. I've moved that internal to the function now. So if the klass >>>>>> is an instance klass or instance klass array, we look for further >>>>>> information. Otherwise (if it's a primitive array), it's from >>>>>> java.base. >>>>>>> >>>>>>> Nit: I don't think these need to be #defines nor have such verbose >>>>>>> names: >>>>>>> >>>>>>> 1936 #define CLASS_LOADER_NAME_DELIMITER "/" >>>>>>> 1937 #define CLASS_LOADER_NAME_DELIMITER_LENGTH >>>>>>> strlen(CLASS_LOADER_NAME_DELIMITER) >>>>>>> >>>>>>> why not just >>>>>>> >>>>>>> const char* delim = "\"; >>>>>>> int delim_len = strlen(delim); >>>>>>> >>>>>>> Plus you are inconsistent with the handling of the "/" delimiter >>>>>>> versus the "@" delimiter. >>>>>> Fixed. >>>>>>> >>>>>>> --- >>>>>>> >>>>>>> test/runtime/modules/CCE_module_msg.java >>>>>>> >>>>>>> Nit: in the catch block from line #95 can you store getMessage() >>>>>>> into a local so it isn't called 5 times. Thanks. :) >>>>>> Fixed too. Thanks! >>>>>> Rachel >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>>> Rachel >>>>>>>> >>>>>>>> On 11/10/2016 5:02 PM, David Holmes wrote: >>>>>>>>> Hi Rachel, >>>>>>>>> >>>>>>>>> Just wondering, in the most complex case, exactly what will this >>>>>>>>> new >>>>>>>>> message look like? We seem to be including module names as >>>>>>>>> well as >>>>>>>>> classloader names, and of course class names. >>>>>>>>> >>>>>>>>> Arguably we should only add the additional info when they differ >>>>>>>>> between the types ie if both in same classloader and same module >>>>>>>>> then >>>>>>>>> only report class name; if same loader but different module >>>>>>>>> report >>>>>>>>> module:class; if different loaders then report everything >>>>>>>>> loader:module:class. Also not sure module version is needed at >>>>>>>>> all in >>>>>>>>> this message. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> David >>>>>>>>> >>>>>>>>> On 11/11/2016 7:38 AM, Rachel Protacio wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Please review this Jigsaw enhancement which adds the new format >>>>>>>>>> of class >>>>>>>>>> loader names to VM ClassCastException messages. Includes an >>>>>>>>>> updated/expanded jtreg test. >>>>>>>>>> >>>>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8165550 >>>>>>>>>> Open webrev: http://cr.openjdk.java.net/~rprotacio/8165550.00/ >>>>>>>>>> >>>>>>>>>> Passes JPRT and jck vm and lang tests. >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> Rachel >>>>>>>> >>>>>> >>>>> >>>> >>> >> From rachel.protacio at oracle.com Thu Nov 17 16:36:18 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Thu, 17 Nov 2016 11:36:18 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> Message-ID: <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> Last one! http://cr.openjdk.java.net/~rprotacio/8165550.03 - Thanks to Lois, I've added a new test case for a custom-named class loader with the unnamed module, and also throw an exception in the test if no CCE is thrown. - Thanks to Mandy and Coleen, I've corrected the part of my code getting the Class Loader Name, which ironically, I was doing incorrectly before. Rachel On 11/15/2016 4:03 PM, Mandy Chung wrote: >> On Nov 15, 2016, at 12:49 PM, Rachel Protacio wrote: >> >> Updated webrev: http://cr.openjdk.java.net/~rprotacio/8165550.02/ >> - added macro in ModuleEntry.hpp for "java.base" to address Lois' comment (opened enhancement https://bugs.openjdk.java.net/browse/JDK-8169734 for other existing cases) >> - added an assert instead of a NULL check for the class loader data >> - fixed NEW_RESOURCE_ARRAY_RETURN_NULL calls > Looks okay to me. It?d be good if you can add a test case testing a custom named class loader with unnamed module. > > Mandy > From lois.foltan at oracle.com Thu Nov 17 19:08:09 2016 From: lois.foltan at oracle.com (Lois Foltan) Date: Thu, 17 Nov 2016 14:08:09 -0500 Subject: RFR(S) 8166304: Skipping access check for classes generated by core reflection In-Reply-To: <105708ff-96eb-84d8-9816-3e7d3d172c8e@oracle.com> References: <582B72D2.6030506@oracle.com> <105708ff-96eb-84d8-9816-3e7d3d172c8e@oracle.com> Message-ID: <582E0019.1080904@oracle.com> On 11/17/2016 10:41 AM, harold seigel wrote: > > Hi Karen, Lois, > > Thanks for your reviews! Please see this new webrev that incorporates > your comments: > > http://cr.openjdk.java.net/~hseigel/bug_8166304.2/ > Thanks for making these changes Harold. Looks good. Lois > Thanks, Harold > > > On 11/16/2016 9:31 AM, Karen Kinnear wrote: >> Harold, >> >> Creative solution to this problem! >> >> I have to agree with Lois? comments. I struggled with the non_reflection_class_loader call which is maybe too subtle >> since you were piggybacking on an existing call. The way I understand it, >> when you call non_reflection_class_loader, IF it is a reflection class loader, then it returns the delegating parent class, >> and therefore the result you come back with is different than the class loader you called it with. >> >> It would be cleaner to just check if the class loader is not null, check against reflect_DelegatingClassLoader_klass() as >> Lois suggested, which is the core check you were extracting from non_reflection_class_loader anyway. >> >> thanks, >> Karen >> >>> On Nov 15, 2016, at 3:40 PM, Lois Foltan wrote: >>> >>> >>> On 11/15/2016 9:09 AM, harold seigel wrote: >>>> Hi, >>>> >>>> Please review this fix for bug JDK-8166304. The fix throws an IllegalAccessError exception when attempting to create a class that is a sub-type of a jdk.internal.reflect class, unless the class loader is either the boot loader or the special reflection class loader. This prevents user classes from being able to extend jdk.internal.reflect classes in order to bypass Reflection.getCallerClass. >>>> >>>> Open webrev:http://cr.openjdk.java.net/~hseigel/bug_8166304/ >>> Hi Harold, >>> >>> A couple of comments. >>> >>> src/share/vm/classfile/classFileParser.cpp >>> line #4360 - I think you should reconfigure the if conditional. Make the check for the boot class loader first so a fast exit from the conditional exists if this is the boot class loader. In other words don't compare the symbol names. Also, use a Symbol::fast_compare instead of "==" to compare the name of the packages. >>> >>> line #4362 - I would like to understand better the following check >>> >>> 4362 this_klass->class_loader() == >>> 4363 java_lang_ClassLoader::non_reflection_class_loader(this_klass->class_loader()) >>> >>> When this_klass's class loader data's _class_loader field is set the non_reflection_class_loader is not called, so I am having difficulty understanding how this conditional could ever be true. I am curious why the check isn't instead, "this_klass->class_loader()->is_a(SystemDictionary::reflect_DelegatingClassLoader_klass())"? >>> >>> Thanks, >>> Lois >>> >>>> JBS bug:https://bugs.openjdk.java.net/browse/JDK-8166304 >>>> >>>> The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, java/io, java/lang, and java/util tests, the nsk co-located and the non-co-located tests, the RBT tier2 tests, and the test in the webrev. >>>> >>>> A sample error message looks like this: >>>> >>>> java.lang.IllegalAccessError: class FakeMethodAccessor loaded by jdk/internal/loader/ClassLoaders$AppClassLoader cannot access jdk/internal/reflect superclass jdk.internal.reflect.MethodAccessorImpl >>>> >>>> Thanks, Harold >>>> > From harold.seigel at oracle.com Thu Nov 17 19:12:33 2016 From: harold.seigel at oracle.com (harold seigel) Date: Thu, 17 Nov 2016 14:12:33 -0500 Subject: RFR(S) 8166304: Skipping access check for classes generated by core reflection In-Reply-To: <582E0019.1080904@oracle.com> References: <582B72D2.6030506@oracle.com> <105708ff-96eb-84d8-9816-3e7d3d172c8e@oracle.com> <582E0019.1080904@oracle.com> Message-ID: <6b3f1e48-5f83-3465-8ed9-c124929a1fad@oracle.com> Thanks Lois! Harold On 11/17/2016 2:08 PM, Lois Foltan wrote: > > On 11/17/2016 10:41 AM, harold seigel wrote: >> >> Hi Karen, Lois, >> >> Thanks for your reviews! Please see this new webrev that >> incorporates your comments: >> >> http://cr.openjdk.java.net/~hseigel/bug_8166304.2/ >> > > Thanks for making these changes Harold. Looks good. > Lois > >> Thanks, Harold >> >> >> On 11/16/2016 9:31 AM, Karen Kinnear wrote: >>> Harold, >>> >>> Creative solution to this problem! >>> >>> I have to agree with Lois? comments. I struggled with the non_reflection_class_loader call which is maybe too subtle >>> since you were piggybacking on an existing call. The way I understand it, >>> when you call non_reflection_class_loader, IF it is a reflection class loader, then it returns the delegating parent class, >>> and therefore the result you come back with is different than the class loader you called it with. >>> >>> It would be cleaner to just check if the class loader is not null, check against reflect_DelegatingClassLoader_klass() as >>> Lois suggested, which is the core check you were extracting from non_reflection_class_loader anyway. >>> >>> thanks, >>> Karen >>> >>>> On Nov 15, 2016, at 3:40 PM, Lois Foltan wrote: >>>> >>>> >>>> On 11/15/2016 9:09 AM, harold seigel wrote: >>>>> Hi, >>>>> >>>>> Please review this fix for bug JDK-8166304. The fix throws an IllegalAccessError exception when attempting to create a class that is a sub-type of a jdk.internal.reflect class, unless the class loader is either the boot loader or the special reflection class loader. This prevents user classes from being able to extend jdk.internal.reflect classes in order to bypass Reflection.getCallerClass. >>>>> >>>>> Open webrev:http://cr.openjdk.java.net/~hseigel/bug_8166304/ >>>> Hi Harold, >>>> >>>> A couple of comments. >>>> >>>> src/share/vm/classfile/classFileParser.cpp >>>> line #4360 - I think you should reconfigure the if conditional. Make the check for the boot class loader first so a fast exit from the conditional exists if this is the boot class loader. In other words don't compare the symbol names. Also, use a Symbol::fast_compare instead of "==" to compare the name of the packages. >>>> >>>> line #4362 - I would like to understand better the following check >>>> >>>> 4362 this_klass->class_loader() == >>>> 4363 java_lang_ClassLoader::non_reflection_class_loader(this_klass->class_loader()) >>>> >>>> When this_klass's class loader data's _class_loader field is set the non_reflection_class_loader is not called, so I am having difficulty understanding how this conditional could ever be true. I am curious why the check isn't instead, "this_klass->class_loader()->is_a(SystemDictionary::reflect_DelegatingClassLoader_klass())"? >>>> >>>> Thanks, >>>> Lois >>>> >>>>> JBS bug:https://bugs.openjdk.java.net/browse/JDK-8166304 >>>>> >>>>> The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, java/io, java/lang, and java/util tests, the nsk co-located and the non-co-located tests, the RBT tier2 tests, and the test in the webrev. >>>>> >>>>> A sample error message looks like this: >>>>> >>>>> java.lang.IllegalAccessError: class FakeMethodAccessor loaded by jdk/internal/loader/ClassLoaders$AppClassLoader cannot access jdk/internal/reflect superclass jdk.internal.reflect.MethodAccessorImpl >>>>> >>>>> Thanks, Harold >>>>> >> > From karen.kinnear at oracle.com Thu Nov 17 19:17:23 2016 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Thu, 17 Nov 2016 14:17:23 -0500 Subject: RFR(S) 8166304: Skipping access check for classes generated by core reflection In-Reply-To: <582E0019.1080904@oracle.com> References: <582B72D2.6030506@oracle.com> <105708ff-96eb-84d8-9816-3e7d3d172c8e@oracle.com> <582E0019.1080904@oracle.com> Message-ID: Harold, Thank you for changing this so cleanly. Looks good. thanks, Karen > On Nov 17, 2016, at 2:08 PM, Lois Foltan wrote: > > > On 11/17/2016 10:41 AM, harold seigel wrote: >> Hi Karen, Lois, >> >> Thanks for your reviews! Please see this new webrev that incorporates your comments: >> http://cr.openjdk.java.net/~hseigel/bug_8166304.2/ > Thanks for making these changes Harold. Looks good. > Lois > >> Thanks, Harold >> >> On 11/16/2016 9:31 AM, Karen Kinnear wrote: >>> Harold, >>> >>> Creative solution to this problem! >>> >>> I have to agree with Lois? comments. I struggled with the non_reflection_class_loader call which is maybe too subtle >>> since you were piggybacking on an existing call. The way I understand it, >>> when you call non_reflection_class_loader, IF it is a reflection class loader, then it returns the delegating parent class, >>> and therefore the result you come back with is different than the class loader you called it with. >>> >>> It would be cleaner to just check if the class loader is not null, check against reflect_DelegatingClassLoader_klass() as >>> Lois suggested, which is the core check you were extracting from non_reflection_class_loader anyway. >>> >>> thanks, >>> Karen >>> >>>> On Nov 15, 2016, at 3:40 PM, Lois Foltan wrote: >>>> >>>> >>>> On 11/15/2016 9:09 AM, harold seigel wrote: >>>>> Hi, >>>>> >>>>> Please review this fix for bug JDK-8166304. The fix throws an IllegalAccessError exception when attempting to create a class that is a sub-type of a jdk.internal.reflect class, unless the class loader is either the boot loader or the special reflection class loader. This prevents user classes from being able to extend jdk.internal.reflect classes in order to bypass Reflection.getCallerClass. >>>>> >>>>> Open webrev: http://cr.openjdk.java.net/~hseigel/bug_8166304/ >>>> Hi Harold, >>>> >>>> A couple of comments. >>>> >>>> src/share/vm/classfile/classFileParser.cpp >>>> line #4360 - I think you should reconfigure the if conditional. Make the check for the boot class loader first so a fast exit from the conditional exists if this is the boot class loader. In other words don't compare the symbol names. Also, use a Symbol::fast_compare instead of "==" to compare the name of the packages. >>>> >>>> line #4362 - I would like to understand better the following check >>>> >>>> 4362 this_klass->class_loader() == >>>> 4363 java_lang_ClassLoader::non_reflection_class_loader(this_klass->class_loader()) >>>> >>>> When this_klass's class loader data's _class_loader field is set the non_reflection_class_loader is not called, so I am having difficulty understanding how this conditional could ever be true. I am curious why the check isn't instead, "this_klass->class_loader()->is_a(SystemDictionary::reflect_DelegatingClassLoader_klass())"? >>>> >>>> Thanks, >>>> Lois >>>> >>>>> JBS bug: https://bugs.openjdk.java.net/browse/JDK-8166304 >>>>> >>>>> The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, java/io, java/lang, and java/util tests, the nsk co-located and the non-co-located tests, the RBT tier2 tests, and the test in the webrev. >>>>> >>>>> A sample error message looks like this: >>>>> >>>>> java.lang.IllegalAccessError: class FakeMethodAccessor loaded by jdk/internal/loader/ClassLoaders$AppClassLoader cannot access jdk/internal/reflect superclass jdk.internal.reflect.MethodAccessorImpl >>>>> >>>>> Thanks, Harold >>>>> >> > From harold.seigel at oracle.com Thu Nov 17 19:18:11 2016 From: harold.seigel at oracle.com (harold seigel) Date: Thu, 17 Nov 2016 14:18:11 -0500 Subject: RFR(S) 8166304: Skipping access check for classes generated by core reflection In-Reply-To: References: <582B72D2.6030506@oracle.com> <105708ff-96eb-84d8-9816-3e7d3d172c8e@oracle.com> <582E0019.1080904@oracle.com> Message-ID: <22d07c90-7429-9edd-bc43-237dbcbb9dbf@oracle.com> Thanks Karen! Harold On 11/17/2016 2:17 PM, Karen Kinnear wrote: > Harold, > > Thank you for changing this so cleanly. Looks good. > > thanks, > Karen > >> On Nov 17, 2016, at 2:08 PM, Lois Foltan > > wrote: >> >> >> On 11/17/2016 10:41 AM, harold seigel wrote: >>> >>> Hi Karen, Lois, >>> >>> Thanks for your reviews! Please see this new webrev that >>> incorporates your comments: >>> >>> http://cr.openjdk.java.net/~hseigel/bug_8166304.2/ >>> >> >> Thanks for making these changes Harold. Looks good. >> Lois >> >>> Thanks, Harold >>> >>> >>> On 11/16/2016 9:31 AM, Karen Kinnear wrote: >>>> Harold, >>>> >>>> Creative solution to this problem! >>>> >>>> I have to agree with Lois? comments. I struggled with the non_reflection_class_loader call which is maybe too subtle >>>> since you were piggybacking on an existing call. The way I understand it, >>>> when you call non_reflection_class_loader, IF it is a reflection class loader, then it returns the delegating parent class, >>>> and therefore the result you come back with is different than the class loader you called it with. >>>> >>>> It would be cleaner to just check if the class loader is not null, check against reflect_DelegatingClassLoader_klass() as >>>> Lois suggested, which is the core check you were extracting from non_reflection_class_loader anyway. >>>> >>>> thanks, >>>> Karen >>>> >>>>> On Nov 15, 2016, at 3:40 PM, Lois Foltan wrote: >>>>> >>>>> >>>>> On 11/15/2016 9:09 AM, harold seigel wrote: >>>>>> Hi, >>>>>> >>>>>> Please review this fix for bug JDK-8166304. The fix throws an IllegalAccessError exception when attempting to create a class that is a sub-type of a jdk.internal.reflect class, unless the class loader is either the boot loader or the special reflection class loader. This prevents user classes from being able to extend jdk.internal.reflect classes in order to bypass Reflection.getCallerClass. >>>>>> >>>>>> Open webrev:http://cr.openjdk.java.net/~hseigel/bug_8166304/ >>>>> Hi Harold, >>>>> >>>>> A couple of comments. >>>>> >>>>> src/share/vm/classfile/classFileParser.cpp >>>>> line #4360 - I think you should reconfigure the if conditional. Make the check for the boot class loader first so a fast exit from the conditional exists if this is the boot class loader. In other words don't compare the symbol names. Also, use a Symbol::fast_compare instead of "==" to compare the name of the packages. >>>>> >>>>> line #4362 - I would like to understand better the following check >>>>> >>>>> 4362 this_klass->class_loader() == >>>>> 4363 java_lang_ClassLoader::non_reflection_class_loader(this_klass->class_loader()) >>>>> >>>>> When this_klass's class loader data's _class_loader field is set the non_reflection_class_loader is not called, so I am having difficulty understanding how this conditional could ever be true. I am curious why the check isn't instead, "this_klass->class_loader()->is_a(SystemDictionary::reflect_DelegatingClassLoader_klass())"? >>>>> >>>>> Thanks, >>>>> Lois >>>>> >>>>>> JBS bug:https://bugs.openjdk.java.net/browse/JDK-8166304 >>>>>> >>>>>> The fix was tested with the JCK Lang and vm tests, the JTreg hotspot, java/io, java/lang, and java/util tests, the nsk co-located and the non-co-located tests, the RBT tier2 tests, and the test in the webrev. >>>>>> >>>>>> A sample error message looks like this: >>>>>> >>>>>> java.lang.IllegalAccessError: class FakeMethodAccessor loaded by jdk/internal/loader/ClassLoaders$AppClassLoader cannot access jdk/internal/reflect superclass jdk.internal.reflect.MethodAccessorImpl >>>>>> >>>>>> Thanks, Harold >>>>>> >>> >> > From lois.foltan at oracle.com Thu Nov 17 19:19:06 2016 From: lois.foltan at oracle.com (Lois Foltan) Date: Thu, 17 Nov 2016 14:19:06 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> Message-ID: <582E02AA.7090008@oracle.com> On 11/17/2016 11:36 AM, Rachel Protacio wrote: > Last one! http://cr.openjdk.java.net/~rprotacio/8165550.03 > - Thanks to Lois, I've added a new test case for a custom-named > class loader with the unnamed module, and also throw an exception in > the test if no CCE is thrown. > - Thanks to Mandy and Coleen, I've corrected the part of my code > getting the Class Loader Name, which ironically, I was doing > incorrectly before. Looks good Rachel! Minor nit that I don't need to see another webrev for, the test case only needs the "2016" date in the copyright, not "2015" as well. Lois > > Rachel > > On 11/15/2016 4:03 PM, Mandy Chung wrote: >>> On Nov 15, 2016, at 12:49 PM, Rachel Protacio >>> wrote: >>> >>> Updated webrev: http://cr.openjdk.java.net/~rprotacio/8165550.02/ >>> - added macro in ModuleEntry.hpp for "java.base" to address >>> Lois' comment (opened enhancement >>> https://bugs.openjdk.java.net/browse/JDK-8169734 for other existing >>> cases) >>> - added an assert instead of a NULL check for the class loader data >>> - fixed NEW_RESOURCE_ARRAY_RETURN_NULL calls >> Looks okay to me. It?d be good if you can add a test case testing a >> custom named class loader with unnamed module. >> >> Mandy >> > From rachel.protacio at oracle.com Thu Nov 17 19:24:53 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Thu, 17 Nov 2016 14:24:53 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <582E02AA.7090008@oracle.com> References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> <582E02AA.7090008@oracle.com> Message-ID: <7b4d5e8f-0a0d-224a-d02c-5b14a38e3f6b@oracle.com> Thanks for the review, Lois! Will fix. Rachel On 11/17/2016 2:19 PM, Lois Foltan wrote: > > On 11/17/2016 11:36 AM, Rachel Protacio wrote: >> Last one! http://cr.openjdk.java.net/~rprotacio/8165550.03 >> - Thanks to Lois, I've added a new test case for a custom-named >> class loader with the unnamed module, and also throw an exception in >> the test if no CCE is thrown. >> - Thanks to Mandy and Coleen, I've corrected the part of my code >> getting the Class Loader Name, which ironically, I was doing >> incorrectly before. > > Looks good Rachel! Minor nit that I don't need to see another webrev > for, the test case only needs the "2016" date in the copyright, not > "2015" as well. > Lois > >> >> Rachel >> >> On 11/15/2016 4:03 PM, Mandy Chung wrote: >>>> On Nov 15, 2016, at 12:49 PM, Rachel Protacio >>>> wrote: >>>> >>>> Updated webrev: http://cr.openjdk.java.net/~rprotacio/8165550.02/ >>>> - added macro in ModuleEntry.hpp for "java.base" to address >>>> Lois' comment (opened enhancement >>>> https://bugs.openjdk.java.net/browse/JDK-8169734 for other existing >>>> cases) >>>> - added an assert instead of a NULL check for the class loader >>>> data >>>> - fixed NEW_RESOURCE_ARRAY_RETURN_NULL calls >>> Looks okay to me. It?d be good if you can add a test case testing a >>> custom named class loader with unnamed module. >>> >>> Mandy >>> >> > From chris.plummer at oracle.com Thu Nov 17 19:54:18 2016 From: chris.plummer at oracle.com (Chris Plummer) Date: Thu, 17 Nov 2016 11:54:18 -0800 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> Message-ID: <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> On 10/18/16 11:49 PM, David Holmes wrote: > > > On 19/10/2016 3:17 PM, Thomas St?fe wrote: >> >> >> On Wed, Oct 19, 2016 at 3:21 AM, David Holmes > > wrote: >> >> On 18/10/2016 3:39 PM, Thomas St?fe wrote: >> >> Hi David, Max, >> >> I changed the asserts according to Max' suggestion. Instead of >> checking >> inside flag_to_index, now I check before callers of this >> function use >> this value to access memory. >> >> >> I don't see where Max suggested that?? >> >> >> Max wrote: " I think the decision on whether to access a slot should >> occur as close to memory accessing code as possible." and proceeded to >> suggest fixing VirtualMemorySnapshot::by_type() as well. > > I did not interpret that comment that way, and was puzzled by the > reference to by_type. >> >> It doesn't make sense to me to have all the callers of flag_to_index >> check what it returned instead of doing it inside flag_to_index. >> >> >> I disagree. Imho it makes sense to either check the Memflags enumeration >> input argument in flag_to_index() or the returned index before >> consumption. In both cases one knows the valid value range. Strictly >> speaking checking the index in flag_to_index() cannot be done because it >> is a faceless int type whose valid values are not yet known. > > The index has to fall in the range 0 <= index <= mt_number_of_types, > and I was suggesting that it makes more sense to verify this once in > flag_to_index() than in all the callers of flag_to_index. Hi Thomas, Just catching up on this thread. This is the same conclusion I came to. I don't understand what you mean by "it is a faceless int type whose valid values are not yet known". BTW, I'll sponsor this fix for you once it is finalized. Please update the copyright dates first. thanks, Chris > > David > >> It is all academical and mostly a matter of taste. >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html >> >> >> As David correctly writes, this is technically not a bug, so >> I guess >> this will have to wait until java 10. >> >> >> Yes, afraid so. >> >> >> The fix is trivial and I will try to get fc extension for this (now that >> Goetz explained to me how to do this :). It seems this is done for many >> other non-bug issues as well. >> >> ..Thomas >> >> >> Thanks, >> David >> >> Kind Regards, Thomas >> >> >> >> On Fri, Oct 14, 2016 at 12:57 AM, David Holmes >> >> > >> wrote: >> >> On 13/10/2016 10:53 PM, Thomas St?fe wrote: >> >> Hi David, >> >> On Thu, Oct 13, 2016 at 12:08 PM, David Holmes >> > > > >> > >> >> > >>> wrote: >> >> Hi Thomas, >> >> On 13/10/2016 3:49 PM, Thomas St?fe wrote: >> >> Hi all, >> >> may I have plase a review for this tiny change? >> It just adds >> some assert to NMT. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8167650 >> >> > > >> >> > >> > >> >> webrev: >> >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ >> >> >> > > >> >> >> > >> >> > >> >> MEMFLAGS/webrev.00/webrev/ >> >> We had an ugly memory overwrite caused by this - >> ultimately our >> fault, because we fed an invalid memory flag to >> NMT - >> but it was >> difficult to find. An assert would have saved >> some time. >> >> >> I'm a little perplexed with asserting that something >> of MEMFLAGS >> type must be an actual MEMFLAGS value - it implies >> the caller is >> coercing plain int to MEMFLAGS, and I don't have >> much >> sympathy if >> they mess that up. Can't help wondering if there >> is some >> clever C++ >> trick to flag bad conversions at compile-time? >> >> >> The error was caused by an uninitialized variable of >> type >> MEMFLAGS. This >> was our fault, we have heavily modified >> allocation.hpp and >> introduced an >> error then merging changes from upstream. Due to a >> merging error >> this >> lead to a case where Arena::_flags was not >> initialized and >> contained a >> very large value. >> >> >> Ah I see. Lack of default initialization can be annoying :) >> >> I admit it looks funny. If it bothers you, I could >> instead check the >> returned index to be in the range for the size of the >> _malloc >> array in >> MallocMemorySnapshot::by_type(). Technically, it would >> mean the >> same. >> >> >> So I just realized that here: >> >> 62 // Map memory type to human readable name >> 63 static const char* flag_to_name(MEMFLAGS flag) { >> 64 assert(flag >= 0 && flag < mt_number_of_types, >> "Invalid >> flag value %d.", (int)flag); >> 65 return _memory_type_names[flag_to_index(flag)]; >> 66 } >> >> we call flag_to_index, so the assert is redundant as it is >> already >> in flag_to_index. Then presumably we change flag_to_index to >> something like this: >> >> static inline int flag_to_index(MEMFLAGS flag) { >> int index = (flag & 0xff); >> assert(index >= 0 && index < mt_number_of_types, >> "Invalid >> flag value %d.", (int)flag); >> return index; >> } >> >> so we're validating the index rather than the flag. >> >> Cheers, >> David >> >> >> >> >> The function that takes the index should validate >> the index, >> so that >> is fine. >> >> Which one were you actually passing the bad value >> to? :) >> >> This isn't a strong objection just musing if we >> can do >> better. And >> as the hs repos are still closed, and likely to >> remain so >> till early >> next week, we have some slack time :) >> >> >> :) Sure. >> >> Kind Regards, Thomas >> >> >> Cheers, >> David >> >> Thank you! >> >> Thomas >> >> >> >> From mandy.chung at oracle.com Thu Nov 17 20:28:10 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 17 Nov 2016 12:28:10 -0800 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> Message-ID: Thanks for adding the test. 2898 const char* SystemDictionary::loader_name(const oop loader) { 2904 const char* SystemDictionary::loader_name(const ClassLoaderData* loader_data) { These functions return the type name of the class loader, right? Would it be better to rename to loader_type_name to avoid confusion? 1957 oop loader = klass->class_loader(); 1959 oopDesc* class_loader_oop = java_lang_ClassLoader::name(loader); I think you mean this be ?loader_name_oop??? Mandy > On Nov 17, 2016, at 8:36 AM, Rachel Protacio wrote: > > Last one! http://cr.openjdk.java.net/~rprotacio/8165550.03 > - Thanks to Lois, I've added a new test case for a custom-named class loader with the unnamed module, and also throw an exception in the test if no CCE is thrown. > - Thanks to Mandy and Coleen, I've corrected the part of my code getting the Class Loader Name, which ironically, I was doing incorrectly before. > > Rachel From rachel.protacio at oracle.com Thu Nov 17 20:37:53 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Thu, 17 Nov 2016 15:37:53 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> Message-ID: <4a94c72b-038b-5569-180c-1edf2277074b@oracle.com> Thanks for the review, Mandy. I'll make those edits and commit. Rachel On 11/17/2016 3:28 PM, Mandy Chung wrote: > Thanks for adding the test. > > 2898 const char* SystemDictionary::loader_name(const oop loader) { > 2904 const char* SystemDictionary::loader_name(const ClassLoaderData* loader_data) { > > These functions return the type name of the class loader, right? Would it be better to rename to loader_type_name to avoid confusion? > > 1957 oop loader = klass->class_loader(); > 1959 oopDesc* class_loader_oop = java_lang_ClassLoader::name(loader); > I think you mean this be ?loader_name_oop??? > > Mandy > >> On Nov 17, 2016, at 8:36 AM, Rachel Protacio wrote: >> >> Last one! http://cr.openjdk.java.net/~rprotacio/8165550.03 >> - Thanks to Lois, I've added a new test case for a custom-named class loader with the unnamed module, and also throw an exception in the test if no CCE is thrown. >> - Thanks to Mandy and Coleen, I've corrected the part of my code getting the Class Loader Name, which ironically, I was doing incorrectly before. >> >> Rachel From rachel.protacio at oracle.com Thu Nov 17 20:47:25 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Thu, 17 Nov 2016 15:47:25 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: <4a94c72b-038b-5569-180c-1edf2277074b@oracle.com> References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> <4a94c72b-038b-5569-180c-1edf2277074b@oracle.com> Message-ID: Actually, I searched for the uses of SystemDictionary::loader_name. There are about 30 places where it's called, and I think changing it in all those places is beyond the scope of this changeset. I'll just do your second suggestion and commit. We can open a separate RFE if anyone wants to change the SystemDictionary function name. Rachel On 11/17/2016 3:37 PM, Rachel Protacio wrote: > Thanks for the review, Mandy. I'll make those edits and commit. > > Rachel > > > On 11/17/2016 3:28 PM, Mandy Chung wrote: >> Thanks for adding the test. >> >> 2898 const char* SystemDictionary::loader_name(const oop loader) { >> 2904 const char* SystemDictionary::loader_name(const ClassLoaderData* >> loader_data) { >> These functions return the type name of the class loader, right? >> Would it be better to rename to loader_type_name to avoid confusion? >> >> 1957 oop loader = klass->class_loader(); >> 1959 oopDesc* class_loader_oop = >> java_lang_ClassLoader::name(loader); >> I think you mean this be ?loader_name_oop??? >> >> Mandy >> >>> On Nov 17, 2016, at 8:36 AM, Rachel Protacio >>> wrote: >>> >>> Last one! http://cr.openjdk.java.net/~rprotacio/8165550.03 >>> - Thanks to Lois, I've added a new test case for a custom-named >>> class loader with the unnamed module, and also throw an exception in >>> the test if no CCE is thrown. >>> - Thanks to Mandy and Coleen, I've corrected the part of my code >>> getting the Class Loader Name, which ironically, I was doing >>> incorrectly before. >>> >>> Rachel > From mandy.chung at oracle.com Thu Nov 17 21:09:40 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 17 Nov 2016 13:09:40 -0800 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> <4a94c72b-038b-5569-180c-1edf2277074b@oracle.com> Message-ID: <503913BF-3A07-435F-B3F3-3090B50FE1FC@oracle.com> It?s fine with me as a separate issue to follow up in a future release. Mandy > On Nov 17, 2016, at 12:47 PM, Rachel Protacio wrote: > > Actually, I searched for the uses of SystemDictionary::loader_name. There are about 30 places where it's called, and I think changing it in all those places is beyond the scope of this changeset. I'll just do your second suggestion and commit. We can open a separate RFE if anyone wants to change the SystemDictionary function name. > > Rachel > > > On 11/17/2016 3:37 PM, Rachel Protacio wrote: >> Thanks for the review, Mandy. I'll make those edits and commit. >> >> Rachel >> >> >> On 11/17/2016 3:28 PM, Mandy Chung wrote: >>> Thanks for adding the test. >>> >>> 2898 const char* SystemDictionary::loader_name(const oop loader) { >>> 2904 const char* SystemDictionary::loader_name(const ClassLoaderData* loader_data) { >>> These functions return the type name of the class loader, right? Would it be better to rename to loader_type_name to avoid confusion? >>> >>> 1957 oop loader = klass->class_loader(); >>> 1959 oopDesc* class_loader_oop = java_lang_ClassLoader::name(loader); >>> I think you mean this be ?loader_name_oop??? >>> >>> Mandy >>> >>>> On Nov 17, 2016, at 8:36 AM, Rachel Protacio wrote: >>>> >>>> Last one! http://cr.openjdk.java.net/~rprotacio/8165550.03 >>>> - Thanks to Lois, I've added a new test case for a custom-named class loader with the unnamed module, and also throw an exception in the test if no CCE is thrown. >>>> - Thanks to Mandy and Coleen, I've corrected the part of my code getting the Class Loader Name, which ironically, I was doing incorrectly before. >>>> >>>> Rachel >> > From coleen.phillimore at oracle.com Thu Nov 17 21:46:56 2016 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Thu, 17 Nov 2016 16:46:56 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> Message-ID: Rachel, I noticed something small from this code review: This should be oop, not oopDesc*. We don't use this type like this except coming into the runtime code from Java. + oopDesc* class_loader_oop = java_lang_ClassLoader::name(loader); In the context that class loaders now have names, a new name for loader_name() function makes sense, but not for now. Can you fix oopDesc* before committing? Thanks, Coleen On 11/17/16 3:28 PM, Mandy Chung wrote: > Thanks for adding the test. > > 2898 const char* SystemDictionary::loader_name(const oop loader) { > 2904 const char* SystemDictionary::loader_name(const ClassLoaderData* loader_data) { > > These functions return the type name of the class loader, right? Would it be better to rename to loader_type_name to avoid confusion? > > 1957 oop loader = klass->class_loader(); > 1959 oopDesc* class_loader_oop = java_lang_ClassLoader::name(loader); > I think you mean this be ?loader_name_oop??? > > Mandy > >> On Nov 17, 2016, at 8:36 AM, Rachel Protacio wrote: >> >> Last one! http://cr.openjdk.java.net/~rprotacio/8165550.03 >> - Thanks to Lois, I've added a new test case for a custom-named class loader with the unnamed module, and also throw an exception in the test if no CCE is thrown. >> - Thanks to Mandy and Coleen, I've corrected the part of my code getting the Class Loader Name, which ironically, I was doing incorrectly before. >> >> Rachel From rachel.protacio at oracle.com Thu Nov 17 21:56:49 2016 From: rachel.protacio at oracle.com (Rachel Protacio) Date: Thu, 17 Nov 2016 16:56:49 -0500 Subject: RFR(M): 8165550: Add class loader names to ClassCastException message In-Reply-To: References: <582B0133.1010001@oracle.com> <582B1F71.2080401@oracle.com> <093ef24d-5d62-150f-6861-5ceba41f5e05@oracle.com> <728cb667-6cd1-882e-bf43-854d3a4c19d6@oracle.com> Message-ID: <7878efec-5d9e-9e3a-e5a0-6c287f4e83d0@oracle.com> Can do. Thanks! Rachel On 11/17/2016 4:46 PM, Coleen Phillimore wrote: > > Rachel, > > I noticed something small from this code review: > > This should be oop, not oopDesc*. We don't use this type like this > except coming into the runtime code from Java. > > + oopDesc* class_loader_oop = java_lang_ClassLoader::name(loader); > > In the context that class loaders now have names, a new name for > loader_name() function makes sense, but not for now. > > Can you fix oopDesc* before committing? > > Thanks, > Coleen > > On 11/17/16 3:28 PM, Mandy Chung wrote: >> Thanks for adding the test. >> >> 2898 const char* SystemDictionary::loader_name(const oop loader) { >> 2904 const char* SystemDictionary::loader_name(const ClassLoaderData* >> loader_data) { >> These functions return the type name of the class loader, right? >> Would it be better to rename to loader_type_name to avoid confusion? >> >> 1957 oop loader = klass->class_loader(); >> 1959 oopDesc* class_loader_oop = >> java_lang_ClassLoader::name(loader); >> I think you mean this be ?loader_name_oop??? >> >> Mandy >> >>> On Nov 17, 2016, at 8:36 AM, Rachel Protacio >>> wrote: >>> >>> Last one! http://cr.openjdk.java.net/~rprotacio/8165550.03 >>> - Thanks to Lois, I've added a new test case for a custom-named >>> class loader with the unnamed module, and also throw an exception in >>> the test if no CCE is thrown. >>> - Thanks to Mandy and Coleen, I've corrected the part of my code >>> getting the Class Loader Name, which ironically, I was doing >>> incorrectly before. >>> >>> Rachel > From david.holmes at oracle.com Fri Nov 18 06:34:45 2016 From: david.holmes at oracle.com (David Holmes) Date: Fri, 18 Nov 2016 16:34:45 +1000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <59f839950a21446fbe26aedfeb8151b8@derote13de14.global.corp.sap> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <5e9b0cf6243e471287da233b16fe1f63@derote13de14.global.corp.sap> <59f839950a21446fbe26aedfeb8151b8@derote13de14.global.corp.sap> Message-ID: <54c40e6f-d055-e711-bfb3-f6339a3870ea@oracle.com> Hi Goetz, On 17/11/2016 1:52 AM, Lindenmaier, Goetz wrote: > Hi, > > I addressed the additional topics, but in an extra change. > > First, I rounded up the NPTL bug workaround: > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.02 > > I fixed os::Posix::describe_pthread_attr() and current_stack_region() > In the os_cpu files. > Interestingly, in zero, current_stack_region() is already fixed. So on Linux, pthread_attr_getstack is also broken - in that it returns the address at the end of the guard section, instead of the usable stack? Can you add error checking on the pthread_attr_getguardsize call please. The duplication of the current_stack_region static method in the os-cpu files is horrible. This should be placed as a private method in the os class and moved to os_linux.cpp. RFE for 10. > Then I edited an extra webrev to fix > - unnecessary OS guard page on Compiler threads > - unpredictable minimal stack sizes because of varying (guard) page sizes: > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.minStSz/ > > Compiler threads now don't get OS guard pages. Saves one page per compiler thread. > > Also, the constants in the os__.cpp files now list minimum space > required for frames. I basically reduced them by the size of the guard zones. > Where minimum stack sizes are computed, the page size is known and the > minimal value is extended by the required space for the guard pages. As compiler threads > don't bang, no space for the shadow zone is needed. > > Should I add this additional change to 8169373? As this only deals with minimal > stack sizes, and reduces these sizes, I don't see a relevant risk. > Only on systems with large pages the minimal sizes now might be bigger, but > then the old sizes would have led to immediate stack overflows. Sorry this is getting a bit too divergent and disruptive to me for this stage of 9. You are redefining what the "minimum stack" values mean as well as changing their actual value. (BTW I'm sure there is a test of these values that would need updating.) I can't tell if it is right/wrong, necessary/unnecessary - but given the recent work done in this area I'm somewhat cautious about changing it again - in 9. David > > Best regards, > Goetz. > > >> -----Original Message----- >> From: David Holmes [mailto:david.holmes at oracle.com] >> Sent: Dienstag, 15. November 2016 00:54 >> To: Lindenmaier, Goetz ; hotspot-runtime- >> dev at openjdk.java.net >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >> >> Hi Goetz, >> >> On 15/11/2016 3:12 AM, Lindenmaier, Goetz wrote: >>> Hi David, >>> >>>> Did you really intend to increase the real minimum stack from 128K to >>>> 256K ? (on a 64K page system) >>> I finally figured why I have to set such high minimum values. >>> >>> A CompilerThread is a JavaThread. It's getting yellow, red etc >>> pages at the top. As these are 64K on 64K systems, the compiler >>> thread stack has to be this big. Thus you actually need similar >>> values once you run a linuxx86_64 VM on a system with 64K pages. >>> Does this exist? >> >> No idea, sorry. >> >>> But, this means that there is no use of the OS guard pages >>> configured in default_guard_size(). This is called with 'compiler_thread' >>> for CompilerThreads. >>> >>> We should probably also return '0' in that case. >>> >>> What do you think? >> >> I think I agree. As you note a CompilerThread is-a JavaThread even >> though the ThreadType enum treats them as completely distinct: >> >> enum ThreadType { >> vm_thread, >> cgc_thread, // Concurrent GC thread >> pgc_thread, // Parallel GC thread >> java_thread, >> compiler_thread, >> watcher_thread, >> os_thread >> }; >> >> so anything with VM supplied guard pages doesn't need the glibc ones as >> well. >> >> I'm struggling to understand why os::Linux::default_guard_size is >> defined per CPU type? I would not expect this to need to vary based on >> CPU at all. ?? >> >> Thanks, >> David >> >>> Best regards, >>> Goetz. >>> >>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>> // Creating guard page is very expensive. Java thread has HotSpot >>> // guard page, only enable glibc guard page for non-Java threads. >>> return (thr_type == java_thread ? 0 : page_size()); >>> } >>> From coleen.phillimore at oracle.com Fri Nov 18 14:40:23 2016 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Fri, 18 Nov 2016 09:40:23 -0500 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <54c40e6f-d055-e711-bfb3-f6339a3870ea@oracle.com> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <5e9b0cf6243e471287da233b16fe1f63@derote13de14.global.corp.sap> <59f839950a21446fbe26aedfeb8151b8@derote13de14.global.corp.sap> <54c40e6f-d055-e711-bfb3-f6339a3870ea@oracle.com> Message-ID: <36f1a912-0038-9399-f85d-7bbf4f56c460@oracle.com> On 11/18/16 1:34 AM, David Holmes wrote: > Hi Goetz, > > On 17/11/2016 1:52 AM, Lindenmaier, Goetz wrote: >> Hi, >> >> I addressed the additional topics, but in an extra change. >> >> First, I rounded up the NPTL bug workaround: >> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.02 >> >> I fixed os::Posix::describe_pthread_attr() and current_stack_region() >> In the os_cpu files. >> Interestingly, in zero, current_stack_region() is already fixed. > > So on Linux, pthread_attr_getstack is also broken - in that it returns > the address at the end of the guard section, instead of the usable stack? > > Can you add error checking on the pthread_attr_getguardsize call please. > > The duplication of the current_stack_region static method in the > os-cpu files is horrible. This should be placed as a private method in > the os class and moved to os_linux.cpp. RFE for 10. By duplication do you mean this bit of code? http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.02/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.udiff.html + // Work around NPTL stack guard error. + size_t guard_size = 0; + pthread_attr_getguardsize(&attr, &guard_size); + *bottom += guard_size; + *size -= guard_size; + This should be cleaned up with this patch and not an RFE for 10. I don't know the NPTL issues well enough to be a reviewer for this, but I don't like something I don't know about distributed in many places. > > >> Then I edited an extra webrev to fix >> - unnecessary OS guard page on Compiler threads >> - unpredictable minimal stack sizes because of varying (guard) page >> sizes: >> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.minStSz/ >> >> >> Compiler threads now don't get OS guard pages. Saves one page per >> compiler thread. >> >> Also, the constants in the os__.cpp files now list minimum >> space >> required for frames. I basically reduced them by the size of the >> guard zones. >> Where minimum stack sizes are computed, the page size is known and the >> minimal value is extended by the required space for the guard pages. >> As compiler threads >> don't bang, no space for the shadow zone is needed. >> >> Should I add this additional change to 8169373? As this only deals >> with minimal >> stack sizes, and reduces these sizes, I don't see a relevant risk. >> Only on systems with large pages the minimal sizes now might be >> bigger, but >> then the old sizes would have led to immediate stack overflows. > > Sorry this is getting a bit too divergent and disruptive to me for > this stage of 9. You are redefining what the "minimum stack" values > mean as well as changing their actual value. (BTW I'm sure there is a > test of these values that would need updating.) > > I can't tell if it is right/wrong, necessary/unnecessary - but given > the recent work done in this area I'm somewhat cautious about changing > it again - in 9. I think the additional changes should also wait for 10. I like the change to not have shadow pages for the compiler threads (not including AOT compiler threads, right?), but they also collide with work that I think Dan is going to do for windows. We don't want the os::Posix functions in these files. http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.minStSz/src/os_cpu/linux_x86/vm/os_linux_x86.cpp.udiff.html Also, the work that Dan and Ron did in this area didn't change the sizes and I'd want Dan to have time to comment on this. Thanks, Coleen > > David > >> >> Best regards, >> Goetz. >> >> >>> -----Original Message----- >>> From: David Holmes [mailto:david.holmes at oracle.com] >>> Sent: Dienstag, 15. November 2016 00:54 >>> To: Lindenmaier, Goetz ; hotspot-runtime- >>> dev at openjdk.java.net >>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>> >>> Hi Goetz, >>> >>> On 15/11/2016 3:12 AM, Lindenmaier, Goetz wrote: >>>> Hi David, >>>> >>>>> Did you really intend to increase the real minimum stack from 128K to >>>>> 256K ? (on a 64K page system) >>>> I finally figured why I have to set such high minimum values. >>>> >>>> A CompilerThread is a JavaThread. It's getting yellow, red etc >>>> pages at the top. As these are 64K on 64K systems, the compiler >>>> thread stack has to be this big. Thus you actually need similar >>>> values once you run a linuxx86_64 VM on a system with 64K pages. >>>> Does this exist? >>> >>> No idea, sorry. >>> >>>> But, this means that there is no use of the OS guard pages >>>> configured in default_guard_size(). This is called with >>>> 'compiler_thread' >>>> for CompilerThreads. >>>> >>>> We should probably also return '0' in that case. >>>> >>>> What do you think? >>> >>> I think I agree. As you note a CompilerThread is-a JavaThread even >>> though the ThreadType enum treats them as completely distinct: >>> >>> enum ThreadType { >>> vm_thread, >>> cgc_thread, // Concurrent GC thread >>> pgc_thread, // Parallel GC thread >>> java_thread, >>> compiler_thread, >>> watcher_thread, >>> os_thread >>> }; >>> >>> so anything with VM supplied guard pages doesn't need the glibc ones as >>> well. >>> >>> I'm struggling to understand why os::Linux::default_guard_size is >>> defined per CPU type? I would not expect this to need to vary based on >>> CPU at all. ?? >>> >>> Thanks, >>> David >>> >>>> Best regards, >>>> Goetz. >>>> >>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>>> // Creating guard page is very expensive. Java thread has HotSpot >>>> // guard page, only enable glibc guard page for non-Java threads. >>>> return (thr_type == java_thread ? 0 : page_size()); >>>> } >>>> From jiangli.zhou at Oracle.COM Sat Nov 19 02:14:19 2016 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Fri, 18 Nov 2016 18:14:19 -0800 Subject: RFR(s): 8169870: CDS: "assert(partition_size <= size()) failed: partition failed" on 32 bit JVM Message-ID: <60904AB8-9732-4D32-86B6-DF3197AF78A5@oracle.com> Please review following fix for JDK-8169870 . http://cr.openjdk.java.net/~jiangli/8169870/webrev.00/ This is an old bug but only discovered recently. Thanks Dmitry for reporting the issue. During CDS dump time, the VM tries to allocate memory (by calling ReservedSpace()) for shared space at the top of the java heap. If ReservedSpace() fails, it tries to allocate the memory at a different location by calling ReservedSpace() again. After the second ReservedSpace() call, it initializes the shared space without checking if the second allocation attempt is successful or not. Added a new case in LargeSharedSpace.java and tested on both 32-bit and 64-bit platforms. Thanks, Jiangli From chris.plummer at oracle.com Sat Nov 19 19:24:35 2016 From: chris.plummer at oracle.com (Chris Plummer) Date: Sat, 19 Nov 2016 11:24:35 -0800 Subject: RFR(s): 8169870: CDS: "assert(partition_size <= size()) failed: partition failed" on 32 bit JVM In-Reply-To: <60904AB8-9732-4D32-86B6-DF3197AF78A5@oracle.com> References: <60904AB8-9732-4D32-86B6-DF3197AF78A5@oracle.com> Message-ID: Hi Jiangli, The changes look fine. BTW, we also have LimitSharedSizes.java. Seems there is some overlap with this test. I'm not sure how we ended up with both. Also not sure how LargeSharedSpace.java caught this problem and LimitSharedSizes.java. I guess it was just unlucky with the size it chose. Chris On 11/18/16 6:14 PM, Jiangli Zhou wrote: > Please review following fix for JDK-8169870 . > > http://cr.openjdk.java.net/~jiangli/8169870/webrev.00/ > > This is an old bug but only discovered recently. Thanks Dmitry for reporting the issue. During CDS dump time, the VM tries to allocate memory (by calling ReservedSpace()) for shared space at the top of the java heap. If ReservedSpace() fails, it tries to allocate the memory at a different location by calling ReservedSpace() again. After the second ReservedSpace() call, it initializes the shared space without checking if the second allocation attempt is successful or not. > > Added a new case in LargeSharedSpace.java and tested on both 32-bit and 64-bit platforms. > > Thanks, > Jiangli From david.holmes at oracle.com Sun Nov 20 01:20:42 2016 From: david.holmes at oracle.com (David Holmes) Date: Sun, 20 Nov 2016 11:20:42 +1000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <36f1a912-0038-9399-f85d-7bbf4f56c460@oracle.com> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <5e9b0cf6243e471287da233b16fe1f63@derote13de14.global.corp.sap> <59f839950a21446fbe26aedfeb8151b8@derote13de14.global.corp.sap> <54c40e6f-d055-e711-bfb3-f6339a3870ea@oracle.com> <36f1a912-0038-9399-f85d-7bbf4f56c460@oracle.com> Message-ID: <2a9cc101-c567-6f7a-c109-1298ff9cc7f9@oracle.com> On 19/11/2016 12:40 AM, Coleen Phillimore wrote: > > > On 11/18/16 1:34 AM, David Holmes wrote: >> Hi Goetz, >> >> On 17/11/2016 1:52 AM, Lindenmaier, Goetz wrote: >>> Hi, >>> >>> I addressed the additional topics, but in an extra change. >>> >>> First, I rounded up the NPTL bug workaround: >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.02 >>> >>> I fixed os::Posix::describe_pthread_attr() and current_stack_region() >>> In the os_cpu files. >>> Interestingly, in zero, current_stack_region() is already fixed. >> >> So on Linux, pthread_attr_getstack is also broken - in that it returns >> the address at the end of the guard section, instead of the usable stack? >> >> Can you add error checking on the pthread_attr_getguardsize call please. >> >> The duplication of the current_stack_region static method in the >> os-cpu files is horrible. This should be placed as a private method in >> the os class and moved to os_linux.cpp. RFE for 10. > By duplication do you mean this bit of code? > > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.02/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.udiff.html > > > + // Work around NPTL stack guard error. > + size_t guard_size = 0; > + pthread_attr_getguardsize(&attr, &guard_size); > + *bottom += guard_size; > + *size -= guard_size; > + > > This should be cleaned up with this patch and not an RFE for 10. I meant the entire current_stack_region method to which this new fragment has been added. David > I don't know the NPTL issues well enough to be a reviewer for this, but > I don't like something I don't know about distributed in many places. > >> >> >>> Then I edited an extra webrev to fix >>> - unnecessary OS guard page on Compiler threads >>> - unpredictable minimal stack sizes because of varying (guard) page >>> sizes: >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.minStSz/ >>> >>> >>> Compiler threads now don't get OS guard pages. Saves one page per >>> compiler thread. >>> >>> Also, the constants in the os__.cpp files now list minimum >>> space >>> required for frames. I basically reduced them by the size of the >>> guard zones. >>> Where minimum stack sizes are computed, the page size is known and the >>> minimal value is extended by the required space for the guard pages. >>> As compiler threads >>> don't bang, no space for the shadow zone is needed. >>> >>> Should I add this additional change to 8169373? As this only deals >>> with minimal >>> stack sizes, and reduces these sizes, I don't see a relevant risk. >>> Only on systems with large pages the minimal sizes now might be >>> bigger, but >>> then the old sizes would have led to immediate stack overflows. >> >> Sorry this is getting a bit too divergent and disruptive to me for >> this stage of 9. You are redefining what the "minimum stack" values >> mean as well as changing their actual value. (BTW I'm sure there is a >> test of these values that would need updating.) >> >> I can't tell if it is right/wrong, necessary/unnecessary - but given >> the recent work done in this area I'm somewhat cautious about changing >> it again - in 9. > > I think the additional changes should also wait for 10. I like the > change to not have shadow pages for the compiler threads (not including > AOT compiler threads, right?), but they also collide with work that I > think Dan is going to do for windows. We don't want the os::Posix > functions in these files. > > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.minStSz/src/os_cpu/linux_x86/vm/os_linux_x86.cpp.udiff.html > > > Also, the work that Dan and Ron did in this area didn't change the sizes > and I'd want Dan to have time to comment on this. > > Thanks, > Coleen > >> >> David >> >>> >>> Best regards, >>> Goetz. >>> >>> >>>> -----Original Message----- >>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>> Sent: Dienstag, 15. November 2016 00:54 >>>> To: Lindenmaier, Goetz ; hotspot-runtime- >>>> dev at openjdk.java.net >>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>>> >>>> Hi Goetz, >>>> >>>> On 15/11/2016 3:12 AM, Lindenmaier, Goetz wrote: >>>>> Hi David, >>>>> >>>>>> Did you really intend to increase the real minimum stack from 128K to >>>>>> 256K ? (on a 64K page system) >>>>> I finally figured why I have to set such high minimum values. >>>>> >>>>> A CompilerThread is a JavaThread. It's getting yellow, red etc >>>>> pages at the top. As these are 64K on 64K systems, the compiler >>>>> thread stack has to be this big. Thus you actually need similar >>>>> values once you run a linuxx86_64 VM on a system with 64K pages. >>>>> Does this exist? >>>> >>>> No idea, sorry. >>>> >>>>> But, this means that there is no use of the OS guard pages >>>>> configured in default_guard_size(). This is called with >>>>> 'compiler_thread' >>>>> for CompilerThreads. >>>>> >>>>> We should probably also return '0' in that case. >>>>> >>>>> What do you think? >>>> >>>> I think I agree. As you note a CompilerThread is-a JavaThread even >>>> though the ThreadType enum treats them as completely distinct: >>>> >>>> enum ThreadType { >>>> vm_thread, >>>> cgc_thread, // Concurrent GC thread >>>> pgc_thread, // Parallel GC thread >>>> java_thread, >>>> compiler_thread, >>>> watcher_thread, >>>> os_thread >>>> }; >>>> >>>> so anything with VM supplied guard pages doesn't need the glibc ones as >>>> well. >>>> >>>> I'm struggling to understand why os::Linux::default_guard_size is >>>> defined per CPU type? I would not expect this to need to vary based on >>>> CPU at all. ?? >>>> >>>> Thanks, >>>> David >>>> >>>>> Best regards, >>>>> Goetz. >>>>> >>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>>>> // Creating guard page is very expensive. Java thread has HotSpot >>>>> // guard page, only enable glibc guard page for non-Java threads. >>>>> return (thr_type == java_thread ? 0 : page_size()); >>>>> } >>>>> > From serguei.spitsyn at oracle.com Mon Nov 21 04:30:41 2016 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Sun, 20 Nov 2016 20:30:41 -0800 Subject: RFR(s): 8169870: CDS: "assert(partition_size <= size()) failed: partition failed" on 32 bit JVM In-Reply-To: <60904AB8-9732-4D32-86B6-DF3197AF78A5@oracle.com> References: <60904AB8-9732-4D32-86B6-DF3197AF78A5@oracle.com> Message-ID: <82ea5297-0ecf-bea5-94ac-2b2bd5897183@oracle.com> Hi Jiangli, This looks good. Thanks, Serguei On 11/18/16 18:14, Jiangli Zhou wrote: > Please review following fix for JDK-8169870 . > > http://cr.openjdk.java.net/~jiangli/8169870/webrev.00/ > > This is an old bug but only discovered recently. Thanks Dmitry for reporting the issue. During CDS dump time, the VM tries to allocate memory (by calling ReservedSpace()) for shared space at the top of the java heap. If ReservedSpace() fails, it tries to allocate the memory at a different location by calling ReservedSpace() again. After the second ReservedSpace() call, it initializes the shared space without checking if the second allocation attempt is successful or not. > > Added a new case in LargeSharedSpace.java and tested on both 32-bit and 64-bit platforms. > > Thanks, > Jiangli From ioi.lam at oracle.com Mon Nov 21 06:17:54 2016 From: ioi.lam at oracle.com (Ioi Lam) Date: Sun, 20 Nov 2016 22:17:54 -0800 Subject: RFR(s): 8169870: CDS: "assert(partition_size <= size()) failed: partition failed" on 32 bit JVM In-Reply-To: <82ea5297-0ecf-bea5-94ac-2b2bd5897183@oracle.com> References: <60904AB8-9732-4D32-86B6-DF3197AF78A5@oracle.com> <82ea5297-0ecf-bea5-94ac-2b2bd5897183@oracle.com> Message-ID: <58329192.3070602@oracle.com> Hi Jiangli, The VM changes look OK to me. For the test case, could you add a comment for these two output strings? "larger than compressed klass limit" "Unable to allocate memory for shared space" under what condition will they happen? Thanks - Ioi On 11/20/16 8:30 PM, serguei.spitsyn at oracle.com wrote: > Hi Jiangli, > > This looks good. > > Thanks, > Serguei > > > On 11/18/16 18:14, Jiangli Zhou wrote: >> Please review following fix for JDK-8169870 >> . >> >> http://cr.openjdk.java.net/~jiangli/8169870/webrev.00/ >> >> >> This is an old bug but only discovered recently. Thanks Dmitry for >> reporting the issue. During CDS dump time, the VM tries to allocate >> memory (by calling ReservedSpace()) for shared space at the top of >> the java heap. If ReservedSpace() fails, it tries to allocate the >> memory at a different location by calling ReservedSpace() again. >> After the second ReservedSpace() call, it initializes the shared >> space without checking if the second allocation attempt is successful >> or not. >> >> Added a new case in LargeSharedSpace.java and tested on both 32-bit >> and 64-bit platforms. >> >> Thanks, >> Jiangli > From erik.helin at oracle.com Mon Nov 21 07:36:26 2016 From: erik.helin at oracle.com (Erik Helin) Date: Mon, 21 Nov 2016 08:36:26 +0100 Subject: Question about VirtualSpace and MPSS support In-Reply-To: References: Message-ID: On 11/17/2016 12:27 PM, Thomas St?fe wrote: > I wonder why VirtualSpace keeps the unaligned high water mark around > instead of, say, just rounding _high up to the next page boundary. A > simpler approach would have been to advance _high to the next page > boundary and just forget the exact increments the caller requested. > Then, caller needs to deal with the fact that an VirtualSpace may expand > by more than the requested size, and for future size calculations always > needs to refer to VirtualSpace::high(). This could be a cleanup for jdk10. Well, you would have to make sure that all code that uses VirtualSpace::expand_by today can deal with this change in semantics. The benefit of the change also hinges on how much of the code that uses VirtualSpace today needs to keep track of how much a VirtualSpace have expanded (without aligning to page size). If a lot of code need this functionality, then you will just be moving it out of VirtualSpace and into a number of places spread throughout the code. OTOH, if very little code requires this knowledge, then the change might be worth it. One could also consider writing a wrapper around VirtualSpace that keeps track of exactly this information. Thanks, Erik > Thanks, Thomas From goetz.lindenmaier at sap.com Mon Nov 21 10:31:07 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Mon, 21 Nov 2016 10:31:07 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <2a9cc101-c567-6f7a-c109-1298ff9cc7f9@oracle.com> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <5e9b0cf6243e471287da233b16fe1f63@derote13de14.global.corp.sap> <59f839950a21446fbe26aedfeb8151b8@derote13de14.global.corp.sap> <54c40e6f-d055-e711-bfb3-f6339a3870ea@oracle.com> <36f1a912-0038-9399-f85d-7bbf4f56c460@oracle.com> <2a9cc101-c567-6f7a-c109-1298ff9cc7f9@oracle.com> Message-ID: <668f5f07c10140f2879a3464f70faea5@DEROTE13DE08.global.corp.sap> Hi Coleen, David, the cleanups we were talking about are in this webrev, which I would like to apply on top of 8169373: http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix The code is identical in all the os_cpu files. (Curently it's edited on top of the other change). I would understand if you don't like the other changes I propose For 9. But I still think this is just how you expect this to work, the minimum stack size is independent of page size and thus should be specified without considering the guard pages. And therefore it should go to 9 where we already changed a lot wrt. stack sizes. Coleen, would you mind pointing me to the bugs Dan and Ron are working on? I'd like to understand the possible conflicts. Best regards, Goetz. > -----Original Message----- > From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- > bounces at openjdk.java.net] On Behalf Of David Holmes > Sent: Sonntag, 20. November 2016 02:21 > To: Coleen Phillimore ; hotspot-runtime- > dev at openjdk.java.net > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > On 19/11/2016 12:40 AM, Coleen Phillimore wrote: > > > > > > On 11/18/16 1:34 AM, David Holmes wrote: > >> Hi Goetz, > >> > >> On 17/11/2016 1:52 AM, Lindenmaier, Goetz wrote: > >>> Hi, > >>> > >>> I addressed the additional topics, but in an extra change. > >>> > >>> First, I rounded up the NPTL bug workaround: > >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > stackFix/webrev.02 > >>> > >>> I fixed os::Posix::describe_pthread_attr() and current_stack_region() > >>> In the os_cpu files. > >>> Interestingly, in zero, current_stack_region() is already fixed. > >> > >> So on Linux, pthread_attr_getstack is also broken - in that it returns > >> the address at the end of the guard section, instead of the usable stack? > >> > >> Can you add error checking on the pthread_attr_getguardsize call please. > >> > >> The duplication of the current_stack_region static method in the > >> os-cpu files is horrible. This should be placed as a private method in > >> the os class and moved to os_linux.cpp. RFE for 10. > > By duplication do you mean this bit of code? > > > > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > stackFix/webrev.02/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.udiff.html > > > > > > + // Work around NPTL stack guard error. > > + size_t guard_size = 0; > > + pthread_attr_getguardsize(&attr, &guard_size); > > + *bottom += guard_size; > > + *size -= guard_size; > > + > > > > This should be cleaned up with this patch and not an RFE for 10. > > I meant the entire current_stack_region method to which this new > fragment has been added. > > David > > > I don't know the NPTL issues well enough to be a reviewer for this, but > > I don't like something I don't know about distributed in many places. > > > >> > >> > >>> Then I edited an extra webrev to fix > >>> - unnecessary OS guard page on Compiler threads > >>> - unpredictable minimal stack sizes because of varying (guard) page > >>> sizes: > >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > stackFix/webrev.minStSz/ > >>> > >>> > >>> Compiler threads now don't get OS guard pages. Saves one page per > >>> compiler thread. > >>> > >>> Also, the constants in the os__.cpp files now list minimum > >>> space > >>> required for frames. I basically reduced them by the size of the > >>> guard zones. > >>> Where minimum stack sizes are computed, the page size is known and > the > >>> minimal value is extended by the required space for the guard pages. > >>> As compiler threads > >>> don't bang, no space for the shadow zone is needed. > >>> > >>> Should I add this additional change to 8169373? As this only deals > >>> with minimal > >>> stack sizes, and reduces these sizes, I don't see a relevant risk. > >>> Only on systems with large pages the minimal sizes now might be > >>> bigger, but > >>> then the old sizes would have led to immediate stack overflows. > >> > >> Sorry this is getting a bit too divergent and disruptive to me for > >> this stage of 9. You are redefining what the "minimum stack" values > >> mean as well as changing their actual value. (BTW I'm sure there is a > >> test of these values that would need updating.) > >> > >> I can't tell if it is right/wrong, necessary/unnecessary - but given > >> the recent work done in this area I'm somewhat cautious about changing > >> it again - in 9. > > > > I think the additional changes should also wait for 10. I like the > > change to not have shadow pages for the compiler threads (not including > > AOT compiler threads, right?), but they also collide with work that I > > think Dan is going to do for windows. We don't want the os::Posix > > functions in these files. > > > > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > stackFix/webrev.minStSz/src/os_cpu/linux_x86/vm/os_linux_x86.cpp.udiff. > html > > > > > > Also, the work that Dan and Ron did in this area didn't change the sizes > > and I'd want Dan to have time to comment on this. > > > > Thanks, > > Coleen > > > >> > >> David > >> > >>> > >>> Best regards, > >>> Goetz. > >>> > >>> > >>>> -----Original Message----- > >>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>> Sent: Dienstag, 15. November 2016 00:54 > >>>> To: Lindenmaier, Goetz ; hotspot- > runtime- > >>>> dev at openjdk.java.net > >>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > error. > >>>> > >>>> Hi Goetz, > >>>> > >>>> On 15/11/2016 3:12 AM, Lindenmaier, Goetz wrote: > >>>>> Hi David, > >>>>> > >>>>>> Did you really intend to increase the real minimum stack from 128K > to > >>>>>> 256K ? (on a 64K page system) > >>>>> I finally figured why I have to set such high minimum values. > >>>>> > >>>>> A CompilerThread is a JavaThread. It's getting yellow, red etc > >>>>> pages at the top. As these are 64K on 64K systems, the compiler > >>>>> thread stack has to be this big. Thus you actually need similar > >>>>> values once you run a linuxx86_64 VM on a system with 64K pages. > >>>>> Does this exist? > >>>> > >>>> No idea, sorry. > >>>> > >>>>> But, this means that there is no use of the OS guard pages > >>>>> configured in default_guard_size(). This is called with > >>>>> 'compiler_thread' > >>>>> for CompilerThreads. > >>>>> > >>>>> We should probably also return '0' in that case. > >>>>> > >>>>> What do you think? > >>>> > >>>> I think I agree. As you note a CompilerThread is-a JavaThread even > >>>> though the ThreadType enum treats them as completely distinct: > >>>> > >>>> enum ThreadType { > >>>> vm_thread, > >>>> cgc_thread, // Concurrent GC thread > >>>> pgc_thread, // Parallel GC thread > >>>> java_thread, > >>>> compiler_thread, > >>>> watcher_thread, > >>>> os_thread > >>>> }; > >>>> > >>>> so anything with VM supplied guard pages doesn't need the glibc ones > as > >>>> well. > >>>> > >>>> I'm struggling to understand why os::Linux::default_guard_size is > >>>> defined per CPU type? I would not expect this to need to vary based > on > >>>> CPU at all. ?? > >>>> > >>>> Thanks, > >>>> David > >>>> > >>>>> Best regards, > >>>>> Goetz. > >>>>> > >>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > >>>>> // Creating guard page is very expensive. Java thread has HotSpot > >>>>> // guard page, only enable glibc guard page for non-Java threads. > >>>>> return (thr_type == java_thread ? 0 : page_size()); > >>>>> } > >>>>> > > From thomas.stuefe at gmail.com Mon Nov 21 11:02:44 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 21 Nov 2016 12:02:44 +0100 Subject: Windows, should we assert CloseHandle errors? Message-ID: Hi all, so, the other day I found a bug (in our code, not in the original OpenJDK) which was caused by a double OS handle close. Similar to a double free, but with handles. On Windows, many OS resources are accessed via kernel handles, which are closed via the generic CloseHandle() API. Numerical values for HANDLE seem to be handed out sequentially, and the chance to reuse a numerical handle value is high. This makes Windows OS coding susceptible to double-handle-free-errors because the chance of a double free freeing another handle in a completely unrelated piece of code is high. In our case, due to an tiny error when merging our additions to OpenJDK upstream sources, we accidentally closed the handle in ProcessImpl twice. This broke completely unrelated code sections, just because they happened to allocate a new handle between the first the the second CloseHandle() call in ProcessImpl, which happened to have the same numerical value as the former ProcessImpl handle. My question: due to the difficulty of tracking down these errors, should we not assert() if CloseHandle() fails with INVALID_HANDLE_VALUE (many call sites of CloseHandle() in hotspot do not check the return code)? For a double free, we would catch all cases where between the first and the second CloseHandle() call this handle was not handed out again. I think this would help a lot in these cases. The argument against asserting is that any code - also third party native code - may cause a double free and hence an assert in our code. What do you think? Kind Regards, Thomas From thomas.stuefe at gmail.com Mon Nov 21 15:17:48 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 21 Nov 2016 16:17:48 +0100 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> Message-ID: Hi Chris, On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer wrote: > On 10/18/16 11:49 PM, David Holmes wrote: > >> >> >> On 19/10/2016 3:17 PM, Thomas St?fe wrote: >> >>> >>> >>> On Wed, Oct 19, 2016 at 3:21 AM, David Holmes >> > wrote: >>> >>> On 18/10/2016 3:39 PM, Thomas St?fe wrote: >>> >>> Hi David, Max, >>> >>> I changed the asserts according to Max' suggestion. Instead of >>> checking >>> inside flag_to_index, now I check before callers of this >>> function use >>> this value to access memory. >>> >>> >>> I don't see where Max suggested that?? >>> >>> >>> Max wrote: " I think the decision on whether to access a slot should >>> occur as close to memory accessing code as possible." and proceeded to >>> suggest fixing VirtualMemorySnapshot::by_type() as well. >>> >> >> I did not interpret that comment that way, and was puzzled by the >> reference to by_type. >> >>> >>> It doesn't make sense to me to have all the callers of flag_to_index >>> check what it returned instead of doing it inside flag_to_index. >>> >>> >>> I disagree. Imho it makes sense to either check the Memflags enumeration >>> input argument in flag_to_index() or the returned index before >>> consumption. In both cases one knows the valid value range. Strictly >>> speaking checking the index in flag_to_index() cannot be done because it >>> is a faceless int type whose valid values are not yet known. >>> >> >> The index has to fall in the range 0 <= index <= mt_number_of_types, and >> I was suggesting that it makes more sense to verify this once in >> flag_to_index() than in all the callers of flag_to_index. >> > Hi Thomas, > > Just catching up on this thread. This is the same conclusion I came to. I > don't understand what you mean by "it is a faceless int type whose valid > values are not yet known". > > Thank you for looking into this! "it is a faceless int type whose valid values are not yet known" was maybe expressed sloppily: My first patch checked the input enum "flag" argument inside NMTUtil::flag_to_index() for correct enum values : http://cr.openjdk.java .net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ David did not like that, he thought it strange to check an enum for correct enum values. I can see his point. He would have preferred instead an assert inside NMTUtil::flag_to_index() which before returning checks the to-be-returned index for 0>=i>=mt_number_of_types. I did not like that because strictly speaking inside of NMTUtil::flag_to_index() it is not known that the returned integer will be used by the caller as an index into an array of mt_number_of_types length. It is all extreme nitpicking and in the end amounts to the same :) The current patch does neither, but follows Max' suggestion of checking the index right before it is consumed to access an array. I think this is a good solution and very clear. > BTW, I'll sponsor this fix for you once it is finalized. Please update the > copyright dates first. > Thank you. I'll update the webrev and repost. > > thanks, > > Chris > > Kind Regards, Thomas > >> David >> >> It is all academical and mostly a matter of taste. >>> >>> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >>> d-check_MEMFLAGS/webrev.01/webrev/index.html >>> >> ld-check_MEMFLAGS/webrev.01/webrev/index.html> >>> >>> As David correctly writes, this is technically not a bug, so I >>> guess >>> this will have to wait until java 10. >>> >>> >>> Yes, afraid so. >>> >>> >>> The fix is trivial and I will try to get fc extension for this (now that >>> Goetz explained to me how to do this :). It seems this is done for many >>> other non-bug issues as well. >>> >>> ..Thomas >>> >>> >>> Thanks, >>> David >>> >>> Kind Regards, Thomas >>> >>> >>> >>> On Fri, Oct 14, 2016 at 12:57 AM, David Holmes >>> >>> >> >> wrote: >>> >>> On 13/10/2016 10:53 PM, Thomas St?fe wrote: >>> >>> Hi David, >>> >>> On Thu, Oct 13, 2016 at 12:08 PM, David Holmes >>> >> >> > >>> >> >>> >>> >> >>> wrote: >>> >>> Hi Thomas, >>> >>> On 13/10/2016 3:49 PM, Thomas St?fe wrote: >>> >>> Hi all, >>> >>> may I have plase a review for this tiny change? >>> It just adds >>> some assert to NMT. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8167650 >>> >>> >> > >>> >>> >> >>> >> >> >>> webrev: >>> >>> >>> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ >>> >>> >>> >> > >>> >>> >>> >> >>> >>> >> >> >>> MEMFLAGS/webrev.00/webrev/ >>> >>> We had an ugly memory overwrite caused by this - >>> ultimately our >>> fault, because we fed an invalid memory flag to >>> NMT - >>> but it was >>> difficult to find. An assert would have saved >>> some time. >>> >>> >>> I'm a little perplexed with asserting that something >>> of MEMFLAGS >>> type must be an actual MEMFLAGS value - it implies >>> the caller is >>> coercing plain int to MEMFLAGS, and I don't have much >>> sympathy if >>> they mess that up. Can't help wondering if there is >>> some >>> clever C++ >>> trick to flag bad conversions at compile-time? >>> >>> >>> The error was caused by an uninitialized variable of type >>> MEMFLAGS. This >>> was our fault, we have heavily modified allocation.hpp >>> and >>> introduced an >>> error then merging changes from upstream. Due to a >>> merging error >>> this >>> lead to a case where Arena::_flags was not initialized >>> and >>> contained a >>> very large value. >>> >>> >>> Ah I see. Lack of default initialization can be annoying :) >>> >>> I admit it looks funny. If it bothers you, I could >>> instead check the >>> returned index to be in the range for the size of the >>> _malloc >>> array in >>> MallocMemorySnapshot::by_type(). Technically, it would >>> mean the >>> same. >>> >>> >>> So I just realized that here: >>> >>> 62 // Map memory type to human readable name >>> 63 static const char* flag_to_name(MEMFLAGS flag) { >>> 64 assert(flag >= 0 && flag < mt_number_of_types, >>> "Invalid >>> flag value %d.", (int)flag); >>> 65 return _memory_type_names[flag_to_index(flag)]; >>> 66 } >>> >>> we call flag_to_index, so the assert is redundant as it is >>> already >>> in flag_to_index. Then presumably we change flag_to_index to >>> something like this: >>> >>> static inline int flag_to_index(MEMFLAGS flag) { >>> int index = (flag & 0xff); >>> assert(index >= 0 && index < mt_number_of_types, >>> "Invalid >>> flag value %d.", (int)flag); >>> return index; >>> } >>> >>> so we're validating the index rather than the flag. >>> >>> Cheers, >>> David >>> >>> >>> >>> >>> The function that takes the index should validate >>> the index, >>> so that >>> is fine. >>> >>> Which one were you actually passing the bad value >>> to? :) >>> >>> This isn't a strong objection just musing if we can >>> do >>> better. And >>> as the hs repos are still closed, and likely to >>> remain so >>> till early >>> next week, we have some slack time :) >>> >>> >>> :) Sure. >>> >>> Kind Regards, Thomas >>> >>> >>> Cheers, >>> David >>> >>> Thank you! >>> >>> Thomas >>> >>> >>> >>> >>> > > From thomas.stuefe at gmail.com Mon Nov 21 15:21:17 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 21 Nov 2016 16:21:17 +0100 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> Message-ID: Hi all, this small issue got fc extension, so lets wrap this up. Thanks to all for reviewing! New webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ No changes to the last one but the updated copyrights Chris did ask for. Solution still follows Max' suggestion of checking the index right before consumption. Thanks, and Kind Regards, Thomas On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe wrote: > Hi Chris, > > On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer > wrote: > >> On 10/18/16 11:49 PM, David Holmes wrote: >> >>> >>> >>> On 19/10/2016 3:17 PM, Thomas St?fe wrote: >>> >>>> >>>> >>>> On Wed, Oct 19, 2016 at 3:21 AM, David Holmes >>> > wrote: >>>> >>>> On 18/10/2016 3:39 PM, Thomas St?fe wrote: >>>> >>>> Hi David, Max, >>>> >>>> I changed the asserts according to Max' suggestion. Instead of >>>> checking >>>> inside flag_to_index, now I check before callers of this >>>> function use >>>> this value to access memory. >>>> >>>> >>>> I don't see where Max suggested that?? >>>> >>>> >>>> Max wrote: " I think the decision on whether to access a slot should >>>> occur as close to memory accessing code as possible." and proceeded to >>>> suggest fixing VirtualMemorySnapshot::by_type() as well. >>>> >>> >>> I did not interpret that comment that way, and was puzzled by the >>> reference to by_type. >>> >>>> >>>> It doesn't make sense to me to have all the callers of flag_to_index >>>> check what it returned instead of doing it inside flag_to_index. >>>> >>>> >>>> I disagree. Imho it makes sense to either check the Memflags enumeration >>>> input argument in flag_to_index() or the returned index before >>>> consumption. In both cases one knows the valid value range. Strictly >>>> speaking checking the index in flag_to_index() cannot be done because it >>>> is a faceless int type whose valid values are not yet known. >>>> >>> >>> The index has to fall in the range 0 <= index <= mt_number_of_types, and >>> I was suggesting that it makes more sense to verify this once in >>> flag_to_index() than in all the callers of flag_to_index. >>> >> Hi Thomas, >> >> Just catching up on this thread. This is the same conclusion I came to. I >> don't understand what you mean by "it is a faceless int type whose valid >> values are not yet known". >> >> > Thank you for looking into this! > > "it is a faceless int type whose valid values are not yet known" was maybe > expressed sloppily: > > My first patch checked the input enum "flag" argument inside > NMTUtil::flag_to_index() for correct enum values : > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT- > should-check_MEMFLAGS/webrev.00/webrev/ > > David did not like that, he thought it strange to check an enum for > correct enum values. I can see his point. He would have preferred instead > an assert inside NMTUtil::flag_to_index() which before returning checks the > to-be-returned index for 0>=i>=mt_number_of_types. I did not like that > because strictly speaking inside of NMTUtil::flag_to_index() it is not > known that the returned integer will be used by the caller as an index into > an array of mt_number_of_types length. > > It is all extreme nitpicking and in the end amounts to the same :) > > The current patch does neither, but follows Max' suggestion of checking > the index right before it is consumed to access an array. I think this is a > good solution and very clear. > > > >> BTW, I'll sponsor this fix for you once it is finalized. Please update >> the copyright dates first. >> > > Thank you. I'll update the webrev and repost. > > >> >> thanks, >> >> Chris >> >> > Kind Regards, Thomas > > >> >>> David >>> >>> It is all academical and mostly a matter of taste. >>>> >>>> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >>>> d-check_MEMFLAGS/webrev.01/webrev/index.html >>>> >>> ld-check_MEMFLAGS/webrev.01/webrev/index.html> >>>> >>>> As David correctly writes, this is technically not a bug, so I >>>> guess >>>> this will have to wait until java 10. >>>> >>>> >>>> Yes, afraid so. >>>> >>>> >>>> The fix is trivial and I will try to get fc extension for this (now that >>>> Goetz explained to me how to do this :). It seems this is done for many >>>> other non-bug issues as well. >>>> >>>> ..Thomas >>>> >>>> >>>> Thanks, >>>> David >>>> >>>> Kind Regards, Thomas >>>> >>>> >>>> >>>> On Fri, Oct 14, 2016 at 12:57 AM, David Holmes >>>> >>>> >>> >> wrote: >>>> >>>> On 13/10/2016 10:53 PM, Thomas St?fe wrote: >>>> >>>> Hi David, >>>> >>>> On Thu, Oct 13, 2016 at 12:08 PM, David Holmes >>>> >>> >>> m >>>> > >>>> >>> >>>> >>>> >>> >>> wrote: >>>> >>>> Hi Thomas, >>>> >>>> On 13/10/2016 3:49 PM, Thomas St?fe wrote: >>>> >>>> Hi all, >>>> >>>> may I have plase a review for this tiny change? >>>> It just adds >>>> some assert to NMT. >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8167650 >>>> >>>> >>> > >>>> >>>> >>> >>>> >>> >> >>>> webrev: >>>> >>>> >>>> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ >>>> >>>> >>>> >>> > >>>> >>>> >>>> >>> >>>> >>>> >>> >>> >>> >>>> MEMFLAGS/webrev.00/webrev/ >>>> >>>> We had an ugly memory overwrite caused by this - >>>> ultimately our >>>> fault, because we fed an invalid memory flag to >>>> NMT - >>>> but it was >>>> difficult to find. An assert would have saved >>>> some time. >>>> >>>> >>>> I'm a little perplexed with asserting that something >>>> of MEMFLAGS >>>> type must be an actual MEMFLAGS value - it implies >>>> the caller is >>>> coercing plain int to MEMFLAGS, and I don't have >>>> much >>>> sympathy if >>>> they mess that up. Can't help wondering if there is >>>> some >>>> clever C++ >>>> trick to flag bad conversions at compile-time? >>>> >>>> >>>> The error was caused by an uninitialized variable of >>>> type >>>> MEMFLAGS. This >>>> was our fault, we have heavily modified allocation.hpp >>>> and >>>> introduced an >>>> error then merging changes from upstream. Due to a >>>> merging error >>>> this >>>> lead to a case where Arena::_flags was not initialized >>>> and >>>> contained a >>>> very large value. >>>> >>>> >>>> Ah I see. Lack of default initialization can be annoying :) >>>> >>>> I admit it looks funny. If it bothers you, I could >>>> instead check the >>>> returned index to be in the range for the size of the >>>> _malloc >>>> array in >>>> MallocMemorySnapshot::by_type(). Technically, it would >>>> mean the >>>> same. >>>> >>>> >>>> So I just realized that here: >>>> >>>> 62 // Map memory type to human readable name >>>> 63 static const char* flag_to_name(MEMFLAGS flag) { >>>> 64 assert(flag >= 0 && flag < mt_number_of_types, >>>> "Invalid >>>> flag value %d.", (int)flag); >>>> 65 return _memory_type_names[flag_to_index(flag)]; >>>> 66 } >>>> >>>> we call flag_to_index, so the assert is redundant as it is >>>> already >>>> in flag_to_index. Then presumably we change flag_to_index to >>>> something like this: >>>> >>>> static inline int flag_to_index(MEMFLAGS flag) { >>>> int index = (flag & 0xff); >>>> assert(index >= 0 && index < mt_number_of_types, >>>> "Invalid >>>> flag value %d.", (int)flag); >>>> return index; >>>> } >>>> >>>> so we're validating the index rather than the flag. >>>> >>>> Cheers, >>>> David >>>> >>>> >>>> >>>> >>>> The function that takes the index should validate >>>> the index, >>>> so that >>>> is fine. >>>> >>>> Which one were you actually passing the bad value >>>> to? :) >>>> >>>> This isn't a strong objection just musing if we can >>>> do >>>> better. And >>>> as the hs repos are still closed, and likely to >>>> remain so >>>> till early >>>> next week, we have some slack time :) >>>> >>>> >>>> :) Sure. >>>> >>>> Kind Regards, Thomas >>>> >>>> >>>> Cheers, >>>> David >>>> >>>> Thank you! >>>> >>>> Thomas >>>> >>>> >>>> >>>> >>>> >> >> > From thomas.stuefe at gmail.com Mon Nov 21 15:24:54 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 21 Nov 2016 16:24:54 +0100 Subject: Question about VirtualSpace and MPSS support In-Reply-To: References: Message-ID: On Mon, Nov 21, 2016 at 8:36 AM, Erik Helin wrote: > On 11/17/2016 12:27 PM, Thomas St?fe wrote: > >> I wonder why VirtualSpace keeps the unaligned high water mark around >> instead of, say, just rounding _high up to the next page boundary. A >> simpler approach would have been to advance _high to the next page >> boundary and just forget the exact increments the caller requested. >> Then, caller needs to deal with the fact that an VirtualSpace may expand >> by more than the requested size, and for future size calculations always >> needs to refer to VirtualSpace::high(). This could be a cleanup for jdk10. >> > > Well, you would have to make sure that all code that uses > VirtualSpace::expand_by today can deal with this change in semantics. The > benefit of the change also hinges on how much of the code that uses > VirtualSpace today needs to keep track of how much a VirtualSpace have > expanded (without aligning to page size). If a lot of code need this > functionality, then you will just be moving it out of VirtualSpace and into > a number of places spread throughout the code. OTOH, if very little code > requires this knowledge, then the change might be worth it. > > You are right, if this change forces the callers to keep track of the unaligned high water mark themselves, nothing is gained. > One could also consider writing a wrapper around VirtualSpace that keeps > track of exactly this information. > Good idea. > Thanks, > Erik > > In the meantime I did find another way to handle the humongous metaspace chunks, so my original idea of allocating from the top of the VirtualSpace may not be needed. Thank you for looking into this! ..Thomas Thanks, Thomas >> > From daniel.daugherty at oracle.com Mon Nov 21 16:28:14 2016 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 21 Nov 2016 09:28:14 -0700 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> Message-ID: <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> Sorry for the delayed responses to this thread. I've been on vacation... One comment/query embedded below... On 11/10/16 8:40 PM, David Holmes wrote: > Hi Goetz, > > On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: >> Hi David, >> >> This issue is different to 6675312, see also my comment in the bug. >> >> It appears running jtreg test runtime/Thread/TooSmallStackSize.java, >> with my patch below you can reproduce it on linuxx86_64. You can not >> do that with 6675312. Also, I would assume there are systems out there >> on x86 that uses 64-K pages, did you run the tests on these? I would >> assume you get hard crashes with stack overflows in the first C2 >> compilation if there is only 64K usable CompilerThreadStack. >> >> My fix does not affect Java threads, which are the largest amount >> of threads used by the VM. It affects only the non-Java threads. >> It adds one page to these threads. The page does not require memory, >> as it's protected. The stack will only require more space if the thread >> ran into a stack overflow before the fix as else the pages are not >> mapped. >> This are stack overflows that cause hard crashes, at least on ppc the VM >> does not properly catch these stack overflows, so any setup working now >> will not run into the additional space. Altogether there should be no >> effect on running systems besides requiring one more entry in the >> page table per non-Java thread. >> >> The problem is caused by a rather recent change (8140520: segfault on >> solaris-amd64 >> with "-XX:VMThreadStackSize=1" option) which was pushed after >> feature-close. As this was a rather recent change, it must be >> possible to >> fix this follow up issue. What else is this period in the project good >> for if not fixing issues? > > So I am seeing a number of factors here. > > First, 8140520, set: > > size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; So I'm confused by the above comment: > The problem is caused by a rather recent change (8140520: segfault > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs-open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html shows this change: @@ -531,19 +531,17 @@ } //////////////////////////////////////////////////////////////////////////////// // thread stack -size_t os::Linux::min_stack_allowed = 128*K; +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * K; so the existing single variable of 'min_stack_allowed' was replaced by three variables: _compiler_thread_min_stack_allowed, _java_thread_min_stack_allowed, and _vm_internal_thread_min_stack_allowed. The old single variable and the three new variables are all initialized to the same value (128K) so the fix for 8140520 did not change stack sizes for this platform. In fact, only one platform had a size change (Solaris X64). So I'm confused about how the fix for 8140520 caused this problem. Based on David's analysis below, it looks to me like this 64K stack guard page problem should also exist prior to the fix for 8140520. Goetz, can you please explain how 8140520 caused this problem? Dan > > Second on linux PPC it is hardwired to use 2 guard pages: > > return 2 * page_size(); > > Third, you had a pagesize of 64K. > > Fourth, NPTL takes the guard space from the stack space - hence with 2 > x 64K guard, and a 128K stack it was all consumed. > > --- > > In the proposed changes you now only use page_size() for the guard, so > that alone would have fixed the observed problem. > > But in addition you want to address the NPTL problem by adding back > the guard space to the stack size requested. That alone would also > have fixed the observed problem. :) > > But in addition you have increased the minimum stack size: > > ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; > > which again, on its own would have fixed the original problem. :) > > Did you really intend to increase the real minimum stack from 128K to > 256K ? (on a 64K page system) > > --- > > Focusing simply on the shared code change to adjust the requested > stacksize by the amount of guard space (if any), this does not seem > unreasonable. As you note it is restricted to non-JavaThreads and only > adds a page to reserved stack space. > > My only query now is whether the minimum stacksize detection logic > will correctly report the real minimum stack size (taking into account > the need for the guard page) ? > > Thanks, > David > >> So I really think this issue should be fixed. >> >> Best regards, >> Goetz. >> >> >>> -----Original Message----- >>> From: David Holmes [mailto:david.holmes at oracle.com] >>> Sent: Thursday, November 10, 2016 10:02 PM >>> To: Lindenmaier, Goetz ; hotspot-runtime- >>> dev at openjdk.java.net >>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>> >>> Hi Goetz, >>> >>> As per the bug report, this issue was already known (6675312) and we >>> chose not to try and address it due to no reported issues at the time. >>> While I see that you have encountered an issue (is it real or >>> fabricated?) I think this change is too intrusive to be applied at this >>> stage of the JDK 9 release cycle, as it will change the stack >>> requirements of every application running on Linux. >>> >>> Thanks, >>> David >>> >>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: >>>> Hi, >>>> >>>> >>>> >>>> Please review this change. I please need a sponsor: >>>> >>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.01/ >>>> >>>> >>>> >>>> In the Linux NPTL pthread implementation the guard size mechanism >>>> is not >>>> implemented properly. The posix standard requires to add the size >>>> of the >>>> guard pages to the stack size, instead Linux takes the space out of >>>> 'stacksize'. >>>> >>>> The Posix standard http://pubs.opengroup.org/onlinepubs/9699919799/ >>>> says "the implementation allocates extra memory at the overflow >>>> end of >>>> the stack". The linux man page >>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As at >>>> glibc >>>> 2.8, the NPTL threading implementation includes the guard area within >>>> the stack size allocation, rather than allocating extra space at >>>> the end >>>> of the stack, as POSIX.1 requires". >>>> >>>> I encounter this problem in runtime/Thread/TooSmallStackSize.java >>>> on ppc >>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K on ppc, and >>>> ppc specifies two OS guard pages. The VM crashes in pthread creation >>>> because there is no usable space in the thread stack after allocating >>>> the guard pages. >>>> >>>> But TooSmallStackSize.java requires that the VM comes up with the >>>> stack >>>> size mentioned in the error message. >>>> >>>> This fix adapts the requested stack size on Linux by the size of the >>>> guard pages to mimick proper behaviour, see change to os_linux.cpp. >>>> >>>> >>>> >>>> The change also streamlines usage of stack_guard_page on linuxppc, >>>> linuxppcle, aixppc and linuxs390. >>>> >>>> >>>> >>>> To reproduce the error on linux_x86_64, apply below patch and call the >>>> VM with -XX:CompilerThreadStackSize=64. >>>> >>>> >>>> >>>> I'm still exploring why I had to choose such big compiler stacks on >>>> ppc >>>> to get -version passing, but I wanted to send the RFR now as people >>>> obviously looked at the bug I opened (Thanks David!). >>>> >>>> >>>> >>>> Best regards, >>>> >>>> Goetz. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>>> >>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 12:37:28 >>> 2016 >>>> +0100 >>>> >>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 16:52:17 >>> 2016 >>>> +0100 >>>> >>>> @@ -701,7 +701,7 @@ >>>> >>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>>> >>>> // Creating guard page is very expensive. Java thread has HotSpot >>>> >>>> // guard page, only enable glibc guard page for non-Java threads. >>>> >>>> - return (thr_type == java_thread ? 0 : page_size()); >>>> >>>> + return (thr_type == java_thread ? 0 : 64*K); >>>> >>>> } >>>> >>>> >>>> >>>> // Java thread: >>>> From daniel.daugherty at oracle.com Mon Nov 21 16:37:09 2016 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 21 Nov 2016 09:37:09 -0700 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <668f5f07c10140f2879a3464f70faea5@DEROTE13DE08.global.corp.sap> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <5e9b0cf6243e471287da233b16fe1f63@derote13de14.global.corp.sap> <59f839950a21446fbe26aedfeb8151b8@derote13de14.global.corp.sap> <54c40e6f-d055-e711-bfb3-f6339a3870ea@oracle.com> <36f1a912-0038-9399-f85d-7bbf4f56c460@oracle.com> <2a9cc101-c567-6f7a-c109-1298ff9cc7f9@oracle.com> <668f5f07c10140f2879a3464f70faea5@DEROTE13DE08.global.corp.sap> Message-ID: <7e83b059-ca5e-8b70-a815-244dbe008134@oracle.com> On 11/21/16 3:31 AM, Lindenmaier, Goetz wrote: > Hi Coleen, David, > > the cleanups we were talking about are in this webrev, which I would like to > apply on top of 8169373: > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix There are four webrevs here: webrev.01/ 2016-Nov-10 15:33:11 - Directory webrev.02/ 2016-Nov-16 14:34:26 - Directory webrev.cleanup/ 2016-Nov-21 10:11:36 - Directory webrev.minStSz/ 2016-Nov-16 14:39:09 - Directory and I'm not sure which I should be looking at... :-) > The code is identical in all the os_cpu files. > (Curently it's edited on top of the other change). > > I would understand if you don't like the other changes I propose > For 9. But I still think this is just how you expect this to work, the > minimum stack size is independent of page size and thus should > be specified without considering the guard pages. And therefore > it should go to 9 where we already changed a lot wrt. stack sizes. > > Coleen, would you mind pointing me to the bugs Dan and Ron are > working on? I'd like to understand the possible conflicts. The fix for 8140520 was not done for Win* because we (Ron and I) could not get reliable stack size behavior out of Win* like we got for the *NIX platforms. I don't remember if I've filed the follow up bugs for 8140520, but, in any case, I'm not working on thread stack size issues at the moment. Dan > > Best regards, > Goetz. > >> -----Original Message----- >> From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- >> bounces at openjdk.java.net] On Behalf Of David Holmes >> Sent: Sonntag, 20. November 2016 02:21 >> To: Coleen Phillimore ; hotspot-runtime- >> dev at openjdk.java.net >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >> >> On 19/11/2016 12:40 AM, Coleen Phillimore wrote: >>> >>> On 11/18/16 1:34 AM, David Holmes wrote: >>>> Hi Goetz, >>>> >>>> On 17/11/2016 1:52 AM, Lindenmaier, Goetz wrote: >>>>> Hi, >>>>> >>>>> I addressed the additional topics, but in an extra change. >>>>> >>>>> First, I rounded up the NPTL bug workaround: >>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >> stackFix/webrev.02 >>>>> I fixed os::Posix::describe_pthread_attr() and current_stack_region() >>>>> In the os_cpu files. >>>>> Interestingly, in zero, current_stack_region() is already fixed. >>>> So on Linux, pthread_attr_getstack is also broken - in that it returns >>>> the address at the end of the guard section, instead of the usable stack? >>>> >>>> Can you add error checking on the pthread_attr_getguardsize call please. >>>> >>>> The duplication of the current_stack_region static method in the >>>> os-cpu files is horrible. This should be placed as a private method in >>>> the os class and moved to os_linux.cpp. RFE for 10. >>> By duplication do you mean this bit of code? >>> >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >> stackFix/webrev.02/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.udiff.html >>> >>> + // Work around NPTL stack guard error. >>> + size_t guard_size = 0; >>> + pthread_attr_getguardsize(&attr, &guard_size); >>> + *bottom += guard_size; >>> + *size -= guard_size; >>> + >>> >>> This should be cleaned up with this patch and not an RFE for 10. >> I meant the entire current_stack_region method to which this new >> fragment has been added. >> >> David >> >>> I don't know the NPTL issues well enough to be a reviewer for this, but >>> I don't like something I don't know about distributed in many places. >>> >>>> >>>>> Then I edited an extra webrev to fix >>>>> - unnecessary OS guard page on Compiler threads >>>>> - unpredictable minimal stack sizes because of varying (guard) page >>>>> sizes: >>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >> stackFix/webrev.minStSz/ >>>>> >>>>> Compiler threads now don't get OS guard pages. Saves one page per >>>>> compiler thread. >>>>> >>>>> Also, the constants in the os__.cpp files now list minimum >>>>> space >>>>> required for frames. I basically reduced them by the size of the >>>>> guard zones. >>>>> Where minimum stack sizes are computed, the page size is known and >> the >>>>> minimal value is extended by the required space for the guard pages. >>>>> As compiler threads >>>>> don't bang, no space for the shadow zone is needed. >>>>> >>>>> Should I add this additional change to 8169373? As this only deals >>>>> with minimal >>>>> stack sizes, and reduces these sizes, I don't see a relevant risk. >>>>> Only on systems with large pages the minimal sizes now might be >>>>> bigger, but >>>>> then the old sizes would have led to immediate stack overflows. >>>> Sorry this is getting a bit too divergent and disruptive to me for >>>> this stage of 9. You are redefining what the "minimum stack" values >>>> mean as well as changing their actual value. (BTW I'm sure there is a >>>> test of these values that would need updating.) >>>> >>>> I can't tell if it is right/wrong, necessary/unnecessary - but given >>>> the recent work done in this area I'm somewhat cautious about changing >>>> it again - in 9. >>> I think the additional changes should also wait for 10. I like the >>> change to not have shadow pages for the compiler threads (not including >>> AOT compiler threads, right?), but they also collide with work that I >>> think Dan is going to do for windows. We don't want the os::Posix >>> functions in these files. >>> >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >> stackFix/webrev.minStSz/src/os_cpu/linux_x86/vm/os_linux_x86.cpp.udiff. >> html >>> >>> Also, the work that Dan and Ron did in this area didn't change the sizes >>> and I'd want Dan to have time to comment on this. >>> >>> Thanks, >>> Coleen >>> >>>> David >>>> >>>>> Best regards, >>>>> Goetz. >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>> Sent: Dienstag, 15. November 2016 00:54 >>>>>> To: Lindenmaier, Goetz ; hotspot- >> runtime- >>>>>> dev at openjdk.java.net >>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >> error. >>>>>> Hi Goetz, >>>>>> >>>>>> On 15/11/2016 3:12 AM, Lindenmaier, Goetz wrote: >>>>>>> Hi David, >>>>>>> >>>>>>>> Did you really intend to increase the real minimum stack from 128K >> to >>>>>>>> 256K ? (on a 64K page system) >>>>>>> I finally figured why I have to set such high minimum values. >>>>>>> >>>>>>> A CompilerThread is a JavaThread. It's getting yellow, red etc >>>>>>> pages at the top. As these are 64K on 64K systems, the compiler >>>>>>> thread stack has to be this big. Thus you actually need similar >>>>>>> values once you run a linuxx86_64 VM on a system with 64K pages. >>>>>>> Does this exist? >>>>>> No idea, sorry. >>>>>> >>>>>>> But, this means that there is no use of the OS guard pages >>>>>>> configured in default_guard_size(). This is called with >>>>>>> 'compiler_thread' >>>>>>> for CompilerThreads. >>>>>>> >>>>>>> We should probably also return '0' in that case. >>>>>>> >>>>>>> What do you think? >>>>>> I think I agree. As you note a CompilerThread is-a JavaThread even >>>>>> though the ThreadType enum treats them as completely distinct: >>>>>> >>>>>> enum ThreadType { >>>>>> vm_thread, >>>>>> cgc_thread, // Concurrent GC thread >>>>>> pgc_thread, // Parallel GC thread >>>>>> java_thread, >>>>>> compiler_thread, >>>>>> watcher_thread, >>>>>> os_thread >>>>>> }; >>>>>> >>>>>> so anything with VM supplied guard pages doesn't need the glibc ones >> as >>>>>> well. >>>>>> >>>>>> I'm struggling to understand why os::Linux::default_guard_size is >>>>>> defined per CPU type? I would not expect this to need to vary based >> on >>>>>> CPU at all. ?? >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>>> Best regards, >>>>>>> Goetz. >>>>>>> >>>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>>>>>> // Creating guard page is very expensive. Java thread has HotSpot >>>>>>> // guard page, only enable glibc guard page for non-Java threads. >>>>>>> return (thr_type == java_thread ? 0 : page_size()); >>>>>>> } >>>>>>> From kirill.zhaldybin at oracle.com Mon Nov 21 16:38:37 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Mon, 21 Nov 2016 19:38:37 +0300 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part In-Reply-To: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> References: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> Message-ID: <85e51138-f3ce-7392-2cc3-ce7840aa3747@oracle.com> Marcus, Thank you for reviewing the fix! >> WebRev: >> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ > > ISO8601 says the decimal point can be either '.' or ',' so the test > should accept either. You could let sscanf read out the decimal point > as a character and just verify that it is one of the two. > > In the UTC test you changed ASSERT_GT to ASSERT_EQ, which means that > we won't accept "Z" suffixed strings. Please revert that. I agree that ISO8601 could add "Z" to time (and as far as I understand date/time without delimiters is legal too) but these are the unit tests. Hence they cover the existing code and they should pass only if the result corresponds to existing code and fail otherwise. The current code from os::iso8601_time format date/time string %04d-%02d-%02dT%02d:%02d:%02d.%03d%c%02d%02d so we should not consider any other format as valid. Could you please let me know your opinion? Thank you. Regards, Kirill > > Thanks, > Marcus > >> CR: https://bugs.openjdk.java.net/browse/JDK-8169003 >> >> Thank you. >> >> Regards, Kirill > From jiangli.zhou at oracle.com Mon Nov 21 18:03:49 2016 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 21 Nov 2016 10:03:49 -0800 Subject: RFR(s): 8169870: CDS: "assert(partition_size <= size()) failed: partition failed" on 32 bit JVM In-Reply-To: References: <60904AB8-9732-4D32-86B6-DF3197AF78A5@oracle.com> Message-ID: Hi Chris, > On Nov 19, 2016, at 11:24 AM, Chris Plummer wrote: > > Hi Jiangli, > > The changes look fine. Thanks for the review! > > BTW, we also have LimitSharedSizes.java. Seems there is some overlap with this test. I'm not sure how we ended up with both. Also not sure how LargeSharedSpace.java caught this problem and LimitSharedSizes.java. I guess it was just unlucky with the size it chose. The ones used by the new LargeSharedSpace.java are extremely large sizes (=> 1G). LimitSharedSizes.java mainly focuses on testing very small space sizes, and makes sure if the space size is too small the VM should not crash. Thanks, Jiangli > > Chris > > On 11/18/16 6:14 PM, Jiangli Zhou wrote: >> Please review following fix for JDK-8169870 . >> >> http://cr.openjdk.java.net/~jiangli/8169870/webrev.00/ >> >> This is an old bug but only discovered recently. Thanks Dmitry for reporting the issue. During CDS dump time, the VM tries to allocate memory (by calling ReservedSpace()) for shared space at the top of the java heap. If ReservedSpace() fails, it tries to allocate the memory at a different location by calling ReservedSpace() again. After the second ReservedSpace() call, it initializes the shared space without checking if the second allocation attempt is successful or not. >> >> Added a new case in LargeSharedSpace.java and tested on both 32-bit and 64-bit platforms. >> >> Thanks, >> Jiangli > > From jiangli.zhou at oracle.com Mon Nov 21 18:04:21 2016 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 21 Nov 2016 10:04:21 -0800 Subject: RFR(s): 8169870: CDS: "assert(partition_size <= size()) failed: partition failed" on 32 bit JVM In-Reply-To: <82ea5297-0ecf-bea5-94ac-2b2bd5897183@oracle.com> References: <60904AB8-9732-4D32-86B6-DF3197AF78A5@oracle.com> <82ea5297-0ecf-bea5-94ac-2b2bd5897183@oracle.com> Message-ID: <450B676F-3AFB-4021-A6AB-D85FF9443A60@oracle.com> Thanks for the review, Serguei! Jiangli > On Nov 20, 2016, at 8:30 PM, serguei.spitsyn at oracle.com wrote: > > Hi Jiangli, > > This looks good. > > Thanks, > Serguei > > > On 11/18/16 18:14, Jiangli Zhou wrote: >> Please review following fix for JDK-8169870 . >> >> http://cr.openjdk.java.net/~jiangli/8169870/webrev.00/ >> >> This is an old bug but only discovered recently. Thanks Dmitry for reporting the issue. During CDS dump time, the VM tries to allocate memory (by calling ReservedSpace()) for shared space at the top of the java heap. If ReservedSpace() fails, it tries to allocate the memory at a different location by calling ReservedSpace() again. After the second ReservedSpace() call, it initializes the shared space without checking if the second allocation attempt is successful or not. >> >> Added a new case in LargeSharedSpace.java and tested on both 32-bit and 64-bit platforms. >> >> Thanks, >> Jiangli > From jiangli.zhou at oracle.com Mon Nov 21 18:05:22 2016 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 21 Nov 2016 10:05:22 -0800 Subject: RFR(s): 8169870: CDS: "assert(partition_size <= size()) failed: partition failed" on 32 bit JVM In-Reply-To: <58329192.3070602@oracle.com> References: <60904AB8-9732-4D32-86B6-DF3197AF78A5@oracle.com> <82ea5297-0ecf-bea5-94ac-2b2bd5897183@oracle.com> <58329192.3070602@oracle.com> Message-ID: <39F05174-1E01-46F7-A70A-627FE06950E8@oracle.com> Hi Ioi, > On Nov 20, 2016, at 10:17 PM, Ioi Lam wrote: > > Hi Jiangli, > > The VM changes look OK to me. Thanks for the review. > > For the test case, could you add a comment for these two output strings? > > "larger than compressed klass limit" > "Unable to allocate memory for shared space" > > under what condition will they happen? Will do. Thanks for the suggestion. Thanks, Jiangli > > Thanks > - Ioi > > On 11/20/16 8:30 PM, serguei.spitsyn at oracle.com wrote: >> Hi Jiangli, >> >> This looks good. >> >> Thanks, >> Serguei >> >> >> On 11/18/16 18:14, Jiangli Zhou wrote: >>> Please review following fix for JDK-8169870 . >>> >>> http://cr.openjdk.java.net/~jiangli/8169870/webrev.00/ >>> >>> This is an old bug but only discovered recently. Thanks Dmitry for reporting the issue. During CDS dump time, the VM tries to allocate memory (by calling ReservedSpace()) for shared space at the top of the java heap. If ReservedSpace() fails, it tries to allocate the memory at a different location by calling ReservedSpace() again. After the second ReservedSpace() call, it initializes the shared space without checking if the second allocation attempt is successful or not. >>> >>> Added a new case in LargeSharedSpace.java and tested on both 32-bit and 64-bit platforms. >>> >>> Thanks, >>> Jiangli >> > From goetz.lindenmaier at sap.com Tue Nov 22 10:55:58 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Tue, 22 Nov 2016 10:55:58 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> Message-ID: <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> Hi Dan, I ran into a row of issues, some errors on the platforms. What I meant with that comment is that os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, JavaThread::stack_guard_zone_size() + JavaThread::stack_shadow_zone_size() + (4*BytesPerWord COMPILER2_PRESENT(+2)) * 4 * K); was executed, and min_stack_allowed used for all stacks. Now, compiler and vm minimum stack sizes are not increased by these sizes. At least the _compiler_thread_min_stack_allowed should be increased similarly by red/yellow zone size. The compiler thread is a Java thread and must have space for these zones. Also, the change added a test that is failing now. Best regards, Goetz. > -----Original Message----- > From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > Sent: Montag, 21. November 2016 17:28 > To: David Holmes ; Lindenmaier, Goetz > ; hotspot-runtime-dev at openjdk.java.net > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > Sorry for the delayed responses to this thread. I've been on vacation... > > One comment/query embedded below... > > > On 11/10/16 8:40 PM, David Holmes wrote: > > Hi Goetz, > > > > On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > >> Hi David, > >> > >> This issue is different to 6675312, see also my comment in the bug. > >> > >> It appears running jtreg test runtime/Thread/TooSmallStackSize.java, > >> with my patch below you can reproduce it on linuxx86_64. You can not > >> do that with 6675312. Also, I would assume there are systems out there > >> on x86 that uses 64-K pages, did you run the tests on these? I would > >> assume you get hard crashes with stack overflows in the first C2 > >> compilation if there is only 64K usable CompilerThreadStack. > >> > >> My fix does not affect Java threads, which are the largest amount > >> of threads used by the VM. It affects only the non-Java threads. > >> It adds one page to these threads. The page does not require memory, > >> as it's protected. The stack will only require more space if the thread > >> ran into a stack overflow before the fix as else the pages are not > >> mapped. > >> This are stack overflows that cause hard crashes, at least on ppc the VM > >> does not properly catch these stack overflows, so any setup working now > >> will not run into the additional space. Altogether there should be no > >> effect on running systems besides requiring one more entry in the > >> page table per non-Java thread. > >> > >> The problem is caused by a rather recent change (8140520: segfault on > >> solaris-amd64 > >> with "-XX:VMThreadStackSize=1" option) which was pushed after > >> feature-close. As this was a rather recent change, it must be > >> possible to > >> fix this follow up issue. What else is this period in the project good > >> for if not fixing issues? > > > > So I am seeing a number of factors here. > > > > First, 8140520, set: > > > > size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > > So I'm confused by the above comment: > > > The problem is caused by a rather recent change (8140520: segfault > > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) > > http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- > open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html > > shows this change: > > @@ -531,19 +531,17 @@ > } > > > ////////////////////////////////////////////////////////////////////////////// > // > // thread stack > > -size_t os::Linux::min_stack_allowed = 128*K; > +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; > +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * K; > > so the existing single variable of 'min_stack_allowed' was > replaced by three variables: _compiler_thread_min_stack_allowed, > _java_thread_min_stack_allowed, and > _vm_internal_thread_min_stack_allowed. > > The old single variable and the three new variables are all > initialized to the same value (128K) so the fix for 8140520 > did not change stack sizes for this platform. In fact, only > one platform had a size change (Solaris X64). > > So I'm confused about how the fix for 8140520 caused this problem. > > Based on David's analysis below, it looks to me like this 64K stack > guard page problem should also exist prior to the fix for 8140520. > > Goetz, can you please explain how 8140520 caused this problem? > > Dan > > > > > > > Second on linux PPC it is hardwired to use 2 guard pages: > > > > return 2 * page_size(); > > > > Third, you had a pagesize of 64K. > > > > Fourth, NPTL takes the guard space from the stack space - hence with 2 > > x 64K guard, and a 128K stack it was all consumed. > > > > --- > > > > In the proposed changes you now only use page_size() for the guard, so > > that alone would have fixed the observed problem. > > > > But in addition you want to address the NPTL problem by adding back > > the guard space to the stack size requested. That alone would also > > have fixed the observed problem. :) > > > > But in addition you have increased the minimum stack size: > > > > ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; > > > > which again, on its own would have fixed the original problem. :) > > > > Did you really intend to increase the real minimum stack from 128K to > > 256K ? (on a 64K page system) > > > > --- > > > > Focusing simply on the shared code change to adjust the requested > > stacksize by the amount of guard space (if any), this does not seem > > unreasonable. As you note it is restricted to non-JavaThreads and only > > adds a page to reserved stack space. > > > > My only query now is whether the minimum stacksize detection logic > > will correctly report the real minimum stack size (taking into account > > the need for the guard page) ? > > > > Thanks, > > David > > > >> So I really think this issue should be fixed. > >> > >> Best regards, > >> Goetz. > >> > >> > >>> -----Original Message----- > >>> From: David Holmes [mailto:david.holmes at oracle.com] > >>> Sent: Thursday, November 10, 2016 10:02 PM > >>> To: Lindenmaier, Goetz ; hotspot- > runtime- > >>> dev at openjdk.java.net > >>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > >>> > >>> Hi Goetz, > >>> > >>> As per the bug report, this issue was already known (6675312) and we > >>> chose not to try and address it due to no reported issues at the time. > >>> While I see that you have encountered an issue (is it real or > >>> fabricated?) I think this change is too intrusive to be applied at this > >>> stage of the JDK 9 release cycle, as it will change the stack > >>> requirements of every application running on Linux. > >>> > >>> Thanks, > >>> David > >>> > >>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > >>>> Hi, > >>>> > >>>> > >>>> > >>>> Please review this change. I please need a sponsor: > >>>> > >>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > stackFix/webrev.01/ > >>>> > >>>> > >>>> > >>>> In the Linux NPTL pthread implementation the guard size mechanism > >>>> is not > >>>> implemented properly. The posix standard requires to add the size > >>>> of the > >>>> guard pages to the stack size, instead Linux takes the space out of > >>>> 'stacksize'. > >>>> > >>>> The Posix standard > http://pubs.opengroup.org/onlinepubs/9699919799/ > >>>> says "the implementation allocates extra memory at the overflow > >>>> end of > >>>> the stack". The linux man page > >>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As at > >>>> glibc > >>>> 2.8, the NPTL threading implementation includes the guard area within > >>>> the stack size allocation, rather than allocating extra space at > >>>> the end > >>>> of the stack, as POSIX.1 requires". > >>>> > >>>> I encounter this problem in runtime/Thread/TooSmallStackSize.java > >>>> on ppc > >>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K on ppc, > and > >>>> ppc specifies two OS guard pages. The VM crashes in pthread creation > >>>> because there is no usable space in the thread stack after allocating > >>>> the guard pages. > >>>> > >>>> But TooSmallStackSize.java requires that the VM comes up with the > >>>> stack > >>>> size mentioned in the error message. > >>>> > >>>> This fix adapts the requested stack size on Linux by the size of the > >>>> guard pages to mimick proper behaviour, see change to os_linux.cpp. > >>>> > >>>> > >>>> > >>>> The change also streamlines usage of stack_guard_page on linuxppc, > >>>> linuxppcle, aixppc and linuxs390. > >>>> > >>>> > >>>> > >>>> To reproduce the error on linux_x86_64, apply below patch and call the > >>>> VM with -XX:CompilerThreadStackSize=64. > >>>> > >>>> > >>>> > >>>> I'm still exploring why I had to choose such big compiler stacks on > >>>> ppc > >>>> to get -version passing, but I wanted to send the RFR now as people > >>>> obviously looked at the bug I opened (Thanks David!). > >>>> > >>>> > >>>> > >>>> Best regards, > >>>> > >>>> Goetz. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp > >>>> > >>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 12:37:28 > >>> 2016 > >>>> +0100 > >>>> > >>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 > 16:52:17 > >>> 2016 > >>>> +0100 > >>>> > >>>> @@ -701,7 +701,7 @@ > >>>> > >>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > >>>> > >>>> // Creating guard page is very expensive. Java thread has HotSpot > >>>> > >>>> // guard page, only enable glibc guard page for non-Java threads. > >>>> > >>>> - return (thr_type == java_thread ? 0 : page_size()); > >>>> > >>>> + return (thr_type == java_thread ? 0 : 64*K); > >>>> > >>>> } > >>>> > >>>> > >>>> > >>>> // Java thread: > >>>> From goetz.lindenmaier at sap.com Tue Nov 22 11:02:09 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Tue, 22 Nov 2016 11:02:09 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <7e83b059-ca5e-8b70-a815-244dbe008134@oracle.com> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <5e9b0cf6243e471287da233b16fe1f63@derote13de14.global.corp.sap> <59f839950a21446fbe26aedfeb8151b8@derote13de14.global.corp.sap> <54c40e6f-d055-e711-bfb3-f6339a3870ea@oracle.com> <36f1a912-0038-9399-f85d-7bbf4f56c460@oracle.com> <2a9cc101-c567-6f7a-c109-1298ff9cc7f9@oracle.com> <668f5f07c10140f2879a3464f70faea5@DEROTE13DE08.global.corp.sap> <7e83b059-ca5e-8b70-a815-244dbe008134@oracle.com> Message-ID: > There are four webrevs here: > While working on this, I detected more and more problems. Because feature complete etc I kept them separate. Webrevs for the original intended fix: NPTL error workaround. Webrev.02 incorporates improvements from review process. It was agreed that this is a bug fix that should go to 9. > webrev.01/ 2016-Nov-10 15:33:11 - Directory > webrev.02/ 2016-Nov-16 14:34:26 - Directory Cleanup of stack handling code proposed by David. No functional changes. > webrev.cleanup/ 2016-Nov-21 10:11:36 - Directory Additional changes to fix the missing space for VM guard pages, removing the wrong OS guard page from compiler threads, and bringing more sense into the handling of minimal sizes (the initial constant should reflect the size needed for stack frames excluding guard pages). I would like to bring these to 9 still. Else I have to do some other, changes (increasing minimal sizes and adding the one check for compiler threads.) > webrev.minStSz/ 2016-Nov-16 14:39:09 - Directory Best regards, Goetz. > > and I'm not sure which I should be looking at... :-) > > > > The code is identical in all the os_cpu files. > > (Curently it's edited on top of the other change). > > > > I would understand if you don't like the other changes I propose > > For 9. But I still think this is just how you expect this to work, the > > minimum stack size is independent of page size and thus should > > be specified without considering the guard pages. And therefore > > it should go to 9 where we already changed a lot wrt. stack sizes. > > > > Coleen, would you mind pointing me to the bugs Dan and Ron are > > working on? I'd like to understand the possible conflicts. > > The fix for 8140520 was not done for Win* because we (Ron and I) > could not get reliable stack size behavior out of Win* like we > got for the *NIX platforms. I don't remember if I've filed the > follow up bugs for 8140520, but, in any case, I'm not working > on thread stack size issues at the moment. > > Dan > > > > > > Best regards, > > Goetz. > > > >> -----Original Message----- > >> From: hotspot-runtime-dev [mailto:hotspot-runtime-dev- > >> bounces at openjdk.java.net] On Behalf Of David Holmes > >> Sent: Sonntag, 20. November 2016 02:21 > >> To: Coleen Phillimore ; hotspot-runtime- > >> dev at openjdk.java.net > >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > >> > >> On 19/11/2016 12:40 AM, Coleen Phillimore wrote: > >>> > >>> On 11/18/16 1:34 AM, David Holmes wrote: > >>>> Hi Goetz, > >>>> > >>>> On 17/11/2016 1:52 AM, Lindenmaier, Goetz wrote: > >>>>> Hi, > >>>>> > >>>>> I addressed the additional topics, but in an extra change. > >>>>> > >>>>> First, I rounded up the NPTL bug workaround: > >>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > >> stackFix/webrev.02 > >>>>> I fixed os::Posix::describe_pthread_attr() and current_stack_region() > >>>>> In the os_cpu files. > >>>>> Interestingly, in zero, current_stack_region() is already fixed. > >>>> So on Linux, pthread_attr_getstack is also broken - in that it returns > >>>> the address at the end of the guard section, instead of the usable > stack? > >>>> > >>>> Can you add error checking on the pthread_attr_getguardsize call > please. > >>>> > >>>> The duplication of the current_stack_region static method in the > >>>> os-cpu files is horrible. This should be placed as a private method in > >>>> the os class and moved to os_linux.cpp. RFE for 10. > >>> By duplication do you mean this bit of code? > >>> > >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > >> > stackFix/webrev.02/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.udiff.html > >>> > >>> + // Work around NPTL stack guard error. > >>> + size_t guard_size = 0; > >>> + pthread_attr_getguardsize(&attr, &guard_size); > >>> + *bottom += guard_size; > >>> + *size -= guard_size; > >>> + > >>> > >>> This should be cleaned up with this patch and not an RFE for 10. > >> I meant the entire current_stack_region method to which this new > >> fragment has been added. > >> > >> David > >> > >>> I don't know the NPTL issues well enough to be a reviewer for this, but > >>> I don't like something I don't know about distributed in many places. > >>> > >>>> > >>>>> Then I edited an extra webrev to fix > >>>>> - unnecessary OS guard page on Compiler threads > >>>>> - unpredictable minimal stack sizes because of varying (guard) page > >>>>> sizes: > >>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > >> stackFix/webrev.minStSz/ > >>>>> > >>>>> Compiler threads now don't get OS guard pages. Saves one page per > >>>>> compiler thread. > >>>>> > >>>>> Also, the constants in the os__.cpp files now list minimum > >>>>> space > >>>>> required for frames. I basically reduced them by the size of the > >>>>> guard zones. > >>>>> Where minimum stack sizes are computed, the page size is known > and > >> the > >>>>> minimal value is extended by the required space for the guard pages. > >>>>> As compiler threads > >>>>> don't bang, no space for the shadow zone is needed. > >>>>> > >>>>> Should I add this additional change to 8169373? As this only deals > >>>>> with minimal > >>>>> stack sizes, and reduces these sizes, I don't see a relevant risk. > >>>>> Only on systems with large pages the minimal sizes now might be > >>>>> bigger, but > >>>>> then the old sizes would have led to immediate stack overflows. > >>>> Sorry this is getting a bit too divergent and disruptive to me for > >>>> this stage of 9. You are redefining what the "minimum stack" values > >>>> mean as well as changing their actual value. (BTW I'm sure there is a > >>>> test of these values that would need updating.) > >>>> > >>>> I can't tell if it is right/wrong, necessary/unnecessary - but given > >>>> the recent work done in this area I'm somewhat cautious about > changing > >>>> it again - in 9. > >>> I think the additional changes should also wait for 10. I like the > >>> change to not have shadow pages for the compiler threads (not > including > >>> AOT compiler threads, right?), but they also collide with work that I > >>> think Dan is going to do for windows. We don't want the os::Posix > >>> functions in these files. > >>> > >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > >> > stackFix/webrev.minStSz/src/os_cpu/linux_x86/vm/os_linux_x86.cpp.udiff. > >> html > >>> > >>> Also, the work that Dan and Ron did in this area didn't change the sizes > >>> and I'd want Dan to have time to comment on this. > >>> > >>> Thanks, > >>> Coleen > >>> > >>>> David > >>>> > >>>>> Best regards, > >>>>> Goetz. > >>>>> > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>>>> Sent: Dienstag, 15. November 2016 00:54 > >>>>>> To: Lindenmaier, Goetz ; hotspot- > >> runtime- > >>>>>> dev at openjdk.java.net > >>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > >> error. > >>>>>> Hi Goetz, > >>>>>> > >>>>>> On 15/11/2016 3:12 AM, Lindenmaier, Goetz wrote: > >>>>>>> Hi David, > >>>>>>> > >>>>>>>> Did you really intend to increase the real minimum stack from > 128K > >> to > >>>>>>>> 256K ? (on a 64K page system) > >>>>>>> I finally figured why I have to set such high minimum values. > >>>>>>> > >>>>>>> A CompilerThread is a JavaThread. It's getting yellow, red etc > >>>>>>> pages at the top. As these are 64K on 64K systems, the compiler > >>>>>>> thread stack has to be this big. Thus you actually need similar > >>>>>>> values once you run a linuxx86_64 VM on a system with 64K pages. > >>>>>>> Does this exist? > >>>>>> No idea, sorry. > >>>>>> > >>>>>>> But, this means that there is no use of the OS guard pages > >>>>>>> configured in default_guard_size(). This is called with > >>>>>>> 'compiler_thread' > >>>>>>> for CompilerThreads. > >>>>>>> > >>>>>>> We should probably also return '0' in that case. > >>>>>>> > >>>>>>> What do you think? > >>>>>> I think I agree. As you note a CompilerThread is-a JavaThread even > >>>>>> though the ThreadType enum treats them as completely distinct: > >>>>>> > >>>>>> enum ThreadType { > >>>>>> vm_thread, > >>>>>> cgc_thread, // Concurrent GC thread > >>>>>> pgc_thread, // Parallel GC thread > >>>>>> java_thread, > >>>>>> compiler_thread, > >>>>>> watcher_thread, > >>>>>> os_thread > >>>>>> }; > >>>>>> > >>>>>> so anything with VM supplied guard pages doesn't need the glibc > ones > >> as > >>>>>> well. > >>>>>> > >>>>>> I'm struggling to understand why os::Linux::default_guard_size is > >>>>>> defined per CPU type? I would not expect this to need to vary based > >> on > >>>>>> CPU at all. ?? > >>>>>> > >>>>>> Thanks, > >>>>>> David > >>>>>> > >>>>>>> Best regards, > >>>>>>> Goetz. > >>>>>>> > >>>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > >>>>>>> // Creating guard page is very expensive. Java thread has HotSpot > >>>>>>> // guard page, only enable glibc guard page for non-Java threads. > >>>>>>> return (thr_type == java_thread ? 0 : page_size()); > >>>>>>> } > >>>>>>> From marcus.larsson at oracle.com Tue Nov 22 12:32:21 2016 From: marcus.larsson at oracle.com (Marcus Larsson) Date: Tue, 22 Nov 2016 13:32:21 +0100 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part In-Reply-To: <85e51138-f3ce-7392-2cc3-ce7840aa3747@oracle.com> References: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> <85e51138-f3ce-7392-2cc3-ce7840aa3747@oracle.com> Message-ID: Hi, On 2016-11-21 17:38, Kirill Zhaldybin wrote: > Marcus, > > Thank you for reviewing the fix! >>> WebRev: >>> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ >> >> ISO8601 says the decimal point can be either '.' or ',' so the test >> should accept either. You could let sscanf read out the decimal point >> as a character and just verify that it is one of the two. >> >> In the UTC test you changed ASSERT_GT to ASSERT_EQ, which means that >> we won't accept "Z" suffixed strings. Please revert that. > I agree that ISO8601 could add "Z" to time (and as far as I understand > date/time without delimiters is legal too) but these are the unit tests. > Hence they cover the existing code and they should pass only if the > result corresponds to existing code and fail otherwise. > The current code from os::iso8601_time format date/time string > %04d-%02d-%02dT%02d:%02d:%02d.%03d%c%02d%02d so we should not consider > any other format as valid. > > Could you please let me know your opinion? I think the test should verify the intended behavior, not the implementation. If we refactor or change something in iso8601_time() we shouldn't be failing the test if it still conforms to ISO8601, IMO. Thanks, Marcus > > Thank you. > > Regards, Kirill > >> >> Thanks, >> Marcus >> >>> CR: https://bugs.openjdk.java.net/browse/JDK-8169003 >>> >>> Thank you. >>> >>> Regards, Kirill >> > From daniel.daugherty at oracle.com Tue Nov 22 13:00:37 2016 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 22 Nov 2016 06:00:37 -0700 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> Message-ID: On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: > Hi Dan, > > I ran into a row of issues, some errors on the platforms. > > What I meant with that comment is that > os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, > JavaThread::stack_guard_zone_size() + > JavaThread::stack_shadow_zone_size() + > (4*BytesPerWord COMPILER2_PRESENT(+2)) * 4 * K); > was executed, and min_stack_allowed used for all stacks. Now, compiler and > vm minimum stack sizes are not increased by these sizes. Now I see what you mean. Thanks for clearing this up. I should have remembered that part of the change because we went back and forth about removing the red/yellow zone pages from the other threads. In particular, we discussed the compiler thread because it is-a JavaThread. Our conclusion was that a compiler thread doesn't execute Java bytecode so we could remove the red/yellow pages... And that decision allowed us to be exposed to the 64K page issue because the "extra" space isn't there anymore. > At least the _compiler_thread_min_stack_allowed should be increased similarly > by red/yellow zone size. The compiler thread is a Java thread and must have space > for these zones. I'm not sure that I completely agree (yet). To me, the red/yellow pages are there for Java thread stack overflow semantics. Yes, the compiler thread needs extra space when 64K pages are used, but I would prefer that we add that space via a different calculation. To put it another way, I'd like to see us add extra space to solve the 64K page issue directly instead of as a side effect of the red/yellow page addition. > Also, the change added a test that is failing now. And that's a "good thing" (TM), right? :-) Again, thanks for clarifying 8140520's role in this issue. Dan > > Best regards, > Goetz. > >> -----Original Message----- >> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >> Sent: Montag, 21. November 2016 17:28 >> To: David Holmes ; Lindenmaier, Goetz >> ; hotspot-runtime-dev at openjdk.java.net >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >> >> Sorry for the delayed responses to this thread. I've been on vacation... >> >> One comment/query embedded below... >> >> >> On 11/10/16 8:40 PM, David Holmes wrote: >>> Hi Goetz, >>> >>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: >>>> Hi David, >>>> >>>> This issue is different to 6675312, see also my comment in the bug. >>>> >>>> It appears running jtreg test runtime/Thread/TooSmallStackSize.java, >>>> with my patch below you can reproduce it on linuxx86_64. You can not >>>> do that with 6675312. Also, I would assume there are systems out there >>>> on x86 that uses 64-K pages, did you run the tests on these? I would >>>> assume you get hard crashes with stack overflows in the first C2 >>>> compilation if there is only 64K usable CompilerThreadStack. >>>> >>>> My fix does not affect Java threads, which are the largest amount >>>> of threads used by the VM. It affects only the non-Java threads. >>>> It adds one page to these threads. The page does not require memory, >>>> as it's protected. The stack will only require more space if the thread >>>> ran into a stack overflow before the fix as else the pages are not >>>> mapped. >>>> This are stack overflows that cause hard crashes, at least on ppc the VM >>>> does not properly catch these stack overflows, so any setup working now >>>> will not run into the additional space. Altogether there should be no >>>> effect on running systems besides requiring one more entry in the >>>> page table per non-Java thread. >>>> >>>> The problem is caused by a rather recent change (8140520: segfault on >>>> solaris-amd64 >>>> with "-XX:VMThreadStackSize=1" option) which was pushed after >>>> feature-close. As this was a rather recent change, it must be >>>> possible to >>>> fix this follow up issue. What else is this period in the project good >>>> for if not fixing issues? >>> So I am seeing a number of factors here. >>> >>> First, 8140520, set: >>> >>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >> So I'm confused by the above comment: >> >> > The problem is caused by a rather recent change (8140520: segfault >> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) >> >> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- >> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html >> >> shows this change: >> >> @@ -531,19 +531,17 @@ >> } >> >> >> ////////////////////////////////////////////////////////////////////////////// >> // >> // thread stack >> >> -size_t os::Linux::min_stack_allowed = 128*K; >> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; >> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * K; >> >> so the existing single variable of 'min_stack_allowed' was >> replaced by three variables: _compiler_thread_min_stack_allowed, >> _java_thread_min_stack_allowed, and >> _vm_internal_thread_min_stack_allowed. >> >> The old single variable and the three new variables are all >> initialized to the same value (128K) so the fix for 8140520 >> did not change stack sizes for this platform. In fact, only >> one platform had a size change (Solaris X64). >> >> So I'm confused about how the fix for 8140520 caused this problem. >> >> Based on David's analysis below, it looks to me like this 64K stack >> guard page problem should also exist prior to the fix for 8140520. >> >> Goetz, can you please explain how 8140520 caused this problem? >> >> Dan >> >> >> >>> Second on linux PPC it is hardwired to use 2 guard pages: >>> >>> return 2 * page_size(); >>> >>> Third, you had a pagesize of 64K. >>> >>> Fourth, NPTL takes the guard space from the stack space - hence with 2 >>> x 64K guard, and a 128K stack it was all consumed. >>> >>> --- >>> >>> In the proposed changes you now only use page_size() for the guard, so >>> that alone would have fixed the observed problem. >>> >>> But in addition you want to address the NPTL problem by adding back >>> the guard space to the stack size requested. That alone would also >>> have fixed the observed problem. :) >>> >>> But in addition you have increased the minimum stack size: >>> >>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; >>> >>> which again, on its own would have fixed the original problem. :) >>> >>> Did you really intend to increase the real minimum stack from 128K to >>> 256K ? (on a 64K page system) >>> >>> --- >>> >>> Focusing simply on the shared code change to adjust the requested >>> stacksize by the amount of guard space (if any), this does not seem >>> unreasonable. As you note it is restricted to non-JavaThreads and only >>> adds a page to reserved stack space. >>> >>> My only query now is whether the minimum stacksize detection logic >>> will correctly report the real minimum stack size (taking into account >>> the need for the guard page) ? >>> >>> Thanks, >>> David >>> >>>> So I really think this issue should be fixed. >>>> >>>> Best regards, >>>> Goetz. >>>> >>>> >>>>> -----Original Message----- >>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>> Sent: Thursday, November 10, 2016 10:02 PM >>>>> To: Lindenmaier, Goetz ; hotspot- >> runtime- >>>>> dev at openjdk.java.net >>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>>>> >>>>> Hi Goetz, >>>>> >>>>> As per the bug report, this issue was already known (6675312) and we >>>>> chose not to try and address it due to no reported issues at the time. >>>>> While I see that you have encountered an issue (is it real or >>>>> fabricated?) I think this change is too intrusive to be applied at this >>>>> stage of the JDK 9 release cycle, as it will change the stack >>>>> requirements of every application running on Linux. >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: >>>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> Please review this change. I please need a sponsor: >>>>>> >>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >> stackFix/webrev.01/ >>>>>> >>>>>> >>>>>> In the Linux NPTL pthread implementation the guard size mechanism >>>>>> is not >>>>>> implemented properly. The posix standard requires to add the size >>>>>> of the >>>>>> guard pages to the stack size, instead Linux takes the space out of >>>>>> 'stacksize'. >>>>>> >>>>>> The Posix standard >> http://pubs.opengroup.org/onlinepubs/9699919799/ >>>>>> says "the implementation allocates extra memory at the overflow >>>>>> end of >>>>>> the stack". The linux man page >>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As at >>>>>> glibc >>>>>> 2.8, the NPTL threading implementation includes the guard area within >>>>>> the stack size allocation, rather than allocating extra space at >>>>>> the end >>>>>> of the stack, as POSIX.1 requires". >>>>>> >>>>>> I encounter this problem in runtime/Thread/TooSmallStackSize.java >>>>>> on ppc >>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K on ppc, >> and >>>>>> ppc specifies two OS guard pages. The VM crashes in pthread creation >>>>>> because there is no usable space in the thread stack after allocating >>>>>> the guard pages. >>>>>> >>>>>> But TooSmallStackSize.java requires that the VM comes up with the >>>>>> stack >>>>>> size mentioned in the error message. >>>>>> >>>>>> This fix adapts the requested stack size on Linux by the size of the >>>>>> guard pages to mimick proper behaviour, see change to os_linux.cpp. >>>>>> >>>>>> >>>>>> >>>>>> The change also streamlines usage of stack_guard_page on linuxppc, >>>>>> linuxppcle, aixppc and linuxs390. >>>>>> >>>>>> >>>>>> >>>>>> To reproduce the error on linux_x86_64, apply below patch and call the >>>>>> VM with -XX:CompilerThreadStackSize=64. >>>>>> >>>>>> >>>>>> >>>>>> I'm still exploring why I had to choose such big compiler stacks on >>>>>> ppc >>>>>> to get -version passing, but I wanted to send the RFR now as people >>>>>> obviously looked at the bug I opened (Thanks David!). >>>>>> >>>>>> >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Goetz. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>>>>> >>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 12:37:28 >>>>> 2016 >>>>>> +0100 >>>>>> >>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 >> 16:52:17 >>>>> 2016 >>>>>> +0100 >>>>>> >>>>>> @@ -701,7 +701,7 @@ >>>>>> >>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>>>>> >>>>>> // Creating guard page is very expensive. Java thread has HotSpot >>>>>> >>>>>> // guard page, only enable glibc guard page for non-Java threads. >>>>>> >>>>>> - return (thr_type == java_thread ? 0 : page_size()); >>>>>> >>>>>> + return (thr_type == java_thread ? 0 : 64*K); >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> // Java thread: >>>>>> From goetz.lindenmaier at sap.com Tue Nov 22 13:19:30 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Tue, 22 Nov 2016 13:19:30 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> Message-ID: <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> Hi Dan, > -----Original Message----- > From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > Sent: Dienstag, 22. November 2016 14:01 > To: Lindenmaier, Goetz ; David Holmes > ; hotspot-runtime-dev at openjdk.java.net > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: > > Hi Dan, > > > > I ran into a row of issues, some errors on the platforms. > > > > What I meant with that comment is that > > os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, > > JavaThread::stack_guard_zone_size() + > > JavaThread::stack_shadow_zone_size() + > > (4*BytesPerWord COMPILER2_PRESENT(+2)) * 4 * K); > > was executed, and min_stack_allowed used for all stacks. Now, compiler > and > > vm minimum stack sizes are not increased by these sizes. > > Now I see what you mean. Thanks for clearing this up. > > I should have remembered that part of the change because we went back > and forth about removing the red/yellow zone pages from the other > threads. In particular, we discussed the compiler thread because it > is-a JavaThread. Our conclusion was that a compiler thread doesn't > execute Java bytecode so we could remove the red/yellow pages... Yes, it does not execute java byte code. Therefore you can remove the shadow pages. There is no code that will bang. But red/yellow pages are protected right during thread startup. Therefore you must have enough space for them. On ppc, we try to protect three 64K pages out of the 128K compiler stack. That obviously fails. Therefore I propose: size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // Set platform dependent. in os::Posix::set_minimum_stack_sizes(): _java_thread_min_stack_allowed = _java_thread_min_stack_allowed + JavaThread::stack_guard_zone_size() + JavaThread::stack_shadow_zone_size(); (Similar for _compiler_thread_min_stack_allowed). The minimal stack size is made up of three components: 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot implementation/platform/os. 2. Sizes of C++ frames: depends on C++ compiler. These are fixed at compile time. 3. Sizes of red/yellow/reserved/shadow pages. Depends on the system the VM is used on. This is not fixed at compile time. (Our ppc machines differ in page size.) Therefore 3. should not be included in a compile time constant. > And that decision allowed us to be exposed to the 64K page issue > because the "extra" space isn't there anymore. > > > At least the _compiler_thread_min_stack_allowed should be increased > > similarly by red/yellow zone size. The compiler thread is a Java > > thread and must have space for these zones. > > I'm not sure that I completely agree (yet). To me, the red/yellow > pages are there for Java thread stack overflow semantics. Yes, the > compiler thread needs extra space when 64K pages are used, but I > would prefer that we add that space via a different calculation. Yes they are. But compiler threads happen tob e a subclass of Java threads and use the same run() method that puts the pages There. I agree that they are not needed for Compiler threads, nor for CodeCacheSweeperThreads. I don't really now about JvmtiAgentThreads and ServiceThreads, but all of the get the guard pages because they are derived from JavaThread. > To put it another way, I'd like to see us add extra space to solve > the 64K page issue directly instead of as a side effect of the > red/yellow page addition. I don't understand. What do you mean by 'directly'? > > Also, the change added a test that is failing now. > > And that's a "good thing" (TM), right? :-) Yes, it showed a bug and thus raised the need to fix it! :) Best regards, Goetz. > Again, thanks for clarifying 8140520's role in this issue. > > Dan > > > > > > Best regards, > > Goetz. > > > >> -----Original Message----- > >> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >> Sent: Montag, 21. November 2016 17:28 > >> To: David Holmes ; Lindenmaier, Goetz > >> ; hotspot-runtime-dev at openjdk.java.net > >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > >> > >> Sorry for the delayed responses to this thread. I've been on vacation... > >> > >> One comment/query embedded below... > >> > >> > >> On 11/10/16 8:40 PM, David Holmes wrote: > >>> Hi Goetz, > >>> > >>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > >>>> Hi David, > >>>> > >>>> This issue is different to 6675312, see also my comment in the bug. > >>>> > >>>> It appears running jtreg test runtime/Thread/TooSmallStackSize.java, > >>>> with my patch below you can reproduce it on linuxx86_64. You can not > >>>> do that with 6675312. Also, I would assume there are systems out there > >>>> on x86 that uses 64-K pages, did you run the tests on these? I would > >>>> assume you get hard crashes with stack overflows in the first C2 > >>>> compilation if there is only 64K usable CompilerThreadStack. > >>>> > >>>> My fix does not affect Java threads, which are the largest amount > >>>> of threads used by the VM. It affects only the non-Java threads. > >>>> It adds one page to these threads. The page does not require memory, > >>>> as it's protected. The stack will only require more space if the thread > >>>> ran into a stack overflow before the fix as else the pages are not > >>>> mapped. > >>>> This are stack overflows that cause hard crashes, at least on ppc the VM > >>>> does not properly catch these stack overflows, so any setup working > now > >>>> will not run into the additional space. Altogether there should be no > >>>> effect on running systems besides requiring one more entry in the > >>>> page table per non-Java thread. > >>>> > >>>> The problem is caused by a rather recent change (8140520: segfault on > >>>> solaris-amd64 > >>>> with "-XX:VMThreadStackSize=1" option) which was pushed after > >>>> feature-close. As this was a rather recent change, it must be > >>>> possible to > >>>> fix this follow up issue. What else is this period in the project good > >>>> for if not fixing issues? > >>> So I am seeing a number of factors here. > >>> > >>> First, 8140520, set: > >>> > >>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >> So I'm confused by the above comment: > >> > >> > The problem is caused by a rather recent change (8140520: segfault > >> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) > >> > >> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- > >> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html > >> > >> shows this change: > >> > >> @@ -531,19 +531,17 @@ > >> } > >> > >> > >> > ////////////////////////////////////////////////////////////////////////////// > >> // > >> // thread stack > >> > >> -size_t os::Linux::min_stack_allowed = 128*K; > >> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; > >> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * K; > >> > >> so the existing single variable of 'min_stack_allowed' was > >> replaced by three variables: _compiler_thread_min_stack_allowed, > >> _java_thread_min_stack_allowed, and > >> _vm_internal_thread_min_stack_allowed. > >> > >> The old single variable and the three new variables are all > >> initialized to the same value (128K) so the fix for 8140520 > >> did not change stack sizes for this platform. In fact, only > >> one platform had a size change (Solaris X64). > >> > >> So I'm confused about how the fix for 8140520 caused this problem. > >> > >> Based on David's analysis below, it looks to me like this 64K stack > >> guard page problem should also exist prior to the fix for 8140520. > >> > >> Goetz, can you please explain how 8140520 caused this problem? > >> > >> Dan > >> > >> > >> > >>> Second on linux PPC it is hardwired to use 2 guard pages: > >>> > >>> return 2 * page_size(); > >>> > >>> Third, you had a pagesize of 64K. > >>> > >>> Fourth, NPTL takes the guard space from the stack space - hence with 2 > >>> x 64K guard, and a 128K stack it was all consumed. > >>> > >>> --- > >>> > >>> In the proposed changes you now only use page_size() for the guard, so > >>> that alone would have fixed the observed problem. > >>> > >>> But in addition you want to address the NPTL problem by adding back > >>> the guard space to the stack size requested. That alone would also > >>> have fixed the observed problem. :) > >>> > >>> But in addition you have increased the minimum stack size: > >>> > >>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; > >>> > >>> which again, on its own would have fixed the original problem. :) > >>> > >>> Did you really intend to increase the real minimum stack from 128K to > >>> 256K ? (on a 64K page system) > >>> > >>> --- > >>> > >>> Focusing simply on the shared code change to adjust the requested > >>> stacksize by the amount of guard space (if any), this does not seem > >>> unreasonable. As you note it is restricted to non-JavaThreads and only > >>> adds a page to reserved stack space. > >>> > >>> My only query now is whether the minimum stacksize detection logic > >>> will correctly report the real minimum stack size (taking into account > >>> the need for the guard page) ? > >>> > >>> Thanks, > >>> David > >>> > >>>> So I really think this issue should be fixed. > >>>> > >>>> Best regards, > >>>> Goetz. > >>>> > >>>> > >>>>> -----Original Message----- > >>>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>>> Sent: Thursday, November 10, 2016 10:02 PM > >>>>> To: Lindenmaier, Goetz ; hotspot- > >> runtime- > >>>>> dev at openjdk.java.net > >>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > error. > >>>>> > >>>>> Hi Goetz, > >>>>> > >>>>> As per the bug report, this issue was already known (6675312) and we > >>>>> chose not to try and address it due to no reported issues at the time. > >>>>> While I see that you have encountered an issue (is it real or > >>>>> fabricated?) I think this change is too intrusive to be applied at this > >>>>> stage of the JDK 9 release cycle, as it will change the stack > >>>>> requirements of every application running on Linux. > >>>>> > >>>>> Thanks, > >>>>> David > >>>>> > >>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > >>>>>> Hi, > >>>>>> > >>>>>> > >>>>>> > >>>>>> Please review this change. I please need a sponsor: > >>>>>> > >>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > >> stackFix/webrev.01/ > >>>>>> > >>>>>> > >>>>>> In the Linux NPTL pthread implementation the guard size mechanism > >>>>>> is not > >>>>>> implemented properly. The posix standard requires to add the size > >>>>>> of the > >>>>>> guard pages to the stack size, instead Linux takes the space out of > >>>>>> 'stacksize'. > >>>>>> > >>>>>> The Posix standard > >> http://pubs.opengroup.org/onlinepubs/9699919799/ > >>>>>> says "the implementation allocates extra memory at the overflow > >>>>>> end of > >>>>>> the stack". The linux man page > >>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As at > >>>>>> glibc > >>>>>> 2.8, the NPTL threading implementation includes the guard area > within > >>>>>> the stack size allocation, rather than allocating extra space at > >>>>>> the end > >>>>>> of the stack, as POSIX.1 requires". > >>>>>> > >>>>>> I encounter this problem in runtime/Thread/TooSmallStackSize.java > >>>>>> on ppc > >>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K on > ppc, > >> and > >>>>>> ppc specifies two OS guard pages. The VM crashes in pthread > creation > >>>>>> because there is no usable space in the thread stack after allocating > >>>>>> the guard pages. > >>>>>> > >>>>>> But TooSmallStackSize.java requires that the VM comes up with the > >>>>>> stack > >>>>>> size mentioned in the error message. > >>>>>> > >>>>>> This fix adapts the requested stack size on Linux by the size of the > >>>>>> guard pages to mimick proper behaviour, see change to > os_linux.cpp. > >>>>>> > >>>>>> > >>>>>> > >>>>>> The change also streamlines usage of stack_guard_page on linuxppc, > >>>>>> linuxppcle, aixppc and linuxs390. > >>>>>> > >>>>>> > >>>>>> > >>>>>> To reproduce the error on linux_x86_64, apply below patch and call > the > >>>>>> VM with -XX:CompilerThreadStackSize=64. > >>>>>> > >>>>>> > >>>>>> > >>>>>> I'm still exploring why I had to choose such big compiler stacks on > >>>>>> ppc > >>>>>> to get -version passing, but I wanted to send the RFR now as people > >>>>>> obviously looked at the bug I opened (Thanks David!). > >>>>>> > >>>>>> > >>>>>> > >>>>>> Best regards, > >>>>>> > >>>>>> Goetz. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp > >>>>>> > >>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 > 12:37:28 > >>>>> 2016 > >>>>>> +0100 > >>>>>> > >>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 > >> 16:52:17 > >>>>> 2016 > >>>>>> +0100 > >>>>>> > >>>>>> @@ -701,7 +701,7 @@ > >>>>>> > >>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > >>>>>> > >>>>>> // Creating guard page is very expensive. Java thread has HotSpot > >>>>>> > >>>>>> // guard page, only enable glibc guard page for non-Java threads. > >>>>>> > >>>>>> - return (thr_type == java_thread ? 0 : page_size()); > >>>>>> > >>>>>> + return (thr_type == java_thread ? 0 : 64*K); > >>>>>> > >>>>>> } > >>>>>> > >>>>>> > >>>>>> > >>>>>> // Java thread: > >>>>>> From kirill.zhaldybin at oracle.com Tue Nov 22 13:24:07 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Tue, 22 Nov 2016 16:24:07 +0300 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part In-Reply-To: References: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> <85e51138-f3ce-7392-2cc3-ce7840aa3747@oracle.com> Message-ID: <23fcd50a-eed0-52a5-8817-244ecf75bb2a@oracle.com> Marcus, Thank you for prompt reply! Could you please read comments inline? I'm looking forward to your reply. Thank you. Regards, Kirill On 22.11.2016 15:32, Marcus Larsson wrote: > Hi, > > > On 2016-11-21 17:38, Kirill Zhaldybin wrote: >> Marcus, >> >> Thank you for reviewing the fix! >>>> WebRev: >>>> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ >>> >>> ISO8601 says the decimal point can be either '.' or ',' so the test >>> should accept either. You could let sscanf read out the decimal >>> point as a character and just verify that it is one of the two. >>> >>> In the UTC test you changed ASSERT_GT to ASSERT_EQ, which means that >>> we won't accept "Z" suffixed strings. Please revert that. >> I agree that ISO8601 could add "Z" to time (and as far as I >> understand date/time without delimiters is legal too) but these are >> the unit tests. >> Hence they cover the existing code and they should pass only if the >> result corresponds to existing code and fail otherwise. >> The current code from os::iso8601_time format date/time string >> %04d-%02d-%02dT%02d:%02d:%02d.%03d%c%02d%02d so we should not >> consider any other format as valid. >> >> Could you please let me know your opinion? > > I think the test should verify the intended behavior, not the > implementation. If we refactor or change something in iso8601_time() > we shouldn't be failing the test if it still conforms to ISO8601, IMO. I would agree with you if we were talking about a functional test. But since it is an unit test I think we should keep it as close to implementation as possible. If the implementation is changed unintentionally the test fails and signals us that something is broken. If it is an intentional change the test must be updated correspondingly. > > Thanks, > Marcus > >> >> Thank you. >> >> Regards, Kirill >> >>> >>> Thanks, >>> Marcus >>> >>>> CR: https://bugs.openjdk.java.net/browse/JDK-8169003 >>>> >>>> Thank you. >>>> >>>> Regards, Kirill >>> >> > From thomas.schatzl at oracle.com Tue Nov 22 13:41:00 2016 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 22 Nov 2016 14:41:00 +0100 Subject: RFR(S): 8168492: Convert CollectedHeap_test to GTest In-Reply-To: <721A3D9B-36D2-41F1-B965-E46CED8F8FA8@oracle.com> References: <489f77cb-f7be-51f5-b602-9e2eab3ee807@oracle.com> <721A3D9B-36D2-41F1-B965-E46CED8F8FA8@oracle.com> Message-ID: <1479822060.2435.33.camel@oracle.com> Hi Kirill, ? in case you need another review... > On Nov 1, 2016, at 5:10 PM, Kirill Zhaldybin > wrote: > > > > Dear all, > > > > Could you please review this fix for 8168492? > > > > WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168492/we > > brev.00/ > > CR: https://bugs.openjdk.java.net/browse/JDK-8168492 In line? ? 47???ASSERT_LE(heap_end, ((uintptr_t) - 1 - epsilon)) the "-" in the "- 1" expression is an unary minus, not the regular minus operator, so the space between it and the "1" should be removed. Further the space between the uintptr_t cast and the unary minus seems not appropriate. I.e. the expression should look like "((uintptr_t)-1 - epsilon)" like in the original code. I do not need a re-review for this. Thanks, ? Thomas From thomas.schatzl at oracle.com Tue Nov 22 13:50:32 2016 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 22 Nov 2016 14:50:32 +0100 Subject: RFR(S): 8168493: Convert TestBitMap_test to GTest In-Reply-To: References: Message-ID: <1479822632.2435.35.camel@oracle.com> Hi Kirill, On Tue, 2016-11-01 at 16:45 +0300, Kirill Zhaldybin wrote: > Dear all, > > Could you please review this fix for 8168493? > > WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168493/webr > ev.00/ > CR: https://bugs.openjdk.java.net/browse/JDK-8168493 > > Changes: > 1. Usual gtest conversion - asserts, file location etc > 2. Simplified the test a bit - deleted unused template parameter,? > refactoring > 3. Deleted friend class from BitMap class since we actually do not > need it > > Thank you. > ? conversion looks good. Thanks, ? Thomas From kirill.zhaldybin at oracle.com Tue Nov 22 15:00:00 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Tue, 22 Nov 2016 18:00:00 +0300 Subject: RFR(S): 8168493: Convert TestBitMap_test to GTest In-Reply-To: <1479822632.2435.35.camel@oracle.com> References: <1479822632.2435.35.camel@oracle.com> Message-ID: Thomas, Thank you for review! Regards, Kirill On 22.11.2016 16:50, Thomas Schatzl wrote: > Hi Kirill, > > On Tue, 2016-11-01 at 16:45 +0300, Kirill Zhaldybin wrote: >> Dear all, >> >> Could you please review this fix for 8168493? >> >> WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168493/webr >> ev.00/ >> CR: https://bugs.openjdk.java.net/browse/JDK-8168493 >> >> Changes: >> 1. Usual gtest conversion - asserts, file location etc >> 2. Simplified the test a bit - deleted unused template parameter, >> refactoring >> 3. Deleted friend class from BitMap class since we actually do not >> need it >> >> Thank you. >> > conversion looks good. > > Thanks, > Thomas > From kirill.zhaldybin at oracle.com Tue Nov 22 15:01:58 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Tue, 22 Nov 2016 18:01:58 +0300 Subject: RFR(S): 8168492: Convert CollectedHeap_test to GTest In-Reply-To: <1479822060.2435.33.camel@oracle.com> References: <489f77cb-f7be-51f5-b602-9e2eab3ee807@oracle.com> <721A3D9B-36D2-41F1-B965-E46CED8F8FA8@oracle.com> <1479822060.2435.33.camel@oracle.com> Message-ID: <6423992a-b649-4605-c8ab-432a53ca827c@oracle.com> Thomas, Thank you for review! I will change the formatting before pushing the fix. Regards, Kirill On 22.11.2016 16:41, Thomas Schatzl wrote: > Hi Kirill, > > in case you need another review... > >> On Nov 1, 2016, at 5:10 PM, Kirill Zhaldybin >> wrote: >>> Dear all, >>> >>> Could you please review this fix for 8168492? >>> >>> WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168492/we >>> brev.00/ >>> CR: https://bugs.openjdk.java.net/browse/JDK-8168492 > In line > > 47 ASSERT_LE(heap_end, ((uintptr_t) - 1 - epsilon)) > > the "-" in the "- 1" expression is an unary minus, not the regular > minus operator, so the space between it and the "1" should be removed. > > Further the space between the uintptr_t cast and the unary minus seems > not appropriate. > > I.e. the expression should look like "((uintptr_t)-1 - epsilon)" like > in the original code. > > I do not need a re-review for this. > > Thanks, > Thomas > From david.holmes at oracle.com Wed Nov 23 01:49:35 2016 From: david.holmes at oracle.com (David Holmes) Date: Wed, 23 Nov 2016 11:49:35 +1000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> Message-ID: On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: > Hi Dan, > >> -----Original Message----- >> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >> Sent: Dienstag, 22. November 2016 14:01 >> To: Lindenmaier, Goetz ; David Holmes >> ; hotspot-runtime-dev at openjdk.java.net >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >> >> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: >>> Hi Dan, >>> >>> I ran into a row of issues, some errors on the platforms. >>> >>> What I meant with that comment is that >>> os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, >>> JavaThread::stack_guard_zone_size() + >>> JavaThread::stack_shadow_zone_size() + >>> (4*BytesPerWord COMPILER2_PRESENT(+2)) * 4 * K); >>> was executed, and min_stack_allowed used for all stacks. Now, compiler >> and >>> vm minimum stack sizes are not increased by these sizes. >> >> Now I see what you mean. Thanks for clearing this up. >> >> I should have remembered that part of the change because we went back >> and forth about removing the red/yellow zone pages from the other >> threads. In particular, we discussed the compiler thread because it >> is-a JavaThread. Our conclusion was that a compiler thread doesn't >> execute Java bytecode so we could remove the red/yellow pages... > Yes, it does not execute java byte code. Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The ability for a CompilerThread to execute Java code (can_call_java()) is now a dynamic property depending on whether the current compiler is the JVMCI compiler. David ----- > Therefore you can remove the shadow pages. There is no code that > will bang. > But red/yellow pages are protected right during thread startup. > Therefore you must have enough space for them. > On ppc, we try to protect three 64K pages out of the 128K compiler stack. > That obviously fails. > > Therefore I propose: > size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // Set platform dependent. > in os::Posix::set_minimum_stack_sizes(): > _java_thread_min_stack_allowed = _java_thread_min_stack_allowed + > JavaThread::stack_guard_zone_size() + > JavaThread::stack_shadow_zone_size(); > > (Similar for _compiler_thread_min_stack_allowed). > > The minimal stack size is made up of three components: > 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot implementation/platform/os. > 2. Sizes of C++ frames: depends on C++ compiler. > These are fixed at compile time. > 3. Sizes of red/yellow/reserved/shadow pages. Depends on the system the > VM is used on. This is not fixed at compile time. (Our ppc machines differ > in page size.) > Therefore 3. should not be included in a compile time constant. > > > >> And that decision allowed us to be exposed to the 64K page issue >> because the "extra" space isn't there anymore. >> >>> At least the _compiler_thread_min_stack_allowed should be increased >>> similarly by red/yellow zone size. The compiler thread is a Java >>> thread and must have space for these zones. >> >> I'm not sure that I completely agree (yet). To me, the red/yellow >> pages are there for Java thread stack overflow semantics. Yes, the >> compiler thread needs extra space when 64K pages are used, but I >> would prefer that we add that space via a different calculation. > Yes they are. But compiler threads happen tob e a subclass of > Java threads and use the same run() method that puts the pages > There. > I agree that they are not needed for Compiler threads, nor for > CodeCacheSweeperThreads. I don't really now about > JvmtiAgentThreads and ServiceThreads, but all of the get the guard pages > because they are derived from JavaThread. > >> To put it another way, I'd like to see us add extra space to solve >> the 64K page issue directly instead of as a side effect of the >> red/yellow page addition. > I don't understand. What do you mean by 'directly'? > >>> Also, the change added a test that is failing now. >> >> And that's a "good thing" (TM), right? :-) > Yes, it showed a bug and thus raised the need to fix it! :) > > Best regards, > Goetz. > > >> Again, thanks for clarifying 8140520's role in this issue. >> >> Dan >> >> >>> >>> Best regards, >>> Goetz. >>> >>>> -----Original Message----- >>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>> Sent: Montag, 21. November 2016 17:28 >>>> To: David Holmes ; Lindenmaier, Goetz >>>> ; hotspot-runtime-dev at openjdk.java.net >>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>>> >>>> Sorry for the delayed responses to this thread. I've been on vacation... >>>> >>>> One comment/query embedded below... >>>> >>>> >>>> On 11/10/16 8:40 PM, David Holmes wrote: >>>>> Hi Goetz, >>>>> >>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: >>>>>> Hi David, >>>>>> >>>>>> This issue is different to 6675312, see also my comment in the bug. >>>>>> >>>>>> It appears running jtreg test runtime/Thread/TooSmallStackSize.java, >>>>>> with my patch below you can reproduce it on linuxx86_64. You can not >>>>>> do that with 6675312. Also, I would assume there are systems out there >>>>>> on x86 that uses 64-K pages, did you run the tests on these? I would >>>>>> assume you get hard crashes with stack overflows in the first C2 >>>>>> compilation if there is only 64K usable CompilerThreadStack. >>>>>> >>>>>> My fix does not affect Java threads, which are the largest amount >>>>>> of threads used by the VM. It affects only the non-Java threads. >>>>>> It adds one page to these threads. The page does not require memory, >>>>>> as it's protected. The stack will only require more space if the thread >>>>>> ran into a stack overflow before the fix as else the pages are not >>>>>> mapped. >>>>>> This are stack overflows that cause hard crashes, at least on ppc the VM >>>>>> does not properly catch these stack overflows, so any setup working >> now >>>>>> will not run into the additional space. Altogether there should be no >>>>>> effect on running systems besides requiring one more entry in the >>>>>> page table per non-Java thread. >>>>>> >>>>>> The problem is caused by a rather recent change (8140520: segfault on >>>>>> solaris-amd64 >>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed after >>>>>> feature-close. As this was a rather recent change, it must be >>>>>> possible to >>>>>> fix this follow up issue. What else is this period in the project good >>>>>> for if not fixing issues? >>>>> So I am seeing a number of factors here. >>>>> >>>>> First, 8140520, set: >>>>> >>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >>>> So I'm confused by the above comment: >>>> >>>> > The problem is caused by a rather recent change (8140520: segfault >>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) >>>> >>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- >>>> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html >>>> >>>> shows this change: >>>> >>>> @@ -531,19 +531,17 @@ >>>> } >>>> >>>> >>>> >> ////////////////////////////////////////////////////////////////////////////// >>>> // >>>> // thread stack >>>> >>>> -size_t os::Linux::min_stack_allowed = 128*K; >>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; >>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * K; >>>> >>>> so the existing single variable of 'min_stack_allowed' was >>>> replaced by three variables: _compiler_thread_min_stack_allowed, >>>> _java_thread_min_stack_allowed, and >>>> _vm_internal_thread_min_stack_allowed. >>>> >>>> The old single variable and the three new variables are all >>>> initialized to the same value (128K) so the fix for 8140520 >>>> did not change stack sizes for this platform. In fact, only >>>> one platform had a size change (Solaris X64). >>>> >>>> So I'm confused about how the fix for 8140520 caused this problem. >>>> >>>> Based on David's analysis below, it looks to me like this 64K stack >>>> guard page problem should also exist prior to the fix for 8140520. >>>> >>>> Goetz, can you please explain how 8140520 caused this problem? >>>> >>>> Dan >>>> >>>> >>>> >>>>> Second on linux PPC it is hardwired to use 2 guard pages: >>>>> >>>>> return 2 * page_size(); >>>>> >>>>> Third, you had a pagesize of 64K. >>>>> >>>>> Fourth, NPTL takes the guard space from the stack space - hence with 2 >>>>> x 64K guard, and a 128K stack it was all consumed. >>>>> >>>>> --- >>>>> >>>>> In the proposed changes you now only use page_size() for the guard, so >>>>> that alone would have fixed the observed problem. >>>>> >>>>> But in addition you want to address the NPTL problem by adding back >>>>> the guard space to the stack size requested. That alone would also >>>>> have fixed the observed problem. :) >>>>> >>>>> But in addition you have increased the minimum stack size: >>>>> >>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; >>>>> >>>>> which again, on its own would have fixed the original problem. :) >>>>> >>>>> Did you really intend to increase the real minimum stack from 128K to >>>>> 256K ? (on a 64K page system) >>>>> >>>>> --- >>>>> >>>>> Focusing simply on the shared code change to adjust the requested >>>>> stacksize by the amount of guard space (if any), this does not seem >>>>> unreasonable. As you note it is restricted to non-JavaThreads and only >>>>> adds a page to reserved stack space. >>>>> >>>>> My only query now is whether the minimum stacksize detection logic >>>>> will correctly report the real minimum stack size (taking into account >>>>> the need for the guard page) ? >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> So I really think this issue should be fixed. >>>>>> >>>>>> Best regards, >>>>>> Goetz. >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>>> Sent: Thursday, November 10, 2016 10:02 PM >>>>>>> To: Lindenmaier, Goetz ; hotspot- >>>> runtime- >>>>>>> dev at openjdk.java.net >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >> error. >>>>>>> >>>>>>> Hi Goetz, >>>>>>> >>>>>>> As per the bug report, this issue was already known (6675312) and we >>>>>>> chose not to try and address it due to no reported issues at the time. >>>>>>> While I see that you have encountered an issue (is it real or >>>>>>> fabricated?) I think this change is too intrusive to be applied at this >>>>>>> stage of the JDK 9 release cycle, as it will change the stack >>>>>>> requirements of every application running on Linux. >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Please review this change. I please need a sponsor: >>>>>>>> >>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >>>> stackFix/webrev.01/ >>>>>>>> >>>>>>>> >>>>>>>> In the Linux NPTL pthread implementation the guard size mechanism >>>>>>>> is not >>>>>>>> implemented properly. The posix standard requires to add the size >>>>>>>> of the >>>>>>>> guard pages to the stack size, instead Linux takes the space out of >>>>>>>> 'stacksize'. >>>>>>>> >>>>>>>> The Posix standard >>>> http://pubs.opengroup.org/onlinepubs/9699919799/ >>>>>>>> says "the implementation allocates extra memory at the overflow >>>>>>>> end of >>>>>>>> the stack". The linux man page >>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As at >>>>>>>> glibc >>>>>>>> 2.8, the NPTL threading implementation includes the guard area >> within >>>>>>>> the stack size allocation, rather than allocating extra space at >>>>>>>> the end >>>>>>>> of the stack, as POSIX.1 requires". >>>>>>>> >>>>>>>> I encounter this problem in runtime/Thread/TooSmallStackSize.java >>>>>>>> on ppc >>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K on >> ppc, >>>> and >>>>>>>> ppc specifies two OS guard pages. The VM crashes in pthread >> creation >>>>>>>> because there is no usable space in the thread stack after allocating >>>>>>>> the guard pages. >>>>>>>> >>>>>>>> But TooSmallStackSize.java requires that the VM comes up with the >>>>>>>> stack >>>>>>>> size mentioned in the error message. >>>>>>>> >>>>>>>> This fix adapts the requested stack size on Linux by the size of the >>>>>>>> guard pages to mimick proper behaviour, see change to >> os_linux.cpp. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> The change also streamlines usage of stack_guard_page on linuxppc, >>>>>>>> linuxppcle, aixppc and linuxs390. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> To reproduce the error on linux_x86_64, apply below patch and call >> the >>>>>>>> VM with -XX:CompilerThreadStackSize=64. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I'm still exploring why I had to choose such big compiler stacks on >>>>>>>> ppc >>>>>>>> to get -version passing, but I wanted to send the RFR now as people >>>>>>>> obviously looked at the bug I opened (Thanks David!). >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> Goetz. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>>>>>>> >>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 >> 12:37:28 >>>>>>> 2016 >>>>>>>> +0100 >>>>>>>> >>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 >>>> 16:52:17 >>>>>>> 2016 >>>>>>>> +0100 >>>>>>>> >>>>>>>> @@ -701,7 +701,7 @@ >>>>>>>> >>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>>>>>>> >>>>>>>> // Creating guard page is very expensive. Java thread has HotSpot >>>>>>>> >>>>>>>> // guard page, only enable glibc guard page for non-Java threads. >>>>>>>> >>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); >>>>>>>> >>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); >>>>>>>> >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> // Java thread: >>>>>>>> > From david.holmes at oracle.com Wed Nov 23 02:08:25 2016 From: david.holmes at oracle.com (David Holmes) Date: Wed, 23 Nov 2016 12:08:25 +1000 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> Message-ID: <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Hi Thomas, > David did not like that, he thought it strange to check an enum for > correct enum values. I can see his point. He would have preferred > instead an assert inside NMTUtil::flag_to_index() which before > returning checks the to-be-returned index for > 0>=i>=mt_number_of_types. I did not like that because strictly > speaking inside of NMTUtil::flag_to_index() it is not known that the > returned integer will be used by the caller as an index into an array > of mt_number_of_types length. Sorry but how is NMTUtil::flag_to_index anything but a mapping from 0 to mt_number_of_types ??? MEMFLAGS aka MemoryType is an enum that spans zero to mt_number_of_types! We're validating the post condition for the method. As I said on Oct 19, and which Chris indicated he agreed with, it makes no sense to me to put the same assert in all the callers of flag_to_index instead of doing the validation inside flag_to_index. Whether you choose to validate the flag or the index I don't care. If we were in a decent language the bug would have been impossible to begin with. David ----- On 22/11/2016 1:21 AM, Thomas St?fe wrote: > Hi all, > > this small issue got fc extension, so lets wrap this up. Thanks to > all for reviewing! > > New webrev: > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ > > No changes to the last one but the updated copyrights Chris did ask > for. Solution still follows Max' suggestion of checking the index > right before consumption. > > Thanks, and Kind Regards, Thomas > > > On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe > > wrote: > > Hi Chris, > > On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer > > wrote: > > On 10/18/16 11:49 PM, David Holmes wrote: > > > > On 19/10/2016 3:17 PM, Thomas St?fe wrote: > > > > On Wed, Oct 19, 2016 at 3:21 AM, David Holmes > > >> > wrote: > > On 18/10/2016 3:39 PM, Thomas St?fe wrote: > > Hi David, Max, > > I changed the asserts according to Max' suggestion. Instead of > checking inside flag_to_index, now I check before callers of this > function use this value to access memory. > > > I don't see where Max suggested that?? > > > Max wrote: " I think the decision on whether to access a slot should > occur as close to memory accessing code as possible." and proceeded > to suggest fixing VirtualMemorySnapshot::by_type() as well. > > > I did not interpret that comment that way, and was puzzled by the > reference to by_type. > > > It doesn't make sense to me to have all the callers of flag_to_index > check what it returned instead of doing it inside flag_to_index. > > > I disagree. Imho it makes sense to either check the Memflags > enumeration input argument in flag_to_index() or the returned index > before consumption. In both cases one knows the valid value range. > Strictly speaking checking the index in flag_to_index() cannot be > done because it is a faceless int type whose valid values are not yet > known. > > > The index has to fall in the range 0 <= index <= mt_number_of_types, > and I was suggesting that it makes more sense to verify this once in > flag_to_index() than in all the callers of flag_to_index. > > Hi Thomas, > > Just catching up on this thread. This is the same conclusion I came > to. I don't understand what you mean by "it is a faceless int type > whose valid values are not yet known". > > > Thank you for looking into this! > > "it is a faceless int type whose valid values are not yet known" was > maybe expressed sloppily: > > My first patch checked the input enum "flag" argument inside > NMTUtil::flag_to_index() for correct enum values : > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > > > David did not like that, he thought it strange to check an enum for > correct enum values. I can see his point. He would have preferred > instead an assert inside NMTUtil::flag_to_index() which before > returning checks the to-be-returned index for > 0>=i>=mt_number_of_types. I did not like that because strictly > speaking inside of NMTUtil::flag_to_index() it is not known that the > returned integer will be used by the caller as an index into an array > of mt_number_of_types length. > > It is all extreme nitpicking and in the end amounts to the same :) > > The current patch does neither, but follows Max' suggestion of > checking the index right before it is consumed to access an array. I > think this is a good solution and very clear. > > > > BTW, I'll sponsor this fix for you once it is finalized. Please > update the copyright dates first. > > > Thank you. I'll update the webrev and repost. > > > > thanks, > > Chris > > > Kind Regards, Thomas > > > > David > > It is all academical and mostly a matter of taste. > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > > > > > > As David correctly writes, this is technically not a bug, so I guess > this will have to wait until java 10. > > > Yes, afraid so. > > > The fix is trivial and I will try to get fc extension for this (now > that Goetz explained to me how to do this :). It seems this is done > for many other non-bug issues as well. > > ..Thomas > > > Thanks, David > > Kind Regards, Thomas > > > > On Fri, Oct 14, 2016 at 12:57 AM, David Holmes > > > > > >>> > wrote: > > On 13/10/2016 10:53 PM, Thomas St?fe wrote: > > Hi David, > > On Thu, Oct 13, 2016 at 12:08 PM, David Holmes > > > > > >> > > > > > > >>>> > wrote: > > Hi Thomas, > > On 13/10/2016 3:49 PM, Thomas St?fe wrote: > > Hi all, > > may I have plase a review for this tiny change? It just adds some > assert to NMT. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8167650 > > > > > > >> > > > > > > > > >>> webrev: > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > > > > > > > > > >> > > > > > > > > > > > > >>> > MEMFLAGS/webrev.00/webrev/ > > We had an ugly memory overwrite caused by this - ultimately our > fault, because we fed an invalid memory flag to NMT - but it was > difficult to find. An assert would have saved some time. > > > I'm a little perplexed with asserting that something of MEMFLAGS type > must be an actual MEMFLAGS value - it implies the caller is coercing > plain int to MEMFLAGS, and I don't have much sympathy if they mess > that up. Can't help wondering if there is some clever C++ trick to > flag bad conversions at compile-time? > > > The error was caused by an uninitialized variable of type MEMFLAGS. > This was our fault, we have heavily modified allocation.hpp and > introduced an error then merging changes from upstream. Due to a > merging error this lead to a case where Arena::_flags was not > initialized and contained a very large value. > > > Ah I see. Lack of default initialization can be annoying :) > > I admit it looks funny. If it bothers you, I could instead check the > returned index to be in the range for the size of the _malloc array > in MallocMemorySnapshot::by_type(). Technically, it would mean the > same. > > > So I just realized that here: > > 62 // Map memory type to human readable name 63 static const > char* flag_to_name(MEMFLAGS flag) { 64 assert(flag >= 0 && flag > < mt_number_of_types, "Invalid flag value %d.", (int)flag); 65 > return _memory_type_names[flag_to_index(flag)]; 66 } > > we call flag_to_index, so the assert is redundant as it is already in > flag_to_index. Then presumably we change flag_to_index to something > like this: > > static inline int flag_to_index(MEMFLAGS flag) { int index = (flag & > 0xff); assert(index >= 0 && index < mt_number_of_types, "Invalid flag > value %d.", (int)flag); return index; } > > so we're validating the index rather than the flag. > > Cheers, David > > > > > The function that takes the index should validate the index, so that > is fine. > > Which one were you actually passing the bad value to? :) > > This isn't a strong objection just musing if we can do better. And as > the hs repos are still closed, and likely to remain so till early > next week, we have some slack time :) > > > :) Sure. > > Kind Regards, Thomas > > > Cheers, David > > Thank you! > > Thomas > > > > > > > > From goetz.lindenmaier at sap.com Wed Nov 23 07:11:17 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Wed, 23 Nov 2016 07:11:17 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> Message-ID: Hi, > Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The > ability for a CompilerThread to execute Java code (can_call_java()) is > now a dynamic property depending on whether the current compiler is the > JVMCI compiler. Ah, then I should also leave space for shadow pages in the minimal stack size of comiler threads. Do we agree on the cleanup and on leaving out the OS guard page on compiler threads? Then I would edit a change comprising the NPTL workaround and these additional changes, and split the other issue into a new bug? I think this will easy the reviewing. Best regards, Goetz. > -----Original Message----- > From: David Holmes [mailto:david.holmes at oracle.com] > Sent: Mittwoch, 23. November 2016 02:50 > To: Lindenmaier, Goetz ; > daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: > > Hi Dan, > > > >> -----Original Message----- > >> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >> Sent: Dienstag, 22. November 2016 14:01 > >> To: Lindenmaier, Goetz ; David Holmes > >> ; hotspot-runtime-dev at openjdk.java.net > >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > >> > >> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: > >>> Hi Dan, > >>> > >>> I ran into a row of issues, some errors on the platforms. > >>> > >>> What I meant with that comment is that > >>> os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, > >>> JavaThread::stack_guard_zone_size() + > >>> JavaThread::stack_shadow_zone_size() + > >>> (4*BytesPerWord COMPILER2_PRESENT(+2)) * 4 * > K); > >>> was executed, and min_stack_allowed used for all stacks. Now, compiler > >> and > >>> vm minimum stack sizes are not increased by these sizes. > >> > >> Now I see what you mean. Thanks for clearing this up. > >> > >> I should have remembered that part of the change because we went back > >> and forth about removing the red/yellow zone pages from the other > >> threads. In particular, we discussed the compiler thread because it > >> is-a JavaThread. Our conclusion was that a compiler thread doesn't > >> execute Java bytecode so we could remove the red/yellow pages... > > Yes, it does not execute java byte code. > > Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The > ability for a CompilerThread to execute Java code (can_call_java()) is > now a dynamic property depending on whether the current compiler is the > JVMCI compiler. > > David > ----- > > > Therefore you can remove the shadow pages. There is no code that > > will bang. > > But red/yellow pages are protected right during thread startup. > > Therefore you must have enough space for them. > > On ppc, we try to protect three 64K pages out of the 128K compiler stack. > > That obviously fails. > > > > Therefore I propose: > > size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // Set > platform dependent. > > in os::Posix::set_minimum_stack_sizes(): > > _java_thread_min_stack_allowed = _java_thread_min_stack_allowed + > > JavaThread::stack_guard_zone_size() + > > JavaThread::stack_shadow_zone_size(); > > > > (Similar for _compiler_thread_min_stack_allowed). > > > > The minimal stack size is made up of three components: > > 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot > implementation/platform/os. > > 2. Sizes of C++ frames: depends on C++ compiler. > > These are fixed at compile time. > > 3. Sizes of red/yellow/reserved/shadow pages. Depends on the system > the > > VM is used on. This is not fixed at compile time. (Our ppc machines > differ > > in page size.) > > Therefore 3. should not be included in a compile time constant. > > > > > > > >> And that decision allowed us to be exposed to the 64K page issue > >> because the "extra" space isn't there anymore. > >> > >>> At least the _compiler_thread_min_stack_allowed should be increased > >>> similarly by red/yellow zone size. The compiler thread is a Java > >>> thread and must have space for these zones. > >> > >> I'm not sure that I completely agree (yet). To me, the red/yellow > >> pages are there for Java thread stack overflow semantics. Yes, the > >> compiler thread needs extra space when 64K pages are used, but I > >> would prefer that we add that space via a different calculation. > > Yes they are. But compiler threads happen tob e a subclass of > > Java threads and use the same run() method that puts the pages > > There. > > I agree that they are not needed for Compiler threads, nor for > > CodeCacheSweeperThreads. I don't really now about > > JvmtiAgentThreads and ServiceThreads, but all of the get the guard pages > > because they are derived from JavaThread. > > > >> To put it another way, I'd like to see us add extra space to solve > >> the 64K page issue directly instead of as a side effect of the > >> red/yellow page addition. > > I don't understand. What do you mean by 'directly'? > > > >>> Also, the change added a test that is failing now. > >> > >> And that's a "good thing" (TM), right? :-) > > Yes, it showed a bug and thus raised the need to fix it! :) > > > > Best regards, > > Goetz. > > > > > >> Again, thanks for clarifying 8140520's role in this issue. > >> > >> Dan > >> > >> > >>> > >>> Best regards, > >>> Goetz. > >>> > >>>> -----Original Message----- > >>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >>>> Sent: Montag, 21. November 2016 17:28 > >>>> To: David Holmes ; Lindenmaier, Goetz > >>>> ; hotspot-runtime- > dev at openjdk.java.net > >>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > error. > >>>> > >>>> Sorry for the delayed responses to this thread. I've been on vacation... > >>>> > >>>> One comment/query embedded below... > >>>> > >>>> > >>>> On 11/10/16 8:40 PM, David Holmes wrote: > >>>>> Hi Goetz, > >>>>> > >>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > >>>>>> Hi David, > >>>>>> > >>>>>> This issue is different to 6675312, see also my comment in the bug. > >>>>>> > >>>>>> It appears running jtreg test > runtime/Thread/TooSmallStackSize.java, > >>>>>> with my patch below you can reproduce it on linuxx86_64. You can > not > >>>>>> do that with 6675312. Also, I would assume there are systems out > there > >>>>>> on x86 that uses 64-K pages, did you run the tests on these? I would > >>>>>> assume you get hard crashes with stack overflows in the first C2 > >>>>>> compilation if there is only 64K usable CompilerThreadStack. > >>>>>> > >>>>>> My fix does not affect Java threads, which are the largest amount > >>>>>> of threads used by the VM. It affects only the non-Java threads. > >>>>>> It adds one page to these threads. The page does not require > memory, > >>>>>> as it's protected. The stack will only require more space if the thread > >>>>>> ran into a stack overflow before the fix as else the pages are not > >>>>>> mapped. > >>>>>> This are stack overflows that cause hard crashes, at least on ppc the > VM > >>>>>> does not properly catch these stack overflows, so any setup working > >> now > >>>>>> will not run into the additional space. Altogether there should be no > >>>>>> effect on running systems besides requiring one more entry in the > >>>>>> page table per non-Java thread. > >>>>>> > >>>>>> The problem is caused by a rather recent change (8140520: segfault > on > >>>>>> solaris-amd64 > >>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed after > >>>>>> feature-close. As this was a rather recent change, it must be > >>>>>> possible to > >>>>>> fix this follow up issue. What else is this period in the project good > >>>>>> for if not fixing issues? > >>>>> So I am seeing a number of factors here. > >>>>> > >>>>> First, 8140520, set: > >>>>> > >>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >>>> So I'm confused by the above comment: > >>>> > >>>> > The problem is caused by a rather recent change (8140520: segfault > >>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) > >>>> > >>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- > >>>> > open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html > >>>> > >>>> shows this change: > >>>> > >>>> @@ -531,19 +531,17 @@ > >>>> } > >>>> > >>>> > >>>> > >> > ////////////////////////////////////////////////////////////////////////////// > >>>> // > >>>> // thread stack > >>>> > >>>> -size_t os::Linux::min_stack_allowed = 128*K; > >>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; > >>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * K; > >>>> > >>>> so the existing single variable of 'min_stack_allowed' was > >>>> replaced by three variables: _compiler_thread_min_stack_allowed, > >>>> _java_thread_min_stack_allowed, and > >>>> _vm_internal_thread_min_stack_allowed. > >>>> > >>>> The old single variable and the three new variables are all > >>>> initialized to the same value (128K) so the fix for 8140520 > >>>> did not change stack sizes for this platform. In fact, only > >>>> one platform had a size change (Solaris X64). > >>>> > >>>> So I'm confused about how the fix for 8140520 caused this problem. > >>>> > >>>> Based on David's analysis below, it looks to me like this 64K stack > >>>> guard page problem should also exist prior to the fix for 8140520. > >>>> > >>>> Goetz, can you please explain how 8140520 caused this problem? > >>>> > >>>> Dan > >>>> > >>>> > >>>> > >>>>> Second on linux PPC it is hardwired to use 2 guard pages: > >>>>> > >>>>> return 2 * page_size(); > >>>>> > >>>>> Third, you had a pagesize of 64K. > >>>>> > >>>>> Fourth, NPTL takes the guard space from the stack space - hence with > 2 > >>>>> x 64K guard, and a 128K stack it was all consumed. > >>>>> > >>>>> --- > >>>>> > >>>>> In the proposed changes you now only use page_size() for the guard, > so > >>>>> that alone would have fixed the observed problem. > >>>>> > >>>>> But in addition you want to address the NPTL problem by adding back > >>>>> the guard space to the stack size requested. That alone would also > >>>>> have fixed the observed problem. :) > >>>>> > >>>>> But in addition you have increased the minimum stack size: > >>>>> > >>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; > >>>>> > >>>>> which again, on its own would have fixed the original problem. :) > >>>>> > >>>>> Did you really intend to increase the real minimum stack from 128K to > >>>>> 256K ? (on a 64K page system) > >>>>> > >>>>> --- > >>>>> > >>>>> Focusing simply on the shared code change to adjust the requested > >>>>> stacksize by the amount of guard space (if any), this does not seem > >>>>> unreasonable. As you note it is restricted to non-JavaThreads and only > >>>>> adds a page to reserved stack space. > >>>>> > >>>>> My only query now is whether the minimum stacksize detection logic > >>>>> will correctly report the real minimum stack size (taking into account > >>>>> the need for the guard page) ? > >>>>> > >>>>> Thanks, > >>>>> David > >>>>> > >>>>>> So I really think this issue should be fixed. > >>>>>> > >>>>>> Best regards, > >>>>>> Goetz. > >>>>>> > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>>>>> Sent: Thursday, November 10, 2016 10:02 PM > >>>>>>> To: Lindenmaier, Goetz ; hotspot- > >>>> runtime- > >>>>>>> dev at openjdk.java.net > >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > >> error. > >>>>>>> > >>>>>>> Hi Goetz, > >>>>>>> > >>>>>>> As per the bug report, this issue was already known (6675312) and > we > >>>>>>> chose not to try and address it due to no reported issues at the > time. > >>>>>>> While I see that you have encountered an issue (is it real or > >>>>>>> fabricated?) I think this change is too intrusive to be applied at this > >>>>>>> stage of the JDK 9 release cycle, as it will change the stack > >>>>>>> requirements of every application running on Linux. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> David > >>>>>>> > >>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Please review this change. I please need a sponsor: > >>>>>>>> > >>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > >>>> stackFix/webrev.01/ > >>>>>>>> > >>>>>>>> > >>>>>>>> In the Linux NPTL pthread implementation the guard size > mechanism > >>>>>>>> is not > >>>>>>>> implemented properly. The posix standard requires to add the > size > >>>>>>>> of the > >>>>>>>> guard pages to the stack size, instead Linux takes the space out of > >>>>>>>> 'stacksize'. > >>>>>>>> > >>>>>>>> The Posix standard > >>>> http://pubs.opengroup.org/onlinepubs/9699919799/ > >>>>>>>> says "the implementation allocates extra memory at the > overflow > >>>>>>>> end of > >>>>>>>> the stack". The linux man page > >>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As > at > >>>>>>>> glibc > >>>>>>>> 2.8, the NPTL threading implementation includes the guard area > >> within > >>>>>>>> the stack size allocation, rather than allocating extra space at > >>>>>>>> the end > >>>>>>>> of the stack, as POSIX.1 requires". > >>>>>>>> > >>>>>>>> I encounter this problem in > runtime/Thread/TooSmallStackSize.java > >>>>>>>> on ppc > >>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K on > >> ppc, > >>>> and > >>>>>>>> ppc specifies two OS guard pages. The VM crashes in pthread > >> creation > >>>>>>>> because there is no usable space in the thread stack after > allocating > >>>>>>>> the guard pages. > >>>>>>>> > >>>>>>>> But TooSmallStackSize.java requires that the VM comes up with > the > >>>>>>>> stack > >>>>>>>> size mentioned in the error message. > >>>>>>>> > >>>>>>>> This fix adapts the requested stack size on Linux by the size of the > >>>>>>>> guard pages to mimick proper behaviour, see change to > >> os_linux.cpp. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> The change also streamlines usage of stack_guard_page on > linuxppc, > >>>>>>>> linuxppcle, aixppc and linuxs390. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> To reproduce the error on linux_x86_64, apply below patch and > call > >> the > >>>>>>>> VM with -XX:CompilerThreadStackSize=64. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> I'm still exploring why I had to choose such big compiler stacks on > >>>>>>>> ppc > >>>>>>>> to get -version passing, but I wanted to send the RFR now as > people > >>>>>>>> obviously looked at the bug I opened (Thanks David!). > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Best regards, > >>>>>>>> > >>>>>>>> Goetz. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp > >>>>>>>> > >>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 > >> 12:37:28 > >>>>>>> 2016 > >>>>>>>> +0100 > >>>>>>>> > >>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 > >>>> 16:52:17 > >>>>>>> 2016 > >>>>>>>> +0100 > >>>>>>>> > >>>>>>>> @@ -701,7 +701,7 @@ > >>>>>>>> > >>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > >>>>>>>> > >>>>>>>> // Creating guard page is very expensive. Java thread has > HotSpot > >>>>>>>> > >>>>>>>> // guard page, only enable glibc guard page for non-Java > threads. > >>>>>>>> > >>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); > >>>>>>>> > >>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); > >>>>>>>> > >>>>>>>> } > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> // Java thread: > >>>>>>>> > > From thomas.stuefe at gmail.com Wed Nov 23 07:29:34 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 23 Nov 2016 08:29:34 +0100 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Message-ID: Hi David, On Wed, Nov 23, 2016 at 3:08 AM, David Holmes wrote: > Hi Thomas, > > David did not like that, he thought it strange to check an enum for >> correct enum values. I can see his point. He would have preferred >> instead an assert inside NMTUtil::flag_to_index() which before >> returning checks the to-be-returned index for >> 0>=i>=mt_number_of_types. I did not like that because strictly >> speaking inside of NMTUtil::flag_to_index() it is not known that the >> returned integer will be used by the caller as an index into an array >> of mt_number_of_types length. >> > > Sorry but how is NMTUtil::flag_to_index anything but a mapping from 0 to > mt_number_of_types ??? MEMFLAGS aka MemoryType is an enum that spans zero > to mt_number_of_types! We're validating the post condition for the method. > Sorry, this may be clear for you but was not clear for me. There is no comment at NMTUtil::flag_to_index explaining the range of the output value, and the callers are spread over multiple files. The fact that the original author found it necessary to add this conversion function and to mask the upper bits out suggested to me that the returned index is something different than MEMFLAGS, or may be different in the future. Otherwise he could have just used the MEMFLAGS enum itself as index. Note that I still do not understand the & 0xff part. Once MEMFLAG values surpass 0xff, what is supposed to happen? > > As I said on Oct 19, and which Chris indicated he agreed with, it makes no > sense to me to put the same assert in all the callers of flag_to_index > instead of doing the validation inside flag_to_index. I understand you. I understood Max' comment otherwise. Faced with two opinions, and as a outsider having no way to weight them against each other, I have to choose one. > Whether you choose to validate the flag or the index I don't care. Then we could have just gone with my very first webrev and I could have saved work. In your very first answer to my original webrev you wrote: "I'm a little perplexed with asserting that something of MEMFLAGS type must be an actual MEMFLAGS value". I understood this as objection. I am fine with either solution. I am also fine with someone else fixing this issue. My only preference would be to some form of assert in this code, because I spent quite some time to find the memory overwriter this caused. If we were in a decent language the bug would have been impossible to begin > with. > > David > ----- > > Thanks, Thomas > On 22/11/2016 1:21 AM, Thomas St?fe wrote: > >> Hi all, >> >> this small issue got fc extension, so lets wrap this up. Thanks to >> all for reviewing! >> >> New webrev: >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >> d-check_MEMFLAGS/webrev.02/webrev/ >> >> No changes to the last one but the updated copyrights Chris did ask >> for. Solution still follows Max' suggestion of checking the index >> right before consumption. >> >> Thanks, and Kind Regards, Thomas >> >> >> On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe >> > wrote: >> >> Hi Chris, >> >> On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer >> > wrote: >> >> On 10/18/16 11:49 PM, David Holmes wrote: >> >> >> >> On 19/10/2016 3:17 PM, Thomas St?fe wrote: >> >> >> >> On Wed, Oct 19, 2016 at 3:21 AM, David Holmes >> >> >> >> wrote: >> >> On 18/10/2016 3:39 PM, Thomas St?fe wrote: >> >> Hi David, Max, >> >> I changed the asserts according to Max' suggestion. Instead of >> checking inside flag_to_index, now I check before callers of this >> function use this value to access memory. >> >> >> I don't see where Max suggested that?? >> >> >> Max wrote: " I think the decision on whether to access a slot should >> occur as close to memory accessing code as possible." and proceeded >> to suggest fixing VirtualMemorySnapshot::by_type() as well. >> >> >> I did not interpret that comment that way, and was puzzled by the >> reference to by_type. >> >> >> It doesn't make sense to me to have all the callers of flag_to_index >> check what it returned instead of doing it inside flag_to_index. >> >> >> I disagree. Imho it makes sense to either check the Memflags >> enumeration input argument in flag_to_index() or the returned index >> before consumption. In both cases one knows the valid value range. >> Strictly speaking checking the index in flag_to_index() cannot be >> done because it is a faceless int type whose valid values are not yet >> known. >> >> >> The index has to fall in the range 0 <= index <= mt_number_of_types, >> and I was suggesting that it makes more sense to verify this once in >> flag_to_index() than in all the callers of flag_to_index. >> >> Hi Thomas, >> >> Just catching up on this thread. This is the same conclusion I came >> to. I don't understand what you mean by "it is a faceless int type >> whose valid values are not yet known". >> >> >> Thank you for looking into this! >> >> "it is a faceless int type whose valid values are not yet known" was >> maybe expressed sloppily: >> >> My first patch checked the input enum "flag" argument inside >> NMTUtil::flag_to_index() for correct enum values : >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >> d-check_MEMFLAGS/webrev.00/webrev/ >> >> >> ld-check_MEMFLAGS/webrev.00/webrev/> > >> >> David did not like that, he thought it strange to check an enum for >> correct enum values. I can see his point. He would have preferred >> instead an assert inside NMTUtil::flag_to_index() which before >> returning checks the to-be-returned index for >> 0>=i>=mt_number_of_types. I did not like that because strictly >> speaking inside of NMTUtil::flag_to_index() it is not known that the >> returned integer will be used by the caller as an index into an array >> of mt_number_of_types length. >> >> It is all extreme nitpicking and in the end amounts to the same :) >> >> The current patch does neither, but follows Max' suggestion of >> checking the index right before it is consumed to access an array. I >> think this is a good solution and very clear. >> >> >> >> BTW, I'll sponsor this fix for you once it is finalized. Please >> update the copyright dates first. >> >> >> Thank you. I'll update the webrev and repost. >> >> >> >> thanks, >> >> Chris >> >> >> Kind Regards, Thomas >> >> >> >> David >> >> It is all academical and mostly a matter of taste. >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >> d-check_MEMFLAGS/webrev.01/webrev/index.html >> >> >> ld-check_MEMFLAGS/webrev.01/webrev/index.html> > >> > ld-check_MEMFLAGS/webrev.01/webrev/index.html >> >> >> ld-check_MEMFLAGS/webrev.01/webrev/index.html>> > >> >> As David correctly writes, this is technically not a bug, so I guess >> this will have to wait until java 10. >> >> >> Yes, afraid so. >> >> >> The fix is trivial and I will try to get fc extension for this (now >> that Goetz explained to me how to do this :). It seems this is done >> for many other non-bug issues as well. >> >> ..Thomas >> >> >> Thanks, David >> >> Kind Regards, Thomas >> >> >> >> On Fri, Oct 14, 2016 at 12:57 AM, David Holmes >> >> > >> >> >>> >> wrote: >> >> On 13/10/2016 10:53 PM, Thomas St?fe wrote: >> >> Hi David, >> >> On Thu, Oct 13, 2016 at 12:08 PM, David Holmes >> >> > >> >> >> >> >> > >> >> >> >>>> >> >> wrote: >> >> Hi Thomas, >> >> On 13/10/2016 3:49 PM, Thomas St?fe wrote: >> >> Hi all, >> >> may I have plase a review for this tiny change? It just adds some >> assert to NMT. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8167650 >> >> >> > > >> > >> > >> >> >> >> > >> >> > > >> > >> > >>> webrev: >> >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ >> >> >> >> >> > >> >> >> > >> >> > >> > >> >> >> > >> >> >> > >> >> > >> > >> >> > > >> >> > >> >> > >> > >> >> >>> > >> MEMFLAGS/webrev.00/webrev/ >> >> We had an ugly memory overwrite caused by this - ultimately our >> fault, because we fed an invalid memory flag to NMT - but it was >> difficult to find. An assert would have saved some time. >> >> >> I'm a little perplexed with asserting that something of MEMFLAGS type >> must be an actual MEMFLAGS value - it implies the caller is coercing >> plain int to MEMFLAGS, and I don't have much sympathy if they mess >> that up. Can't help wondering if there is some clever C++ trick to >> flag bad conversions at compile-time? >> >> >> The error was caused by an uninitialized variable of type MEMFLAGS. >> This was our fault, we have heavily modified allocation.hpp and >> introduced an error then merging changes from upstream. Due to a >> merging error this lead to a case where Arena::_flags was not >> initialized and contained a very large value. >> >> >> Ah I see. Lack of default initialization can be annoying :) >> >> I admit it looks funny. If it bothers you, I could instead check the >> returned index to be in the range for the size of the _malloc array >> in MallocMemorySnapshot::by_type(). Technically, it would mean the >> same. >> >> >> So I just realized that here: >> >> 62 // Map memory type to human readable name 63 static const >> char* flag_to_name(MEMFLAGS flag) { 64 assert(flag >= 0 && flag >> < mt_number_of_types, "Invalid flag value %d.", (int)flag); 65 >> return _memory_type_names[flag_to_index(flag)]; 66 } >> >> we call flag_to_index, so the assert is redundant as it is already in >> flag_to_index. Then presumably we change flag_to_index to something >> like this: >> >> static inline int flag_to_index(MEMFLAGS flag) { int index = (flag & >> 0xff); assert(index >= 0 && index < mt_number_of_types, "Invalid flag >> value %d.", (int)flag); return index; } >> >> so we're validating the index rather than the flag. >> >> Cheers, David >> >> >> >> >> The function that takes the index should validate the index, so that >> is fine. >> >> Which one were you actually passing the bad value to? :) >> >> This isn't a strong objection just musing if we can do better. And as >> the hs repos are still closed, and likely to remain so till early >> next week, we have some slack time :) >> >> >> :) Sure. >> >> Kind Regards, Thomas >> >> >> Cheers, David >> >> Thank you! >> >> Thomas >> >> >> >> >> >> >> >> >> From dmitry.fazunenko at oracle.com Wed Nov 23 08:25:01 2016 From: dmitry.fazunenko at oracle.com (Dmitry Fazunenenko) Date: Wed, 23 Nov 2016 11:25:01 +0300 Subject: RFR(S): 8168492: Convert CollectedHeap_test to GTest In-Reply-To: <6423992a-b649-4605-c8ab-432a53ca827c@oracle.com> References: <489f77cb-f7be-51f5-b602-9e2eab3ee807@oracle.com> <721A3D9B-36D2-41F1-B965-E46CED8F8FA8@oracle.com> <1479822060.2435.33.camel@oracle.com> <6423992a-b649-4605-c8ab-432a53ca827c@oracle.com> Message-ID: <61e61b6f-bc77-6a0d-cd66-eda9f61d81be@oracle.com> Hi Kirill, the fix looks to me. Thanks, Dima On 22.11.2016 18:01, Kirill Zhaldybin wrote: > Thomas, > > Thank you for review! > > I will change the formatting before pushing the fix. > > Regards, Kirill > > On 22.11.2016 16:41, Thomas Schatzl wrote: >> Hi Kirill, >> >> in case you need another review... >> >>> On Nov 1, 2016, at 5:10 PM, Kirill Zhaldybin >>> wrote: >>>> Dear all, >>>> >>>> Could you please review this fix for 8168492? >>>> >>>> WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168492/we >>>> brev.00/ >>>> CR: https://bugs.openjdk.java.net/browse/JDK-8168492 >> In line >> >> 47 ASSERT_LE(heap_end, ((uintptr_t) - 1 - epsilon)) >> >> the "-" in the "- 1" expression is an unary minus, not the regular >> minus operator, so the space between it and the "1" should be removed. >> >> Further the space between the uintptr_t cast and the unary minus seems >> not appropriate. >> >> I.e. the expression should look like "((uintptr_t)-1 - epsilon)" like >> in the original code. >> >> I do not need a re-review for this. >> >> Thanks, >> Thomas >> > From kirill.zhaldybin at oracle.com Wed Nov 23 10:20:51 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Wed, 23 Nov 2016 13:20:51 +0300 Subject: RFR(S): 8168492: Convert CollectedHeap_test to GTest In-Reply-To: <61e61b6f-bc77-6a0d-cd66-eda9f61d81be@oracle.com> References: <489f77cb-f7be-51f5-b602-9e2eab3ee807@oracle.com> <721A3D9B-36D2-41F1-B965-E46CED8F8FA8@oracle.com> <1479822060.2435.33.camel@oracle.com> <6423992a-b649-4605-c8ab-432a53ca827c@oracle.com> <61e61b6f-bc77-6a0d-cd66-eda9f61d81be@oracle.com> Message-ID: <58356D83.1010801@oracle.com> Dmitry, Thank you for review! Regards, Kirill On 23.11.2016 11:25, Dmitry Fazunenenko wrote: > Hi Kirill, > > the fix looks to me. > > Thanks, > Dima > > On 22.11.2016 18:01, Kirill Zhaldybin wrote: >> Thomas, >> >> Thank you for review! >> >> I will change the formatting before pushing the fix. >> >> Regards, Kirill >> >> On 22.11.2016 16:41, Thomas Schatzl wrote: >>> Hi Kirill, >>> >>> in case you need another review... >>> >>>> On Nov 1, 2016, at 5:10 PM, Kirill Zhaldybin >>>> wrote: >>>>> Dear all, >>>>> >>>>> Could you please review this fix for 8168492? >>>>> >>>>> WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8168492/we >>>>> brev.00/ >>>>> CR: https://bugs.openjdk.java.net/browse/JDK-8168492 >>> In line >>> >>> 47 ASSERT_LE(heap_end, ((uintptr_t) - 1 - epsilon)) >>> >>> the "-" in the "- 1" expression is an unary minus, not the regular >>> minus operator, so the space between it and the "1" should be removed. >>> >>> Further the space between the uintptr_t cast and the unary minus seems >>> not appropriate. >>> >>> I.e. the expression should look like "((uintptr_t)-1 - epsilon)" like >>> in the original code. >>> >>> I do not need a re-review for this. >>> >>> Thanks, >>> Thomas >>> >> > From david.holmes at oracle.com Wed Nov 23 11:20:47 2016 From: david.holmes at oracle.com (David Holmes) Date: Wed, 23 Nov 2016 21:20:47 +1000 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Message-ID: Hi Thomas, On 23/11/2016 5:29 PM, Thomas St?fe wrote: > Hi David, > > On Wed, Nov 23, 2016 at 3:08 AM, David Holmes > wrote: > > Hi Thomas, > > David did not like that, he thought it strange to check an enum for > correct enum values. I can see his point. He would have preferred > instead an assert inside NMTUtil::flag_to_index() which before > returning checks the to-be-returned index for > 0>=i>=mt_number_of_types. I did not like that because strictly > speaking inside of NMTUtil::flag_to_index() it is not known that the > returned integer will be used by the caller as an index into an > array > of mt_number_of_types length. > > > Sorry but how is NMTUtil::flag_to_index anything but a mapping from > 0 to mt_number_of_types ??? MEMFLAGS aka MemoryType is an enum that > spans zero to mt_number_of_types! We're validating the post > condition for the method. > > > Sorry, this may be clear for you but was not clear for me. There is no > comment at NMTUtil::flag_to_index explaining the range of the output > value, and the callers are spread over multiple files. The fact that the > original author found it necessary to add this conversion function and > to mask the upper bits out suggested to me that the returned index is > something different than MEMFLAGS, or may be different in the future. > Otherwise he could have just used the MEMFLAGS enum itself as index. Can't answer that - perhaps just for cleanliness of the type system? Perhaps he thought the enum was strongly-typed? Or perhaps it was to hide the actual mapping function to allow for a different scheme. If anything the & 0xff suggests that the input might be something other than a MEMFLAGS value - despite it being typed as MEMFLAGS - that needs to be brought back into range. > Note that I still do not understand the & 0xff part. Once MEMFLAG values > surpass 0xff, what is supposed to happen? I don't think there is any expectation that we will ever go that high: enum MemoryType { // Memory type by sub systems. It occupies lower byte. though why this was restricted to be a byte-size value I can't say. I also don't know why we then typedef MemoryType to MEMFLAGS. > As I said on Oct 19, and which Chris indicated he agreed with, it > makes no sense to me to put the same assert in all the callers of > flag_to_index instead of doing the validation inside flag_to_index. > > > I understand you. I understood Max' comment otherwise. Faced with two > opinions, and as a outsider having no way to weight them against each > other, I have to choose one. > Fair enough, but Chris made it 2-1. :) > Whether you choose to validate the flag or the index I don't care. > > > Then we could have just gone with my very first webrev and I could have > saved work. In your very first answer to my original webrev you wrote: > "I'm a little perplexed with asserting that something of MEMFLAGS type > must be an actual MEMFLAGS value". I understood this as objection. Yes but I've now reached a point where I'd accept that if it is the only way to have the assertion inside the function. When I made that objection I did not expect this to take the path it did. Sorry. > I am fine with either solution. I am also fine with someone else fixing > this issue. My only preference would be to some form of assert in this > code, because I spent quite some time to find the memory overwriter this > caused. My preference, obviously, is an assert inside flag_to_index, on the value of index to be returned. I did not expect this to become so prolonged, and I'm sorry about that, but I really object to the callers of the function validating its output. Thanks, David > If we were in a decent language the bug would have been impossible > to begin with. > > David > ----- > > > Thanks, Thomas > > > > On 22/11/2016 1:21 AM, Thomas St?fe wrote: > > Hi all, > > this small issue got fc extension, so lets wrap this up. Thanks to > all for reviewing! > > New webrev: > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ > > > No changes to the last one but the updated copyrights Chris did ask > for. Solution still follows Max' suggestion of checking the index > right before consumption. > > Thanks, and Kind Regards, Thomas > > > On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe > > >> wrote: > > Hi Chris, > > On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer > > >> wrote: > > On 10/18/16 11:49 PM, David Holmes wrote: > > > > On 19/10/2016 3:17 PM, Thomas St?fe wrote: > > > > On Wed, Oct 19, 2016 at 3:21 AM, David Holmes > > > > > >>> > wrote: > > On 18/10/2016 3:39 PM, Thomas St?fe wrote: > > Hi David, Max, > > I changed the asserts according to Max' suggestion. Instead of > checking inside flag_to_index, now I check before callers of this > function use this value to access memory. > > > I don't see where Max suggested that?? > > > Max wrote: " I think the decision on whether to access a slot should > occur as close to memory accessing code as possible." and proceeded > to suggest fixing VirtualMemorySnapshot::by_type() as well. > > > I did not interpret that comment that way, and was puzzled by the > reference to by_type. > > > It doesn't make sense to me to have all the callers of flag_to_index > check what it returned instead of doing it inside flag_to_index. > > > I disagree. Imho it makes sense to either check the Memflags > enumeration input argument in flag_to_index() or the returned index > before consumption. In both cases one knows the valid value range. > Strictly speaking checking the index in flag_to_index() cannot be > done because it is a faceless int type whose valid values are > not yet > known. > > > The index has to fall in the range 0 <= index <= mt_number_of_types, > and I was suggesting that it makes more sense to verify this once in > flag_to_index() than in all the callers of flag_to_index. > > Hi Thomas, > > Just catching up on this thread. This is the same conclusion I came > to. I don't understand what you mean by "it is a faceless int type > whose valid values are not yet known". > > > Thank you for looking into this! > > "it is a faceless int type whose valid values are not yet known" was > maybe expressed sloppily: > > My first patch checked the input enum "flag" argument inside > NMTUtil::flag_to_index() for correct enum values : > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > > > > > > > David did not like that, he thought it strange to check an enum for > correct enum values. I can see his point. He would have preferred > instead an assert inside NMTUtil::flag_to_index() which before > returning checks the to-be-returned index for > 0>=i>=mt_number_of_types. I did not like that because strictly > speaking inside of NMTUtil::flag_to_index() it is not known that the > returned integer will be used by the caller as an index into an > array > of mt_number_of_types length. > > It is all extreme nitpicking and in the end amounts to the same :) > > The current patch does neither, but follows Max' suggestion of > checking the index right before it is consumed to access an array. I > think this is a good solution and very clear. > > > > BTW, I'll sponsor this fix for you once it is finalized. Please > update the copyright dates first. > > > Thank you. I'll update the webrev and repost. > > > > thanks, > > Chris > > > Kind Regards, Thomas > > > > David > > It is all academical and mostly a matter of taste. > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > > > > > > > > > >> > > > As David correctly writes, this is technically not a bug, so I guess > this will have to wait until java 10. > > > Yes, afraid so. > > > The fix is trivial and I will try to get fc extension for this (now > that Goetz explained to me how to do this :). It seems this is done > for many other non-bug issues as well. > > ..Thomas > > > Thanks, David > > Kind Regards, Thomas > > > > On Fri, Oct 14, 2016 at 12:57 AM, David Holmes > > > > > >> > > > > > >>>> > wrote: > > On 13/10/2016 10:53 PM, Thomas St?fe wrote: > > Hi David, > > On Thu, Oct 13, 2016 at 12:08 PM, David Holmes > > > > > >> > > > > > >>> > > > > > >> > > > > > > >>>>> > > wrote: > > Hi Thomas, > > On 13/10/2016 3:49 PM, Thomas St?fe wrote: > > Hi all, > > may I have plase a review for this tiny change? It just adds some > assert to NMT. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8167650 > > > > > > >> > > > > > >>> > > > > > > > > >> > > > > > >>>> webrev: > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > > > > > > > >> > > > > > > > > > > > > >>> > > > > > > > > > > > > > >> > > > > > > > > > > > > >>>> > > MEMFLAGS/webrev.00/webrev/ > > We had an ugly memory overwrite caused by this - ultimately our > fault, because we fed an invalid memory flag to NMT - but it was > difficult to find. An assert would have saved some time. > > > I'm a little perplexed with asserting that something of MEMFLAGS > type > must be an actual MEMFLAGS value - it implies the caller is coercing > plain int to MEMFLAGS, and I don't have much sympathy if they mess > that up. Can't help wondering if there is some clever C++ trick to > flag bad conversions at compile-time? > > > The error was caused by an uninitialized variable of type MEMFLAGS. > This was our fault, we have heavily modified allocation.hpp and > introduced an error then merging changes from upstream. Due to a > merging error this lead to a case where Arena::_flags was not > initialized and contained a very large value. > > > Ah I see. Lack of default initialization can be annoying :) > > I admit it looks funny. If it bothers you, I could instead check the > returned index to be in the range for the size of the _malloc array > in MallocMemorySnapshot::by_type(). Technically, it would mean the > same. > > > So I just realized that here: > > 62 // Map memory type to human readable name 63 static const > char* flag_to_name(MEMFLAGS flag) { 64 assert(flag >= 0 && flag > < mt_number_of_types, "Invalid flag value %d.", (int)flag); 65 > return _memory_type_names[flag_to_index(flag)]; 66 } > > we call flag_to_index, so the assert is redundant as it is > already in > flag_to_index. Then presumably we change flag_to_index to something > like this: > > static inline int flag_to_index(MEMFLAGS flag) { int index = (flag & > 0xff); assert(index >= 0 && index < mt_number_of_types, "Invalid > flag > value %d.", (int)flag); return index; } > > so we're validating the index rather than the flag. > > Cheers, David > > > > > The function that takes the index should validate the index, so that > is fine. > > Which one were you actually passing the bad value to? :) > > This isn't a strong objection just musing if we can do better. > And as > the hs repos are still closed, and likely to remain so till early > next week, we have some slack time :) > > > :) Sure. > > Kind Regards, Thomas > > > Cheers, David > > Thank you! > > Thomas > > > > > > > > > From thomas.stuefe at gmail.com Wed Nov 23 13:47:46 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 23 Nov 2016 14:47:46 +0100 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Message-ID: Hi David, On Wed, Nov 23, 2016 at 12:20 PM, David Holmes wrote: > Hi Thomas, > > On 23/11/2016 5:29 PM, Thomas St?fe wrote: > >> Hi David, >> > ... > My preference, obviously, is an assert inside flag_to_index, on the value > of index to be returned. > > My latest webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.03/webrev/ I did it the way you preferred, by checking the return value of flag_to_index(). > I did not expect this to become so prolonged, and I'm sorry about that, > but I really object to the callers of the function validating its output. > > No problem. I'm happy to have this issue closed. Kind Regards, Thomas > Thanks, > David > > If we were in a decent language the bug would have been impossible >> to begin with. >> >> David >> ----- >> >> >> Thanks, Thomas >> >> >> >> On 22/11/2016 1:21 AM, Thomas St?fe wrote: >> >> Hi all, >> >> this small issue got fc extension, so lets wrap this up. Thanks to >> all for reviewing! >> >> New webrev: >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >> d-check_MEMFLAGS/webrev.02/webrev/ >> > ld-check_MEMFLAGS/webrev.02/webrev/> >> >> No changes to the last one but the updated copyrights Chris did >> ask >> for. Solution still follows Max' suggestion of checking the index >> right before consumption. >> >> Thanks, and Kind Regards, Thomas >> >> >> On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe >> >> > >> wrote: >> >> Hi Chris, >> >> On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer >> >> > >> wrote: >> >> On 10/18/16 11:49 PM, David Holmes wrote: >> >> >> >> On 19/10/2016 3:17 PM, Thomas St?fe wrote: >> >> >> >> On Wed, Oct 19, 2016 at 3:21 AM, David Holmes >> >> > >> >> > >>>> >> wrote: >> >> On 18/10/2016 3:39 PM, Thomas St?fe wrote: >> >> Hi David, Max, >> >> I changed the asserts according to Max' suggestion. Instead of >> checking inside flag_to_index, now I check before callers of this >> function use this value to access memory. >> >> >> I don't see where Max suggested that?? >> >> >> Max wrote: " I think the decision on whether to access a slot >> should >> occur as close to memory accessing code as possible." and >> proceeded >> to suggest fixing VirtualMemorySnapshot::by_type() as well. >> >> >> I did not interpret that comment that way, and was puzzled by the >> reference to by_type. >> >> >> It doesn't make sense to me to have all the callers of >> flag_to_index >> check what it returned instead of doing it inside flag_to_index. >> >> >> I disagree. Imho it makes sense to either check the Memflags >> enumeration input argument in flag_to_index() or the returned >> index >> before consumption. In both cases one knows the valid value range. >> Strictly speaking checking the index in flag_to_index() cannot be >> done because it is a faceless int type whose valid values are >> not yet >> known. >> >> >> The index has to fall in the range 0 <= index <= >> mt_number_of_types, >> and I was suggesting that it makes more sense to verify this once >> in >> flag_to_index() than in all the callers of flag_to_index. >> >> Hi Thomas, >> >> Just catching up on this thread. This is the same conclusion I >> came >> to. I don't understand what you mean by "it is a faceless int type >> whose valid values are not yet known". >> >> >> Thank you for looking into this! >> >> "it is a faceless int type whose valid values are not yet known" >> was >> maybe expressed sloppily: >> >> My first patch checked the input enum "flag" argument inside >> NMTUtil::flag_to_index() for correct enum values : >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >> d-check_MEMFLAGS/webrev.00/webrev/ >> > ld-check_MEMFLAGS/webrev.00/webrev/> >> >> >> > ld-check_MEMFLAGS/webrev.00/webrev/ >> > ld-check_MEMFLAGS/webrev.00/webrev/>> >> >> >> David did not like that, he thought it strange to check an enum >> for >> correct enum values. I can see his point. He would have preferred >> instead an assert inside NMTUtil::flag_to_index() which before >> returning checks the to-be-returned index for >> 0>=i>=mt_number_of_types. I did not like that because strictly >> speaking inside of NMTUtil::flag_to_index() it is not known that >> the >> returned integer will be used by the caller as an index into an >> array >> of mt_number_of_types length. >> >> It is all extreme nitpicking and in the end amounts to the same :) >> >> The current patch does neither, but follows Max' suggestion of >> checking the index right before it is consumed to access an >> array. I >> think this is a good solution and very clear. >> >> >> >> BTW, I'll sponsor this fix for you once it is finalized. Please >> update the copyright dates first. >> >> >> Thank you. I'll update the webrev and repost. >> >> >> >> thanks, >> >> Chris >> >> >> Kind Regards, Thomas >> >> >> >> David >> >> It is all academical and mostly a matter of taste. >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >> d-check_MEMFLAGS/webrev.01/webrev/index.html >> > ld-check_MEMFLAGS/webrev.01/webrev/index.html> >> >> >> > ld-check_MEMFLAGS/webrev.01/webrev/index.html >> > ld-check_MEMFLAGS/webrev.01/webrev/index.html>> >> >> > ld-check_MEMFLAGS/webrev.01/webrev/index.html >> > ld-check_MEMFLAGS/webrev.01/webrev/index.html> >> >> >> > ld-check_MEMFLAGS/webrev.01/webrev/index.html >> > ld-check_MEMFLAGS/webrev.01/webrev/index.html>>> >> >> >> As David correctly writes, this is technically not a bug, so I >> guess >> this will have to wait until java 10. >> >> >> Yes, afraid so. >> >> >> The fix is trivial and I will try to get fc extension for this >> (now >> that Goetz explained to me how to do this :). It seems this is >> done >> for many other non-bug issues as well. >> >> ..Thomas >> >> >> Thanks, David >> >> Kind Regards, Thomas >> >> >> >> On Fri, Oct 14, 2016 at 12:57 AM, David Holmes >> >> > >> >> > >>> >> >> > >> >> > >>>>> >> wrote: >> >> On 13/10/2016 10:53 PM, Thomas St?fe wrote: >> >> Hi David, >> >> On Thu, Oct 13, 2016 at 12:08 PM, David Holmes >> >> > >> >> > >>> >> >> > >> >> > >>>> >> >> > >> >> > >>> >> >> >> > >> >> > >>>>> >> >> wrote: >> >> Hi Thomas, >> >> On 13/10/2016 3:49 PM, Thomas St?fe wrote: >> >> Hi all, >> >> may I have plase a review for this tiny change? It just adds some >> assert to NMT. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8167650 >> >> > > >> >> > >> > >> >> > >> > > >> > >> > >>> >> >> >> > >> > > >> >> > >> > >> >> > >> > > >> > >> > >>>> webrev: >> >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >> d-check_ >> > ld-check_> >> > ld-check_ >> > ld-check_>> >> >> >> > >> >> > ld-check_ >> > ld-check_>>> >> >> >> > ld-check_ >> > ld-check_> >> >> >> > > >> >> > ld-check_ >> > ld-check_> >> >> >> > >>> >> >> >> >> > ld-check_ >> > ld-check_> >> >> >> > > >> >> > ld-check_ >> > ld-check_> >> >> >> > >> >> >> >> > ld-check_ >> > ld-check_> >> >> >> > > >> >> > ld-check_ >> > ld-check_> >> >> >> > > >>>>> >> >> MEMFLAGS/webrev.00/webrev/ >> >> We had an ugly memory overwrite caused by this - ultimately our >> fault, because we fed an invalid memory flag to NMT - but it was >> difficult to find. An assert would have saved some time. >> >> >> I'm a little perplexed with asserting that something of MEMFLAGS >> type >> must be an actual MEMFLAGS value - it implies the caller is >> coercing >> plain int to MEMFLAGS, and I don't have much sympathy if they mess >> that up. Can't help wondering if there is some clever C++ trick to >> flag bad conversions at compile-time? >> >> >> The error was caused by an uninitialized variable of type >> MEMFLAGS. >> This was our fault, we have heavily modified allocation.hpp and >> introduced an error then merging changes from upstream. Due to a >> merging error this lead to a case where Arena::_flags was not >> initialized and contained a very large value. >> >> >> Ah I see. Lack of default initialization can be annoying :) >> >> I admit it looks funny. If it bothers you, I could instead check >> the >> returned index to be in the range for the size of the _malloc >> array >> in MallocMemorySnapshot::by_type(). Technically, it would mean >> the >> same. >> >> >> So I just realized that here: >> >> 62 // Map memory type to human readable name 63 static const >> char* flag_to_name(MEMFLAGS flag) { 64 assert(flag >= 0 && >> flag >> < mt_number_of_types, "Invalid flag value %d.", (int)flag); 65 >> return _memory_type_names[flag_to_index(flag)]; 66 } >> >> we call flag_to_index, so the assert is redundant as it is >> already in >> flag_to_index. Then presumably we change flag_to_index to >> something >> like this: >> >> static inline int flag_to_index(MEMFLAGS flag) { int index = >> (flag & >> 0xff); assert(index >= 0 && index < mt_number_of_types, "Invalid >> flag >> value %d.", (int)flag); return index; } >> >> so we're validating the index rather than the flag. >> >> Cheers, David >> >> >> >> >> The function that takes the index should validate the index, so >> that >> is fine. >> >> Which one were you actually passing the bad value to? :) >> >> This isn't a strong objection just musing if we can do better. >> And as >> the hs repos are still closed, and likely to remain so till early >> next week, we have some slack time :) >> >> >> :) Sure. >> >> Kind Regards, Thomas >> >> >> Cheers, David >> >> Thank you! >> >> Thomas >> >> >> >> >> >> >> >> >> >> From zgu at redhat.com Wed Nov 23 13:58:48 2016 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 23 Nov 2016 08:58:48 -0500 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Message-ID: Sorry for participating late, I did not expect to be a lengthy discussion. On 11/23/2016 06:20 AM, David Holmes wrote: > Hi Thomas, > > On 23/11/2016 5:29 PM, Thomas St?fe wrote: >> Hi David, >> >> On Wed, Nov 23, 2016 at 3:08 AM, David Holmes > > wrote: >> >> Hi Thomas, >> >> David did not like that, he thought it strange to check an >> enum for >> correct enum values. I can see his point. He would have >> preferred >> instead an assert inside NMTUtil::flag_to_index() which before >> returning checks the to-be-returned index for >> 0>=i>=mt_number_of_types. I did not like that because strictly >> speaking inside of NMTUtil::flag_to_index() it is not known >> that the >> returned integer will be used by the caller as an index into an >> array >> of mt_number_of_types length. >> >> >> Sorry but how is NMTUtil::flag_to_index anything but a mapping from >> 0 to mt_number_of_types ??? MEMFLAGS aka MemoryType is an enum that >> spans zero to mt_number_of_types! We're validating the post >> condition for the method. >> >> >> Sorry, this may be clear for you but was not clear for me. There is no >> comment at NMTUtil::flag_to_index explaining the range of the output >> value, and the callers are spread over multiple files. The fact that the >> original author found it necessary to add this conversion function and >> to mask the upper bits out suggested to me that the returned index is >> something different than MEMFLAGS, or may be different in the future. >> Otherwise he could have just used the MEMFLAGS enum itself as index. > > Can't answer that - perhaps just for cleanliness of the type system? > Perhaps he thought the enum was strongly-typed? Or perhaps it was to > hide the actual mapping function to allow for a different scheme. Yes, I was assuming the type system should ensure correct MEMFLAGS value here. > > If anything the & 0xff suggests that the input might be something > other than a MEMFLAGS value - despite it being typed as MEMFLAGS - > that needs to be brought back into range. > >> Note that I still do not understand the & 0xff part. Once MEMFLAG values >> surpass 0xff, what is supposed to happen? > > I don't think there is any expectation that we will ever go that high: > > enum MemoryType { > // Memory type by sub systems. It occupies lower byte. > > though why this was restricted to be a byte-size value I can't say. I > also don't know why we then typedef MemoryType to MEMFLAGS. NMTUtil::flag_to_index() converts flag to byte size for encoding the value into malloc header (services/mallocTracker.hpp). At the time of implementation, we don't have some types, such as u1, etc. maybe change the return type to u1 (maybe refactor MallocHeader) to make the code more explicit. > >> As I said on Oct 19, and which Chris indicated he agreed with, it >> makes no sense to me to put the same assert in all the callers of >> flag_to_index instead of doing the validation inside flag_to_index. >> >> >> I understand you. I understood Max' comment otherwise. Faced with two >> opinions, and as a outsider having no way to weight them against each >> other, I have to choose one. >> > > Fair enough, but Chris made it 2-1. :) I am also with David and Chris. mt_number_of_types just for working around imperfect of enum type system, I would rather it not to leak everywhere. Thanks, -Zhengyu > >> Whether you choose to validate the flag or the index I don't care. >> >> >> Then we could have just gone with my very first webrev and I could have >> saved work. In your very first answer to my original webrev you wrote: >> "I'm a little perplexed with asserting that something of MEMFLAGS type >> must be an actual MEMFLAGS value". I understood this as objection. > > Yes but I've now reached a point where I'd accept that if it is the > only way to have the assertion inside the function. When I made that > objection I did not expect this to take the path it did. Sorry. > >> I am fine with either solution. I am also fine with someone else fixing >> this issue. My only preference would be to some form of assert in this >> code, because I spent quite some time to find the memory overwriter this >> caused. > > My preference, obviously, is an assert inside flag_to_index, on the > value of index to be returned. > > I did not expect this to become so prolonged, and I'm sorry about > that, but I really object to the callers of the function validating > its output. > > Thanks, > David > >> If we were in a decent language the bug would have been impossible >> to begin with. >> >> David >> ----- >> >> >> Thanks, Thomas >> >> >> >> On 22/11/2016 1:21 AM, Thomas St?fe wrote: >> >> Hi all, >> >> this small issue got fc extension, so lets wrap this up. >> Thanks to >> all for reviewing! >> >> New webrev: >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ >> >> >> No changes to the last one but the updated copyrights Chris >> did ask >> for. Solution still follows Max' suggestion of checking the >> index >> right before consumption. >> >> Thanks, and Kind Regards, Thomas >> >> >> On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe >> >> > >> wrote: >> >> Hi Chris, >> >> On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer >> >> > >> wrote: >> >> On 10/18/16 11:49 PM, David Holmes wrote: >> >> >> >> On 19/10/2016 3:17 PM, Thomas St?fe wrote: >> >> >> >> On Wed, Oct 19, 2016 at 3:21 AM, David Holmes >> >> > > >> >> > >>> >> wrote: >> >> On 18/10/2016 3:39 PM, Thomas St?fe wrote: >> >> Hi David, Max, >> >> I changed the asserts according to Max' suggestion. Instead of >> checking inside flag_to_index, now I check before callers of >> this >> function use this value to access memory. >> >> >> I don't see where Max suggested that?? >> >> >> Max wrote: " I think the decision on whether to access a slot >> should >> occur as close to memory accessing code as possible." and >> proceeded >> to suggest fixing VirtualMemorySnapshot::by_type() as well. >> >> >> I did not interpret that comment that way, and was puzzled by >> the >> reference to by_type. >> >> >> It doesn't make sense to me to have all the callers of >> flag_to_index >> check what it returned instead of doing it inside flag_to_index. >> >> >> I disagree. Imho it makes sense to either check the Memflags >> enumeration input argument in flag_to_index() or the returned >> index >> before consumption. In both cases one knows the valid value >> range. >> Strictly speaking checking the index in flag_to_index() >> cannot be >> done because it is a faceless int type whose valid values are >> not yet >> known. >> >> >> The index has to fall in the range 0 <= index <= >> mt_number_of_types, >> and I was suggesting that it makes more sense to verify this >> once in >> flag_to_index() than in all the callers of flag_to_index. >> >> Hi Thomas, >> >> Just catching up on this thread. This is the same conclusion >> I came >> to. I don't understand what you mean by "it is a faceless int >> type >> whose valid values are not yet known". >> >> >> Thank you for looking into this! >> >> "it is a faceless int type whose valid values are not yet >> known" was >> maybe expressed sloppily: >> >> My first patch checked the input enum "flag" argument inside >> NMTUtil::flag_to_index() for correct enum values : >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ >> >> >> >> > > >> >> >> David did not like that, he thought it strange to check an >> enum for >> correct enum values. I can see his point. He would have >> preferred >> instead an assert inside NMTUtil::flag_to_index() which before >> returning checks the to-be-returned index for >> 0>=i>=mt_number_of_types. I did not like that because strictly >> speaking inside of NMTUtil::flag_to_index() it is not known >> that the >> returned integer will be used by the caller as an index into an >> array >> of mt_number_of_types length. >> >> It is all extreme nitpicking and in the end amounts to the >> same :) >> >> The current patch does neither, but follows Max' suggestion of >> checking the index right before it is consumed to access an >> array. I >> think this is a good solution and very clear. >> >> >> >> BTW, I'll sponsor this fix for you once it is finalized. Please >> update the copyright dates first. >> >> >> Thank you. I'll update the webrev and repost. >> >> >> >> thanks, >> >> Chris >> >> >> Kind Regards, Thomas >> >> >> >> David >> >> It is all academical and mostly a matter of taste. >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html >> >> >> >> > > >> >> > >> >> >> > >> >> >> >> As David correctly writes, this is technically not a bug, so >> I guess >> this will have to wait until java 10. >> >> >> Yes, afraid so. >> >> >> The fix is trivial and I will try to get fc extension for >> this (now >> that Goetz explained to me how to do this :). It seems this >> is done >> for many other non-bug issues as well. >> >> ..Thomas >> >> >> Thanks, David >> >> Kind Regards, Thomas >> >> >> >> On Fri, Oct 14, 2016 at 12:57 AM, David Holmes >> >> > > >> >> > >> >> >> > > >> >> > >>>> >> wrote: >> >> On 13/10/2016 10:53 PM, Thomas St?fe wrote: >> >> Hi David, >> >> On Thu, Oct 13, 2016 at 12:08 PM, David Holmes >> >> > > >> >> > >> >> >> > > >> >> > >>> >> >> > > >> >> > >> >> >> >> > > >> >> > >>>>> >> >> wrote: >> >> Hi Thomas, >> >> On 13/10/2016 3:49 PM, Thomas St?fe wrote: >> >> Hi all, >> >> may I have plase a review for this tiny change? It just adds >> some >> assert to NMT. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8167650 >> >> > > >> >> > >> > >> >> > >> > > >> > >> > >>> >> >> >> > >> > > >> >> > >> > >> >> > >> > > >> > >> > >>>> webrev: >> >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ >> >> > > >> >> >> > >> >> > >> >> >> >> > >> >> >> > > >> >> > >> >> >> > >>> >> >> >> >> >> > >> >> >> > > >> >> > >> >> >> > >> >> >> >> > >> >> >> > > >> >> > >> >> >> > >>>> >> >> >> MEMFLAGS/webrev.00/webrev/ >> >> We had an ugly memory overwrite caused by this - ultimately our >> fault, because we fed an invalid memory flag to NMT - but it was >> difficult to find. An assert would have saved some time. >> >> >> I'm a little perplexed with asserting that something of MEMFLAGS >> type >> must be an actual MEMFLAGS value - it implies the caller is >> coercing >> plain int to MEMFLAGS, and I don't have much sympathy if they >> mess >> that up. Can't help wondering if there is some clever C++ >> trick to >> flag bad conversions at compile-time? >> >> >> The error was caused by an uninitialized variable of type >> MEMFLAGS. >> This was our fault, we have heavily modified allocation.hpp and >> introduced an error then merging changes from upstream. Due to a >> merging error this lead to a case where Arena::_flags was not >> initialized and contained a very large value. >> >> >> Ah I see. Lack of default initialization can be annoying :) >> >> I admit it looks funny. If it bothers you, I could instead >> check the >> returned index to be in the range for the size of the _malloc >> array >> in MallocMemorySnapshot::by_type(). Technically, it would >> mean the >> same. >> >> >> So I just realized that here: >> >> 62 // Map memory type to human readable name 63 static const >> char* flag_to_name(MEMFLAGS flag) { 64 assert(flag >= 0 >> && flag >> < mt_number_of_types, "Invalid flag value %d.", (int)flag); 65 >> return _memory_type_names[flag_to_index(flag)]; 66 } >> >> we call flag_to_index, so the assert is redundant as it is >> already in >> flag_to_index. Then presumably we change flag_to_index to >> something >> like this: >> >> static inline int flag_to_index(MEMFLAGS flag) { int index = >> (flag & >> 0xff); assert(index >= 0 && index < mt_number_of_types, "Invalid >> flag >> value %d.", (int)flag); return index; } >> >> so we're validating the index rather than the flag. >> >> Cheers, David >> >> >> >> >> The function that takes the index should validate the index, >> so that >> is fine. >> >> Which one were you actually passing the bad value to? :) >> >> This isn't a strong objection just musing if we can do better. >> And as >> the hs repos are still closed, and likely to remain so till >> early >> next week, we have some slack time :) >> >> >> :) Sure. >> >> Kind Regards, Thomas >> >> >> Cheers, David >> >> Thank you! >> >> Thomas >> >> >> >> >> >> >> >> >> From thomas.stuefe at gmail.com Wed Nov 23 14:15:45 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 23 Nov 2016 15:15:45 +0100 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Message-ID: Hi Zhengyu, thanks for looking at this. See my answer to David, I did the last rev as you guys requested. See: http://cr.openjdk.java.net/~stuefe/webrevs/ 8167650-NMT-should-check_MEMFLAGS/webrev.03/webrev/ Kind Regards, Thomas On Wed, Nov 23, 2016 at 2:58 PM, Zhengyu Gu wrote: > Sorry for participating late, I did not expect to be a lengthy discussion. > > > > On 11/23/2016 06:20 AM, David Holmes wrote: > >> Hi Thomas, >> >> On 23/11/2016 5:29 PM, Thomas St?fe wrote: >> >>> Hi David, >>> >>> On Wed, Nov 23, 2016 at 3:08 AM, David Holmes >> > wrote: >>> >>> Hi Thomas, >>> >>> David did not like that, he thought it strange to check an enum >>> for >>> correct enum values. I can see his point. He would have preferred >>> instead an assert inside NMTUtil::flag_to_index() which before >>> returning checks the to-be-returned index for >>> 0>=i>=mt_number_of_types. I did not like that because strictly >>> speaking inside of NMTUtil::flag_to_index() it is not known that >>> the >>> returned integer will be used by the caller as an index into an >>> array >>> of mt_number_of_types length. >>> >>> >>> Sorry but how is NMTUtil::flag_to_index anything but a mapping from >>> 0 to mt_number_of_types ??? MEMFLAGS aka MemoryType is an enum that >>> spans zero to mt_number_of_types! We're validating the post >>> condition for the method. >>> >>> >>> Sorry, this may be clear for you but was not clear for me. There is no >>> comment at NMTUtil::flag_to_index explaining the range of the output >>> value, and the callers are spread over multiple files. The fact that the >>> original author found it necessary to add this conversion function and >>> to mask the upper bits out suggested to me that the returned index is >>> something different than MEMFLAGS, or may be different in the future. >>> Otherwise he could have just used the MEMFLAGS enum itself as index. >>> >> >> Can't answer that - perhaps just for cleanliness of the type system? >> Perhaps he thought the enum was strongly-typed? Or perhaps it was to hide >> the actual mapping function to allow for a different scheme. >> > > Yes, I was assuming the type system should ensure correct MEMFLAGS value > here. > > > >> If anything the & 0xff suggests that the input might be something other >> than a MEMFLAGS value - despite it being typed as MEMFLAGS - that needs to >> be brought back into range. >> >> Note that I still do not understand the & 0xff part. Once MEMFLAG values >>> surpass 0xff, what is supposed to happen? >>> >> >> I don't think there is any expectation that we will ever go that high: >> >> enum MemoryType { >> // Memory type by sub systems. It occupies lower byte. >> >> though why this was restricted to be a byte-size value I can't say. I >> also don't know why we then typedef MemoryType to MEMFLAGS. >> > > NMTUtil::flag_to_index() converts flag to byte size for encoding the value > into malloc header (services/mallocTracker.hpp). > > At the time of implementation, we don't have some types, such as u1, etc. > maybe change the return type to u1 (maybe refactor MallocHeader) > to make the code more explicit. > > >> As I said on Oct 19, and which Chris indicated he agreed with, it >>> makes no sense to me to put the same assert in all the callers of >>> flag_to_index instead of doing the validation inside flag_to_index. >>> >>> >>> I understand you. I understood Max' comment otherwise. Faced with two >>> opinions, and as a outsider having no way to weight them against each >>> other, I have to choose one. >>> >>> >> Fair enough, but Chris made it 2-1. :) >> > > I am also with David and Chris. mt_number_of_types just for working around > imperfect of enum type system, > I would rather it not to leak everywhere. > > Thanks, > > -Zhengyu > > > >> Whether you choose to validate the flag or the index I don't care. >>> >>> >>> Then we could have just gone with my very first webrev and I could have >>> saved work. In your very first answer to my original webrev you wrote: >>> "I'm a little perplexed with asserting that something of MEMFLAGS type >>> must be an actual MEMFLAGS value". I understood this as objection. >>> >> >> Yes but I've now reached a point where I'd accept that if it is the only >> way to have the assertion inside the function. When I made that objection I >> did not expect this to take the path it did. Sorry. >> >> I am fine with either solution. I am also fine with someone else fixing >>> this issue. My only preference would be to some form of assert in this >>> code, because I spent quite some time to find the memory overwriter this >>> caused. >>> >> >> My preference, obviously, is an assert inside flag_to_index, on the value >> of index to be returned. >> >> I did not expect this to become so prolonged, and I'm sorry about that, >> but I really object to the callers of the function validating its output. >> >> Thanks, >> David >> >> If we were in a decent language the bug would have been impossible >>> to begin with. >>> >>> David >>> ----- >>> >>> >>> Thanks, Thomas >>> >>> >>> >>> On 22/11/2016 1:21 AM, Thomas St?fe wrote: >>> >>> Hi all, >>> >>> this small issue got fc extension, so lets wrap this up. Thanks >>> to >>> all for reviewing! >>> >>> New webrev: >>> >>> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >>> d-check_MEMFLAGS/webrev.02/webrev/ >>> >> ld-check_MEMFLAGS/webrev.02/webrev/> >>> >>> No changes to the last one but the updated copyrights Chris did >>> ask >>> for. Solution still follows Max' suggestion of checking the index >>> right before consumption. >>> >>> Thanks, and Kind Regards, Thomas >>> >>> >>> On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe >>> >>> >> >> wrote: >>> >>> Hi Chris, >>> >>> On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer >>> >>> >> >> wrote: >>> >>> On 10/18/16 11:49 PM, David Holmes wrote: >>> >>> >>> >>> On 19/10/2016 3:17 PM, Thomas St?fe wrote: >>> >>> >>> >>> On Wed, Oct 19, 2016 at 3:21 AM, David Holmes >>> >>> >> >> >>> >>> >> >>>> >>> wrote: >>> >>> On 18/10/2016 3:39 PM, Thomas St?fe wrote: >>> >>> Hi David, Max, >>> >>> I changed the asserts according to Max' suggestion. Instead of >>> checking inside flag_to_index, now I check before callers of this >>> function use this value to access memory. >>> >>> >>> I don't see where Max suggested that?? >>> >>> >>> Max wrote: " I think the decision on whether to access a slot >>> should >>> occur as close to memory accessing code as possible." and >>> proceeded >>> to suggest fixing VirtualMemorySnapshot::by_type() as well. >>> >>> >>> I did not interpret that comment that way, and was puzzled by the >>> reference to by_type. >>> >>> >>> It doesn't make sense to me to have all the callers of >>> flag_to_index >>> check what it returned instead of doing it inside flag_to_index. >>> >>> >>> I disagree. Imho it makes sense to either check the Memflags >>> enumeration input argument in flag_to_index() or the returned >>> index >>> before consumption. In both cases one knows the valid value >>> range. >>> Strictly speaking checking the index in flag_to_index() cannot be >>> done because it is a faceless int type whose valid values are >>> not yet >>> known. >>> >>> >>> The index has to fall in the range 0 <= index <= >>> mt_number_of_types, >>> and I was suggesting that it makes more sense to verify this >>> once in >>> flag_to_index() than in all the callers of flag_to_index. >>> >>> Hi Thomas, >>> >>> Just catching up on this thread. This is the same conclusion I >>> came >>> to. I don't understand what you mean by "it is a faceless int >>> type >>> whose valid values are not yet known". >>> >>> >>> Thank you for looking into this! >>> >>> "it is a faceless int type whose valid values are not yet known" >>> was >>> maybe expressed sloppily: >>> >>> My first patch checked the input enum "flag" argument inside >>> NMTUtil::flag_to_index() for correct enum values : >>> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >>> d-check_MEMFLAGS/webrev.00/webrev/ >>> >> ld-check_MEMFLAGS/webrev.00/webrev/> >>> >>> >>> >> ld-check_MEMFLAGS/webrev.00/webrev/ >>> >> ld-check_MEMFLAGS/webrev.00/webrev/>> >>> >>> >>> David did not like that, he thought it strange to check an enum >>> for >>> correct enum values. I can see his point. He would have preferred >>> instead an assert inside NMTUtil::flag_to_index() which before >>> returning checks the to-be-returned index for >>> 0>=i>=mt_number_of_types. I did not like that because strictly >>> speaking inside of NMTUtil::flag_to_index() it is not known that >>> the >>> returned integer will be used by the caller as an index into an >>> array >>> of mt_number_of_types length. >>> >>> It is all extreme nitpicking and in the end amounts to the same >>> :) >>> >>> The current patch does neither, but follows Max' suggestion of >>> checking the index right before it is consumed to access an >>> array. I >>> think this is a good solution and very clear. >>> >>> >>> >>> BTW, I'll sponsor this fix for you once it is finalized. Please >>> update the copyright dates first. >>> >>> >>> Thank you. I'll update the webrev and repost. >>> >>> >>> >>> thanks, >>> >>> Chris >>> >>> >>> Kind Regards, Thomas >>> >>> >>> >>> David >>> >>> It is all academical and mostly a matter of taste. >>> >>> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-shoul >>> d-check_MEMFLAGS/webrev.01/webrev/index.html >>> >> ld-check_MEMFLAGS/webrev.01/webrev/index.html> >>> >>> >>> >> ld-check_MEMFLAGS/webrev.01/webrev/index.html >>> >> ld-check_MEMFLAGS/webrev.01/webrev/index.html>> >>> >>> >> ld-check_MEMFLAGS/webrev.01/webrev/index.html >>> >> ld-check_MEMFLAGS/webrev.01/webrev/index.html> >>> >>> >>> >> ld-check_MEMFLAGS/webrev.01/webrev/index.html >>> >> ld-check_MEMFLAGS/webrev.01/webrev/index.html>>> >>> >>> >>> As David correctly writes, this is technically not a bug, so I >>> guess >>> this will have to wait until java 10. >>> >>> >>> Yes, afraid so. >>> >>> >>> The fix is trivial and I will try to get fc extension for this >>> (now >>> that Goetz explained to me how to do this :). It seems this is >>> done >>> for many other non-bug issues as well. >>> >>> ..Thomas >>> >>> >>> Thanks, David >>> >>> Kind Regards, Thomas >>> >>> >>> >>> On Fri, Oct 14, 2016 at 12:57 AM, David Holmes >>> >>> >> >> >>> >>> >> >>> >>> >>> >> >> >>> >>> >> >>>>> >>> wrote: >>> >>> On 13/10/2016 10:53 PM, Thomas St?fe wrote: >>> >>> Hi David, >>> >>> On Thu, Oct 13, 2016 at 12:08 PM, David Holmes >>> >>> >> >> >>> >>> >> >>> >>> >>> >> >> >>> >>> >> >>>> >>> >>> >> >> >>> >>> >> >>> >>> >>> >>> >> >> >>> >>> >> >>>>> >>> >>> wrote: >>> >>> Hi Thomas, >>> >>> On 13/10/2016 3:49 PM, Thomas St?fe wrote: >>> >>> Hi all, >>> >>> may I have plase a review for this tiny change? It just adds some >>> assert to NMT. >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8167650 >>> >>> >> > >>> >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>> >>> >>> >>> >> >>> >> > >>> >>> >> >>> >> >> >>> >> >>> >> > >>> >> >>> >> >>>> webrev: >>> >>> >>> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ >>> >>> >> > >>> >>> >>> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_>> >>> >>> >> >>> >>> >>> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_>>>> >>> >>> >>> >>> >>> >> >>> >>> >>> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_>> >>> >>> >> >>> >>> >>> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_>>> >>> >>> >>> >> >>> >>> >>> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_>> >>> >>> >> >>> >>> >>> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_>>>>> >>> >>> >>> MEMFLAGS/webrev.00/webrev/ >>> >>> We had an ugly memory overwrite caused by this - ultimately our >>> fault, because we fed an invalid memory flag to NMT - but it was >>> difficult to find. An assert would have saved some time. >>> >>> >>> I'm a little perplexed with asserting that something of MEMFLAGS >>> type >>> must be an actual MEMFLAGS value - it implies the caller is >>> coercing >>> plain int to MEMFLAGS, and I don't have much sympathy if they >>> mess >>> that up. Can't help wondering if there is some clever C++ trick >>> to >>> flag bad conversions at compile-time? >>> >>> >>> The error was caused by an uninitialized variable of type >>> MEMFLAGS. >>> This was our fault, we have heavily modified allocation.hpp and >>> introduced an error then merging changes from upstream. Due to a >>> merging error this lead to a case where Arena::_flags was not >>> initialized and contained a very large value. >>> >>> >>> Ah I see. Lack of default initialization can be annoying :) >>> >>> I admit it looks funny. If it bothers you, I could instead check >>> the >>> returned index to be in the range for the size of the _malloc >>> array >>> in MallocMemorySnapshot::by_type(). Technically, it would mean >>> the >>> same. >>> >>> >>> So I just realized that here: >>> >>> 62 // Map memory type to human readable name 63 static const >>> char* flag_to_name(MEMFLAGS flag) { 64 assert(flag >= 0 && >>> flag >>> < mt_number_of_types, "Invalid flag value %d.", (int)flag); 65 >>> return _memory_type_names[flag_to_index(flag)]; 66 } >>> >>> we call flag_to_index, so the assert is redundant as it is >>> already in >>> flag_to_index. Then presumably we change flag_to_index to >>> something >>> like this: >>> >>> static inline int flag_to_index(MEMFLAGS flag) { int index = >>> (flag & >>> 0xff); assert(index >= 0 && index < mt_number_of_types, "Invalid >>> flag >>> value %d.", (int)flag); return index; } >>> >>> so we're validating the index rather than the flag. >>> >>> Cheers, David >>> >>> >>> >>> >>> The function that takes the index should validate the index, so >>> that >>> is fine. >>> >>> Which one were you actually passing the bad value to? :) >>> >>> This isn't a strong objection just musing if we can do better. >>> And as >>> the hs repos are still closed, and likely to remain so till early >>> next week, we have some slack time :) >>> >>> >>> :) Sure. >>> >>> Kind Regards, Thomas >>> >>> >>> Cheers, David >>> >>> Thank you! >>> >>> Thomas >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> > From zgu at redhat.com Wed Nov 23 14:40:25 2016 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 23 Nov 2016 09:40:25 -0500 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Message-ID: <9472a0ce-8af4-6238-4b57-eeaf4f5529bd@redhat.com> Looks good to me. Thanks, -Zhengyu On 11/23/2016 09:15 AM, Thomas St?fe wrote: > Hi Zhengyu, > > thanks for looking at this. See my answer to David, I did the last rev > as you guys requested. See: > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.03/webrev/ > > > Kind Regards, Thomas > > On Wed, Nov 23, 2016 at 2:58 PM, Zhengyu Gu > wrote: > > Sorry for participating late, I did not expect to be a lengthy > discussion. > > > > On 11/23/2016 06:20 AM, David Holmes wrote: > > Hi Thomas, > > On 23/11/2016 5:29 PM, Thomas St?fe wrote: > > Hi David, > > On Wed, Nov 23, 2016 at 3:08 AM, David Holmes > > >> wrote: > > Hi Thomas, > > David did not like that, he thought it strange to > check an enum for > correct enum values. I can see his point. He would > have preferred > instead an assert inside NMTUtil::flag_to_index() > which before > returning checks the to-be-returned index for > 0>=i>=mt_number_of_types. I did not like that > because strictly > speaking inside of NMTUtil::flag_to_index() it is > not known that the > returned integer will be used by the caller as an > index into an > array > of mt_number_of_types length. > > > Sorry but how is NMTUtil::flag_to_index anything but a > mapping from > 0 to mt_number_of_types ??? MEMFLAGS aka MemoryType is > an enum that > spans zero to mt_number_of_types! We're validating the > post > condition for the method. > > > Sorry, this may be clear for you but was not clear for me. > There is no > comment at NMTUtil::flag_to_index explaining the range of > the output > value, and the callers are spread over multiple files. The > fact that the > original author found it necessary to add this conversion > function and > to mask the upper bits out suggested to me that the > returned index is > something different than MEMFLAGS, or may be different in > the future. > Otherwise he could have just used the MEMFLAGS enum itself > as index. > > > Can't answer that - perhaps just for cleanliness of the type > system? Perhaps he thought the enum was strongly-typed? Or > perhaps it was to hide the actual mapping function to allow > for a different scheme. > > > Yes, I was assuming the type system should ensure correct MEMFLAGS > value here. > > > > If anything the & 0xff suggests that the input might be > something other than a MEMFLAGS value - despite it being typed > as MEMFLAGS - that needs to be brought back into range. > > Note that I still do not understand the & 0xff part. Once > MEMFLAG values > surpass 0xff, what is supposed to happen? > > > I don't think there is any expectation that we will ever go > that high: > > enum MemoryType { > // Memory type by sub systems. It occupies lower byte. > > though why this was restricted to be a byte-size value I can't > say. I also don't know why we then typedef MemoryType to MEMFLAGS. > > > NMTUtil::flag_to_index() converts flag to byte size for encoding > the value into malloc header (services/mallocTracker.hpp). > > At the time of implementation, we don't have some types, such as > u1, etc. maybe change the return type to u1 (maybe refactor > MallocHeader) > to make the code more explicit. > > > As I said on Oct 19, and which Chris indicated he > agreed with, it > makes no sense to me to put the same assert in all the > callers of > flag_to_index instead of doing the validation inside > flag_to_index. > > > I understand you. I understood Max' comment otherwise. > Faced with two > opinions, and as a outsider having no way to weight them > against each > other, I have to choose one. > > > Fair enough, but Chris made it 2-1. :) > > > I am also with David and Chris. mt_number_of_types just for > working around imperfect of enum type system, > I would rather it not to leak everywhere. > > Thanks, > > -Zhengyu > > > > Whether you choose to validate the flag or the index I > don't care. > > > Then we could have just gone with my very first webrev and > I could have > saved work. In your very first answer to my original > webrev you wrote: > "I'm a little perplexed with asserting that something of > MEMFLAGS type > must be an actual MEMFLAGS value". I understood this as > objection. > > > Yes but I've now reached a point where I'd accept that if it > is the only way to have the assertion inside the function. > When I made that objection I did not expect this to take the > path it did. Sorry. > > I am fine with either solution. I am also fine with > someone else fixing > this issue. My only preference would be to some form of > assert in this > code, because I spent quite some time to find the memory > overwriter this > caused. > > > My preference, obviously, is an assert inside flag_to_index, > on the value of index to be returned. > > I did not expect this to become so prolonged, and I'm sorry > about that, but I really object to the callers of the function > validating its output. > > Thanks, > David > > If we were in a decent language the bug would have > been impossible > to begin with. > > David > ----- > > > Thanks, Thomas > > > > On 22/11/2016 1:21 AM, Thomas St?fe wrote: > > Hi all, > > this small issue got fc extension, so lets wrap > this up. Thanks to > all for reviewing! > > New webrev: > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ > > > > > No changes to the last one but the updated > copyrights Chris did ask > for. Solution still follows Max' suggestion of > checking the index > right before consumption. > > Thanks, and Kind Regards, Thomas > > > On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe > > > > > >>> wrote: > > Hi Chris, > > On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer > > > > > >>> wrote: > > On 10/18/16 11:49 PM, David Holmes wrote: > > > > On 19/10/2016 3:17 PM, Thomas St?fe wrote: > > > > On Wed, Oct 19, 2016 at 3:21 AM, David Holmes > > > > > >> > > > > > >>>> > wrote: > > On 18/10/2016 3:39 PM, Thomas St?fe wrote: > > Hi David, Max, > > I changed the asserts according to Max' > suggestion. Instead of > checking inside flag_to_index, now I check before > callers of this > function use this value to access memory. > > > I don't see where Max suggested that?? > > > Max wrote: " I think the decision on whether to > access a slot should > occur as close to memory accessing code as > possible." and proceeded > to suggest fixing VirtualMemorySnapshot::by_type() > as well. > > > I did not interpret that comment that way, and was > puzzled by the > reference to by_type. > > > It doesn't make sense to me to have all the > callers of flag_to_index > check what it returned instead of doing it inside > flag_to_index. > > > I disagree. Imho it makes sense to either check > the Memflags > enumeration input argument in flag_to_index() or > the returned index > before consumption. In both cases one knows the > valid value range. > Strictly speaking checking the index in > flag_to_index() cannot be > done because it is a faceless int type whose valid > values are > not yet > known. > > > The index has to fall in the range 0 <= index <= > mt_number_of_types, > and I was suggesting that it makes more sense to > verify this once in > flag_to_index() than in all the callers of > flag_to_index. > > Hi Thomas, > > Just catching up on this thread. This is the same > conclusion I came > to. I don't understand what you mean by "it is a > faceless int type > whose valid values are not yet known". > > > Thank you for looking into this! > > "it is a faceless int type whose valid values are > not yet known" was > maybe expressed sloppily: > > My first patch checked the input enum "flag" > argument inside > NMTUtil::flag_to_index() for correct enum values : > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > > > > > > >> > > > David did not like that, he thought it strange to > check an enum for > correct enum values. I can see his point. He would > have preferred > instead an assert inside NMTUtil::flag_to_index() > which before > returning checks the to-be-returned index for > 0>=i>=mt_number_of_types. I did not like that > because strictly > speaking inside of NMTUtil::flag_to_index() it is > not known that the > returned integer will be used by the caller as an > index into an > array > of mt_number_of_types length. > > It is all extreme nitpicking and in the end > amounts to the same :) > > The current patch does neither, but follows Max' > suggestion of > checking the index right before it is consumed to > access an array. I > think this is a good solution and very clear. > > > > BTW, I'll sponsor this fix for you once it is > finalized. Please > update the copyright dates first. > > > Thank you. I'll update the webrev and repost. > > > > thanks, > > Chris > > > Kind Regards, Thomas > > > > David > > It is all academical and mostly a matter of taste. > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > > > > > > >> > > > > > > > > >>> > > > As David correctly writes, this is technically not > a bug, so I guess > this will have to wait until java 10. > > > Yes, afraid so. > > > The fix is trivial and I will try to get fc > extension for this (now > that Goetz explained to me how to do this :). It > seems this is done > for many other non-bug issues as well. > > ..Thomas > > > Thanks, David > > Kind Regards, Thomas > > > > On Fri, Oct 14, 2016 at 12:57 AM, David Holmes > > > > > >> > > > > > >>> > > > > > >> > > > > > >>>>> > wrote: > > On 13/10/2016 10:53 PM, Thomas St?fe wrote: > > Hi David, > > On Thu, Oct 13, 2016 at 12:08 PM, David Holmes > > > > > >> > > > > > >>> > > > > > >> > > > > > >>>> > > > > > >> > > > > > >>> > > > > > > >> > > > > > >>>>>> > > wrote: > > Hi Thomas, > > On 13/10/2016 3:49 PM, Thomas St?fe wrote: > > Hi all, > > may I have plase a review for this tiny change? It > just adds some > assert to NMT. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8167650 > > > > > >> > > > > > > >>> > > > > > >> > > > > > >>>> > > > > > > > >> > > > > > > >>> > > > > > >> > > > > > >>>>> > webrev: > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > > > > >> > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > >>>> > > > > > > > > > > > >> > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > >>>>> > > > MEMFLAGS/webrev.00/webrev/ > > We had an ugly memory overwrite caused by this - > ultimately our > fault, because we fed an invalid memory flag to > NMT - but it was > difficult to find. An assert would have saved some > time. > > > I'm a little perplexed with asserting that > something of MEMFLAGS > type > must be an actual MEMFLAGS value - it implies the > caller is coercing > plain int to MEMFLAGS, and I don't have much > sympathy if they mess > that up. Can't help wondering if there is some > clever C++ trick to > flag bad conversions at compile-time? > > > The error was caused by an uninitialized variable > of type MEMFLAGS. > This was our fault, we have heavily modified > allocation.hpp and > introduced an error then merging changes from > upstream. Due to a > merging error this lead to a case where > Arena::_flags was not > initialized and contained a very large value. > > > Ah I see. Lack of default initialization can be > annoying :) > > I admit it looks funny. If it bothers you, I could > instead check the > returned index to be in the range for the size of > the _malloc array > in MallocMemorySnapshot::by_type(). Technically, > it would mean the > same. > > > So I just realized that here: > > 62 // Map memory type to human readable name 63 > static const > char* flag_to_name(MEMFLAGS flag) { 64 > assert(flag >= 0 && flag > < mt_number_of_types, "Invalid flag value %d.", > (int)flag); 65 > return _memory_type_names[flag_to_index(flag)]; 66 } > > we call flag_to_index, so the assert is redundant > as it is > already in > flag_to_index. Then presumably we change > flag_to_index to something > like this: > > static inline int flag_to_index(MEMFLAGS flag) { > int index = (flag & > 0xff); assert(index >= 0 && index < > mt_number_of_types, "Invalid > flag > value %d.", (int)flag); return index; } > > so we're validating the index rather than the flag. > > Cheers, David > > > > > The function that takes the index should validate > the index, so that > is fine. > > Which one were you actually passing the bad value > to? :) > > This isn't a strong objection just musing if we > can do better. > And as > the hs repos are still closed, and likely to > remain so till early > next week, we have some slack time :) > > > :) Sure. > > Kind Regards, Thomas > > > Cheers, David > > Thank you! > > Thomas > > > > > > > > > > > From max.ockner at oracle.com Wed Nov 23 16:09:03 2016 From: max.ockner at oracle.com (Max Ockner) Date: Wed, 23 Nov 2016 11:09:03 -0500 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Message-ID: <5835BF1F.7060600@oracle.com> I retract any strong opinion that I may seem to have held about the suggestion I offered. I don't think it's worth it to continue this debate. I will gladly switch sides to join the majority on this issue. Max On 11/23/2016 6:20 AM, David Holmes wrote: > Hi Thomas, > > On 23/11/2016 5:29 PM, Thomas St?fe wrote: >> Hi David, >> >> On Wed, Nov 23, 2016 at 3:08 AM, David Holmes > > wrote: >> >> Hi Thomas, >> >> David did not like that, he thought it strange to check an >> enum for >> correct enum values. I can see his point. He would have >> preferred >> instead an assert inside NMTUtil::flag_to_index() which before >> returning checks the to-be-returned index for >> 0>=i>=mt_number_of_types. I did not like that because strictly >> speaking inside of NMTUtil::flag_to_index() it is not known >> that the >> returned integer will be used by the caller as an index into an >> array >> of mt_number_of_types length. >> >> >> Sorry but how is NMTUtil::flag_to_index anything but a mapping from >> 0 to mt_number_of_types ??? MEMFLAGS aka MemoryType is an enum that >> spans zero to mt_number_of_types! We're validating the post >> condition for the method. >> >> >> Sorry, this may be clear for you but was not clear for me. There is no >> comment at NMTUtil::flag_to_index explaining the range of the output >> value, and the callers are spread over multiple files. The fact that the >> original author found it necessary to add this conversion function and >> to mask the upper bits out suggested to me that the returned index is >> something different than MEMFLAGS, or may be different in the future. >> Otherwise he could have just used the MEMFLAGS enum itself as index. > > Can't answer that - perhaps just for cleanliness of the type system? > Perhaps he thought the enum was strongly-typed? Or perhaps it was to > hide the actual mapping function to allow for a different scheme. > > If anything the & 0xff suggests that the input might be something > other than a MEMFLAGS value - despite it being typed as MEMFLAGS - > that needs to be brought back into range. > >> Note that I still do not understand the & 0xff part. Once MEMFLAG values >> surpass 0xff, what is supposed to happen? > > I don't think there is any expectation that we will ever go that high: > > enum MemoryType { > // Memory type by sub systems. It occupies lower byte. > > though why this was restricted to be a byte-size value I can't say. I > also don't know why we then typedef MemoryType to MEMFLAGS. > >> As I said on Oct 19, and which Chris indicated he agreed with, it >> makes no sense to me to put the same assert in all the callers of >> flag_to_index instead of doing the validation inside flag_to_index. >> >> >> I understand you. I understood Max' comment otherwise. Faced with two >> opinions, and as a outsider having no way to weight them against each >> other, I have to choose one. >> > > Fair enough, but Chris made it 2-1. :) > >> Whether you choose to validate the flag or the index I don't care. >> >> >> Then we could have just gone with my very first webrev and I could have >> saved work. In your very first answer to my original webrev you wrote: >> "I'm a little perplexed with asserting that something of MEMFLAGS type >> must be an actual MEMFLAGS value". I understood this as objection. > > Yes but I've now reached a point where I'd accept that if it is the > only way to have the assertion inside the function. When I made that > objection I did not expect this to take the path it did. Sorry. > Me too. >> I am fine with either solution. I am also fine with someone else fixing >> this issue. My only preference would be to some form of assert in this >> code, because I spent quite some time to find the memory overwriter this >> caused. > Me too. > My preference, obviously, is an assert inside flag_to_index, on the > value of index to be returned. > > I did not expect this to become so prolonged, and I'm sorry about > that, but I really object to the callers of the function validating > its output. Sorry about that... > > Thanks, > David > >> If we were in a decent language the bug would have been impossible >> to begin with. >> >> David >> ----- >> >> >> Thanks, Thomas >> >> >> >> On 22/11/2016 1:21 AM, Thomas St?fe wrote: >> >> Hi all, >> >> this small issue got fc extension, so lets wrap this up. >> Thanks to >> all for reviewing! >> >> New webrev: >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ >> >> >> No changes to the last one but the updated copyrights Chris >> did ask >> for. Solution still follows Max' suggestion of checking the >> index >> right before consumption. >> >> Thanks, and Kind Regards, Thomas >> >> >> On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe >> >> > >> wrote: >> >> Hi Chris, >> >> On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer >> >> > >> wrote: >> >> On 10/18/16 11:49 PM, David Holmes wrote: >> >> >> >> On 19/10/2016 3:17 PM, Thomas St?fe wrote: >> >> >> >> On Wed, Oct 19, 2016 at 3:21 AM, David Holmes >> >> > > >> >> > >>> >> wrote: >> >> On 18/10/2016 3:39 PM, Thomas St?fe wrote: >> >> Hi David, Max, >> >> I changed the asserts according to Max' suggestion. Instead of >> checking inside flag_to_index, now I check before callers of >> this >> function use this value to access memory. >> >> >> I don't see where Max suggested that?? >> >> >> Max wrote: " I think the decision on whether to access a slot >> should >> occur as close to memory accessing code as possible." and >> proceeded >> to suggest fixing VirtualMemorySnapshot::by_type() as well. >> >> >> I did not interpret that comment that way, and was puzzled by >> the >> reference to by_type. >> >> >> It doesn't make sense to me to have all the callers of >> flag_to_index >> check what it returned instead of doing it inside flag_to_index. >> >> >> I disagree. Imho it makes sense to either check the Memflags >> enumeration input argument in flag_to_index() or the returned >> index >> before consumption. In both cases one knows the valid value >> range. >> Strictly speaking checking the index in flag_to_index() >> cannot be >> done because it is a faceless int type whose valid values are >> not yet >> known. >> >> >> The index has to fall in the range 0 <= index <= >> mt_number_of_types, >> and I was suggesting that it makes more sense to verify this >> once in >> flag_to_index() than in all the callers of flag_to_index. >> >> Hi Thomas, >> >> Just catching up on this thread. This is the same conclusion >> I came >> to. I don't understand what you mean by "it is a faceless int >> type >> whose valid values are not yet known". >> >> >> Thank you for looking into this! >> >> "it is a faceless int type whose valid values are not yet >> known" was >> maybe expressed sloppily: >> >> My first patch checked the input enum "flag" argument inside >> NMTUtil::flag_to_index() for correct enum values : >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ >> >> >> >> > > >> >> >> David did not like that, he thought it strange to check an >> enum for >> correct enum values. I can see his point. He would have >> preferred >> instead an assert inside NMTUtil::flag_to_index() which before >> returning checks the to-be-returned index for >> 0>=i>=mt_number_of_types. I did not like that because strictly >> speaking inside of NMTUtil::flag_to_index() it is not known >> that the >> returned integer will be used by the caller as an index into an >> array >> of mt_number_of_types length. >> >> It is all extreme nitpicking and in the end amounts to the >> same :) >> >> The current patch does neither, but follows Max' suggestion of >> checking the index right before it is consumed to access an >> array. I >> think this is a good solution and very clear. >> >> >> >> BTW, I'll sponsor this fix for you once it is finalized. Please >> update the copyright dates first. >> >> >> Thank you. I'll update the webrev and repost. >> >> >> >> thanks, >> >> Chris >> >> >> Kind Regards, Thomas >> >> >> >> David >> >> It is all academical and mostly a matter of taste. >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html >> >> >> >> > > >> >> > >> >> >> > >> >> >> >> As David correctly writes, this is technically not a bug, so >> I guess >> this will have to wait until java 10. >> >> >> Yes, afraid so. >> >> >> The fix is trivial and I will try to get fc extension for >> this (now >> that Goetz explained to me how to do this :). It seems this >> is done >> for many other non-bug issues as well. >> >> ..Thomas >> >> >> Thanks, David >> >> Kind Regards, Thomas >> >> >> >> On Fri, Oct 14, 2016 at 12:57 AM, David Holmes >> >> > > >> >> > >> >> >> > > >> >> > >>>> >> wrote: >> >> On 13/10/2016 10:53 PM, Thomas St?fe wrote: >> >> Hi David, >> >> On Thu, Oct 13, 2016 at 12:08 PM, David Holmes >> >> > > >> >> > >> >> >> > > >> >> > >>> >> >> > > >> >> > >> >> >> >> > > >> >> > >>>>> >> >> wrote: >> >> Hi Thomas, >> >> On 13/10/2016 3:49 PM, Thomas St?fe wrote: >> >> Hi all, >> >> may I have plase a review for this tiny change? It just adds >> some >> assert to NMT. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8167650 >> >> > > >> >> > >> > >> >> > >> > > >> > >> > >>> >> >> >> > >> > > >> >> > >> > >> >> > >> > > >> > >> > >>>> webrev: >> >> >> http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ >> >> > > >> >> >> > >> >> > >> >> >> >> > >> >> >> > > >> >> > >> >> >> > >>> >> >> >> >> >> > >> >> >> > > >> >> > >> >> >> > >> >> >> >> > >> >> >> > > >> >> > >> >> >> > >>>> >> >> >> MEMFLAGS/webrev.00/webrev/ >> >> We had an ugly memory overwrite caused by this - ultimately our >> fault, because we fed an invalid memory flag to NMT - but it was >> difficult to find. An assert would have saved some time. >> >> >> I'm a little perplexed with asserting that something of MEMFLAGS >> type >> must be an actual MEMFLAGS value - it implies the caller is >> coercing >> plain int to MEMFLAGS, and I don't have much sympathy if they >> mess >> that up. Can't help wondering if there is some clever C++ >> trick to >> flag bad conversions at compile-time? >> >> >> The error was caused by an uninitialized variable of type >> MEMFLAGS. >> This was our fault, we have heavily modified allocation.hpp and >> introduced an error then merging changes from upstream. Due to a >> merging error this lead to a case where Arena::_flags was not >> initialized and contained a very large value. >> >> >> Ah I see. Lack of default initialization can be annoying :) >> >> I admit it looks funny. If it bothers you, I could instead >> check the >> returned index to be in the range for the size of the _malloc >> array >> in MallocMemorySnapshot::by_type(). Technically, it would >> mean the >> same. >> >> >> So I just realized that here: >> >> 62 // Map memory type to human readable name 63 static const >> char* flag_to_name(MEMFLAGS flag) { 64 assert(flag >= 0 >> && flag >> < mt_number_of_types, "Invalid flag value %d.", (int)flag); 65 >> return _memory_type_names[flag_to_index(flag)]; 66 } >> >> we call flag_to_index, so the assert is redundant as it is >> already in >> flag_to_index. Then presumably we change flag_to_index to >> something >> like this: >> >> static inline int flag_to_index(MEMFLAGS flag) { int index = >> (flag & >> 0xff); assert(index >= 0 && index < mt_number_of_types, "Invalid >> flag >> value %d.", (int)flag); return index; } >> >> so we're validating the index rather than the flag. >> >> Cheers, David >> >> >> >> >> The function that takes the index should validate the index, >> so that >> is fine. >> >> Which one were you actually passing the bad value to? :) >> >> This isn't a strong objection just musing if we can do better. >> And as >> the hs repos are still closed, and likely to remain so till >> early >> next week, we have some slack time :) >> >> >> :) Sure. >> >> Kind Regards, Thomas >> >> >> Cheers, David >> >> Thank you! >> >> Thomas >> >> >> >> >> >> >> >> >> From chris.plummer at oracle.com Wed Nov 23 17:45:37 2016 From: chris.plummer at oracle.com (Chris Plummer) Date: Wed, 23 Nov 2016 09:45:37 -0800 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Message-ID: On 11/23/16 5:47 AM, Thomas St?fe wrote: > Hi David, > > > > On Wed, Nov 23, 2016 at 12:20 PM, David Holmes > > wrote: > > Hi Thomas, > > On 23/11/2016 5:29 PM, Thomas St?fe wrote: > > Hi David, > > ... > > My preference, obviously, is an assert inside flag_to_index, on > the value of index to be returned. > > > My latest webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.03/webrev/ > > > > I did it the way you preferred, by checking the return value of > flag_to_index(). Looks good to me. :) I'll run this through some internal testing and push it for you. Chris > > I did not expect this to become so prolonged, and I'm sorry about > that, but I really object to the callers of the function > validating its output. > > > No problem. I'm happy to have this issue closed. > > Kind Regards, Thomas > > Thanks, > David > > If we were in a decent language the bug would have been > impossible > to begin with. > > David > ----- > > > Thanks, Thomas > > > > On 22/11/2016 1:21 AM, Thomas St?fe wrote: > > Hi all, > > this small issue got fc extension, so lets wrap this > up. Thanks to > all for reviewing! > > New webrev: > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ > > > > > > No changes to the last one but the updated copyrights > Chris did ask > for. Solution still follows Max' suggestion of > checking the index > right before consumption. > > Thanks, and Kind Regards, Thomas > > > On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe > > > > > >>> wrote: > > Hi Chris, > > On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer > > > > > >>> wrote: > > On 10/18/16 11:49 PM, David Holmes wrote: > > > > On 19/10/2016 3:17 PM, Thomas St?fe wrote: > > > > On Wed, Oct 19, 2016 at 3:21 AM, David Holmes > > > > > >> > > > > > >>>> > wrote: > > On 18/10/2016 3:39 PM, Thomas St?fe wrote: > > Hi David, Max, > > I changed the asserts according to Max' suggestion. > Instead of > checking inside flag_to_index, now I check before > callers of this > function use this value to access memory. > > > I don't see where Max suggested that?? > > > Max wrote: " I think the decision on whether to access > a slot should > occur as close to memory accessing code as possible." > and proceeded > to suggest fixing VirtualMemorySnapshot::by_type() as > well. > > > I did not interpret that comment that way, and was > puzzled by the > reference to by_type. > > > It doesn't make sense to me to have all the callers of > flag_to_index > check what it returned instead of doing it inside > flag_to_index. > > > I disagree. Imho it makes sense to either check the > Memflags > enumeration input argument in flag_to_index() or the > returned index > before consumption. In both cases one knows the valid > value range. > Strictly speaking checking the index in > flag_to_index() cannot be > done because it is a faceless int type whose valid > values are > not yet > known. > > > The index has to fall in the range 0 <= index <= > mt_number_of_types, > and I was suggesting that it makes more sense to > verify this once in > flag_to_index() than in all the callers of flag_to_index. > > Hi Thomas, > > Just catching up on this thread. This is the same > conclusion I came > to. I don't understand what you mean by "it is a > faceless int type > whose valid values are not yet known". > > > Thank you for looking into this! > > "it is a faceless int type whose valid values are not > yet known" was > maybe expressed sloppily: > > My first patch checked the input enum "flag" argument > inside > NMTUtil::flag_to_index() for correct enum values : > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > > > > > > > > > >> > > > David did not like that, he thought it strange to > check an enum for > correct enum values. I can see his point. He would > have preferred > instead an assert inside NMTUtil::flag_to_index() > which before > returning checks the to-be-returned index for > 0>=i>=mt_number_of_types. I did not like that because > strictly > speaking inside of NMTUtil::flag_to_index() it is not > known that the > returned integer will be used by the caller as an > index into an > array > of mt_number_of_types length. > > It is all extreme nitpicking and in the end amounts to > the same :) > > The current patch does neither, but follows Max' > suggestion of > checking the index right before it is consumed to > access an array. I > think this is a good solution and very clear. > > > > BTW, I'll sponsor this fix for you once it is > finalized. Please > update the copyright dates first. > > > Thank you. I'll update the webrev and repost. > > > > thanks, > > Chris > > > Kind Regards, Thomas > > > > David > > It is all academical and mostly a matter of taste. > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > > > > > > > > > >> > > > > > > > > > > > > >>> > > > As David correctly writes, this is technically not a > bug, so I guess > this will have to wait until java 10. > > > Yes, afraid so. > > > The fix is trivial and I will try to get fc extension > for this (now > that Goetz explained to me how to do this :). It seems > this is done > for many other non-bug issues as well. > > ..Thomas > > > Thanks, David > > Kind Regards, Thomas > > > > On Fri, Oct 14, 2016 at 12:57 AM, David Holmes > > > > > >> > > > > > >>> > > > > > >> > > > > > >>>>> > wrote: > > On 13/10/2016 10:53 PM, Thomas St?fe wrote: > > Hi David, > > On Thu, Oct 13, 2016 at 12:08 PM, David Holmes > > > > > >> > > > > > >>> > > > > > >> > > > > > >>>> > > > > > >> > > > > > >>> > > > > > > >> > > > > > >>>>>> > > wrote: > > Hi Thomas, > > On 13/10/2016 3:49 PM, Thomas St?fe wrote: > > Hi all, > > may I have plase a review for this tiny change? It > just adds some > assert to NMT. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8167650 > > > > > >> > > > > > > >>> > > > > > >> > > > > > >>>> > > > > > > > >> > > > > > > >>> > > > > > >> > > > > > >>>>> webrev: > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > > > > > > > >> > > > > > > > > > > > >>> > > > > > > > > > > > > >> > > > > > > > > > > > >>>> > > > > > > > > > > > > > >> > > > > > > > > > > > >>> > > > > > > > > > > > > >> > > > > > > > > > > > >>>>> > > MEMFLAGS/webrev.00/webrev/ > > We had an ugly memory overwrite caused by this - > ultimately our > fault, because we fed an invalid memory flag to NMT - > but it was > difficult to find. An assert would have saved some time. > > > I'm a little perplexed with asserting that something > of MEMFLAGS > type > must be an actual MEMFLAGS value - it implies the > caller is coercing > plain int to MEMFLAGS, and I don't have much sympathy > if they mess > that up. Can't help wondering if there is some clever > C++ trick to > flag bad conversions at compile-time? > > > The error was caused by an uninitialized variable of > type MEMFLAGS. > This was our fault, we have heavily modified > allocation.hpp and > introduced an error then merging changes from > upstream. Due to a > merging error this lead to a case where Arena::_flags > was not > initialized and contained a very large value. > > > Ah I see. Lack of default initialization can be > annoying :) > > I admit it looks funny. If it bothers you, I could > instead check the > returned index to be in the range for the size of the > _malloc array > in MallocMemorySnapshot::by_type(). Technically, it > would mean the > same. > > > So I just realized that here: > > 62 // Map memory type to human readable name 63 > static const > char* flag_to_name(MEMFLAGS flag) { 64 assert(flag >= > 0 && flag > < mt_number_of_types, "Invalid flag value %d.", > (int)flag); 65 > return _memory_type_names[flag_to_index(flag)]; 66 } > > we call flag_to_index, so the assert is redundant as it is > already in > flag_to_index. Then presumably we change flag_to_index > to something > like this: > > static inline int flag_to_index(MEMFLAGS flag) { int > index = (flag & > 0xff); assert(index >= 0 && index < > mt_number_of_types, "Invalid > flag > value %d.", (int)flag); return index; } > > so we're validating the index rather than the flag. > > Cheers, David > > > > > The function that takes the index should validate the > index, so that > is fine. > > Which one were you actually passing the bad value to? :) > > This isn't a strong objection just musing if we can do > better. > And as > the hs repos are still closed, and likely to remain so > till early > next week, we have some slack time :) > > > :) Sure. > > Kind Regards, Thomas > > > Cheers, David > > Thank you! > > Thomas > > > > > > > > > > From thomas.stuefe at gmail.com Wed Nov 23 19:56:38 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 23 Nov 2016 19:56:38 +0000 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Message-ID: Thanks, Chris! On Wed, 23 Nov 2016 at 18:45, Chris Plummer wrote: > On 11/23/16 5:47 AM, Thomas St?fe wrote: > > Hi David, > > > > > > > > On Wed, Nov 23, 2016 at 12:20 PM, David Holmes > > > wrote: > > > > Hi Thomas, > > > > On 23/11/2016 5:29 PM, Thomas St?fe wrote: > > > > Hi David, > > > > ... > > > > My preference, obviously, is an assert inside flag_to_index, on > > the value of index to be returned. > > > > > > My latest webrev: > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.03/webrev/ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.03/webrev/ > > > > > > > > I did it the way you preferred, by checking the return value of > > flag_to_index(). > Looks good to me. :) > > I'll run this through some internal testing and push it for you. > > Chris > > > > I did not expect this to become so prolonged, and I'm sorry about > > that, but I really object to the callers of the function > > validating its output. > > > > > > No problem. I'm happy to have this issue closed. > > > > Kind Regards, Thomas > > > > Thanks, > > David > > > > If we were in a decent language the bug would have been > > impossible > > to begin with. > > > > David > > ----- > > > > > > Thanks, Thomas > > > > > > > > On 22/11/2016 1:21 AM, Thomas St?fe wrote: > > > > Hi all, > > > > this small issue got fc extension, so lets wrap this > > up. Thanks to > > all for reviewing! > > > > New webrev: > > > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ > >> > > > > No changes to the last one but the updated copyrights > > Chris did ask > > for. Solution still follows Max' suggestion of > > checking the index > > right before consumption. > > > > Thanks, and Kind Regards, Thomas > > > > > > On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe > > > > > >> > > > > > > >>> wrote: > > > > Hi Chris, > > > > On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer > > > > > > > > > > > > > >>> wrote: > > > > On 10/18/16 11:49 PM, David Holmes wrote: > > > > > > > > On 19/10/2016 3:17 PM, Thomas St?fe wrote: > > > > > > > > On Wed, Oct 19, 2016 at 3:21 AM, David Holmes > > > > > >> > > > > > >>> > > > > > >> > > > > > > >>>> > > wrote: > > > > On 18/10/2016 3:39 PM, Thomas St?fe wrote: > > > > Hi David, Max, > > > > I changed the asserts according to Max' suggestion. > > Instead of > > checking inside flag_to_index, now I check before > > callers of this > > function use this value to access memory. > > > > > > I don't see where Max suggested that?? > > > > > > Max wrote: " I think the decision on whether to access > > a slot should > > occur as close to memory accessing code as possible." > > and proceeded > > to suggest fixing VirtualMemorySnapshot::by_type() as > > well. > > > > > > I did not interpret that comment that way, and was > > puzzled by the > > reference to by_type. > > > > > > It doesn't make sense to me to have all the callers of > > flag_to_index > > check what it returned instead of doing it inside > > flag_to_index. > > > > > > I disagree. Imho it makes sense to either check the > > Memflags > > enumeration input argument in flag_to_index() or the > > returned index > > before consumption. In both cases one knows the valid > > value range. > > Strictly speaking checking the index in > > flag_to_index() cannot be > > done because it is a faceless int type whose valid > > values are > > not yet > > known. > > > > > > The index has to fall in the range 0 <= index <= > > mt_number_of_types, > > and I was suggesting that it makes more sense to > > verify this once in > > flag_to_index() than in all the callers of flag_to_index. > > > > Hi Thomas, > > > > Just catching up on this thread. This is the same > > conclusion I came > > to. I don't understand what you mean by "it is a > > faceless int type > > whose valid values are not yet known". > > > > > > Thank you for looking into this! > > > > "it is a faceless int type whose valid values are not > > yet known" was > > maybe expressed sloppily: > > > > My first patch checked the input enum "flag" argument > > inside > > NMTUtil::flag_to_index() for correct enum values : > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > >> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > >>> > > > > > > David did not like that, he thought it strange to > > check an enum for > > correct enum values. I can see his point. He would > > have preferred > > instead an assert inside NMTUtil::flag_to_index() > > which before > > returning checks the to-be-returned index for > > 0>=i>=mt_number_of_types. I did not like that because > > strictly > > speaking inside of NMTUtil::flag_to_index() it is not > > known that the > > returned integer will be used by the caller as an > > index into an > > array > > of mt_number_of_types length. > > > > It is all extreme nitpicking and in the end amounts to > > the same :) > > > > The current patch does neither, but follows Max' > > suggestion of > > checking the index right before it is consumed to > > access an array. I > > think this is a good solution and very clear. > > > > > > > > BTW, I'll sponsor this fix for you once it is > > finalized. Please > > update the copyright dates first. > > > > > > Thank you. I'll update the webrev and repost. > > > > > > > > thanks, > > > > Chris > > > > > > Kind Regards, Thomas > > > > > > > > David > > > > It is all academical and mostly a matter of taste. > > > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > >> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > >>> > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > >> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > >>>> > > > > > > As David correctly writes, this is technically not a > > bug, so I guess > > this will have to wait until java 10. > > > > > > Yes, afraid so. > > > > > > The fix is trivial and I will try to get fc extension > > for this (now > > that Goetz explained to me how to do this :). It seems > > this is done > > for many other non-bug issues as well. > > > > ..Thomas > > > > > > Thanks, David > > > > Kind Regards, Thomas > > > > > > > > On Fri, Oct 14, 2016 at 12:57 AM, David Holmes > > > > > >> > > > > > >>> > > > > > >> > > > > > > >>> > > > > > >> > > > > > >>> > > > > > >> > > > > > > >>>>> > > wrote: > > > > On 13/10/2016 10:53 PM, Thomas St?fe wrote: > > > > Hi David, > > > > On Thu, Oct 13, 2016 at 12:08 PM, David Holmes > > > > > >> > > > > > >>> > > > > > >> > > > > > > >>> > > > > > >> > > > > > >>> > > > > > >> > > > > > > >>>> > > > > > >> > > > > > >>> > > > > > >> > > > > > > >>> > > > > > > > >> > > > > > >>> > > > > > >> > > > > > > >>>>>> > > > > wrote: > > > > Hi Thomas, > > > > On 13/10/2016 3:49 PM, Thomas St?fe wrote: > > > > Hi all, > > > > may I have plase a review for this tiny change? It > > just adds some > > assert to NMT. > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8167650 > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >>>>> webrev: > > > > > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>> > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>>> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>> > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>>>> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>>> > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_ > >>>>> > > > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>>> > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>>>> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>>> > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_>> > > > > > > > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_> > > < > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > < > http://cr.openjdk.java.net/%7Estuefe/webrevs/8167650-NMT-should-check_ > >>>>>> > > > > MEMFLAGS/webrev.00/webrev/ > > > > We had an ugly memory overwrite caused by this - > > ultimately our > > fault, because we fed an invalid memory flag to NMT - > > but it was > > difficult to find. An assert would have saved some time. > > > > > > I'm a little perplexed with asserting that something > > of MEMFLAGS > > type > > must be an actual MEMFLAGS value - it implies the > > caller is coercing > > plain int to MEMFLAGS, and I don't have much sympathy > > if they mess > > that up. Can't help wondering if there is some clever > > C++ trick to > > flag bad conversions at compile-time? > > > > > > The error was caused by an uninitialized variable of > > type MEMFLAGS. > > This was our fault, we have heavily modified > > allocation.hpp and > > introduced an error then merging changes from > > upstream. Due to a > > merging error this lead to a case where Arena::_flags > > was not > > initialized and contained a very large value. > > > > > > Ah I see. Lack of default initialization can be > > annoying :) > > > > I admit it looks funny. If it bothers you, I could > > instead check the > > returned index to be in the range for the size of the > > _malloc array > > in MallocMemorySnapshot::by_type(). Technically, it > > would mean the > > same. > > > > > > So I just realized that here: > > > > 62 // Map memory type to human readable name 63 > > static const > > char* flag_to_name(MEMFLAGS flag) { 64 assert(flag >= > > 0 && flag > > < mt_number_of_types, "Invalid flag value %d.", > > (int)flag); 65 > > return _memory_type_names[flag_to_index(flag)]; 66 } > > > > we call flag_to_index, so the assert is redundant as it > is > > already in > > flag_to_index. Then presumably we change flag_to_index > > to something > > like this: > > > > static inline int flag_to_index(MEMFLAGS flag) { int > > index = (flag & > > 0xff); assert(index >= 0 && index < > > mt_number_of_types, "Invalid > > flag > > value %d.", (int)flag); return index; } > > > > so we're validating the index rather than the flag. > > > > Cheers, David > > > > > > > > > > The function that takes the index should validate the > > index, so that > > is fine. > > > > Which one were you actually passing the bad value to? :) > > > > This isn't a strong objection just musing if we can do > > better. > > And as > > the hs repos are still closed, and likely to remain so > > till early > > next week, we have some slack time :) > > > > > > :) Sure. > > > > Kind Regards, Thomas > > > > > > Cheers, David > > > > Thank you! > > > > Thomas > > > > > > > > > > > > > > > > > > > > > > > From coleen.phillimore at oracle.com Wed Nov 23 20:02:58 2016 From: coleen.phillimore at oracle.com (Coleen Phillimore) Date: Wed, 23 Nov 2016 15:02:58 -0500 Subject: Request Review: JDK-6479237 (cl) Add support for classloader names In-Reply-To: <52930077-0002-cb83-f58d-d4ea6040076a@oracle.com> References: <2C30243B-71D2-49E2-A8B6-2C33B82DB104@oracle.com> <5d494d3a-f8ec-0456-8d04-f8998d840fc7@oracle.com> <52E8F822-2D4E-4BD7-BE4E-748E47901625@oracle.com> <931BF9A2-6F22-48FF-855E-287BAF10FDC0@oracle.com> <52930077-0002-cb83-f58d-d4ea6040076a@oracle.com> Message-ID: Sorry Brent, I just found these in the hotspot-runtime-dev list administrator mail among all the spam. Hopefully, your messages will get through without supervision after this. Coleen On 10/28/16 8:15 PM, Brent Christian wrote: > On 10/28/16 1:44 PM, Mandy Chung wrote: >> >>> On Oct 28, 2016, at 11:11 AM, Brent Christian >>> wrote: >>> >>> Should something be done for STEs returned from >>> StackFrameInfo.toStackTraceElement() ? >> >> Good catch - I missed it. I added package-private static methods in >> StackTraceElement class for both Throwable and StackFrameInfo to get >> StackTraceElement(s). >> >> http://cr.openjdk.java.net/~mchung/jdk9/webrevs/6479237/webrev.02/ >> > > Looks good. > > -Brent From david.holmes at oracle.com Wed Nov 23 23:26:18 2016 From: david.holmes at oracle.com (David Holmes) Date: Thu, 24 Nov 2016 09:26:18 +1000 Subject: RFR(xxs): 8167650: NMT should check for invalid MEMFLAGS. In-Reply-To: References: <98572424-37a1-90b9-30ef-3be0691e7bf0@oracle.com> <60cafd21-1945-b7d4-b0bf-db92a93dfdcf@oracle.com> <97c9d091-73b2-d82f-de69-55cd79bf17b8@oracle.com> <64345190-1d47-31ee-7daf-1c6821ef0ca0@oracle.com> Message-ID: On 23/11/2016 11:47 PM, Thomas St?fe wrote: > Hi David, > > On Wed, Nov 23, 2016 at 12:20 PM, David Holmes > wrote: > > Hi Thomas, > > On 23/11/2016 5:29 PM, Thomas St?fe wrote: > > Hi David, > > ... > > > My preference, obviously, is an assert inside flag_to_index, on the > value of index to be returned. > > > My latest > webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.03/webrev/ > > I did it the way you preferred, by checking the return value of > flag_to_index(). Looks good! Many thanks. David ----- > > I did not expect this to become so prolonged, and I'm sorry about > that, but I really object to the callers of the function validating > its output. > > > No problem. I'm happy to have this issue closed. > > Kind Regards, Thomas > > > > Thanks, > David > > If we were in a decent language the bug would have been > impossible > to begin with. > > David > ----- > > > Thanks, Thomas > > > > On 22/11/2016 1:21 AM, Thomas St?fe wrote: > > Hi all, > > this small issue got fc extension, so lets wrap this up. > Thanks to > all for reviewing! > > New webrev: > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.02/webrev/ > > > > > > No changes to the last one but the updated copyrights > Chris did ask > for. Solution still follows Max' suggestion of checking > the index > right before consumption. > > Thanks, and Kind Regards, Thomas > > > On Mon, Nov 21, 2016 at 4:17 PM, Thomas St?fe > > > > >>> wrote: > > Hi Chris, > > On Thu, Nov 17, 2016 at 8:54 PM, Chris Plummer > > > > > >>> wrote: > > On 10/18/16 11:49 PM, David Holmes wrote: > > > > On 19/10/2016 3:17 PM, Thomas St?fe wrote: > > > > On Wed, Oct 19, 2016 at 3:21 AM, David Holmes > > > >> > > > >>>> > wrote: > > On 18/10/2016 3:39 PM, Thomas St?fe wrote: > > Hi David, Max, > > I changed the asserts according to Max' suggestion. > Instead of > checking inside flag_to_index, now I check before > callers of this > function use this value to access memory. > > > I don't see where Max suggested that?? > > > Max wrote: " I think the decision on whether to access a > slot should > occur as close to memory accessing code as possible." > and proceeded > to suggest fixing VirtualMemorySnapshot::by_type() as well. > > > I did not interpret that comment that way, and was > puzzled by the > reference to by_type. > > > It doesn't make sense to me to have all the callers of > flag_to_index > check what it returned instead of doing it inside > flag_to_index. > > > I disagree. Imho it makes sense to either check the Memflags > enumeration input argument in flag_to_index() or the > returned index > before consumption. In both cases one knows the valid > value range. > Strictly speaking checking the index in flag_to_index() > cannot be > done because it is a faceless int type whose valid > values are > not yet > known. > > > The index has to fall in the range 0 <= index <= > mt_number_of_types, > and I was suggesting that it makes more sense to verify > this once in > flag_to_index() than in all the callers of flag_to_index. > > Hi Thomas, > > Just catching up on this thread. This is the same > conclusion I came > to. I don't understand what you mean by "it is a > faceless int type > whose valid values are not yet known". > > > Thank you for looking into this! > > "it is a faceless int type whose valid values are not > yet known" was > maybe expressed sloppily: > > My first patch checked the input enum "flag" argument inside > NMTUtil::flag_to_index() for correct enum values : > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.00/webrev/ > > > > > > > > > > >> > > > David did not like that, he thought it strange to check > an enum for > correct enum values. I can see his point. He would have > preferred > instead an assert inside NMTUtil::flag_to_index() which > before > returning checks the to-be-returned index for > 0>=i>=mt_number_of_types. I did not like that because > strictly > speaking inside of NMTUtil::flag_to_index() it is not > known that the > returned integer will be used by the caller as an index > into an > array > of mt_number_of_types length. > > It is all extreme nitpicking and in the end amounts to > the same :) > > The current patch does neither, but follows Max' > suggestion of > checking the index right before it is consumed to access > an array. I > think this is a good solution and very clear. > > > > BTW, I'll sponsor this fix for you once it is finalized. > Please > update the copyright dates first. > > > Thank you. I'll update the webrev and repost. > > > > thanks, > > Chris > > > Kind Regards, Thomas > > > > David > > It is all academical and mostly a matter of taste. > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_MEMFLAGS/webrev.01/webrev/index.html > > > > > > > > > > >> > > > > > > > > > > > > >>> > > > As David correctly writes, this is technically not a > bug, so I guess > this will have to wait until java 10. > > > Yes, afraid so. > > > The fix is trivial and I will try to get fc extension > for this (now > that Goetz explained to me how to do this :). It seems > this is done > for many other non-bug issues as well. > > ..Thomas > > > Thanks, David > > Kind Regards, Thomas > > > > On Fri, Oct 14, 2016 at 12:57 AM, David Holmes > > > >> > > > >>> > > > >> > > > >>>>> > wrote: > > On 13/10/2016 10:53 PM, Thomas St?fe wrote: > > Hi David, > > On Thu, Oct 13, 2016 at 12:08 PM, David Holmes > > > >> > > > >>> > > > >> > > > >>>> > > > >> > > > >>> > > > > >> > > > > >>>>>> > > wrote: > > Hi Thomas, > > On 13/10/2016 3:49 PM, Thomas St?fe wrote: > > Hi all, > > may I have plase a review for this tiny change? It just > adds some > assert to NMT. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8167650 > > > > > >> > > > > > > >>> > > > > > >> > > > > > >>>> > > > > > > > >> > > > > > > >>> > > > > > >> > > > > > >>>>> webrev: > > > > http://cr.openjdk.java.net/~stuefe/webrevs/8167650-NMT-should-check_ > > > > > > > > >> > > > > > > > > > > > >>> > > > > > > > > > > > > >> > > > > > > > > > > > >>>> > > > > > > > > > > > > > >> > > > > > > > > > > > >>> > > > > > > > > > > > > >> > > > > > > > > > > > >>>>> > > MEMFLAGS/webrev.00/webrev/ > > We had an ugly memory overwrite caused by this - > ultimately our > fault, because we fed an invalid memory flag to NMT - > but it was > difficult to find. An assert would have saved some time. > > > I'm a little perplexed with asserting that something of > MEMFLAGS > type > must be an actual MEMFLAGS value - it implies the caller > is coercing > plain int to MEMFLAGS, and I don't have much sympathy if > they mess > that up. Can't help wondering if there is some clever > C++ trick to > flag bad conversions at compile-time? > > > The error was caused by an uninitialized variable of > type MEMFLAGS. > This was our fault, we have heavily modified > allocation.hpp and > introduced an error then merging changes from upstream. > Due to a > merging error this lead to a case where Arena::_flags > was not > initialized and contained a very large value. > > > Ah I see. Lack of default initialization can be annoying :) > > I admit it looks funny. If it bothers you, I could > instead check the > returned index to be in the range for the size of the > _malloc array > in MallocMemorySnapshot::by_type(). Technically, it > would mean the > same. > > > So I just realized that here: > > 62 // Map memory type to human readable name 63 > static const > char* flag_to_name(MEMFLAGS flag) { 64 assert(flag > >= 0 && flag > < mt_number_of_types, "Invalid flag value %d.", > (int)flag); 65 > return _memory_type_names[flag_to_index(flag)]; 66 } > > we call flag_to_index, so the assert is redundant as it is > already in > flag_to_index. Then presumably we change flag_to_index > to something > like this: > > static inline int flag_to_index(MEMFLAGS flag) { int > index = (flag & > 0xff); assert(index >= 0 && index < mt_number_of_types, > "Invalid > flag > value %d.", (int)flag); return index; } > > so we're validating the index rather than the flag. > > Cheers, David > > > > > The function that takes the index should validate the > index, so that > is fine. > > Which one were you actually passing the bad value to? :) > > This isn't a strong objection just musing if we can do > better. > And as > the hs repos are still closed, and likely to remain so > till early > next week, we have some slack time :) > > > :) Sure. > > Kind Regards, Thomas > > > Cheers, David > > Thank you! > > Thomas > > > > > > > > > > From goetz.lindenmaier at sap.com Thu Nov 24 12:15:34 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Thu, 24 Nov 2016 12:15:34 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> Message-ID: Hi, I now edited the stuff I had proposed below: http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.03/ This includes - the NPTL fix from webrev.02 - merging code on linux - not adding OS guard to compiler threads. I think this should be pushed for this bug ID. For the other changes I'll make another bug. Best regards, Goetz. > -----Original Message----- > From: Lindenmaier, Goetz > Sent: Wednesday, November 23, 2016 8:11 AM > To: David Holmes ; > daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > Subject: RE: RFR(M): 8169373: Work around linux NPTL stack guard error. > > Hi, > > > Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The > > ability for a CompilerThread to execute Java code (can_call_java()) is > > now a dynamic property depending on whether the current compiler is the > > JVMCI compiler. > Ah, then I should also leave space for shadow pages in the minimal stack size > of comiler threads. > > Do we agree on the cleanup and on leaving out the OS guard page on > compiler threads? > Then I would edit a change comprising the NPTL workaround and these > additional changes, and split the other issue into a new bug? I think this > will easy the reviewing. > > Best regards, > Goetz. > > > -----Original Message----- > > From: David Holmes [mailto:david.holmes at oracle.com] > > Sent: Mittwoch, 23. November 2016 02:50 > > To: Lindenmaier, Goetz ; > > daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > > > On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: > > > Hi Dan, > > > > > >> -----Original Message----- > > >> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > > >> Sent: Dienstag, 22. November 2016 14:01 > > >> To: Lindenmaier, Goetz ; David Holmes > > >> ; hotspot-runtime-dev at openjdk.java.net > > >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > error. > > >> > > >> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: > > >>> Hi Dan, > > >>> > > >>> I ran into a row of issues, some errors on the platforms. > > >>> > > >>> What I meant with that comment is that > > >>> os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, > > >>> JavaThread::stack_guard_zone_size() + > > >>> JavaThread::stack_shadow_zone_size() + > > >>> (4*BytesPerWord COMPILER2_PRESENT(+2)) * 4 > * > > K); > > >>> was executed, and min_stack_allowed used for all stacks. Now, > compiler > > >> and > > >>> vm minimum stack sizes are not increased by these sizes. > > >> > > >> Now I see what you mean. Thanks for clearing this up. > > >> > > >> I should have remembered that part of the change because we went > back > > >> and forth about removing the red/yellow zone pages from the other > > >> threads. In particular, we discussed the compiler thread because it > > >> is-a JavaThread. Our conclusion was that a compiler thread doesn't > > >> execute Java bytecode so we could remove the red/yellow pages... > > > Yes, it does not execute java byte code. > > > > Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The > > ability for a CompilerThread to execute Java code (can_call_java()) is > > now a dynamic property depending on whether the current compiler is the > > JVMCI compiler. > > > > David > > ----- > > > > > Therefore you can remove the shadow pages. There is no code that > > > will bang. > > > But red/yellow pages are protected right during thread startup. > > > Therefore you must have enough space for them. > > > On ppc, we try to protect three 64K pages out of the 128K compiler stack. > > > That obviously fails. > > > > > > Therefore I propose: > > > size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // Set > > platform dependent. > > > in os::Posix::set_minimum_stack_sizes(): > > > _java_thread_min_stack_allowed = _java_thread_min_stack_allowed + > > > JavaThread::stack_guard_zone_size() + > > > JavaThread::stack_shadow_zone_size(); > > > > > > (Similar for _compiler_thread_min_stack_allowed). > > > > > > The minimal stack size is made up of three components: > > > 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot > > implementation/platform/os. > > > 2. Sizes of C++ frames: depends on C++ compiler. > > > These are fixed at compile time. > > > 3. Sizes of red/yellow/reserved/shadow pages. Depends on the system > > the > > > VM is used on. This is not fixed at compile time. (Our ppc machines > > differ > > > in page size.) > > > Therefore 3. should not be included in a compile time constant. > > > > > > > > > > > >> And that decision allowed us to be exposed to the 64K page issue > > >> because the "extra" space isn't there anymore. > > >> > > >>> At least the _compiler_thread_min_stack_allowed should be increased > > >>> similarly by red/yellow zone size. The compiler thread is a Java > > >>> thread and must have space for these zones. > > >> > > >> I'm not sure that I completely agree (yet). To me, the red/yellow > > >> pages are there for Java thread stack overflow semantics. Yes, the > > >> compiler thread needs extra space when 64K pages are used, but I > > >> would prefer that we add that space via a different calculation. > > > Yes they are. But compiler threads happen tob e a subclass of > > > Java threads and use the same run() method that puts the pages > > > There. > > > I agree that they are not needed for Compiler threads, nor for > > > CodeCacheSweeperThreads. I don't really now about > > > JvmtiAgentThreads and ServiceThreads, but all of the get the guard > pages > > > because they are derived from JavaThread. > > > > > >> To put it another way, I'd like to see us add extra space to solve > > >> the 64K page issue directly instead of as a side effect of the > > >> red/yellow page addition. > > > I don't understand. What do you mean by 'directly'? > > > > > >>> Also, the change added a test that is failing now. > > >> > > >> And that's a "good thing" (TM), right? :-) > > > Yes, it showed a bug and thus raised the need to fix it! :) > > > > > > Best regards, > > > Goetz. > > > > > > > > >> Again, thanks for clarifying 8140520's role in this issue. > > >> > > >> Dan > > >> > > >> > > >>> > > >>> Best regards, > > >>> Goetz. > > >>> > > >>>> -----Original Message----- > > >>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > > >>>> Sent: Montag, 21. November 2016 17:28 > > >>>> To: David Holmes ; Lindenmaier, Goetz > > >>>> ; hotspot-runtime- > > dev at openjdk.java.net > > >>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > > error. > > >>>> > > >>>> Sorry for the delayed responses to this thread. I've been on vacation... > > >>>> > > >>>> One comment/query embedded below... > > >>>> > > >>>> > > >>>> On 11/10/16 8:40 PM, David Holmes wrote: > > >>>>> Hi Goetz, > > >>>>> > > >>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > > >>>>>> Hi David, > > >>>>>> > > >>>>>> This issue is different to 6675312, see also my comment in the bug. > > >>>>>> > > >>>>>> It appears running jtreg test > > runtime/Thread/TooSmallStackSize.java, > > >>>>>> with my patch below you can reproduce it on linuxx86_64. You can > > not > > >>>>>> do that with 6675312. Also, I would assume there are systems out > > there > > >>>>>> on x86 that uses 64-K pages, did you run the tests on these? I > would > > >>>>>> assume you get hard crashes with stack overflows in the first C2 > > >>>>>> compilation if there is only 64K usable CompilerThreadStack. > > >>>>>> > > >>>>>> My fix does not affect Java threads, which are the largest amount > > >>>>>> of threads used by the VM. It affects only the non-Java threads. > > >>>>>> It adds one page to these threads. The page does not require > > memory, > > >>>>>> as it's protected. The stack will only require more space if the > thread > > >>>>>> ran into a stack overflow before the fix as else the pages are not > > >>>>>> mapped. > > >>>>>> This are stack overflows that cause hard crashes, at least on ppc > the > > VM > > >>>>>> does not properly catch these stack overflows, so any setup > working > > >> now > > >>>>>> will not run into the additional space. Altogether there should be > no > > >>>>>> effect on running systems besides requiring one more entry in the > > >>>>>> page table per non-Java thread. > > >>>>>> > > >>>>>> The problem is caused by a rather recent change (8140520: > segfault > > on > > >>>>>> solaris-amd64 > > >>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed after > > >>>>>> feature-close. As this was a rather recent change, it must be > > >>>>>> possible to > > >>>>>> fix this follow up issue. What else is this period in the project good > > >>>>>> for if not fixing issues? > > >>>>> So I am seeing a number of factors here. > > >>>>> > > >>>>> First, 8140520, set: > > >>>>> > > >>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > > >>>> So I'm confused by the above comment: > > >>>> > > >>>> > The problem is caused by a rather recent change (8140520: > segfault > > >>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) > > >>>> > > >>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- > > >>>> > > open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html > > >>>> > > >>>> shows this change: > > >>>> > > >>>> @@ -531,19 +531,17 @@ > > >>>> } > > >>>> > > >>>> > > >>>> > > >> > > ////////////////////////////////////////////////////////////////////////////// > > >>>> // > > >>>> // thread stack > > >>>> > > >>>> -size_t os::Linux::min_stack_allowed = 128*K; > > >>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > > >>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; > > >>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * K; > > >>>> > > >>>> so the existing single variable of 'min_stack_allowed' was > > >>>> replaced by three variables: _compiler_thread_min_stack_allowed, > > >>>> _java_thread_min_stack_allowed, and > > >>>> _vm_internal_thread_min_stack_allowed. > > >>>> > > >>>> The old single variable and the three new variables are all > > >>>> initialized to the same value (128K) so the fix for 8140520 > > >>>> did not change stack sizes for this platform. In fact, only > > >>>> one platform had a size change (Solaris X64). > > >>>> > > >>>> So I'm confused about how the fix for 8140520 caused this problem. > > >>>> > > >>>> Based on David's analysis below, it looks to me like this 64K stack > > >>>> guard page problem should also exist prior to the fix for 8140520. > > >>>> > > >>>> Goetz, can you please explain how 8140520 caused this problem? > > >>>> > > >>>> Dan > > >>>> > > >>>> > > >>>> > > >>>>> Second on linux PPC it is hardwired to use 2 guard pages: > > >>>>> > > >>>>> return 2 * page_size(); > > >>>>> > > >>>>> Third, you had a pagesize of 64K. > > >>>>> > > >>>>> Fourth, NPTL takes the guard space from the stack space - hence > with > > 2 > > >>>>> x 64K guard, and a 128K stack it was all consumed. > > >>>>> > > >>>>> --- > > >>>>> > > >>>>> In the proposed changes you now only use page_size() for the guard, > > so > > >>>>> that alone would have fixed the observed problem. > > >>>>> > > >>>>> But in addition you want to address the NPTL problem by adding > back > > >>>>> the guard space to the stack size requested. That alone would also > > >>>>> have fixed the observed problem. :) > > >>>>> > > >>>>> But in addition you have increased the minimum stack size: > > >>>>> > > >>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; > > >>>>> > > >>>>> which again, on its own would have fixed the original problem. :) > > >>>>> > > >>>>> Did you really intend to increase the real minimum stack from 128K > to > > >>>>> 256K ? (on a 64K page system) > > >>>>> > > >>>>> --- > > >>>>> > > >>>>> Focusing simply on the shared code change to adjust the requested > > >>>>> stacksize by the amount of guard space (if any), this does not seem > > >>>>> unreasonable. As you note it is restricted to non-JavaThreads and > only > > >>>>> adds a page to reserved stack space. > > >>>>> > > >>>>> My only query now is whether the minimum stacksize detection logic > > >>>>> will correctly report the real minimum stack size (taking into account > > >>>>> the need for the guard page) ? > > >>>>> > > >>>>> Thanks, > > >>>>> David > > >>>>> > > >>>>>> So I really think this issue should be fixed. > > >>>>>> > > >>>>>> Best regards, > > >>>>>> Goetz. > > >>>>>> > > >>>>>> > > >>>>>>> -----Original Message----- > > >>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] > > >>>>>>> Sent: Thursday, November 10, 2016 10:02 PM > > >>>>>>> To: Lindenmaier, Goetz ; hotspot- > > >>>> runtime- > > >>>>>>> dev at openjdk.java.net > > >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack > guard > > >> error. > > >>>>>>> > > >>>>>>> Hi Goetz, > > >>>>>>> > > >>>>>>> As per the bug report, this issue was already known (6675312) and > > we > > >>>>>>> chose not to try and address it due to no reported issues at the > > time. > > >>>>>>> While I see that you have encountered an issue (is it real or > > >>>>>>> fabricated?) I think this change is too intrusive to be applied at this > > >>>>>>> stage of the JDK 9 release cycle, as it will change the stack > > >>>>>>> requirements of every application running on Linux. > > >>>>>>> > > >>>>>>> Thanks, > > >>>>>>> David > > >>>>>>> > > >>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > > >>>>>>>> Hi, > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Please review this change. I please need a sponsor: > > >>>>>>>> > > >>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > > >>>> stackFix/webrev.01/ > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> In the Linux NPTL pthread implementation the guard size > > mechanism > > >>>>>>>> is not > > >>>>>>>> implemented properly. The posix standard requires to add the > > size > > >>>>>>>> of the > > >>>>>>>> guard pages to the stack size, instead Linux takes the space out > of > > >>>>>>>> 'stacksize'. > > >>>>>>>> > > >>>>>>>> The Posix standard > > >>>> http://pubs.opengroup.org/onlinepubs/9699919799/ > > >>>>>>>> says "the implementation allocates extra memory at the > > overflow > > >>>>>>>> end of > > >>>>>>>> the stack". The linux man page > > >>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As > > at > > >>>>>>>> glibc > > >>>>>>>> 2.8, the NPTL threading implementation includes the guard area > > >> within > > >>>>>>>> the stack size allocation, rather than allocating extra space at > > >>>>>>>> the end > > >>>>>>>> of the stack, as POSIX.1 requires". > > >>>>>>>> > > >>>>>>>> I encounter this problem in > > runtime/Thread/TooSmallStackSize.java > > >>>>>>>> on ppc > > >>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K > on > > >> ppc, > > >>>> and > > >>>>>>>> ppc specifies two OS guard pages. The VM crashes in pthread > > >> creation > > >>>>>>>> because there is no usable space in the thread stack after > > allocating > > >>>>>>>> the guard pages. > > >>>>>>>> > > >>>>>>>> But TooSmallStackSize.java requires that the VM comes up with > > the > > >>>>>>>> stack > > >>>>>>>> size mentioned in the error message. > > >>>>>>>> > > >>>>>>>> This fix adapts the requested stack size on Linux by the size of > the > > >>>>>>>> guard pages to mimick proper behaviour, see change to > > >> os_linux.cpp. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> The change also streamlines usage of stack_guard_page on > > linuxppc, > > >>>>>>>> linuxppcle, aixppc and linuxs390. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> To reproduce the error on linux_x86_64, apply below patch and > > call > > >> the > > >>>>>>>> VM with -XX:CompilerThreadStackSize=64. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> I'm still exploring why I had to choose such big compiler stacks > on > > >>>>>>>> ppc > > >>>>>>>> to get -version passing, but I wanted to send the RFR now as > > people > > >>>>>>>> obviously looked at the bug I opened (Thanks David!). > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Best regards, > > >>>>>>>> > > >>>>>>>> Goetz. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp > > >>>>>>>> > > >>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 > > >> 12:37:28 > > >>>>>>> 2016 > > >>>>>>>> +0100 > > >>>>>>>> > > >>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 > > >>>> 16:52:17 > > >>>>>>> 2016 > > >>>>>>>> +0100 > > >>>>>>>> > > >>>>>>>> @@ -701,7 +701,7 @@ > > >>>>>>>> > > >>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > > >>>>>>>> > > >>>>>>>> // Creating guard page is very expensive. Java thread has > > HotSpot > > >>>>>>>> > > >>>>>>>> // guard page, only enable glibc guard page for non-Java > > threads. > > >>>>>>>> > > >>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); > > >>>>>>>> > > >>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); > > >>>>>>>> > > >>>>>>>> } > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> // Java thread: > > >>>>>>>> > > > From marcus.larsson at oracle.com Thu Nov 24 14:35:37 2016 From: marcus.larsson at oracle.com (Marcus Larsson) Date: Thu, 24 Nov 2016 15:35:37 +0100 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part In-Reply-To: <23fcd50a-eed0-52a5-8817-244ecf75bb2a@oracle.com> References: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> <85e51138-f3ce-7392-2cc3-ce7840aa3747@oracle.com> <23fcd50a-eed0-52a5-8817-244ecf75bb2a@oracle.com> Message-ID: Hi, On 11/22/2016 02:24 PM, Kirill Zhaldybin wrote: > Marcus, > > Thank you for prompt reply! > > Could you please read comments inline? > I'm looking forward to your reply. > > Thank you. > > Regards, Kirill > > On 22.11.2016 15:32, Marcus Larsson wrote: >> Hi, >> >> >> On 2016-11-21 17:38, Kirill Zhaldybin wrote: >>> Marcus, >>> >>> Thank you for reviewing the fix! >>>>> WebRev: >>>>> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ >>>> >>>> ISO8601 says the decimal point can be either '.' or ',' so the test >>>> should accept either. You could let sscanf read out the decimal >>>> point as a character and just verify that it is one of the two. >>>> >>>> In the UTC test you changed ASSERT_GT to ASSERT_EQ, which means >>>> that we won't accept "Z" suffixed strings. Please revert that. >>> I agree that ISO8601 could add "Z" to time (and as far as I >>> understand date/time without delimiters is legal too) but these are >>> the unit tests. >>> Hence they cover the existing code and they should pass only if the >>> result corresponds to existing code and fail otherwise. >>> The current code from os::iso8601_time format date/time string >>> %04d-%02d-%02dT%02d:%02d:%02d.%03d%c%02d%02d so we should not >>> consider any other format as valid. >>> >>> Could you please let me know your opinion? >> >> I think the test should verify the intended behavior, not the >> implementation. If we refactor or change something in iso8601_time() >> we shouldn't be failing the test if it still conforms to ISO8601, IMO. > I would agree with you if we were talking about a functional test. But > since it is an unit test I think we should keep it as close to > implementation as possible. > If the implementation is changed unintentionally the test fails and > signals us that something is broken. > If it is an intentional change the test must be updated correspondingly. I still think it's unnecessary noise, but if you insist I'm fine with it. If we're not going to accept anything else than the current implementation then you should also remove the if-case for the Z suffix, since the test will fail for that anyway. Thanks, Marcus > >> >> Thanks, >> Marcus >> >>> >>> Thank you. >>> >>> Regards, Kirill >>> >>>> >>>> Thanks, >>>> Marcus >>>> >>>>> CR: https://bugs.openjdk.java.net/browse/JDK-8169003 >>>>> >>>>> Thank you. >>>>> >>>>> Regards, Kirill >>>> >>> >> > From Alan.Bateman at oracle.com Thu Nov 24 15:25:20 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 24 Nov 2016 15:25:20 +0000 Subject: Code review for jigsaw/jake -> jdk9/dev sync up Message-ID: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> Folks on jigsaw-dev will know that we are on a mission to bring the changes accumulated in the jake forest to jdk9/dev. We can think of this as a refresh of the module system in JDK 9, the last big refresh was in May with many small updates since then. The focus this time is to bring the changes that are tied to JSR issues into jdk9/dev, specifically the issues that are tracked on the JSR issues list [1] as: #CompileTimeDependences #AddExportsInManifest #ClassFileModuleName #ClassFileAccPublic #ServiceLoaderEnhancements #ResourceEncapsulation/#ClassFilesAsResources #ReflectiveAccessToNonExportedTypes #AwkwardStrongEncapsulation #ReadabilityAddedByLayerCreator #IndirectQualifiedReflectiveAccess (partial) #VersionsInModuleNames #NonHierarchicalLayers #ModuleAnnotations/#ModuleDeprecation #ReflectiveAccessByInstrumentationAgents Some of these issues are not "Resolved" yet, meaning there is still ongoing discussion on the EG mailing list. That is okay, there is nothing final here. If there are changes to these proposals then the implementation changes will follow. Also, as I said in a mail to jigsaw-dev yesterday [2], is that we will keep the jake forest open for ongoing prototyping and iteration, also ongoing implementation improvements where iteration or bake time is important. For the code review then the focus is therefore on sanity checking the changes that we would like to bring into jdk9/dev. We will not use this review thread to debate alternative designs or other big implementation changes that are more appropriate to bake in jake. To get going, I've put the webrevs with a snapshot of the changes in jake here: http://cr.openjdk.java.net/~alanb/8169069/0/ The changes are currently sync'ed against jdk-9+146 and will be rebased (and re-tested) against jdk9/dev prior to integration. There are a number of small changes that need to be added to this in the coming days, I will refresh the webrev every few days to take account of these updates. A few important points to mention, even if you aren't reviewing the changes: 1. This refresh requires a new version of jtreg to run the tests. The changes for this new version are in the code-tools/jtreg repository and the plan is to tag a new build (jtreg4.2-b04) next week. Once the tag has been added then we'll update the requiredVersion property in each TEST.ROOT to force everyone to update. 2. For developers trying out modules with the main line JDK 9 builds then be aware that `requires public` changes to `requires transitive` and the `provides` clause changes to require all providers for a specific service type to be in the same clause. Also be aware that the binary form of the module declaration (module-info.class) changes so you will need to recompile any modules. 3. Those running existing code on JDK 9 and ignoring modules will need to be aware of a disruptive change in this refresh. The disruptive change is #AwkwardStrongEncapsulation where setAccessible(true) is changed so that it can't be used to break into non-public fields/methods of JDK classes. This change is going to expose a lot of hacks in existing code. We plan to send mail to jdk9-dev in advance of this integration to create awareness of this change. As per the original introduction of strong encapsulation then command line options (and now the manifest of application JAR files) can be used to keep existing code working. The new option is `--add-opens` to open a package in a module for deep reflection by other modules. As an example, if you find yourself with code that hacks into the private `comparator` field in java.util.TreeMap then running with `--add-opens java.base/java.util=ALL-UNNAMED` will keep that code working. A few miscellaneous notes for those that are reviewing: 1. We have some temporary/transition code in the top-level repo to deal with the importing of the JavaFX modules. This will be removed once the changes are in JDK 9 for the OpenJFX project to use. 2. In the jdk repo then it's important to understand that the module system is initialized at startup and there are many places where we need to keep startup performance in mind. This sometimes means less elegant code than might be used if startup wasn't such a big concern. 3. The changes in the jaxws repo make use of new APIs that means the code doesn't compile with JDK 7 or JDK 8. Our intention is to work with the JAXB and JAX-WS maintainers to address the issues in the upstream project and then bring those changes into jdk9/dev to replace the patches that we are forced to push for the short term. 4. You will see several tests where the value of the @modules tag has `:open` or `:+open`. This is new jtreg speak. The former means the test is run with --add-opens to open the package, the latter means the test is exported at compile-time and exported + open at run-time (the latter usage will be rare, it's where tests have static references to JDK internal types and are also doing deep reflection with setAccessible). In terms of dates then we are aiming to integrate these changes into jdk9/dev in early December. I will send a follow-up mail next week on this as we work through the logistics. -Alan [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html From kirill.zhaldybin at oracle.com Fri Nov 25 17:23:52 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Fri, 25 Nov 2016 20:23:52 +0300 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part In-Reply-To: References: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> <85e51138-f3ce-7392-2cc3-ce7840aa3747@oracle.com> <23fcd50a-eed0-52a5-8817-244ecf75bb2a@oracle.com> Message-ID: <583873A8.8000106@oracle.com> Marcus, Here are a new webrev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.01/ I addressed your comment about "if-case for the Z suffix". Could you please let me know your opinion? Thank you. Regards, Kirill On 24.11.2016 17:35, Marcus Larsson wrote: > Hi, > > > On 11/22/2016 02:24 PM, Kirill Zhaldybin wrote: >> Marcus, >> >> Thank you for prompt reply! >> >> Could you please read comments inline? >> I'm looking forward to your reply. >> >> Thank you. >> >> Regards, Kirill >> >> On 22.11.2016 15:32, Marcus Larsson wrote: >>> Hi, >>> >>> >>> On 2016-11-21 17:38, Kirill Zhaldybin wrote: >>>> Marcus, >>>> >>>> Thank you for reviewing the fix! >>>>>> WebRev: >>>>>> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ >>>>> >>>>> ISO8601 says the decimal point can be either '.' or ',' so the test >>>>> should accept either. You could let sscanf read out the decimal >>>>> point as a character and just verify that it is one of the two. >>>>> >>>>> In the UTC test you changed ASSERT_GT to ASSERT_EQ, which means >>>>> that we won't accept "Z" suffixed strings. Please revert that. >>>> I agree that ISO8601 could add "Z" to time (and as far as I >>>> understand date/time without delimiters is legal too) but these are >>>> the unit tests. >>>> Hence they cover the existing code and they should pass only if the >>>> result corresponds to existing code and fail otherwise. >>>> The current code from os::iso8601_time format date/time string >>>> %04d-%02d-%02dT%02d:%02d:%02d.%03d%c%02d%02d so we should not >>>> consider any other format as valid. >>>> >>>> Could you please let me know your opinion? >>> >>> I think the test should verify the intended behavior, not the >>> implementation. If we refactor or change something in iso8601_time() >>> we shouldn't be failing the test if it still conforms to ISO8601, IMO. >> I would agree with you if we were talking about a functional test. But >> since it is an unit test I think we should keep it as close to >> implementation as possible. >> If the implementation is changed unintentionally the test fails and >> signals us that something is broken. >> If it is an intentional change the test must be updated correspondingly. > > I still think it's unnecessary noise, but if you insist I'm fine with it. > > If we're not going to accept anything else than the current > implementation then you should also remove the if-case for the Z suffix, > since the test will fail for that anyway. > > Thanks, > Marcus > >> >>> >>> Thanks, >>> Marcus >>> >>>> >>>> Thank you. >>>> >>>> Regards, Kirill >>>> >>>>> >>>>> Thanks, >>>>> Marcus >>>>> >>>>>> CR: https://bugs.openjdk.java.net/browse/JDK-8169003 >>>>>> >>>>>> Thank you. >>>>>> >>>>>> Regards, Kirill >>>>> >>>> >>> >> > From kirill.zhaldybin at oracle.com Fri Nov 25 19:55:56 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Fri, 25 Nov 2016 22:55:56 +0300 Subject: RFR(S): 8166156: Convert test_semaphore to GTest Message-ID: <5838974C.9000603@oracle.com> Dear all, Could you please review this fix for 8166156? Pretty straight-forward conversion. WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.00/ CR: https://bugs.openjdk.java.net/browse/JDK-8166156 Thank you. Regards, Kirill From kim.barrett at oracle.com Sat Nov 26 05:25:13 2016 From: kim.barrett at oracle.com (Kim Barrett) Date: Sat, 26 Nov 2016 00:25:13 -0500 Subject: RFR(S): 8166156: Convert test_semaphore to GTest In-Reply-To: <5838974C.9000603@oracle.com> References: <5838974C.9000603@oracle.com> Message-ID: <3878A570-5ECF-4D81-87D1-A17DFFF5ABFF@oracle.com> > On Nov 25, 2016, at 2:55 PM, Kirill Zhaldybin wrote: > > Dear all, > > Could you please review this fix for 8166156? > Pretty straight-forward conversion. > > WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.00/ > CR: https://bugs.openjdk.java.net/browse/JDK-8166156 > > Thank you. > > Regards, Kirill Looks good. From kirill.zhaldybin at oracle.com Sat Nov 26 05:56:46 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldbybin) Date: Sat, 26 Nov 2016 08:56:46 +0300 Subject: RFR(S): 8166156: Convert test_semaphore to GTest In-Reply-To: <3878A570-5ECF-4D81-87D1-A17DFFF5ABFF@oracle.com> References: <5838974C.9000603@oracle.com> <3878A570-5ECF-4D81-87D1-A17DFFF5ABFF@oracle.com> Message-ID: <5839241E.2030802@oracle.com> Kim, Thank you for review! Regards, Kirill On 11/26/2016 08:25 AM, Kim Barrett wrote: >> On Nov 25, 2016, at 2:55 PM, Kirill Zhaldybin wrote: >> >> Dear all, >> >> Could you please review this fix for 8166156? >> Pretty straight-forward conversion. >> >> WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.00/ >> CR: https://bugs.openjdk.java.net/browse/JDK-8166156 >> >> Thank you. >> >> Regards, Kirill > Looks good. > From david.holmes at oracle.com Sun Nov 27 23:24:43 2016 From: david.holmes at oracle.com (David Holmes) Date: Mon, 28 Nov 2016 09:24:43 +1000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> Message-ID: <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> Hi Goetz, On 24/11/2016 10:15 PM, Lindenmaier, Goetz wrote: > Hi, > > I now edited the stuff I had proposed below: > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.03/ > This includes > - the NPTL fix from webrev.02 Okay in principle. As discussed this only impacts non-JavaThreads so the change should be minimal. > - merging code on linux Went a bit further than I had expected but if this truly isn't CPU dependent code then great! > - not adding OS guard to compiler threads. Okay in principle. IIUC we will now save the OS guard page for compiler thread stacks. Is that the only impact? The hotspot-compiler-dev folk may want to sign off on this part. A few minor comments: src/os/linux/vm/os_linux.cpp 2854 return ((thr_type == java_thread || thr_type == os::compiler_thread) ... os:: should be used for both types or none. 6153 pthread_attr_getguardsize(&attr, &guard_size); Can you at least verify a zero return code in an assert/assert_status please. --- src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp src/os_cpu/linux_s390/vm/os_linux_s390.cpp Are the changes to min_stack_allowed just fixing an existing bug? --- Thanks, David ----- > I think this should be pushed for this bug ID. For the other changes I'll > make another bug. > > Best regards, > Goetz. > >> -----Original Message----- >> From: Lindenmaier, Goetz >> Sent: Wednesday, November 23, 2016 8:11 AM >> To: David Holmes ; >> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net >> Subject: RE: RFR(M): 8169373: Work around linux NPTL stack guard error. >> >> Hi, >> >>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The >>> ability for a CompilerThread to execute Java code (can_call_java()) is >>> now a dynamic property depending on whether the current compiler is the >>> JVMCI compiler. >> Ah, then I should also leave space for shadow pages in the minimal stack size >> of comiler threads. >> >> Do we agree on the cleanup and on leaving out the OS guard page on >> compiler threads? >> Then I would edit a change comprising the NPTL workaround and these >> additional changes, and split the other issue into a new bug? I think this >> will easy the reviewing. >> >> Best regards, >> Goetz. >> >>> -----Original Message----- >>> From: David Holmes [mailto:david.holmes at oracle.com] >>> Sent: Mittwoch, 23. November 2016 02:50 >>> To: Lindenmaier, Goetz ; >>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net >>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>> >>> On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: >>>> Hi Dan, >>>> >>>>> -----Original Message----- >>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>>> Sent: Dienstag, 22. November 2016 14:01 >>>>> To: Lindenmaier, Goetz ; David Holmes >>>>> ; hotspot-runtime-dev at openjdk.java.net >>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >> error. >>>>> >>>>> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: >>>>>> Hi Dan, >>>>>> >>>>>> I ran into a row of issues, some errors on the platforms. >>>>>> >>>>>> What I meant with that comment is that >>>>>> os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, >>>>>> JavaThread::stack_guard_zone_size() + >>>>>> JavaThread::stack_shadow_zone_size() + >>>>>> (4*BytesPerWord COMPILER2_PRESENT(+2)) * 4 >> * >>> K); >>>>>> was executed, and min_stack_allowed used for all stacks. Now, >> compiler >>>>> and >>>>>> vm minimum stack sizes are not increased by these sizes. >>>>> >>>>> Now I see what you mean. Thanks for clearing this up. >>>>> >>>>> I should have remembered that part of the change because we went >> back >>>>> and forth about removing the red/yellow zone pages from the other >>>>> threads. In particular, we discussed the compiler thread because it >>>>> is-a JavaThread. Our conclusion was that a compiler thread doesn't >>>>> execute Java bytecode so we could remove the red/yellow pages... >>>> Yes, it does not execute java byte code. >>> >>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The >>> ability for a CompilerThread to execute Java code (can_call_java()) is >>> now a dynamic property depending on whether the current compiler is the >>> JVMCI compiler. >>> >>> David >>> ----- >>> >>>> Therefore you can remove the shadow pages. There is no code that >>>> will bang. >>>> But red/yellow pages are protected right during thread startup. >>>> Therefore you must have enough space for them. >>>> On ppc, we try to protect three 64K pages out of the 128K compiler stack. >>>> That obviously fails. >>>> >>>> Therefore I propose: >>>> size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // Set >>> platform dependent. >>>> in os::Posix::set_minimum_stack_sizes(): >>>> _java_thread_min_stack_allowed = _java_thread_min_stack_allowed + >>>> JavaThread::stack_guard_zone_size() + >>>> JavaThread::stack_shadow_zone_size(); >>>> >>>> (Similar for _compiler_thread_min_stack_allowed). >>>> >>>> The minimal stack size is made up of three components: >>>> 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot >>> implementation/platform/os. >>>> 2. Sizes of C++ frames: depends on C++ compiler. >>>> These are fixed at compile time. >>>> 3. Sizes of red/yellow/reserved/shadow pages. Depends on the system >>> the >>>> VM is used on. This is not fixed at compile time. (Our ppc machines >>> differ >>>> in page size.) >>>> Therefore 3. should not be included in a compile time constant. >>>> >>>> >>>> >>>>> And that decision allowed us to be exposed to the 64K page issue >>>>> because the "extra" space isn't there anymore. >>>>> >>>>>> At least the _compiler_thread_min_stack_allowed should be increased >>>>>> similarly by red/yellow zone size. The compiler thread is a Java >>>>>> thread and must have space for these zones. >>>>> >>>>> I'm not sure that I completely agree (yet). To me, the red/yellow >>>>> pages are there for Java thread stack overflow semantics. Yes, the >>>>> compiler thread needs extra space when 64K pages are used, but I >>>>> would prefer that we add that space via a different calculation. >>>> Yes they are. But compiler threads happen tob e a subclass of >>>> Java threads and use the same run() method that puts the pages >>>> There. >>>> I agree that they are not needed for Compiler threads, nor for >>>> CodeCacheSweeperThreads. I don't really now about >>>> JvmtiAgentThreads and ServiceThreads, but all of the get the guard >> pages >>>> because they are derived from JavaThread. >>>> >>>>> To put it another way, I'd like to see us add extra space to solve >>>>> the 64K page issue directly instead of as a side effect of the >>>>> red/yellow page addition. >>>> I don't understand. What do you mean by 'directly'? >>>> >>>>>> Also, the change added a test that is failing now. >>>>> >>>>> And that's a "good thing" (TM), right? :-) >>>> Yes, it showed a bug and thus raised the need to fix it! :) >>>> >>>> Best regards, >>>> Goetz. >>>> >>>> >>>>> Again, thanks for clarifying 8140520's role in this issue. >>>>> >>>>> Dan >>>>> >>>>> >>>>>> >>>>>> Best regards, >>>>>> Goetz. >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>>>>> Sent: Montag, 21. November 2016 17:28 >>>>>>> To: David Holmes ; Lindenmaier, Goetz >>>>>>> ; hotspot-runtime- >>> dev at openjdk.java.net >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >>> error. >>>>>>> >>>>>>> Sorry for the delayed responses to this thread. I've been on vacation... >>>>>>> >>>>>>> One comment/query embedded below... >>>>>>> >>>>>>> >>>>>>> On 11/10/16 8:40 PM, David Holmes wrote: >>>>>>>> Hi Goetz, >>>>>>>> >>>>>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: >>>>>>>>> Hi David, >>>>>>>>> >>>>>>>>> This issue is different to 6675312, see also my comment in the bug. >>>>>>>>> >>>>>>>>> It appears running jtreg test >>> runtime/Thread/TooSmallStackSize.java, >>>>>>>>> with my patch below you can reproduce it on linuxx86_64. You can >>> not >>>>>>>>> do that with 6675312. Also, I would assume there are systems out >>> there >>>>>>>>> on x86 that uses 64-K pages, did you run the tests on these? I >> would >>>>>>>>> assume you get hard crashes with stack overflows in the first C2 >>>>>>>>> compilation if there is only 64K usable CompilerThreadStack. >>>>>>>>> >>>>>>>>> My fix does not affect Java threads, which are the largest amount >>>>>>>>> of threads used by the VM. It affects only the non-Java threads. >>>>>>>>> It adds one page to these threads. The page does not require >>> memory, >>>>>>>>> as it's protected. The stack will only require more space if the >> thread >>>>>>>>> ran into a stack overflow before the fix as else the pages are not >>>>>>>>> mapped. >>>>>>>>> This are stack overflows that cause hard crashes, at least on ppc >> the >>> VM >>>>>>>>> does not properly catch these stack overflows, so any setup >> working >>>>> now >>>>>>>>> will not run into the additional space. Altogether there should be >> no >>>>>>>>> effect on running systems besides requiring one more entry in the >>>>>>>>> page table per non-Java thread. >>>>>>>>> >>>>>>>>> The problem is caused by a rather recent change (8140520: >> segfault >>> on >>>>>>>>> solaris-amd64 >>>>>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed after >>>>>>>>> feature-close. As this was a rather recent change, it must be >>>>>>>>> possible to >>>>>>>>> fix this follow up issue. What else is this period in the project good >>>>>>>>> for if not fixing issues? >>>>>>>> So I am seeing a number of factors here. >>>>>>>> >>>>>>>> First, 8140520, set: >>>>>>>> >>>>>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >>>>>>> So I'm confused by the above comment: >>>>>>> >>>>>>> > The problem is caused by a rather recent change (8140520: >> segfault >>>>>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) >>>>>>> >>>>>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- >>>>>>> >>> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html >>>>>>> >>>>>>> shows this change: >>>>>>> >>>>>>> @@ -531,19 +531,17 @@ >>>>>>> } >>>>>>> >>>>>>> >>>>>>> >>>>> >>> ////////////////////////////////////////////////////////////////////////////// >>>>>>> // >>>>>>> // thread stack >>>>>>> >>>>>>> -size_t os::Linux::min_stack_allowed = 128*K; >>>>>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >>>>>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; >>>>>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * K; >>>>>>> >>>>>>> so the existing single variable of 'min_stack_allowed' was >>>>>>> replaced by three variables: _compiler_thread_min_stack_allowed, >>>>>>> _java_thread_min_stack_allowed, and >>>>>>> _vm_internal_thread_min_stack_allowed. >>>>>>> >>>>>>> The old single variable and the three new variables are all >>>>>>> initialized to the same value (128K) so the fix for 8140520 >>>>>>> did not change stack sizes for this platform. In fact, only >>>>>>> one platform had a size change (Solaris X64). >>>>>>> >>>>>>> So I'm confused about how the fix for 8140520 caused this problem. >>>>>>> >>>>>>> Based on David's analysis below, it looks to me like this 64K stack >>>>>>> guard page problem should also exist prior to the fix for 8140520. >>>>>>> >>>>>>> Goetz, can you please explain how 8140520 caused this problem? >>>>>>> >>>>>>> Dan >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Second on linux PPC it is hardwired to use 2 guard pages: >>>>>>>> >>>>>>>> return 2 * page_size(); >>>>>>>> >>>>>>>> Third, you had a pagesize of 64K. >>>>>>>> >>>>>>>> Fourth, NPTL takes the guard space from the stack space - hence >> with >>> 2 >>>>>>>> x 64K guard, and a 128K stack it was all consumed. >>>>>>>> >>>>>>>> --- >>>>>>>> >>>>>>>> In the proposed changes you now only use page_size() for the guard, >>> so >>>>>>>> that alone would have fixed the observed problem. >>>>>>>> >>>>>>>> But in addition you want to address the NPTL problem by adding >> back >>>>>>>> the guard space to the stack size requested. That alone would also >>>>>>>> have fixed the observed problem. :) >>>>>>>> >>>>>>>> But in addition you have increased the minimum stack size: >>>>>>>> >>>>>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; >>>>>>>> >>>>>>>> which again, on its own would have fixed the original problem. :) >>>>>>>> >>>>>>>> Did you really intend to increase the real minimum stack from 128K >> to >>>>>>>> 256K ? (on a 64K page system) >>>>>>>> >>>>>>>> --- >>>>>>>> >>>>>>>> Focusing simply on the shared code change to adjust the requested >>>>>>>> stacksize by the amount of guard space (if any), this does not seem >>>>>>>> unreasonable. As you note it is restricted to non-JavaThreads and >> only >>>>>>>> adds a page to reserved stack space. >>>>>>>> >>>>>>>> My only query now is whether the minimum stacksize detection logic >>>>>>>> will correctly report the real minimum stack size (taking into account >>>>>>>> the need for the guard page) ? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> David >>>>>>>> >>>>>>>>> So I really think this issue should be fixed. >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> Goetz. >>>>>>>>> >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>>>>>> Sent: Thursday, November 10, 2016 10:02 PM >>>>>>>>>> To: Lindenmaier, Goetz ; hotspot- >>>>>>> runtime- >>>>>>>>>> dev at openjdk.java.net >>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack >> guard >>>>> error. >>>>>>>>>> >>>>>>>>>> Hi Goetz, >>>>>>>>>> >>>>>>>>>> As per the bug report, this issue was already known (6675312) and >>> we >>>>>>>>>> chose not to try and address it due to no reported issues at the >>> time. >>>>>>>>>> While I see that you have encountered an issue (is it real or >>>>>>>>>> fabricated?) I think this change is too intrusive to be applied at this >>>>>>>>>> stage of the JDK 9 release cycle, as it will change the stack >>>>>>>>>> requirements of every application running on Linux. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> David >>>>>>>>>> >>>>>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Please review this change. I please need a sponsor: >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >>>>>>> stackFix/webrev.01/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> In the Linux NPTL pthread implementation the guard size >>> mechanism >>>>>>>>>>> is not >>>>>>>>>>> implemented properly. The posix standard requires to add the >>> size >>>>>>>>>>> of the >>>>>>>>>>> guard pages to the stack size, instead Linux takes the space out >> of >>>>>>>>>>> 'stacksize'. >>>>>>>>>>> >>>>>>>>>>> The Posix standard >>>>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ >>>>>>>>>>> says "the implementation allocates extra memory at the >>> overflow >>>>>>>>>>> end of >>>>>>>>>>> the stack". The linux man page >>>>>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As >>> at >>>>>>>>>>> glibc >>>>>>>>>>> 2.8, the NPTL threading implementation includes the guard area >>>>> within >>>>>>>>>>> the stack size allocation, rather than allocating extra space at >>>>>>>>>>> the end >>>>>>>>>>> of the stack, as POSIX.1 requires". >>>>>>>>>>> >>>>>>>>>>> I encounter this problem in >>> runtime/Thread/TooSmallStackSize.java >>>>>>>>>>> on ppc >>>>>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K >> on >>>>> ppc, >>>>>>> and >>>>>>>>>>> ppc specifies two OS guard pages. The VM crashes in pthread >>>>> creation >>>>>>>>>>> because there is no usable space in the thread stack after >>> allocating >>>>>>>>>>> the guard pages. >>>>>>>>>>> >>>>>>>>>>> But TooSmallStackSize.java requires that the VM comes up with >>> the >>>>>>>>>>> stack >>>>>>>>>>> size mentioned in the error message. >>>>>>>>>>> >>>>>>>>>>> This fix adapts the requested stack size on Linux by the size of >> the >>>>>>>>>>> guard pages to mimick proper behaviour, see change to >>>>> os_linux.cpp. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The change also streamlines usage of stack_guard_page on >>> linuxppc, >>>>>>>>>>> linuxppcle, aixppc and linuxs390. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> To reproduce the error on linux_x86_64, apply below patch and >>> call >>>>> the >>>>>>>>>>> VM with -XX:CompilerThreadStackSize=64. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I'm still exploring why I had to choose such big compiler stacks >> on >>>>>>>>>>> ppc >>>>>>>>>>> to get -version passing, but I wanted to send the RFR now as >>> people >>>>>>>>>>> obviously looked at the bug I opened (Thanks David!). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best regards, >>>>>>>>>>> >>>>>>>>>>> Goetz. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>>>>>>>>>> >>>>>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 >>>>> 12:37:28 >>>>>>>>>> 2016 >>>>>>>>>>> +0100 >>>>>>>>>>> >>>>>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 >>>>>>> 16:52:17 >>>>>>>>>> 2016 >>>>>>>>>>> +0100 >>>>>>>>>>> >>>>>>>>>>> @@ -701,7 +701,7 @@ >>>>>>>>>>> >>>>>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>>>>>>>>>> >>>>>>>>>>> // Creating guard page is very expensive. Java thread has >>> HotSpot >>>>>>>>>>> >>>>>>>>>>> // guard page, only enable glibc guard page for non-Java >>> threads. >>>>>>>>>>> >>>>>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); >>>>>>>>>>> >>>>>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> // Java thread: >>>>>>>>>>> >>>> From goetz.lindenmaier at sap.com Mon Nov 28 08:36:34 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Mon, 28 Nov 2016 08:36:34 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> Message-ID: <76aaef03caaf43c99247c948c2acae57@DEROTE13DE08.global.corp.sap> Hi David, thanks for looking at the webrev again! New webrev: http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.04/ > Okay in principle. IIUC we will now save the OS guard page for compiler > thread stacks. Is that the only impact? The hotspot-compiler-dev folk > may want to sign off on this part. Yes, that's the only impact. I'll post to hotspot-compier. > src/os/linux/vm/os_linux.cpp os:: should be used for both types or none. I removed os:: . > 6153 pthread_attr_getguardsize(&attr, &guard_size); > Can you at least verify a zero return code in an assert/assert_status > please. Yes, I should check the error. I would even do a fatal, as this is done above, too. > src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp > src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp > src/os_cpu/linux_s390/vm/os_linux_s390.cpp > Are the changes to min_stack_allowed just fixing an existing bug? These are the numbers I need to get through the test added in 8140520. I can reduce them again with the followup change I intend to make. But the way minimum size checks are implemented now it needs to be that big. Very bad for 4K page ppc systems. Best regards, Goetz. > -----Original Message----- > From: David Holmes [mailto:david.holmes at oracle.com] > Sent: Montag, 28. November 2016 00:25 > To: Lindenmaier, Goetz ; > 'daniel.daugherty at oracle.com' ; 'hotspot- > runtime-dev at openjdk.java.net' > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > Hi Goetz, > > On 24/11/2016 10:15 PM, Lindenmaier, Goetz wrote: > > Hi, > > > > I now edited the stuff I had proposed below: > > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.03/ > > This includes > > - the NPTL fix from webrev.02 > > Okay in principle. As discussed this only impacts non-JavaThreads so the > change should be minimal. > > > - merging code on linux > > Went a bit further than I had expected but if this truly isn't CPU > dependent code then great! > > > - not adding OS guard to compiler threads. > > Okay in principle. IIUC we will now save the OS guard page for compiler > thread stacks. Is that the only impact? The hotspot-compiler-dev folk > may want to sign off on this part. > > > A few minor comments: > > src/os/linux/vm/os_linux.cpp > > 2854 return ((thr_type == java_thread || thr_type == > os::compiler_thread) ... > > os:: should be used for both types or none. > > 6153 pthread_attr_getguardsize(&attr, &guard_size); > > Can you at least verify a zero return code in an assert/assert_status > please. > > --- > > src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp > src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp > src/os_cpu/linux_s390/vm/os_linux_s390.cpp > > Are the changes to min_stack_allowed just fixing an existing bug? > > --- > > Thanks, > David > ----- > > > > I think this should be pushed for this bug ID. For the other changes I'll > > make another bug. > > > > Best regards, > > Goetz. > > > >> -----Original Message----- > >> From: Lindenmaier, Goetz > >> Sent: Wednesday, November 23, 2016 8:11 AM > >> To: David Holmes ; > >> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > >> Subject: RE: RFR(M): 8169373: Work around linux NPTL stack guard error. > >> > >> Hi, > >> > >>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The > >>> ability for a CompilerThread to execute Java code (can_call_java()) is > >>> now a dynamic property depending on whether the current compiler is > the > >>> JVMCI compiler. > >> Ah, then I should also leave space for shadow pages in the minimal stack > size > >> of comiler threads. > >> > >> Do we agree on the cleanup and on leaving out the OS guard page on > >> compiler threads? > >> Then I would edit a change comprising the NPTL workaround and these > >> additional changes, and split the other issue into a new bug? I think this > >> will easy the reviewing. > >> > >> Best regards, > >> Goetz. > >> > >>> -----Original Message----- > >>> From: David Holmes [mailto:david.holmes at oracle.com] > >>> Sent: Mittwoch, 23. November 2016 02:50 > >>> To: Lindenmaier, Goetz ; > >>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > >>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > >>> > >>> On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: > >>>> Hi Dan, > >>>> > >>>>> -----Original Message----- > >>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >>>>> Sent: Dienstag, 22. November 2016 14:01 > >>>>> To: Lindenmaier, Goetz ; David Holmes > >>>>> ; hotspot-runtime- > dev at openjdk.java.net > >>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > >> error. > >>>>> > >>>>> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: > >>>>>> Hi Dan, > >>>>>> > >>>>>> I ran into a row of issues, some errors on the platforms. > >>>>>> > >>>>>> What I meant with that comment is that > >>>>>> os::Linux::min_stack_allowed = > MAX2(os::Linux::min_stack_allowed, > >>>>>> JavaThread::stack_guard_zone_size() + > >>>>>> JavaThread::stack_shadow_zone_size() + > >>>>>> (4*BytesPerWord COMPILER2_PRESENT(+2)) * 4 > >> * > >>> K); > >>>>>> was executed, and min_stack_allowed used for all stacks. Now, > >> compiler > >>>>> and > >>>>>> vm minimum stack sizes are not increased by these sizes. > >>>>> > >>>>> Now I see what you mean. Thanks for clearing this up. > >>>>> > >>>>> I should have remembered that part of the change because we went > >> back > >>>>> and forth about removing the red/yellow zone pages from the other > >>>>> threads. In particular, we discussed the compiler thread because it > >>>>> is-a JavaThread. Our conclusion was that a compiler thread doesn't > >>>>> execute Java bytecode so we could remove the red/yellow pages... > >>>> Yes, it does not execute java byte code. > >>> > >>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The > >>> ability for a CompilerThread to execute Java code (can_call_java()) is > >>> now a dynamic property depending on whether the current compiler is > the > >>> JVMCI compiler. > >>> > >>> David > >>> ----- > >>> > >>>> Therefore you can remove the shadow pages. There is no code that > >>>> will bang. > >>>> But red/yellow pages are protected right during thread startup. > >>>> Therefore you must have enough space for them. > >>>> On ppc, we try to protect three 64K pages out of the 128K compiler > stack. > >>>> That obviously fails. > >>>> > >>>> Therefore I propose: > >>>> size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // Set > >>> platform dependent. > >>>> in os::Posix::set_minimum_stack_sizes(): > >>>> _java_thread_min_stack_allowed = > _java_thread_min_stack_allowed + > >>>> JavaThread::stack_guard_zone_size() + > >>>> JavaThread::stack_shadow_zone_size(); > >>>> > >>>> (Similar for _compiler_thread_min_stack_allowed). > >>>> > >>>> The minimal stack size is made up of three components: > >>>> 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot > >>> implementation/platform/os. > >>>> 2. Sizes of C++ frames: depends on C++ compiler. > >>>> These are fixed at compile time. > >>>> 3. Sizes of red/yellow/reserved/shadow pages. Depends on the > system > >>> the > >>>> VM is used on. This is not fixed at compile time. (Our ppc machines > >>> differ > >>>> in page size.) > >>>> Therefore 3. should not be included in a compile time constant. > >>>> > >>>> > >>>> > >>>>> And that decision allowed us to be exposed to the 64K page issue > >>>>> because the "extra" space isn't there anymore. > >>>>> > >>>>>> At least the _compiler_thread_min_stack_allowed should be > increased > >>>>>> similarly by red/yellow zone size. The compiler thread is a Java > >>>>>> thread and must have space for these zones. > >>>>> > >>>>> I'm not sure that I completely agree (yet). To me, the red/yellow > >>>>> pages are there for Java thread stack overflow semantics. Yes, the > >>>>> compiler thread needs extra space when 64K pages are used, but I > >>>>> would prefer that we add that space via a different calculation. > >>>> Yes they are. But compiler threads happen tob e a subclass of > >>>> Java threads and use the same run() method that puts the pages > >>>> There. > >>>> I agree that they are not needed for Compiler threads, nor for > >>>> CodeCacheSweeperThreads. I don't really now about > >>>> JvmtiAgentThreads and ServiceThreads, but all of the get the guard > >> pages > >>>> because they are derived from JavaThread. > >>>> > >>>>> To put it another way, I'd like to see us add extra space to solve > >>>>> the 64K page issue directly instead of as a side effect of the > >>>>> red/yellow page addition. > >>>> I don't understand. What do you mean by 'directly'? > >>>> > >>>>>> Also, the change added a test that is failing now. > >>>>> > >>>>> And that's a "good thing" (TM), right? :-) > >>>> Yes, it showed a bug and thus raised the need to fix it! :) > >>>> > >>>> Best regards, > >>>> Goetz. > >>>> > >>>> > >>>>> Again, thanks for clarifying 8140520's role in this issue. > >>>>> > >>>>> Dan > >>>>> > >>>>> > >>>>>> > >>>>>> Best regards, > >>>>>> Goetz. > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >>>>>>> Sent: Montag, 21. November 2016 17:28 > >>>>>>> To: David Holmes ; Lindenmaier, > Goetz > >>>>>>> ; hotspot-runtime- > >>> dev at openjdk.java.net > >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > >>> error. > >>>>>>> > >>>>>>> Sorry for the delayed responses to this thread. I've been on > vacation... > >>>>>>> > >>>>>>> One comment/query embedded below... > >>>>>>> > >>>>>>> > >>>>>>> On 11/10/16 8:40 PM, David Holmes wrote: > >>>>>>>> Hi Goetz, > >>>>>>>> > >>>>>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > >>>>>>>>> Hi David, > >>>>>>>>> > >>>>>>>>> This issue is different to 6675312, see also my comment in the > bug. > >>>>>>>>> > >>>>>>>>> It appears running jtreg test > >>> runtime/Thread/TooSmallStackSize.java, > >>>>>>>>> with my patch below you can reproduce it on linuxx86_64. You > can > >>> not > >>>>>>>>> do that with 6675312. Also, I would assume there are systems > out > >>> there > >>>>>>>>> on x86 that uses 64-K pages, did you run the tests on these? I > >> would > >>>>>>>>> assume you get hard crashes with stack overflows in the first C2 > >>>>>>>>> compilation if there is only 64K usable CompilerThreadStack. > >>>>>>>>> > >>>>>>>>> My fix does not affect Java threads, which are the largest > amount > >>>>>>>>> of threads used by the VM. It affects only the non-Java threads. > >>>>>>>>> It adds one page to these threads. The page does not require > >>> memory, > >>>>>>>>> as it's protected. The stack will only require more space if the > >> thread > >>>>>>>>> ran into a stack overflow before the fix as else the pages are not > >>>>>>>>> mapped. > >>>>>>>>> This are stack overflows that cause hard crashes, at least on ppc > >> the > >>> VM > >>>>>>>>> does not properly catch these stack overflows, so any setup > >> working > >>>>> now > >>>>>>>>> will not run into the additional space. Altogether there should be > >> no > >>>>>>>>> effect on running systems besides requiring one more entry in > the > >>>>>>>>> page table per non-Java thread. > >>>>>>>>> > >>>>>>>>> The problem is caused by a rather recent change (8140520: > >> segfault > >>> on > >>>>>>>>> solaris-amd64 > >>>>>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed > after > >>>>>>>>> feature-close. As this was a rather recent change, it must be > >>>>>>>>> possible to > >>>>>>>>> fix this follow up issue. What else is this period in the project > good > >>>>>>>>> for if not fixing issues? > >>>>>>>> So I am seeing a number of factors here. > >>>>>>>> > >>>>>>>> First, 8140520, set: > >>>>>>>> > >>>>>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >>>>>>> So I'm confused by the above comment: > >>>>>>> > >>>>>>> > The problem is caused by a rather recent change (8140520: > >> segfault > >>>>>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) > >>>>>>> > >>>>>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- > >>>>>>> > >>> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html > >>>>>>> > >>>>>>> shows this change: > >>>>>>> > >>>>>>> @@ -531,19 +531,17 @@ > >>>>>>> } > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>> > ////////////////////////////////////////////////////////////////////////////// > >>>>>>> // > >>>>>>> // thread stack > >>>>>>> > >>>>>>> -size_t os::Linux::min_stack_allowed = 128*K; > >>>>>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >>>>>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; > >>>>>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 > * K; > >>>>>>> > >>>>>>> so the existing single variable of 'min_stack_allowed' was > >>>>>>> replaced by three variables: > _compiler_thread_min_stack_allowed, > >>>>>>> _java_thread_min_stack_allowed, and > >>>>>>> _vm_internal_thread_min_stack_allowed. > >>>>>>> > >>>>>>> The old single variable and the three new variables are all > >>>>>>> initialized to the same value (128K) so the fix for 8140520 > >>>>>>> did not change stack sizes for this platform. In fact, only > >>>>>>> one platform had a size change (Solaris X64). > >>>>>>> > >>>>>>> So I'm confused about how the fix for 8140520 caused this problem. > >>>>>>> > >>>>>>> Based on David's analysis below, it looks to me like this 64K stack > >>>>>>> guard page problem should also exist prior to the fix for 8140520. > >>>>>>> > >>>>>>> Goetz, can you please explain how 8140520 caused this problem? > >>>>>>> > >>>>>>> Dan > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Second on linux PPC it is hardwired to use 2 guard pages: > >>>>>>>> > >>>>>>>> return 2 * page_size(); > >>>>>>>> > >>>>>>>> Third, you had a pagesize of 64K. > >>>>>>>> > >>>>>>>> Fourth, NPTL takes the guard space from the stack space - hence > >> with > >>> 2 > >>>>>>>> x 64K guard, and a 128K stack it was all consumed. > >>>>>>>> > >>>>>>>> --- > >>>>>>>> > >>>>>>>> In the proposed changes you now only use page_size() for the > guard, > >>> so > >>>>>>>> that alone would have fixed the observed problem. > >>>>>>>> > >>>>>>>> But in addition you want to address the NPTL problem by adding > >> back > >>>>>>>> the guard space to the stack size requested. That alone would also > >>>>>>>> have fixed the observed problem. :) > >>>>>>>> > >>>>>>>> But in addition you have increased the minimum stack size: > >>>>>>>> > >>>>>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * > K; > >>>>>>>> > >>>>>>>> which again, on its own would have fixed the original problem. :) > >>>>>>>> > >>>>>>>> Did you really intend to increase the real minimum stack from > 128K > >> to > >>>>>>>> 256K ? (on a 64K page system) > >>>>>>>> > >>>>>>>> --- > >>>>>>>> > >>>>>>>> Focusing simply on the shared code change to adjust the > requested > >>>>>>>> stacksize by the amount of guard space (if any), this does not > seem > >>>>>>>> unreasonable. As you note it is restricted to non-JavaThreads and > >> only > >>>>>>>> adds a page to reserved stack space. > >>>>>>>> > >>>>>>>> My only query now is whether the minimum stacksize detection > logic > >>>>>>>> will correctly report the real minimum stack size (taking into > account > >>>>>>>> the need for the guard page) ? > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> David > >>>>>>>> > >>>>>>>>> So I really think this issue should be fixed. > >>>>>>>>> > >>>>>>>>> Best regards, > >>>>>>>>> Goetz. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> -----Original Message----- > >>>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>>>>>>>> Sent: Thursday, November 10, 2016 10:02 PM > >>>>>>>>>> To: Lindenmaier, Goetz ; > hotspot- > >>>>>>> runtime- > >>>>>>>>>> dev at openjdk.java.net > >>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack > >> guard > >>>>> error. > >>>>>>>>>> > >>>>>>>>>> Hi Goetz, > >>>>>>>>>> > >>>>>>>>>> As per the bug report, this issue was already known (6675312) > and > >>> we > >>>>>>>>>> chose not to try and address it due to no reported issues at the > >>> time. > >>>>>>>>>> While I see that you have encountered an issue (is it real or > >>>>>>>>>> fabricated?) I think this change is too intrusive to be applied at > this > >>>>>>>>>> stage of the JDK 9 release cycle, as it will change the stack > >>>>>>>>>> requirements of every application running on Linux. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> David > >>>>>>>>>> > >>>>>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Please review this change. I please need a sponsor: > >>>>>>>>>>> > >>>>>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > >>>>>>> stackFix/webrev.01/ > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> In the Linux NPTL pthread implementation the guard size > >>> mechanism > >>>>>>>>>>> is not > >>>>>>>>>>> implemented properly. The posix standard requires to add > the > >>> size > >>>>>>>>>>> of the > >>>>>>>>>>> guard pages to the stack size, instead Linux takes the space > out > >> of > >>>>>>>>>>> 'stacksize'. > >>>>>>>>>>> > >>>>>>>>>>> The Posix standard > >>>>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ > >>>>>>>>>>> says "the implementation allocates extra memory at the > >>> overflow > >>>>>>>>>>> end of > >>>>>>>>>>> the stack". The linux man page > >>>>>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says > "As > >>> at > >>>>>>>>>>> glibc > >>>>>>>>>>> 2.8, the NPTL threading implementation includes the guard > area > >>>>> within > >>>>>>>>>>> the stack size allocation, rather than allocating extra space at > >>>>>>>>>>> the end > >>>>>>>>>>> of the stack, as POSIX.1 requires". > >>>>>>>>>>> > >>>>>>>>>>> I encounter this problem in > >>> runtime/Thread/TooSmallStackSize.java > >>>>>>>>>>> on ppc > >>>>>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K > >> on > >>>>> ppc, > >>>>>>> and > >>>>>>>>>>> ppc specifies two OS guard pages. The VM crashes in pthread > >>>>> creation > >>>>>>>>>>> because there is no usable space in the thread stack after > >>> allocating > >>>>>>>>>>> the guard pages. > >>>>>>>>>>> > >>>>>>>>>>> But TooSmallStackSize.java requires that the VM comes up > with > >>> the > >>>>>>>>>>> stack > >>>>>>>>>>> size mentioned in the error message. > >>>>>>>>>>> > >>>>>>>>>>> This fix adapts the requested stack size on Linux by the size of > >> the > >>>>>>>>>>> guard pages to mimick proper behaviour, see change to > >>>>> os_linux.cpp. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> The change also streamlines usage of stack_guard_page on > >>> linuxppc, > >>>>>>>>>>> linuxppcle, aixppc and linuxs390. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> To reproduce the error on linux_x86_64, apply below patch > and > >>> call > >>>>> the > >>>>>>>>>>> VM with -XX:CompilerThreadStackSize=64. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I'm still exploring why I had to choose such big compiler stacks > >> on > >>>>>>>>>>> ppc > >>>>>>>>>>> to get -version passing, but I wanted to send the RFR now as > >>> people > >>>>>>>>>>> obviously looked at the bug I opened (Thanks David!). > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Best regards, > >>>>>>>>>>> > >>>>>>>>>>> Goetz. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> diff -r b7ae012c55c3 > src/os_cpu/linux_x86/vm/os_linux_x86.cpp > >>>>>>>>>>> > >>>>>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 > >>>>> 12:37:28 > >>>>>>>>>> 2016 > >>>>>>>>>>> +0100 > >>>>>>>>>>> > >>>>>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov > 10 > >>>>>>> 16:52:17 > >>>>>>>>>> 2016 > >>>>>>>>>>> +0100 > >>>>>>>>>>> > >>>>>>>>>>> @@ -701,7 +701,7 @@ > >>>>>>>>>>> > >>>>>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType > thr_type) { > >>>>>>>>>>> > >>>>>>>>>>> // Creating guard page is very expensive. Java thread has > >>> HotSpot > >>>>>>>>>>> > >>>>>>>>>>> // guard page, only enable glibc guard page for non-Java > >>> threads. > >>>>>>>>>>> > >>>>>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); > >>>>>>>>>>> > >>>>>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); > >>>>>>>>>>> > >>>>>>>>>>> } > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> // Java thread: > >>>>>>>>>>> > >>>> From goetz.lindenmaier at sap.com Mon Nov 28 09:08:35 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Mon, 28 Nov 2016 09:08:35 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> Message-ID: Hi, I'm working on a fix for OS guard pages on stacks. I figured there are VM guard pages (reserved, yellow, red) on the compiler stacks _and_ OS guard pages. For Java threads, the OS guard pages are left out. I think this should be done for compiler threads, too. Please confirm. Webrev: http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.04/ The change affecting the compier threads is in os_linux.cpp, default_guard_size(), where '|| thr_type == compiler_thread' has been added. The function was also moved from the os_cpu files, as it's identical on all cpus. Best regards, Goetz. > -----Original Message----- > From: David Holmes [mailto:david.holmes at oracle.com] > Sent: Montag, 28. November 2016 00:25 > To: Lindenmaier, Goetz ; > 'daniel.daugherty at oracle.com' ; 'hotspot- > runtime-dev at openjdk.java.net' > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > Hi Goetz, > > On 24/11/2016 10:15 PM, Lindenmaier, Goetz wrote: > > Hi, > > > > I now edited the stuff I had proposed below: > > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.03/ > > This includes > > - the NPTL fix from webrev.02 > > Okay in principle. As discussed this only impacts non-JavaThreads so the > change should be minimal. > > > - merging code on linux > > Went a bit further than I had expected but if this truly isn't CPU > dependent code then great! > > > - not adding OS guard to compiler threads. > > Okay in principle. IIUC we will now save the OS guard page for compiler > thread stacks. Is that the only impact? The hotspot-compiler-dev folk > may want to sign off on this part. > > > A few minor comments: > > src/os/linux/vm/os_linux.cpp > > 2854 return ((thr_type == java_thread || thr_type == > os::compiler_thread) ... > > os:: should be used for both types or none. > > 6153 pthread_attr_getguardsize(&attr, &guard_size); > > Can you at least verify a zero return code in an assert/assert_status > please. > > --- > > src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp > src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp > src/os_cpu/linux_s390/vm/os_linux_s390.cpp > > Are the changes to min_stack_allowed just fixing an existing bug? > > --- > > Thanks, > David > ----- > > > > I think this should be pushed for this bug ID. For the other changes I'll > > make another bug. > > > > Best regards, > > Goetz. > > > >> -----Original Message----- > >> From: Lindenmaier, Goetz > >> Sent: Wednesday, November 23, 2016 8:11 AM > >> To: David Holmes ; > >> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > >> Subject: RE: RFR(M): 8169373: Work around linux NPTL stack guard error. > >> > >> Hi, > >> > >>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The > >>> ability for a CompilerThread to execute Java code (can_call_java()) is > >>> now a dynamic property depending on whether the current compiler is the > >>> JVMCI compiler. > >> Ah, then I should also leave space for shadow pages in the minimal stack > size > >> of comiler threads. > >> > >> Do we agree on the cleanup and on leaving out the OS guard page on > >> compiler threads? > >> Then I would edit a change comprising the NPTL workaround and these > >> additional changes, and split the other issue into a new bug? I think this > >> will easy the reviewing. > >> > >> Best regards, > >> Goetz. > >> > >>> -----Original Message----- > >>> From: David Holmes [mailto:david.holmes at oracle.com] > >>> Sent: Mittwoch, 23. November 2016 02:50 > >>> To: Lindenmaier, Goetz ; > >>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > >>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > >>> > >>> On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: > >>>> Hi Dan, > >>>> > >>>>> -----Original Message----- > >>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >>>>> Sent: Dienstag, 22. November 2016 14:01 > >>>>> To: Lindenmaier, Goetz ; David Holmes > >>>>> ; hotspot-runtime-dev at openjdk.java.net > >>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > >> error. > >>>>> > >>>>> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: > >>>>>> Hi Dan, > >>>>>> > >>>>>> I ran into a row of issues, some errors on the platforms. > >>>>>> > >>>>>> What I meant with that comment is that > >>>>>> os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, > >>>>>> JavaThread::stack_guard_zone_size() + > >>>>>> JavaThread::stack_shadow_zone_size() + > >>>>>> (4*BytesPerWord COMPILER2_PRESENT(+2)) * > 4 > >> * > >>> K); > >>>>>> was executed, and min_stack_allowed used for all stacks. Now, > >> compiler > >>>>> and > >>>>>> vm minimum stack sizes are not increased by these sizes. > >>>>> > >>>>> Now I see what you mean. Thanks for clearing this up. > >>>>> > >>>>> I should have remembered that part of the change because we went > >> back > >>>>> and forth about removing the red/yellow zone pages from the other > >>>>> threads. In particular, we discussed the compiler thread because it > >>>>> is-a JavaThread. Our conclusion was that a compiler thread doesn't > >>>>> execute Java bytecode so we could remove the red/yellow pages... > >>>> Yes, it does not execute java byte code. > >>> > >>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The > >>> ability for a CompilerThread to execute Java code (can_call_java()) is > >>> now a dynamic property depending on whether the current compiler is the > >>> JVMCI compiler. > >>> > >>> David > >>> ----- > >>> > >>>> Therefore you can remove the shadow pages. There is no code that > >>>> will bang. > >>>> But red/yellow pages are protected right during thread startup. > >>>> Therefore you must have enough space for them. > >>>> On ppc, we try to protect three 64K pages out of the 128K compiler stack. > >>>> That obviously fails. > >>>> > >>>> Therefore I propose: > >>>> size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // Set > >>> platform dependent. > >>>> in os::Posix::set_minimum_stack_sizes(): > >>>> _java_thread_min_stack_allowed = _java_thread_min_stack_allowed + > >>>> JavaThread::stack_guard_zone_size() + > >>>> JavaThread::stack_shadow_zone_size(); > >>>> > >>>> (Similar for _compiler_thread_min_stack_allowed). > >>>> > >>>> The minimal stack size is made up of three components: > >>>> 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot > >>> implementation/platform/os. > >>>> 2. Sizes of C++ frames: depends on C++ compiler. > >>>> These are fixed at compile time. > >>>> 3. Sizes of red/yellow/reserved/shadow pages. Depends on the system > >>> the > >>>> VM is used on. This is not fixed at compile time. (Our ppc machines > >>> differ > >>>> in page size.) > >>>> Therefore 3. should not be included in a compile time constant. > >>>> > >>>> > >>>> > >>>>> And that decision allowed us to be exposed to the 64K page issue > >>>>> because the "extra" space isn't there anymore. > >>>>> > >>>>>> At least the _compiler_thread_min_stack_allowed should be increased > >>>>>> similarly by red/yellow zone size. The compiler thread is a Java > >>>>>> thread and must have space for these zones. > >>>>> > >>>>> I'm not sure that I completely agree (yet). To me, the red/yellow > >>>>> pages are there for Java thread stack overflow semantics. Yes, the > >>>>> compiler thread needs extra space when 64K pages are used, but I > >>>>> would prefer that we add that space via a different calculation. > >>>> Yes they are. But compiler threads happen tob e a subclass of > >>>> Java threads and use the same run() method that puts the pages > >>>> There. > >>>> I agree that they are not needed for Compiler threads, nor for > >>>> CodeCacheSweeperThreads. I don't really now about > >>>> JvmtiAgentThreads and ServiceThreads, but all of the get the guard > >> pages > >>>> because they are derived from JavaThread. > >>>> > >>>>> To put it another way, I'd like to see us add extra space to solve > >>>>> the 64K page issue directly instead of as a side effect of the > >>>>> red/yellow page addition. > >>>> I don't understand. What do you mean by 'directly'? > >>>> > >>>>>> Also, the change added a test that is failing now. > >>>>> > >>>>> And that's a "good thing" (TM), right? :-) > >>>> Yes, it showed a bug and thus raised the need to fix it! :) > >>>> > >>>> Best regards, > >>>> Goetz. > >>>> > >>>> > >>>>> Again, thanks for clarifying 8140520's role in this issue. > >>>>> > >>>>> Dan > >>>>> > >>>>> > >>>>>> > >>>>>> Best regards, > >>>>>> Goetz. > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >>>>>>> Sent: Montag, 21. November 2016 17:28 > >>>>>>> To: David Holmes ; Lindenmaier, Goetz > >>>>>>> ; hotspot-runtime- > >>> dev at openjdk.java.net > >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > >>> error. > >>>>>>> > >>>>>>> Sorry for the delayed responses to this thread. I've been on vacation... > >>>>>>> > >>>>>>> One comment/query embedded below... > >>>>>>> > >>>>>>> > >>>>>>> On 11/10/16 8:40 PM, David Holmes wrote: > >>>>>>>> Hi Goetz, > >>>>>>>> > >>>>>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > >>>>>>>>> Hi David, > >>>>>>>>> > >>>>>>>>> This issue is different to 6675312, see also my comment in the bug. > >>>>>>>>> > >>>>>>>>> It appears running jtreg test > >>> runtime/Thread/TooSmallStackSize.java, > >>>>>>>>> with my patch below you can reproduce it on linuxx86_64. You can > >>> not > >>>>>>>>> do that with 6675312. Also, I would assume there are systems out > >>> there > >>>>>>>>> on x86 that uses 64-K pages, did you run the tests on these? I > >> would > >>>>>>>>> assume you get hard crashes with stack overflows in the first C2 > >>>>>>>>> compilation if there is only 64K usable CompilerThreadStack. > >>>>>>>>> > >>>>>>>>> My fix does not affect Java threads, which are the largest amount > >>>>>>>>> of threads used by the VM. It affects only the non-Java threads. > >>>>>>>>> It adds one page to these threads. The page does not require > >>> memory, > >>>>>>>>> as it's protected. The stack will only require more space if the > >> thread > >>>>>>>>> ran into a stack overflow before the fix as else the pages are not > >>>>>>>>> mapped. > >>>>>>>>> This are stack overflows that cause hard crashes, at least on ppc > >> the > >>> VM > >>>>>>>>> does not properly catch these stack overflows, so any setup > >> working > >>>>> now > >>>>>>>>> will not run into the additional space. Altogether there should be > >> no > >>>>>>>>> effect on running systems besides requiring one more entry in the > >>>>>>>>> page table per non-Java thread. > >>>>>>>>> > >>>>>>>>> The problem is caused by a rather recent change (8140520: > >> segfault > >>> on > >>>>>>>>> solaris-amd64 > >>>>>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed after > >>>>>>>>> feature-close. As this was a rather recent change, it must be > >>>>>>>>> possible to > >>>>>>>>> fix this follow up issue. What else is this period in the project good > >>>>>>>>> for if not fixing issues? > >>>>>>>> So I am seeing a number of factors here. > >>>>>>>> > >>>>>>>> First, 8140520, set: > >>>>>>>> > >>>>>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >>>>>>> So I'm confused by the above comment: > >>>>>>> > >>>>>>> > The problem is caused by a rather recent change (8140520: > >> segfault > >>>>>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) > >>>>>>> > >>>>>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- > >>>>>>> > >>> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html > >>>>>>> > >>>>>>> shows this change: > >>>>>>> > >>>>>>> @@ -531,19 +531,17 @@ > >>>>>>> } > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>> ////////////////////////////////////////////////////////////////////////////// > >>>>>>> // > >>>>>>> // thread stack > >>>>>>> > >>>>>>> -size_t os::Linux::min_stack_allowed = 128*K; > >>>>>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >>>>>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; > >>>>>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * K; > >>>>>>> > >>>>>>> so the existing single variable of 'min_stack_allowed' was > >>>>>>> replaced by three variables: _compiler_thread_min_stack_allowed, > >>>>>>> _java_thread_min_stack_allowed, and > >>>>>>> _vm_internal_thread_min_stack_allowed. > >>>>>>> > >>>>>>> The old single variable and the three new variables are all > >>>>>>> initialized to the same value (128K) so the fix for 8140520 > >>>>>>> did not change stack sizes for this platform. In fact, only > >>>>>>> one platform had a size change (Solaris X64). > >>>>>>> > >>>>>>> So I'm confused about how the fix for 8140520 caused this problem. > >>>>>>> > >>>>>>> Based on David's analysis below, it looks to me like this 64K stack > >>>>>>> guard page problem should also exist prior to the fix for 8140520. > >>>>>>> > >>>>>>> Goetz, can you please explain how 8140520 caused this problem? > >>>>>>> > >>>>>>> Dan > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Second on linux PPC it is hardwired to use 2 guard pages: > >>>>>>>> > >>>>>>>> return 2 * page_size(); > >>>>>>>> > >>>>>>>> Third, you had a pagesize of 64K. > >>>>>>>> > >>>>>>>> Fourth, NPTL takes the guard space from the stack space - hence > >> with > >>> 2 > >>>>>>>> x 64K guard, and a 128K stack it was all consumed. > >>>>>>>> > >>>>>>>> --- > >>>>>>>> > >>>>>>>> In the proposed changes you now only use page_size() for the guard, > >>> so > >>>>>>>> that alone would have fixed the observed problem. > >>>>>>>> > >>>>>>>> But in addition you want to address the NPTL problem by adding > >> back > >>>>>>>> the guard space to the stack size requested. That alone would also > >>>>>>>> have fixed the observed problem. :) > >>>>>>>> > >>>>>>>> But in addition you have increased the minimum stack size: > >>>>>>>> > >>>>>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; > >>>>>>>> > >>>>>>>> which again, on its own would have fixed the original problem. :) > >>>>>>>> > >>>>>>>> Did you really intend to increase the real minimum stack from 128K > >> to > >>>>>>>> 256K ? (on a 64K page system) > >>>>>>>> > >>>>>>>> --- > >>>>>>>> > >>>>>>>> Focusing simply on the shared code change to adjust the requested > >>>>>>>> stacksize by the amount of guard space (if any), this does not seem > >>>>>>>> unreasonable. As you note it is restricted to non-JavaThreads and > >> only > >>>>>>>> adds a page to reserved stack space. > >>>>>>>> > >>>>>>>> My only query now is whether the minimum stacksize detection logic > >>>>>>>> will correctly report the real minimum stack size (taking into account > >>>>>>>> the need for the guard page) ? > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> David > >>>>>>>> > >>>>>>>>> So I really think this issue should be fixed. > >>>>>>>>> > >>>>>>>>> Best regards, > >>>>>>>>> Goetz. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> -----Original Message----- > >>>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>>>>>>>> Sent: Thursday, November 10, 2016 10:02 PM > >>>>>>>>>> To: Lindenmaier, Goetz ; hotspot- > >>>>>>> runtime- > >>>>>>>>>> dev at openjdk.java.net > >>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack > >> guard > >>>>> error. > >>>>>>>>>> > >>>>>>>>>> Hi Goetz, > >>>>>>>>>> > >>>>>>>>>> As per the bug report, this issue was already known (6675312) and > >>> we > >>>>>>>>>> chose not to try and address it due to no reported issues at the > >>> time. > >>>>>>>>>> While I see that you have encountered an issue (is it real or > >>>>>>>>>> fabricated?) I think this change is too intrusive to be applied at this > >>>>>>>>>> stage of the JDK 9 release cycle, as it will change the stack > >>>>>>>>>> requirements of every application running on Linux. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> David > >>>>>>>>>> > >>>>>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Please review this change. I please need a sponsor: > >>>>>>>>>>> > >>>>>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > >>>>>>> stackFix/webrev.01/ > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> In the Linux NPTL pthread implementation the guard size > >>> mechanism > >>>>>>>>>>> is not > >>>>>>>>>>> implemented properly. The posix standard requires to add the > >>> size > >>>>>>>>>>> of the > >>>>>>>>>>> guard pages to the stack size, instead Linux takes the space out > >> of > >>>>>>>>>>> 'stacksize'. > >>>>>>>>>>> > >>>>>>>>>>> The Posix standard > >>>>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ > >>>>>>>>>>> says "the implementation allocates extra memory at the > >>> overflow > >>>>>>>>>>> end of > >>>>>>>>>>> the stack". The linux man page > >>>>>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As > >>> at > >>>>>>>>>>> glibc > >>>>>>>>>>> 2.8, the NPTL threading implementation includes the guard area > >>>>> within > >>>>>>>>>>> the stack size allocation, rather than allocating extra space at > >>>>>>>>>>> the end > >>>>>>>>>>> of the stack, as POSIX.1 requires". > >>>>>>>>>>> > >>>>>>>>>>> I encounter this problem in > >>> runtime/Thread/TooSmallStackSize.java > >>>>>>>>>>> on ppc > >>>>>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K > >> on > >>>>> ppc, > >>>>>>> and > >>>>>>>>>>> ppc specifies two OS guard pages. The VM crashes in pthread > >>>>> creation > >>>>>>>>>>> because there is no usable space in the thread stack after > >>> allocating > >>>>>>>>>>> the guard pages. > >>>>>>>>>>> > >>>>>>>>>>> But TooSmallStackSize.java requires that the VM comes up with > >>> the > >>>>>>>>>>> stack > >>>>>>>>>>> size mentioned in the error message. > >>>>>>>>>>> > >>>>>>>>>>> This fix adapts the requested stack size on Linux by the size of > >> the > >>>>>>>>>>> guard pages to mimick proper behaviour, see change to > >>>>> os_linux.cpp. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> The change also streamlines usage of stack_guard_page on > >>> linuxppc, > >>>>>>>>>>> linuxppcle, aixppc and linuxs390. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> To reproduce the error on linux_x86_64, apply below patch and > >>> call > >>>>> the > >>>>>>>>>>> VM with -XX:CompilerThreadStackSize=64. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I'm still exploring why I had to choose such big compiler stacks > >> on > >>>>>>>>>>> ppc > >>>>>>>>>>> to get -version passing, but I wanted to send the RFR now as > >>> people > >>>>>>>>>>> obviously looked at the bug I opened (Thanks David!). > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Best regards, > >>>>>>>>>>> > >>>>>>>>>>> Goetz. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp > >>>>>>>>>>> > >>>>>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 > >>>>> 12:37:28 > >>>>>>>>>> 2016 > >>>>>>>>>>> +0100 > >>>>>>>>>>> > >>>>>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 > >>>>>>> 16:52:17 > >>>>>>>>>> 2016 > >>>>>>>>>>> +0100 > >>>>>>>>>>> > >>>>>>>>>>> @@ -701,7 +701,7 @@ > >>>>>>>>>>> > >>>>>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > >>>>>>>>>>> > >>>>>>>>>>> // Creating guard page is very expensive. Java thread has > >>> HotSpot > >>>>>>>>>>> > >>>>>>>>>>> // guard page, only enable glibc guard page for non-Java > >>> threads. > >>>>>>>>>>> > >>>>>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); > >>>>>>>>>>> > >>>>>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); > >>>>>>>>>>> > >>>>>>>>>>> } > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> // Java thread: > >>>>>>>>>>> > >>>> From thomas.schatzl at oracle.com Mon Nov 28 09:54:30 2016 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 28 Nov 2016 10:54:30 +0100 Subject: RFR(S): 8166156: Convert test_semaphore to GTest In-Reply-To: <5839241E.2030802@oracle.com> References: <5838974C.9000603@oracle.com> <3878A570-5ECF-4D81-87D1-A17DFFF5ABFF@oracle.com> <5839241E.2030802@oracle.com> Message-ID: <1480326870.2915.0.camel@oracle.com> Hi, On Sat, 2016-11-26 at 08:56 +0300, Kirill Zhaldbybin wrote: > Kim, > > Thank you for review! > > Regards, Kirill > > On 11/26/2016 08:25 AM, Kim Barrett wrote: > > > > > > > > On Nov 25, 2016, at 2:55 PM, Kirill Zhaldybin > > racle.com> wrote: > > > > > > Dear all, > > > > > > Could you please review this fix for 8166156? > > > Pretty straight-forward conversion. > > > > > > WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/ > > > webrev.00/ > > > CR: https://bugs.openjdk.java.net/browse/JDK-8166156 > > > > > > Thank you. > > > > > > Regards, Kirill > > Looks good. > > ? looks good. Thomas From marcus.larsson at oracle.com Mon Nov 28 10:06:27 2016 From: marcus.larsson at oracle.com (Marcus Larsson) Date: Mon, 28 Nov 2016 11:06:27 +0100 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part In-Reply-To: <583873A8.8000106@oracle.com> References: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> <85e51138-f3ce-7392-2cc3-ce7840aa3747@oracle.com> <23fcd50a-eed0-52a5-8817-244ecf75bb2a@oracle.com> <583873A8.8000106@oracle.com> Message-ID: Hi, On 11/25/2016 06:23 PM, Kirill Zhaldybin wrote: > Marcus, > > Here are a new webrev: > http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.01/ Looks ok. Thanks, Marcus > > I addressed your comment about "if-case for the Z suffix". > > Could you please let me know your opinion? > > Thank you. > > Regards, Kirill > > On 24.11.2016 17:35, Marcus Larsson wrote: >> Hi, >> >> >> On 11/22/2016 02:24 PM, Kirill Zhaldybin wrote: >>> Marcus, >>> >>> Thank you for prompt reply! >>> >>> Could you please read comments inline? >>> I'm looking forward to your reply. >>> >>> Thank you. >>> >>> Regards, Kirill >>> >>> On 22.11.2016 15:32, Marcus Larsson wrote: >>>> Hi, >>>> >>>> >>>> On 2016-11-21 17:38, Kirill Zhaldybin wrote: >>>>> Marcus, >>>>> >>>>> Thank you for reviewing the fix! >>>>>>> WebRev: >>>>>>> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ >>>>>> >>>>>> ISO8601 says the decimal point can be either '.' or ',' so the test >>>>>> should accept either. You could let sscanf read out the decimal >>>>>> point as a character and just verify that it is one of the two. >>>>>> >>>>>> In the UTC test you changed ASSERT_GT to ASSERT_EQ, which means >>>>>> that we won't accept "Z" suffixed strings. Please revert that. >>>>> I agree that ISO8601 could add "Z" to time (and as far as I >>>>> understand date/time without delimiters is legal too) but these are >>>>> the unit tests. >>>>> Hence they cover the existing code and they should pass only if the >>>>> result corresponds to existing code and fail otherwise. >>>>> The current code from os::iso8601_time format date/time string >>>>> %04d-%02d-%02dT%02d:%02d:%02d.%03d%c%02d%02d so we should not >>>>> consider any other format as valid. >>>>> >>>>> Could you please let me know your opinion? >>>> >>>> I think the test should verify the intended behavior, not the >>>> implementation. If we refactor or change something in iso8601_time() >>>> we shouldn't be failing the test if it still conforms to ISO8601, IMO. >>> I would agree with you if we were talking about a functional test. But >>> since it is an unit test I think we should keep it as close to >>> implementation as possible. >>> If the implementation is changed unintentionally the test fails and >>> signals us that something is broken. >>> If it is an intentional change the test must be updated >>> correspondingly. >> >> I still think it's unnecessary noise, but if you insist I'm fine with >> it. >> >> If we're not going to accept anything else than the current >> implementation then you should also remove the if-case for the Z suffix, >> since the test will fail for that anyway. >> >> Thanks, >> Marcus >> >>> >>>> >>>> Thanks, >>>> Marcus >>>> >>>>> >>>>> Thank you. >>>>> >>>>> Regards, Kirill >>>>> >>>>>> >>>>>> Thanks, >>>>>> Marcus >>>>>> >>>>>>> CR: https://bugs.openjdk.java.net/browse/JDK-8169003 >>>>>>> >>>>>>> Thank you. >>>>>>> >>>>>>> Regards, Kirill >>>>>> >>>>> >>>> >>> >> > From igor.ignatyev at oracle.com Mon Nov 28 12:19:05 2016 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Mon, 28 Nov 2016 15:19:05 +0300 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part In-Reply-To: References: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> <85e51138-f3ce-7392-2cc3-ce7840aa3747@oracle.com> <23fcd50a-eed0-52a5-8817-244ecf75bb2a@oracle.com> <583873A8.8000106@oracle.com> Message-ID: Hi Kirill, looks good to me, thanks for fixing that. Cheers, ? Igor > On Nov 28, 2016, at 1:06 PM, Marcus Larsson wrote: > > Hi, > > > On 11/25/2016 06:23 PM, Kirill Zhaldybin wrote: >> Marcus, >> >> Here are a new webrev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.01/ > > Looks ok. > > Thanks, > Marcus > >> >> I addressed your comment about "if-case for the Z suffix". >> >> Could you please let me know your opinion? >> >> Thank you. >> >> Regards, Kirill >> >> On 24.11.2016 17:35, Marcus Larsson wrote: >>> Hi, >>> >>> >>> On 11/22/2016 02:24 PM, Kirill Zhaldybin wrote: >>>> Marcus, >>>> >>>> Thank you for prompt reply! >>>> >>>> Could you please read comments inline? >>>> I'm looking forward to your reply. >>>> >>>> Thank you. >>>> >>>> Regards, Kirill >>>> >>>> On 22.11.2016 15:32, Marcus Larsson wrote: >>>>> Hi, >>>>> >>>>> >>>>> On 2016-11-21 17:38, Kirill Zhaldybin wrote: >>>>>> Marcus, >>>>>> >>>>>> Thank you for reviewing the fix! >>>>>>>> WebRev: >>>>>>>> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ >>>>>>> >>>>>>> ISO8601 says the decimal point can be either '.' or ',' so the test >>>>>>> should accept either. You could let sscanf read out the decimal >>>>>>> point as a character and just verify that it is one of the two. >>>>>>> >>>>>>> In the UTC test you changed ASSERT_GT to ASSERT_EQ, which means >>>>>>> that we won't accept "Z" suffixed strings. Please revert that. >>>>>> I agree that ISO8601 could add "Z" to time (and as far as I >>>>>> understand date/time without delimiters is legal too) but these are >>>>>> the unit tests. >>>>>> Hence they cover the existing code and they should pass only if the >>>>>> result corresponds to existing code and fail otherwise. >>>>>> The current code from os::iso8601_time format date/time string >>>>>> %04d-%02d-%02dT%02d:%02d:%02d.%03d%c%02d%02d so we should not >>>>>> consider any other format as valid. >>>>>> >>>>>> Could you please let me know your opinion? >>>>> >>>>> I think the test should verify the intended behavior, not the >>>>> implementation. If we refactor or change something in iso8601_time() >>>>> we shouldn't be failing the test if it still conforms to ISO8601, IMO. >>>> I would agree with you if we were talking about a functional test. But >>>> since it is an unit test I think we should keep it as close to >>>> implementation as possible. >>>> If the implementation is changed unintentionally the test fails and >>>> signals us that something is broken. >>>> If it is an intentional change the test must be updated correspondingly. >>> >>> I still think it's unnecessary noise, but if you insist I'm fine with it. >>> >>> If we're not going to accept anything else than the current >>> implementation then you should also remove the if-case for the Z suffix, >>> since the test will fail for that anyway. >>> >>> Thanks, >>> Marcus >>> >>>> >>>>> >>>>> Thanks, >>>>> Marcus >>>>> >>>>>> >>>>>> Thank you. >>>>>> >>>>>> Regards, Kirill >>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Marcus >>>>>>> >>>>>>>> CR: https://bugs.openjdk.java.net/browse/JDK-8169003 >>>>>>>> >>>>>>>> Thank you. >>>>>>>> >>>>>>>> Regards, Kirill >>>>>>> >>>>>> >>>>> >>>> >>> >> > From kirill.zhaldybin at oracle.com Mon Nov 28 13:01:25 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Mon, 28 Nov 2016 16:01:25 +0300 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part In-Reply-To: References: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> <85e51138-f3ce-7392-2cc3-ce7840aa3747@oracle.com> <23fcd50a-eed0-52a5-8817-244ecf75bb2a@oracle.com> <583873A8.8000106@oracle.com> Message-ID: Markus, Thank you for review! Regards, Kirill On 28.11.2016 13:06, Marcus Larsson wrote: > Hi, > > > On 11/25/2016 06:23 PM, Kirill Zhaldybin wrote: >> Marcus, >> >> Here are a new webrev: >> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.01/ > > Looks ok. > > Thanks, > Marcus > >> >> I addressed your comment about "if-case for the Z suffix". >> >> Could you please let me know your opinion? >> >> Thank you. >> >> Regards, Kirill >> >> On 24.11.2016 17:35, Marcus Larsson wrote: >>> Hi, >>> >>> >>> On 11/22/2016 02:24 PM, Kirill Zhaldybin wrote: >>>> Marcus, >>>> >>>> Thank you for prompt reply! >>>> >>>> Could you please read comments inline? >>>> I'm looking forward to your reply. >>>> >>>> Thank you. >>>> >>>> Regards, Kirill >>>> >>>> On 22.11.2016 15:32, Marcus Larsson wrote: >>>>> Hi, >>>>> >>>>> >>>>> On 2016-11-21 17:38, Kirill Zhaldybin wrote: >>>>>> Marcus, >>>>>> >>>>>> Thank you for reviewing the fix! >>>>>>>> WebRev: >>>>>>>> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ >>>>>>>> >>>>>>> >>>>>>> ISO8601 says the decimal point can be either '.' or ',' so the test >>>>>>> should accept either. You could let sscanf read out the decimal >>>>>>> point as a character and just verify that it is one of the two. >>>>>>> >>>>>>> In the UTC test you changed ASSERT_GT to ASSERT_EQ, which means >>>>>>> that we won't accept "Z" suffixed strings. Please revert that. >>>>>> I agree that ISO8601 could add "Z" to time (and as far as I >>>>>> understand date/time without delimiters is legal too) but these are >>>>>> the unit tests. >>>>>> Hence they cover the existing code and they should pass only if the >>>>>> result corresponds to existing code and fail otherwise. >>>>>> The current code from os::iso8601_time format date/time string >>>>>> %04d-%02d-%02dT%02d:%02d:%02d.%03d%c%02d%02d so we should not >>>>>> consider any other format as valid. >>>>>> >>>>>> Could you please let me know your opinion? >>>>> >>>>> I think the test should verify the intended behavior, not the >>>>> implementation. If we refactor or change something in iso8601_time() >>>>> we shouldn't be failing the test if it still conforms to ISO8601, >>>>> IMO. >>>> I would agree with you if we were talking about a functional test. But >>>> since it is an unit test I think we should keep it as close to >>>> implementation as possible. >>>> If the implementation is changed unintentionally the test fails and >>>> signals us that something is broken. >>>> If it is an intentional change the test must be updated >>>> correspondingly. >>> >>> I still think it's unnecessary noise, but if you insist I'm fine >>> with it. >>> >>> If we're not going to accept anything else than the current >>> implementation then you should also remove the if-case for the Z >>> suffix, >>> since the test will fail for that anyway. >>> >>> Thanks, >>> Marcus >>> >>>> >>>>> >>>>> Thanks, >>>>> Marcus >>>>> >>>>>> >>>>>> Thank you. >>>>>> >>>>>> Regards, Kirill >>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Marcus >>>>>>> >>>>>>>> CR: https://bugs.openjdk.java.net/browse/JDK-8169003 >>>>>>>> >>>>>>>> Thank you. >>>>>>>> >>>>>>>> Regards, Kirill >>>>>>> >>>>>> >>>>> >>>> >>> >> > From kirill.zhaldybin at oracle.com Mon Nov 28 13:01:53 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Mon, 28 Nov 2016 16:01:53 +0300 Subject: RFR(XS): 8169003: LogDecorations.iso8601_utctime_test fails if numeric locale uses ", " as separator between integer and fraction part In-Reply-To: References: <60ea633b-23b7-ee2f-27c6-9f0c754a7ec6@oracle.com> <85e51138-f3ce-7392-2cc3-ce7840aa3747@oracle.com> <23fcd50a-eed0-52a5-8817-244ecf75bb2a@oracle.com> <583873A8.8000106@oracle.com> Message-ID: <0b1cf3ff-5317-cd1c-e5e3-e43ef215bdb7@oracle.com> Igor, Thank you for review! Regards, Kirill On 28.11.2016 15:19, Igor Ignatyev wrote: > Hi Kirill, > > looks good to me, thanks for fixing that. > > Cheers, > ? Igor > >> On Nov 28, 2016, at 1:06 PM, Marcus Larsson wrote: >> >> Hi, >> >> >> On 11/25/2016 06:23 PM, Kirill Zhaldybin wrote: >>> Marcus, >>> >>> Here are a new webrev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.01/ >> Looks ok. >> >> Thanks, >> Marcus >> >>> I addressed your comment about "if-case for the Z suffix". >>> >>> Could you please let me know your opinion? >>> >>> Thank you. >>> >>> Regards, Kirill >>> >>> On 24.11.2016 17:35, Marcus Larsson wrote: >>>> Hi, >>>> >>>> >>>> On 11/22/2016 02:24 PM, Kirill Zhaldybin wrote: >>>>> Marcus, >>>>> >>>>> Thank you for prompt reply! >>>>> >>>>> Could you please read comments inline? >>>>> I'm looking forward to your reply. >>>>> >>>>> Thank you. >>>>> >>>>> Regards, Kirill >>>>> >>>>> On 22.11.2016 15:32, Marcus Larsson wrote: >>>>>> Hi, >>>>>> >>>>>> >>>>>> On 2016-11-21 17:38, Kirill Zhaldybin wrote: >>>>>>> Marcus, >>>>>>> >>>>>>> Thank you for reviewing the fix! >>>>>>>>> WebRev: >>>>>>>>> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8169003/webrev.00/ >>>>>>>> ISO8601 says the decimal point can be either '.' or ',' so the test >>>>>>>> should accept either. You could let sscanf read out the decimal >>>>>>>> point as a character and just verify that it is one of the two. >>>>>>>> >>>>>>>> In the UTC test you changed ASSERT_GT to ASSERT_EQ, which means >>>>>>>> that we won't accept "Z" suffixed strings. Please revert that. >>>>>>> I agree that ISO8601 could add "Z" to time (and as far as I >>>>>>> understand date/time without delimiters is legal too) but these are >>>>>>> the unit tests. >>>>>>> Hence they cover the existing code and they should pass only if the >>>>>>> result corresponds to existing code and fail otherwise. >>>>>>> The current code from os::iso8601_time format date/time string >>>>>>> %04d-%02d-%02dT%02d:%02d:%02d.%03d%c%02d%02d so we should not >>>>>>> consider any other format as valid. >>>>>>> >>>>>>> Could you please let me know your opinion? >>>>>> I think the test should verify the intended behavior, not the >>>>>> implementation. If we refactor or change something in iso8601_time() >>>>>> we shouldn't be failing the test if it still conforms to ISO8601, IMO. >>>>> I would agree with you if we were talking about a functional test. But >>>>> since it is an unit test I think we should keep it as close to >>>>> implementation as possible. >>>>> If the implementation is changed unintentionally the test fails and >>>>> signals us that something is broken. >>>>> If it is an intentional change the test must be updated correspondingly. >>>> I still think it's unnecessary noise, but if you insist I'm fine with it. >>>> >>>> If we're not going to accept anything else than the current >>>> implementation then you should also remove the if-case for the Z suffix, >>>> since the test will fail for that anyway. >>>> >>>> Thanks, >>>> Marcus >>>> >>>>>> Thanks, >>>>>> Marcus >>>>>> >>>>>>> Thank you. >>>>>>> >>>>>>> Regards, Kirill >>>>>>> >>>>>>>> Thanks, >>>>>>>> Marcus >>>>>>>> >>>>>>>>> CR: https://bugs.openjdk.java.net/browse/JDK-8169003 >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> >>>>>>>>> Regards, Kirill From kirill.zhaldybin at oracle.com Mon Nov 28 13:02:17 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Mon, 28 Nov 2016 16:02:17 +0300 Subject: RFR(S): 8166156: Convert test_semaphore to GTest In-Reply-To: <1480326870.2915.0.camel@oracle.com> References: <5838974C.9000603@oracle.com> <3878A570-5ECF-4D81-87D1-A17DFFF5ABFF@oracle.com> <5839241E.2030802@oracle.com> <1480326870.2915.0.camel@oracle.com> Message-ID: Thomas, Thank you for review! Regards, Kirill On 28.11.2016 12:54, Thomas Schatzl wrote: > Hi, > > On Sat, 2016-11-26 at 08:56 +0300, Kirill Zhaldbybin wrote: >> Kim, >> >> Thank you for review! >> >> Regards, Kirill >> >> On 11/26/2016 08:25 AM, Kim Barrett wrote: >>>> On Nov 25, 2016, at 2:55 PM, Kirill Zhaldybin >>> racle.com> wrote: >>>> >>>> Dear all, >>>> >>>> Could you please review this fix for 8166156? >>>> Pretty straight-forward conversion. >>>> >>>> WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/ >>>> webrev.00/ >>>> CR: https://bugs.openjdk.java.net/browse/JDK-8166156 >>>> >>>> Thank you. >>>> >>>> Regards, Kirill >>> Looks good. >>> > looks good. > > Thomas > From stefan.karlsson at oracle.com Mon Nov 28 13:01:10 2016 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Mon, 28 Nov 2016 14:01:10 +0100 Subject: RFR(S): 8166156: Convert test_semaphore to GTest In-Reply-To: <5838974C.9000603@oracle.com> References: <5838974C.9000603@oracle.com> Message-ID: Hi Kirill, On 2016-11-25 20:55, Kirill Zhaldybin wrote: > Dear all, > > Could you please review this fix for 8166156? > Pretty straight-forward conversion. > > WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.00/ > CR: https://bugs.openjdk.java.net/browse/JDK-8166156 Thanks for porting this test. I think it would be preferable if this test was using TEST instead of TEST_VM. I don't see anything in the test that requires the VM to run. Thanks, StefanK > > Thank you. > > Regards, Kirill From lois.foltan at oracle.com Mon Nov 28 14:32:57 2016 From: lois.foltan at oracle.com (Lois Foltan) Date: Mon, 28 Nov 2016 09:32:57 -0500 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> Message-ID: <583C4019.8040506@oracle.com> Hi Alan, I have reviewed the hotspot changes and they look good. Minor nit, src/share/vm/classfile/javaClasses.cpp only differs by the addition of a blank line. Thanks, Lois On 11/24/2016 10:25 AM, Alan Bateman wrote: > Folks on jigsaw-dev will know that we are on a mission to bring the > changes accumulated in the jake forest to jdk9/dev. We can think of > this as a refresh of the module system in JDK 9, the last big refresh > was in May with many small updates since then. > > The focus this time is to bring the changes that are tied to JSR > issues into jdk9/dev, specifically the issues that are tracked on the > JSR issues list [1] as: > > #CompileTimeDependences > #AddExportsInManifest > #ClassFileModuleName > #ClassFileAccPublic > #ServiceLoaderEnhancements > #ResourceEncapsulation/#ClassFilesAsResources > #ReflectiveAccessToNonExportedTypes > #AwkwardStrongEncapsulation > #ReadabilityAddedByLayerCreator > #IndirectQualifiedReflectiveAccess (partial) > #VersionsInModuleNames > #NonHierarchicalLayers > #ModuleAnnotations/#ModuleDeprecation > #ReflectiveAccessByInstrumentationAgents > > Some of these issues are not "Resolved" yet, meaning there is still > ongoing discussion on the EG mailing list. That is okay, there is > nothing final here. If there are changes to these proposals then the > implementation changes will follow. Also, as I said in a mail to > jigsaw-dev yesterday [2], is that we will keep the jake forest open > for ongoing prototyping and iteration, also ongoing implementation > improvements where iteration or bake time is important. > > For the code review then the focus is therefore on sanity checking the > changes that we would like to bring into jdk9/dev. We will not use > this review thread to debate alternative designs or other big > implementation changes that are more appropriate to bake in jake. > > To get going, I've put the webrevs with a snapshot of the changes in > jake here: > http://cr.openjdk.java.net/~alanb/8169069/0/ > > The changes are currently sync'ed against jdk-9+146 and will be > rebased (and re-tested) against jdk9/dev prior to integration. There > are a number of small changes that need to be added to this in the > coming days, I will refresh the webrev every few days to take account > of these updates. > > > A few important points to mention, even if you aren't reviewing the > changes: > > 1. This refresh requires a new version of jtreg to run the tests. The > changes for this new version are in the code-tools/jtreg repository > and the plan is to tag a new build (jtreg4.2-b04) next week. Once the > tag has been added then we'll update the requiredVersion property in > each TEST.ROOT to force everyone to update. > > 2. For developers trying out modules with the main line JDK 9 builds > then be aware that `requires public` changes to `requires transitive` > and the `provides` clause changes to require all providers for a > specific service type to be in the same clause. Also be aware that the > binary form of the module declaration (module-info.class) changes so > you will need to recompile any modules. > > 3. Those running existing code on JDK 9 and ignoring modules will need > to be aware of a disruptive change in this refresh. The disruptive > change is #AwkwardStrongEncapsulation where setAccessible(true) is > changed so that it can't be used to break into non-public > fields/methods of JDK classes. This change is going to expose a lot of > hacks in existing code. We plan to send mail to jdk9-dev in advance of > this integration to create awareness of this change. As per the > original introduction of strong encapsulation then command line > options (and now the manifest of application JAR files) can be used to > keep existing code working. The new option is `--add-opens` to open a > package in a module for deep reflection by other modules. As an > example, if you find yourself with code that hacks into the private > `comparator` field in java.util.TreeMap then running with `--add-opens > java.base/java.util=ALL-UNNAMED` will keep that code working. > > > A few miscellaneous notes for those that are reviewing: > > 1. We have some temporary/transition code in the top-level repo to > deal with the importing of the JavaFX modules. This will be removed > once the changes are in JDK 9 for the OpenJFX project to use. > > 2. In the jdk repo then it's important to understand that the module > system is initialized at startup and there are many places where we > need to keep startup performance in mind. This sometimes means less > elegant code than might be used if startup wasn't such a big concern. > > 3. The changes in the jaxws repo make use of new APIs that means the > code doesn't compile with JDK 7 or JDK 8. Our intention is to work > with the JAXB and JAX-WS maintainers to address the issues in the > upstream project and then bring those changes into jdk9/dev to replace > the patches that we are forced to push for the short term. > > 4. You will see several tests where the value of the @modules tag has > `:open` or `:+open`. This is new jtreg speak. The former means the > test is run with --add-opens to open the package, the latter means the > test is exported at compile-time and exported + open at run-time (the > latter usage will be rare, it's where tests have static references to > JDK internal types and are also doing deep reflection with > setAccessible). > > > In terms of dates then we are aiming to integrate these changes into > jdk9/dev in early December. I will send a follow-up mail next week on > this as we work through the logistics. > > -Alan > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ > [2] > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html From karen.kinnear at oracle.com Mon Nov 28 14:47:03 2016 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Mon, 28 Nov 2016 09:47:03 -0500 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <583C4019.8040506@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> <583C4019.8040506@oracle.com> Message-ID: Alan, I reviewed all the hotspot runtime changes - except the tests (Christian will review those) - and jvmti - which Dmitry Samersoff will review. They look good. thanks, Karen > On Nov 28, 2016, at 9:32 AM, Lois Foltan wrote: > > Hi Alan, > > I have reviewed the hotspot changes and they look good. Minor nit, src/share/vm/classfile/javaClasses.cpp only differs by the addition of a blank line. > > Thanks, > Lois > > On 11/24/2016 10:25 AM, Alan Bateman wrote: >> Folks on jigsaw-dev will know that we are on a mission to bring the changes accumulated in the jake forest to jdk9/dev. We can think of this as a refresh of the module system in JDK 9, the last big refresh was in May with many small updates since then. >> >> The focus this time is to bring the changes that are tied to JSR issues into jdk9/dev, specifically the issues that are tracked on the JSR issues list [1] as: >> >> #CompileTimeDependences >> #AddExportsInManifest >> #ClassFileModuleName >> #ClassFileAccPublic >> #ServiceLoaderEnhancements >> #ResourceEncapsulation/#ClassFilesAsResources >> #ReflectiveAccessToNonExportedTypes >> #AwkwardStrongEncapsulation >> #ReadabilityAddedByLayerCreator >> #IndirectQualifiedReflectiveAccess (partial) >> #VersionsInModuleNames >> #NonHierarchicalLayers >> #ModuleAnnotations/#ModuleDeprecation >> #ReflectiveAccessByInstrumentationAgents >> >> Some of these issues are not "Resolved" yet, meaning there is still ongoing discussion on the EG mailing list. That is okay, there is nothing final here. If there are changes to these proposals then the implementation changes will follow. Also, as I said in a mail to jigsaw-dev yesterday [2], is that we will keep the jake forest open for ongoing prototyping and iteration, also ongoing implementation improvements where iteration or bake time is important. >> >> For the code review then the focus is therefore on sanity checking the changes that we would like to bring into jdk9/dev. We will not use this review thread to debate alternative designs or other big implementation changes that are more appropriate to bake in jake. >> >> To get going, I've put the webrevs with a snapshot of the changes in jake here: >> http://cr.openjdk.java.net/~alanb/8169069/0/ >> >> The changes are currently sync'ed against jdk-9+146 and will be rebased (and re-tested) against jdk9/dev prior to integration. There are a number of small changes that need to be added to this in the coming days, I will refresh the webrev every few days to take account of these updates. >> >> >> A few important points to mention, even if you aren't reviewing the changes: >> >> 1. This refresh requires a new version of jtreg to run the tests. The changes for this new version are in the code-tools/jtreg repository and the plan is to tag a new build (jtreg4.2-b04) next week. Once the tag has been added then we'll update the requiredVersion property in each TEST.ROOT to force everyone to update. >> >> 2. For developers trying out modules with the main line JDK 9 builds then be aware that `requires public` changes to `requires transitive` and the `provides` clause changes to require all providers for a specific service type to be in the same clause. Also be aware that the binary form of the module declaration (module-info.class) changes so you will need to recompile any modules. >> >> 3. Those running existing code on JDK 9 and ignoring modules will need to be aware of a disruptive change in this refresh. The disruptive change is #AwkwardStrongEncapsulation where setAccessible(true) is changed so that it can't be used to break into non-public fields/methods of JDK classes. This change is going to expose a lot of hacks in existing code. We plan to send mail to jdk9-dev in advance of this integration to create awareness of this change. As per the original introduction of strong encapsulation then command line options (and now the manifest of application JAR files) can be used to keep existing code working. The new option is `--add-opens` to open a package in a module for deep reflection by other modules. As an example, if you find yourself with code that hacks into the private `comparator` field in java.util.TreeMap then running with `--add-opens java.base/java.util=ALL-UNNAMED` will keep that code working. >> >> >> A few miscellaneous notes for those that are reviewing: >> >> 1. We have some temporary/transition code in the top-level repo to deal with the importing of the JavaFX modules. This will be removed once the changes are in JDK 9 for the OpenJFX project to use. >> >> 2. In the jdk repo then it's important to understand that the module system is initialized at startup and there are many places where we need to keep startup performance in mind. This sometimes means less elegant code than might be used if startup wasn't such a big concern. >> >> 3. The changes in the jaxws repo make use of new APIs that means the code doesn't compile with JDK 7 or JDK 8. Our intention is to work with the JAXB and JAX-WS maintainers to address the issues in the upstream project and then bring those changes into jdk9/dev to replace the patches that we are forced to push for the short term. >> >> 4. You will see several tests where the value of the @modules tag has `:open` or `:+open`. This is new jtreg speak. The former means the test is run with --add-opens to open the package, the latter means the test is exported at compile-time and exported + open at run-time (the latter usage will be rare, it's where tests have static references to JDK internal types and are also doing deep reflection with setAccessible). >> >> >> In terms of dates then we are aiming to integrate these changes into jdk9/dev in early December. I will send a follow-up mail next week on this as we work through the logistics. >> >> -Alan >> >> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ >> [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html > From chris.hegarty at oracle.com Mon Nov 28 14:47:24 2016 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 28 Nov 2016 14:47:24 +0000 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> Message-ID: <79ECD41A-D3B4-40C2-AEDE-976C67BFE844@oracle.com> On 24 Nov 2016, at 15:25, Alan Bateman wrote: > > ... > To get going, I've put the webrevs with a snapshot of the changes in jake here: > http://cr.openjdk.java.net/~alanb/8169069/0/ Overall this look very good. I ran through most of the changes in the jdk repo, just a few small comments. 1) SuppressWarnings.java typo element - > elementS 38 * However, note that if a warning is suppressed in a {@code 39 * module-info} file, the suppression applies to elementS within the 40 * file and not to types contained within the module. 2) jartool Main.java Maybe concealedPackages should have a comment about its use ( it is used in the Validator, and not by Main at all ). 3) MethodHandles.java privateLookupIn It might be clearer if the third bullet used {@code lookup}, or 'caller lookup?, or ?given lookup'? The CALLER lookup has the {@link Lookup#MODULE MODULE} lookup mode. 4) ServiceLoader {@code ExtendedCodecsFactory} 111 * will be treated as a provider factory and {@code 112 * ExtendedCodecsFactory.provider()} will be invoked to INSTANTIATE the 113 * provider. Is 'instantiate' strictly true here? Should it simply be ?return' 206 *
  • If a named module declares more than one provider then the providers 207 * are located in the order that they appear in the {@code provides} table of 208 * the {@code Module} class file attribute ({@code module-info.class}).
  • Wow. I assume the JLS, or otherwise, will specify that the order in which the providers are listed in the module-info be preserved, no? Maybe this item could mention that. The class file reference can still be kept, but seems a bit low-level for developers. -Chris. From maurizio.cimadamore at oracle.com Mon Nov 28 11:28:56 2016 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 28 Nov 2016 11:28:56 +0000 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> Message-ID: <117bda90-cf1f-8d33-ef81-b86624f486f6@oracle.com> Hi, the langtools code looks generally ok. Few questions: * Why doesn't 'open' get its own directive in Directive.java - instead of relying on a 'mode' set on an export directive? * ClassReader: should we have checks regarding an open module containing no open directives in the classfile? This seems to be called out in the spec [1] - see section 2.2 * At some point we should investigate better sharing strategy between ClassReader and ModuleNameReader * Names.dynamic seems unused * I note that the classfile attribute name changes are not captured in the spec (but I might be referring to a slightly older version). Maurizio [1] - http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html#jigsaw-2.2 On 24/11/16 15:25, Alan Bateman wrote: > Folks on jigsaw-dev will know that we are on a mission to bring the > changes accumulated in the jake forest to jdk9/dev. We can think of > this as a refresh of the module system in JDK 9, the last big refresh > was in May with many small updates since then. > > The focus this time is to bring the changes that are tied to JSR > issues into jdk9/dev, specifically the issues that are tracked on the > JSR issues list [1] as: > > #CompileTimeDependences > #AddExportsInManifest > #ClassFileModuleName > #ClassFileAccPublic > #ServiceLoaderEnhancements > #ResourceEncapsulation/#ClassFilesAsResources > #ReflectiveAccessToNonExportedTypes > #AwkwardStrongEncapsulation > #ReadabilityAddedByLayerCreator > #IndirectQualifiedReflectiveAccess (partial) > #VersionsInModuleNames > #NonHierarchicalLayers > #ModuleAnnotations/#ModuleDeprecation > #ReflectiveAccessByInstrumentationAgents > > Some of these issues are not "Resolved" yet, meaning there is still > ongoing discussion on the EG mailing list. That is okay, there is > nothing final here. If there are changes to these proposals then the > implementation changes will follow. Also, as I said in a mail to > jigsaw-dev yesterday [2], is that we will keep the jake forest open > for ongoing prototyping and iteration, also ongoing implementation > improvements where iteration or bake time is important. > > For the code review then the focus is therefore on sanity checking the > changes that we would like to bring into jdk9/dev. We will not use > this review thread to debate alternative designs or other big > implementation changes that are more appropriate to bake in jake. > > To get going, I've put the webrevs with a snapshot of the changes in > jake here: > http://cr.openjdk.java.net/~alanb/8169069/0/ > > The changes are currently sync'ed against jdk-9+146 and will be > rebased (and re-tested) against jdk9/dev prior to integration. There > are a number of small changes that need to be added to this in the > coming days, I will refresh the webrev every few days to take account > of these updates. > > > A few important points to mention, even if you aren't reviewing the > changes: > > 1. This refresh requires a new version of jtreg to run the tests. The > changes for this new version are in the code-tools/jtreg repository > and the plan is to tag a new build (jtreg4.2-b04) next week. Once the > tag has been added then we'll update the requiredVersion property in > each TEST.ROOT to force everyone to update. > > 2. For developers trying out modules with the main line JDK 9 builds > then be aware that `requires public` changes to `requires transitive` > and the `provides` clause changes to require all providers for a > specific service type to be in the same clause. Also be aware that the > binary form of the module declaration (module-info.class) changes so > you will need to recompile any modules. > > 3. Those running existing code on JDK 9 and ignoring modules will need > to be aware of a disruptive change in this refresh. The disruptive > change is #AwkwardStrongEncapsulation where setAccessible(true) is > changed so that it can't be used to break into non-public > fields/methods of JDK classes. This change is going to expose a lot of > hacks in existing code. We plan to send mail to jdk9-dev in advance of > this integration to create awareness of this change. As per the > original introduction of strong encapsulation then command line > options (and now the manifest of application JAR files) can be used to > keep existing code working. The new option is `--add-opens` to open a > package in a module for deep reflection by other modules. As an > example, if you find yourself with code that hacks into the private > `comparator` field in java.util.TreeMap then running with `--add-opens > java.base/java.util=ALL-UNNAMED` will keep that code working. > > > A few miscellaneous notes for those that are reviewing: > > 1. We have some temporary/transition code in the top-level repo to > deal with the importing of the JavaFX modules. This will be removed > once the changes are in JDK 9 for the OpenJFX project to use. > > 2. In the jdk repo then it's important to understand that the module > system is initialized at startup and there are many places where we > need to keep startup performance in mind. This sometimes means less > elegant code than might be used if startup wasn't such a big concern. > > 3. The changes in the jaxws repo make use of new APIs that means the > code doesn't compile with JDK 7 or JDK 8. Our intention is to work > with the JAXB and JAX-WS maintainers to address the issues in the > upstream project and then bring those changes into jdk9/dev to replace > the patches that we are forced to push for the short term. > > 4. You will see several tests where the value of the @modules tag has > `:open` or `:+open`. This is new jtreg speak. The former means the > test is run with --add-opens to open the package, the latter means the > test is exported at compile-time and exported + open at run-time (the > latter usage will be rare, it's where tests have static references to > JDK internal types and are also doing deep reflection with > setAccessible). > > > In terms of dates then we are aiming to integrate these changes into > jdk9/dev in early December. I will send a follow-up mail next week on > this as we work through the logistics. > > -Alan > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ > [2] > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html From jan.lahoda at oracle.com Mon Nov 28 14:53:59 2016 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Mon, 28 Nov 2016 15:53:59 +0100 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <117bda90-cf1f-8d33-ef81-b86624f486f6@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> <117bda90-cf1f-8d33-ef81-b86624f486f6@oracle.com> Message-ID: <583C4507.9010702@oracle.com> Thanks for the comments Maurizio. On 28.11.2016 12:28, Maurizio Cimadamore wrote: > Hi, > the langtools code looks generally ok. Few questions: > > * Why doesn't 'open' get its own directive in Directive.java - instead > of relying on a 'mode' set on an export directive? It seemed to me that having two directive interfaces in the API for directives that have the same structure was unnecessary (as we don't have MethodElement and ConstructorElement, but just ExecutableElement, or TypeElement that represents a class, an interface, an annotation type or an enum type). If you think it would be better to have separate interfaces for exports and opens, I am OK with that as well. > > * ClassReader: should we have checks regarding an open module containing > no open directives in the classfile? This seems to be called out in the > spec [1] - see section 2.2 I think such checks would be fine, working on a patch. > > * At some point we should investigate better sharing strategy between > ClassReader and ModuleNameReader > > * Names.dynamic seems unused Fixed: http://hg.openjdk.java.net/jigsaw/jake/langtools/rev/8156e205fcb4 Jan > > * I note that the classfile attribute name changes are not captured in > the spec (but I might be referring to a slightly older version). > > Maurizio > > [1] - http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html#jigsaw-2.2 > > > > > > > On 24/11/16 15:25, Alan Bateman wrote: >> Folks on jigsaw-dev will know that we are on a mission to bring the >> changes accumulated in the jake forest to jdk9/dev. We can think of >> this as a refresh of the module system in JDK 9, the last big refresh >> was in May with many small updates since then. >> >> The focus this time is to bring the changes that are tied to JSR >> issues into jdk9/dev, specifically the issues that are tracked on the >> JSR issues list [1] as: >> >> #CompileTimeDependences >> #AddExportsInManifest >> #ClassFileModuleName >> #ClassFileAccPublic >> #ServiceLoaderEnhancements >> #ResourceEncapsulation/#ClassFilesAsResources >> #ReflectiveAccessToNonExportedTypes >> #AwkwardStrongEncapsulation >> #ReadabilityAddedByLayerCreator >> #IndirectQualifiedReflectiveAccess (partial) >> #VersionsInModuleNames >> #NonHierarchicalLayers >> #ModuleAnnotations/#ModuleDeprecation >> #ReflectiveAccessByInstrumentationAgents >> >> Some of these issues are not "Resolved" yet, meaning there is still >> ongoing discussion on the EG mailing list. That is okay, there is >> nothing final here. If there are changes to these proposals then the >> implementation changes will follow. Also, as I said in a mail to >> jigsaw-dev yesterday [2], is that we will keep the jake forest open >> for ongoing prototyping and iteration, also ongoing implementation >> improvements where iteration or bake time is important. >> >> For the code review then the focus is therefore on sanity checking the >> changes that we would like to bring into jdk9/dev. We will not use >> this review thread to debate alternative designs or other big >> implementation changes that are more appropriate to bake in jake. >> >> To get going, I've put the webrevs with a snapshot of the changes in >> jake here: >> http://cr.openjdk.java.net/~alanb/8169069/0/ >> >> The changes are currently sync'ed against jdk-9+146 and will be >> rebased (and re-tested) against jdk9/dev prior to integration. There >> are a number of small changes that need to be added to this in the >> coming days, I will refresh the webrev every few days to take account >> of these updates. >> >> >> A few important points to mention, even if you aren't reviewing the >> changes: >> >> 1. This refresh requires a new version of jtreg to run the tests. The >> changes for this new version are in the code-tools/jtreg repository >> and the plan is to tag a new build (jtreg4.2-b04) next week. Once the >> tag has been added then we'll update the requiredVersion property in >> each TEST.ROOT to force everyone to update. >> >> 2. For developers trying out modules with the main line JDK 9 builds >> then be aware that `requires public` changes to `requires transitive` >> and the `provides` clause changes to require all providers for a >> specific service type to be in the same clause. Also be aware that the >> binary form of the module declaration (module-info.class) changes so >> you will need to recompile any modules. >> >> 3. Those running existing code on JDK 9 and ignoring modules will need >> to be aware of a disruptive change in this refresh. The disruptive >> change is #AwkwardStrongEncapsulation where setAccessible(true) is >> changed so that it can't be used to break into non-public >> fields/methods of JDK classes. This change is going to expose a lot of >> hacks in existing code. We plan to send mail to jdk9-dev in advance of >> this integration to create awareness of this change. As per the >> original introduction of strong encapsulation then command line >> options (and now the manifest of application JAR files) can be used to >> keep existing code working. The new option is `--add-opens` to open a >> package in a module for deep reflection by other modules. As an >> example, if you find yourself with code that hacks into the private >> `comparator` field in java.util.TreeMap then running with `--add-opens >> java.base/java.util=ALL-UNNAMED` will keep that code working. >> >> >> A few miscellaneous notes for those that are reviewing: >> >> 1. We have some temporary/transition code in the top-level repo to >> deal with the importing of the JavaFX modules. This will be removed >> once the changes are in JDK 9 for the OpenJFX project to use. >> >> 2. In the jdk repo then it's important to understand that the module >> system is initialized at startup and there are many places where we >> need to keep startup performance in mind. This sometimes means less >> elegant code than might be used if startup wasn't such a big concern. >> >> 3. The changes in the jaxws repo make use of new APIs that means the >> code doesn't compile with JDK 7 or JDK 8. Our intention is to work >> with the JAXB and JAX-WS maintainers to address the issues in the >> upstream project and then bring those changes into jdk9/dev to replace >> the patches that we are forced to push for the short term. >> >> 4. You will see several tests where the value of the @modules tag has >> `:open` or `:+open`. This is new jtreg speak. The former means the >> test is run with --add-opens to open the package, the latter means the >> test is exported at compile-time and exported + open at run-time (the >> latter usage will be rare, it's where tests have static references to >> JDK internal types and are also doing deep reflection with >> setAccessible). >> >> >> In terms of dates then we are aiming to integrate these changes into >> jdk9/dev in early December. I will send a follow-up mail next week on >> this as we work through the logistics. >> >> -Alan >> >> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ >> [2] >> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html >> > From kirill.zhaldybin at oracle.com Mon Nov 28 15:59:22 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Mon, 28 Nov 2016 18:59:22 +0300 Subject: RFR(S): 8166156: Convert test_semaphore to GTest In-Reply-To: References: <5838974C.9000603@oracle.com> Message-ID: Stefan, On 28.11.2016 16:01, Stefan Karlsson wrote: > Hi Kirill, > > On 2016-11-25 20:55, Kirill Zhaldybin wrote: >> Dear all, >> >> Could you please review this fix for 8166156? >> Pretty straight-forward conversion. >> >> WebRev: >> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.00/ >> CR: https://bugs.openjdk.java.net/browse/JDK-8166156 > > Thanks for porting this test. > > I think it would be preferable if this test was using TEST instead of > TEST_VM. I don't see anything in the test that requires the VM to run. Thank you for noticing this. Here are a new WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.01/ Could you please let me know your opinion? Regards, Kirill > > Thanks, > StefanK > >> >> Thank you. >> >> Regards, Kirill From stefan.karlsson at oracle.com Mon Nov 28 15:58:01 2016 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Mon, 28 Nov 2016 16:58:01 +0100 Subject: RFR(S): 8166156: Convert test_semaphore to GTest In-Reply-To: References: <5838974C.9000603@oracle.com> Message-ID: <731f97e6-9439-95d5-a199-e655df76fc63@oracle.com> Looks good. Thanks, StefanK On 2016-11-28 16:59, Kirill Zhaldybin wrote: > Stefan, > > On 28.11.2016 16:01, Stefan Karlsson wrote: >> Hi Kirill, >> >> On 2016-11-25 20:55, Kirill Zhaldybin wrote: >>> Dear all, >>> >>> Could you please review this fix for 8166156? >>> Pretty straight-forward conversion. >>> >>> WebRev: >>> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.00/ >>> CR: https://bugs.openjdk.java.net/browse/JDK-8166156 >> >> Thanks for porting this test. >> >> I think it would be preferable if this test was using TEST instead of >> TEST_VM. I don't see anything in the test that requires the VM to run. > Thank you for noticing this. > > Here are a new WebRev: > http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.01/ > > Could you please let me know your opinion? > > Regards, Kirill >> >> Thanks, >> StefanK >> >>> >>> Thank you. >>> >>> Regards, Kirill > From maurizio.cimadamore at oracle.com Mon Nov 28 16:01:05 2016 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 28 Nov 2016 16:01:05 +0000 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <583C4507.9010702@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> <117bda90-cf1f-8d33-ef81-b86624f486f6@oracle.com> <583C4507.9010702@oracle.com> Message-ID: On 28/11/16 14:53, Jan Lahoda wrote: > Thanks for the comments Maurizio. > > On 28.11.2016 12:28, Maurizio Cimadamore wrote: >> Hi, >> the langtools code looks generally ok. Few questions: >> >> * Why doesn't 'open' get its own directive in Directive.java - instead >> of relying on a 'mode' set on an export directive? > > It seemed to me that having two directive interfaces in the API for > directives that have the same structure was unnecessary (as we don't > have MethodElement and ConstructorElement, but just ExecutableElement, > or TypeElement that represents a class, an interface, an annotation > type or an enum type). > > If you think it would be better to have separate interfaces for > exports and opens, I am OK with that as well. I agree that it would be redundant - perhaps the two directive can both share a common superclass which defines the common fields? I said that because it looks like for everything else, 'opens' and 'exports' are really two separate directives, with separate bytecode encodings and such (that is, the Module attribute has separate entries for 'opens' and 'exports'). > >> >> * ClassReader: should we have checks regarding an open module containing >> no open directives in the classfile? This seems to be called out in the >> spec [1] - see section 2.2 > > I think such checks would be fine, working on a patch. ok > >> >> * At some point we should investigate better sharing strategy between >> ClassReader and ModuleNameReader >> >> * Names.dynamic seems unused > > Fixed: > http://hg.openjdk.java.net/jigsaw/jake/langtools/rev/8156e205fcb4 Thanks Maurizio > Jan > >> >> * I note that the classfile attribute name changes are not captured in >> the spec (but I might be referring to a slightly older version). >> >> Maurizio >> >> [1] - http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html#jigsaw-2.2 >> >> >> >> >> >> >> On 24/11/16 15:25, Alan Bateman wrote: >>> Folks on jigsaw-dev will know that we are on a mission to bring the >>> changes accumulated in the jake forest to jdk9/dev. We can think of >>> this as a refresh of the module system in JDK 9, the last big refresh >>> was in May with many small updates since then. >>> >>> The focus this time is to bring the changes that are tied to JSR >>> issues into jdk9/dev, specifically the issues that are tracked on the >>> JSR issues list [1] as: >>> >>> #CompileTimeDependences >>> #AddExportsInManifest >>> #ClassFileModuleName >>> #ClassFileAccPublic >>> #ServiceLoaderEnhancements >>> #ResourceEncapsulation/#ClassFilesAsResources >>> #ReflectiveAccessToNonExportedTypes >>> #AwkwardStrongEncapsulation >>> #ReadabilityAddedByLayerCreator >>> #IndirectQualifiedReflectiveAccess (partial) >>> #VersionsInModuleNames >>> #NonHierarchicalLayers >>> #ModuleAnnotations/#ModuleDeprecation >>> #ReflectiveAccessByInstrumentationAgents >>> >>> Some of these issues are not "Resolved" yet, meaning there is still >>> ongoing discussion on the EG mailing list. That is okay, there is >>> nothing final here. If there are changes to these proposals then the >>> implementation changes will follow. Also, as I said in a mail to >>> jigsaw-dev yesterday [2], is that we will keep the jake forest open >>> for ongoing prototyping and iteration, also ongoing implementation >>> improvements where iteration or bake time is important. >>> >>> For the code review then the focus is therefore on sanity checking the >>> changes that we would like to bring into jdk9/dev. We will not use >>> this review thread to debate alternative designs or other big >>> implementation changes that are more appropriate to bake in jake. >>> >>> To get going, I've put the webrevs with a snapshot of the changes in >>> jake here: >>> http://cr.openjdk.java.net/~alanb/8169069/0/ >>> >>> The changes are currently sync'ed against jdk-9+146 and will be >>> rebased (and re-tested) against jdk9/dev prior to integration. There >>> are a number of small changes that need to be added to this in the >>> coming days, I will refresh the webrev every few days to take account >>> of these updates. >>> >>> >>> A few important points to mention, even if you aren't reviewing the >>> changes: >>> >>> 1. This refresh requires a new version of jtreg to run the tests. The >>> changes for this new version are in the code-tools/jtreg repository >>> and the plan is to tag a new build (jtreg4.2-b04) next week. Once the >>> tag has been added then we'll update the requiredVersion property in >>> each TEST.ROOT to force everyone to update. >>> >>> 2. For developers trying out modules with the main line JDK 9 builds >>> then be aware that `requires public` changes to `requires transitive` >>> and the `provides` clause changes to require all providers for a >>> specific service type to be in the same clause. Also be aware that the >>> binary form of the module declaration (module-info.class) changes so >>> you will need to recompile any modules. >>> >>> 3. Those running existing code on JDK 9 and ignoring modules will need >>> to be aware of a disruptive change in this refresh. The disruptive >>> change is #AwkwardStrongEncapsulation where setAccessible(true) is >>> changed so that it can't be used to break into non-public >>> fields/methods of JDK classes. This change is going to expose a lot of >>> hacks in existing code. We plan to send mail to jdk9-dev in advance of >>> this integration to create awareness of this change. As per the >>> original introduction of strong encapsulation then command line >>> options (and now the manifest of application JAR files) can be used to >>> keep existing code working. The new option is `--add-opens` to open a >>> package in a module for deep reflection by other modules. As an >>> example, if you find yourself with code that hacks into the private >>> `comparator` field in java.util.TreeMap then running with `--add-opens >>> java.base/java.util=ALL-UNNAMED` will keep that code working. >>> >>> >>> A few miscellaneous notes for those that are reviewing: >>> >>> 1. We have some temporary/transition code in the top-level repo to >>> deal with the importing of the JavaFX modules. This will be removed >>> once the changes are in JDK 9 for the OpenJFX project to use. >>> >>> 2. In the jdk repo then it's important to understand that the module >>> system is initialized at startup and there are many places where we >>> need to keep startup performance in mind. This sometimes means less >>> elegant code than might be used if startup wasn't such a big concern. >>> >>> 3. The changes in the jaxws repo make use of new APIs that means the >>> code doesn't compile with JDK 7 or JDK 8. Our intention is to work >>> with the JAXB and JAX-WS maintainers to address the issues in the >>> upstream project and then bring those changes into jdk9/dev to replace >>> the patches that we are forced to push for the short term. >>> >>> 4. You will see several tests where the value of the @modules tag has >>> `:open` or `:+open`. This is new jtreg speak. The former means the >>> test is run with --add-opens to open the package, the latter means the >>> test is exported at compile-time and exported + open at run-time (the >>> latter usage will be rare, it's where tests have static references to >>> JDK internal types and are also doing deep reflection with >>> setAccessible). >>> >>> >>> In terms of dates then we are aiming to integrate these changes into >>> jdk9/dev in early December. I will send a follow-up mail next week on >>> this as we work through the logistics. >>> >>> -Alan >>> >>> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ >>> [2] >>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html >>> >>> >> From kirill.zhaldybin at oracle.com Mon Nov 28 16:08:53 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Mon, 28 Nov 2016 19:08:53 +0300 Subject: RFR(S): 8166156: Convert test_semaphore to GTest In-Reply-To: <731f97e6-9439-95d5-a199-e655df76fc63@oracle.com> References: <5838974C.9000603@oracle.com> <731f97e6-9439-95d5-a199-e655df76fc63@oracle.com> Message-ID: <83439ee5-c351-0bff-b00c-408b68ed7554@oracle.com> Stefan, Thank you for review! Regards, Kirill On 28.11.2016 18:58, Stefan Karlsson wrote: > Looks good. > > Thanks, > StefanK > > On 2016-11-28 16:59, Kirill Zhaldybin wrote: >> Stefan, >> >> On 28.11.2016 16:01, Stefan Karlsson wrote: >>> Hi Kirill, >>> >>> On 2016-11-25 20:55, Kirill Zhaldybin wrote: >>>> Dear all, >>>> >>>> Could you please review this fix for 8166156? >>>> Pretty straight-forward conversion. >>>> >>>> WebRev: >>>> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.00/ >>>> CR: https://bugs.openjdk.java.net/browse/JDK-8166156 >>> >>> Thanks for porting this test. >>> >>> I think it would be preferable if this test was using TEST instead of >>> TEST_VM. I don't see anything in the test that requires the VM to run. >> Thank you for noticing this. >> >> Here are a new WebRev: >> http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.01/ >> >> Could you please let me know your opinion? >> >> Regards, Kirill >>> >>> Thanks, >>> StefanK >>> >>>> >>>> Thank you. >>>> >>>> Regards, Kirill >> From kim.barrett at oracle.com Mon Nov 28 16:23:25 2016 From: kim.barrett at oracle.com (Kim Barrett) Date: Mon, 28 Nov 2016 11:23:25 -0500 Subject: RFR(S): 8166156: Convert test_semaphore to GTest In-Reply-To: References: <5838974C.9000603@oracle.com> Message-ID: > On Nov 28, 2016, at 10:59 AM, Kirill Zhaldybin wrote: > > Stefan, > > On 28.11.2016 16:01, Stefan Karlsson wrote: >> Hi Kirill, >> >> On 2016-11-25 20:55, Kirill Zhaldybin wrote: >>> Dear all, >>> >>> Could you please review this fix for 8166156? >>> Pretty straight-forward conversion. >>> >>> WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.00/ >>> CR: https://bugs.openjdk.java.net/browse/JDK-8166156 >> >> Thanks for porting this test. >> >> I think it would be preferable if this test was using TEST instead of TEST_VM. I don't see anything in the test that requires the VM to run. > Thank you for noticing this. > > Here are a new WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.01/ > > Could you please let me know your opinion? Yes, that?s better. From kirill.zhaldybin at oracle.com Mon Nov 28 17:10:41 2016 From: kirill.zhaldybin at oracle.com (Kirill Zhaldybin) Date: Mon, 28 Nov 2016 20:10:41 +0300 Subject: RFR(S): 8166156: Convert test_semaphore to GTest In-Reply-To: References: <5838974C.9000603@oracle.com> Message-ID: <121e8158-c852-1688-7014-e4cd79adc562@oracle.com> Kim, Thank you! Regards, Kirill On 28.11.2016 19:23, Kim Barrett wrote: >> On Nov 28, 2016, at 10:59 AM, Kirill Zhaldybin wrote: >> >> Stefan, >> >> On 28.11.2016 16:01, Stefan Karlsson wrote: >>> Hi Kirill, >>> >>> On 2016-11-25 20:55, Kirill Zhaldybin wrote: >>>> Dear all, >>>> >>>> Could you please review this fix for 8166156? >>>> Pretty straight-forward conversion. >>>> >>>> WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.00/ >>>> CR: https://bugs.openjdk.java.net/browse/JDK-8166156 >>> Thanks for porting this test. >>> >>> I think it would be preferable if this test was using TEST instead of TEST_VM. I don't see anything in the test that requires the VM to run. >> Thank you for noticing this. >> >> Here are a new WebRev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8166156/webrev.01/ >> >> Could you please let me know your opinion? > Yes, that?s better. > From jiangli.zhou at Oracle.COM Mon Nov 28 21:52:58 2016 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Mon, 28 Nov 2016 13:52:58 -0800 Subject: RFR: 8170297: runtime/SharedArchiveFile/LargeSharedSpace.java didn't run out of memory Message-ID: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> Please review the following fix for test bug JDK-8170297 . http://cr.openjdk.java.net/~jiangli/8170297/webrev.00/ On some test platforms, the archive might be dumped successfully with the specified large size. Added an extra level of checking in the test. Tested with linux-64, linux-32 and aarch64. Thanks, Jiangli From calvin.cheung at oracle.com Mon Nov 28 22:05:17 2016 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Mon, 28 Nov 2016 14:05:17 -0800 Subject: RFR: 8170297: runtime/SharedArchiveFile/LargeSharedSpace.java didn't run out of memory In-Reply-To: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> References: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> Message-ID: <583CAA1D.6000902@oracle.com> Hi Jiangli, The fix looks good to me. thanks, Calvin On 11/28/16, 1:52 PM, Jiangli Zhou wrote: > Please review the following fix for test bug JDK-8170297. > > http://cr.openjdk.java.net/~jiangli/8170297/webrev.00/ > > On some test platforms, the archive might be dumped successfully with the specified large size. Added an extra level of checking in the test. Tested with linux-64, linux-32 and aarch64. > > Thanks, > Jiangli From jiangli.zhou at oracle.com Mon Nov 28 22:11:40 2016 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 28 Nov 2016 14:11:40 -0800 Subject: RFR: 8170297: runtime/SharedArchiveFile/LargeSharedSpace.java didn't run out of memory In-Reply-To: <583CAA1D.6000902@oracle.com> References: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> <583CAA1D.6000902@oracle.com> Message-ID: <4799E03B-EFC8-491E-BF15-5FF3806C2104@oracle.com> Thanks, Calvin! Thanks, Jiangli > On Nov 28, 2016, at 2:05 PM, Calvin Cheung wrote: > > Hi Jiangli, > > The fix looks good to me. > > thanks, > Calvin > >> On 11/28/16, 1:52 PM, Jiangli Zhou wrote: >> Please review the following fix for test bug JDK-8170297. >> >> http://cr.openjdk.java.net/~jiangli/8170297/webrev.00/ >> >> On some test platforms, the archive might be dumped successfully with the specified large size. Added an extra level of checking in the test. Tested with linux-64, linux-32 and aarch64. >> >> Thanks, >> Jiangli From david.holmes at oracle.com Mon Nov 28 22:16:12 2016 From: david.holmes at oracle.com (David Holmes) Date: Tue, 29 Nov 2016 08:16:12 +1000 Subject: RFR: 8170297: runtime/SharedArchiveFile/LargeSharedSpace.java didn't run out of memory In-Reply-To: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> References: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> Message-ID: <64cebc75-ef4c-16c7-2a01-e1204a792c30@oracle.com> Hi Jiangli, On 29/11/2016 7:52 AM, Jiangli Zhou wrote: > Please review the following fix for test bug JDK-8170297 . > > http://cr.openjdk.java.net/~jiangli/8170297/webrev.00/ > > On some test platforms, the archive might be dumped successfully with the specified large size. Added an extra level of checking in the test. Tested with linux-64, linux-32 and aarch64. I have a problem with this test. :) The test says "the size is too large, exceeding the limit for compressed klass support" - so how can it pass? Is there a limit or not? If this size is too small to fail should we be using a bigger size? Does it need to be dynamic based on the host machine? Do we need to calculate a size based on available memory? Style nit: you added a few // comments but this test is using /* ... */ comments. The mix looks odd to me especially when comments run into each other. Thanks, David > Thanks, > Jiangli > From ioi.lam at oracle.com Mon Nov 28 22:47:06 2016 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 28 Nov 2016 14:47:06 -0800 Subject: RFR: 8170297: runtime/SharedArchiveFile/LargeSharedSpace.java didn't run out of memory In-Reply-To: <64cebc75-ef4c-16c7-2a01-e1204a792c30@oracle.com> References: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> <64cebc75-ef4c-16c7-2a01-e1204a792c30@oracle.com> Message-ID: <583CB3EA.2020803@oracle.com> On 11/28/16 2:16 PM, David Holmes wrote: > Hi Jiangli, > > On 29/11/2016 7:52 AM, Jiangli Zhou wrote: >> Please review the following fix for test bug JDK-8170297 >> . >> >> http://cr.openjdk.java.net/~jiangli/8170297/webrev.00/ >> >> >> On some test platforms, the archive might be dumped successfully with >> the specified large size. Added an extra level of checking in the >> test. Tested with linux-64, linux-32 and aarch64. > > I have a problem with this test. :) The test says "the size is too > large, exceeding the limit for compressed klass support" - so how can > it pass? Is there a limit or not? If this size is too small to fail > should we be using a bigger size? Does it need to be dynamic based on > the host machine? Do we need to calculate a size based on available > memory? > Maybe the comments should spell out the exact same size of the compress klass size limit, in both hex and decimal forms? Otherwise it will be hard to know what 1066924031 would actually lead to. I would also suggest splitting the test into two different portions, instead of using the loop. It's hard to understand how the 3 output message would related to the 2 input numbers.I think a test case should be clear about exactly what it intends to test, and not just be "here's bunch of input parameters and it should just work". E.g, for the second case (1600386047): "Loading classes to share" should not happen on (64-bit platforms && compressed pointers.) In this case (1600386047), since the intention is to test the effect on the compress klass size limit, you should explicitly pass -XX:+UseCompressedOops -XX:+UseCompressedClassPointersto the end of your command-line, so that they would override any VM options specified via JTREG. Also, LargeSharedSpace.java can be run on 32-bit platforms, but the message " larger than compressed klass limit" only happens in LP64 code. So I think the second test (1600386047) should be excluded from 32-bit platforms by using Platform.is32bit(). #ifdef _LP64 if (cds_total + compressed_class_space_size() > UnscaledClassSpaceMax) { vm_exit_during_initialization("Unable to dump shared archive.", err_msg("Size of archive (" SIZE_FORMAT ") + compressed class space (" SIZE_FORMAT ") == total (" SIZE_FORMAT ") is larger than compressed " "klass limit: " UINT64_FORMAT, cds_total, compressed_class_space_size(), cds_total + compressed_class_space_size(), UnscaledClassSpaceMax)); } ... Thanks - Ioi > Style nit: you added a few // comments but this test is using /* ... > */ comments. The mix looks odd to me especially when comments run into > each other. > > Thanks, > David > >> Thanks, >> Jiangli >> From jiangli.zhou at Oracle.COM Mon Nov 28 22:57:20 2016 From: jiangli.zhou at Oracle.COM (Jiangli Zhou) Date: Mon, 28 Nov 2016 14:57:20 -0800 Subject: RFR: 8170297: runtime/SharedArchiveFile/LargeSharedSpace.java didn't run out of memory In-Reply-To: <583CB3EA.2020803@oracle.com> References: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> <64cebc75-ef4c-16c7-2a01-e1204a792c30@oracle.com> <583CB3EA.2020803@oracle.com> Message-ID: Hi David and Ioi, Thank you so much for the comments! For the compressed klass limit, it is determined by "shared space size + compressed klass space size". I agree, the test case and comments are not well defined for that. I?ll update the test to incorporate your suggestions. Thanks, Jiangli > On Nov 28, 2016, at 2:47 PM, Ioi Lam wrote: > > > > On 11/28/16 2:16 PM, David Holmes wrote: >> Hi Jiangli, >> >> On 29/11/2016 7:52 AM, Jiangli Zhou wrote: >>> Please review the following fix for test bug JDK-8170297 . >>> >>> http://cr.openjdk.java.net/~jiangli/8170297/webrev.00/ >>> >>> On some test platforms, the archive might be dumped successfully with the specified large size. Added an extra level of checking in the test. Tested with linux-64, linux-32 and aarch64. >> >> I have a problem with this test. :) The test says "the size is too large, exceeding the limit for compressed klass support" - so how can it pass? Is there a limit or not? If this size is too small to fail should we be using a bigger size? Does it need to be dynamic based on the host machine? Do we need to calculate a size based on available memory? >> > > Maybe the comments should spell out the exact same size of the compress klass > size limit, in both hex and decimal forms? Otherwise it will be hard to know > what 1066924031 would actually lead to. > > I would also suggest splitting the test into two different portions, instead > of using the loop. It's hard to understand how the 3 output message would related > to the 2 input numbers.I think a test case should be clear about exactly what it > intends to test, and not just be "here's bunch of input parameters and it > should just work". > > > E.g, for the second case (1600386047): > > "Loading classes to share" should not happen on (64-bit platforms && compressed > pointers.) > > In this case (1600386047), since the intention is to test the effect on > the compress klass size limit, you should explicitly pass -XX:+UseCompressedOops > -XX:+UseCompressedClassPointersto the end of your command-line, so that > they would override any VM options specified via JTREG. > > Also, LargeSharedSpace.java can be run on 32-bit platforms, but the > message " larger than compressed klass limit" only happens in LP64 code. > So I think the second test (1600386047) should be excluded from 32-bit platforms > by using Platform.is32bit(). > > #ifdef _LP64 > if (cds_total + compressed_class_space_size() > UnscaledClassSpaceMax) { > vm_exit_during_initialization("Unable to dump shared archive.", > err_msg("Size of archive (" SIZE_FORMAT ") + compressed class space (" > SIZE_FORMAT ") == total (" SIZE_FORMAT ") is larger than compressed " > "klass limit: " UINT64_FORMAT, cds_total, compressed_class_space_size(), > cds_total + compressed_class_space_size(), UnscaledClassSpaceMax)); > } > ... > > Thanks > - Ioi > > >> Style nit: you added a few // comments but this test is using /* ... */ comments. The mix looks odd to me especially when comments run into each other. >> >> Thanks, >> David >> >>> Thanks, >>> Jiangli >>> > From daniel.daugherty at oracle.com Tue Nov 29 00:38:21 2016 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Mon, 28 Nov 2016 17:38:21 -0700 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> Message-ID: <4dfa4ae6-4fa6-994c-f2cf-8fc309d8c520@oracle.com> On 11/28/16 2:08 AM, Lindenmaier, Goetz wrote: > Hi, > > I'm working on a fix for OS guard pages on stacks. I figured there > are VM guard pages (reserved, yellow, red) on the compiler stacks > _and_ OS guard pages. For Java threads, the OS guard pages are left > out. I think this should be done for compiler threads, too. Please > confirm. > Webrev: > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.04/ src/os/aix/vm/os_aix.cpp L888: pthread_attr_setguardsize(&attr, os::Aix::default_guard_size(thr_type)); No check or assert on the return status of this call. Is one needed? L3044: // guard pages, so only enable libc guard pages for non-Java threads. src/os/linux/vm/os_linux.cpp also has this comment: // (Remember: compiler thread is a Java thread, too!) L3051: return ((thr_type == java_thread || thr_type == compiler_thread) ? 0 : 4*K); nit - please add spaces around the '*' so '4 * K'.' src/os/aix/vm/os_aix.hpp No comments. src/os/linux/vm/os_linux.cpp L729: // is not implemented properly. The posix standard requires to add Typo: 'to add' -> 'adding' L738: pthread_attr_setguardsize(&attr, os::Linux::default_guard_size(thr_type)); No check or assert on the return status of this call. Is one needed? L2851: // Creating guard page is very expensive. Java thread has HotSpot L2852: // guard page, only enable glibc guard page for non-Java threads. L2853: // (Remember: compiler thread is a java thread, too!) Typo: "java thread" -> "Java thread" (consistency) This comment block should be common to all the platforms that define default_guard_size(). Yes, I can see that AIX needs to add another paragraph, but we should make the core comment common if possible. L6090: // Java/Compiler thread: Thanks for making this common in os_linux.cpp. L6095: // | glibc guard page | - guard, attached Java thread usually has Clarity: "guard," -> "guard page," Typo: "Java thread" -> "JavaThread" (consistency) L6099: // | HotSpot Guard Pages | - red and yellow pages The fairly recently added reserved page should be mentioned here also? L6120 // ** P1 (aka bottom) and size ( P2 = P1 - size) are the address and stack size returned from Typo: "( P2 = ..." -> "(P2 = ..." L6148: fatal("Can not locate current stack attributes!"); Typo: "Can not" -> "Cannot" L6175: // stack size includes normal stack and HotSpot guard pages Perhaps add to the comment: "for the threads that have HotSpot guard pages." src/os/posix/vm/os_posix.cpp L1097: pthread_attr_getstacksize(attr, &stack_size); L1098: pthread_attr_getguardsize(attr, &guard_size); Do these two calls need to have their result checked? Now I'm starting to wonder if all the uses of these two APIs need to be checked? Separate bug? src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp L540: size_t os::Posix::_compiler_thread_min_stack_allowed = 512 * K; L541: size_t os::Posix::_java_thread_min_stack_allowed = 512 * K; So prior to the fix for 8140520, src/os/aix/vm/os_aix.cpp had this single min_stack_allowed value: L3601: os::Aix::min_stack_allowed = MAX2(os::Aix::min_stack_allowed, L3602: JavaThread::stack_guard_zone_size() + L3603: JavaThread::stack_shadow_zone_size() + L3604: (4*BytesPerWord COMPILER2_PRESENT(+2)) * 4 * K); and the fix for 8140520 changed that for *NIX platforms to three mins in src/os/posix/vm/os_posix.cpp: L1108: _java_thread_min_stack_allowed = MAX2(_java_thread_min_stack_allowed, L1109: JavaThread::stack_guard_zone_size() + L1110: JavaThread::stack_shadow_zone_size() + L1111: (4 * BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K); L1150: _compiler_thread_min_stack_allowed = align_size_up(_compiler_thread_min_stack_allowed, vm_page_size()); L1161 _vm_internal_thread_min_stack_allowed = align_size_up(_vm_internal_thread_min_stack_allowed, vm_page_size()); Which means that the compiler_thread no longer benefits from the extra space for quard and shadow pages. The thinking in 8140520 was that the compiler_thread and vm_internal_threads don't need the quard and shadow pages since they don't run Java code (ignoring JVMCI for now). So I can see bumping _compiler_thread_min_stack_allowed from 128 -> 512 as one solution for getting that extra space back. However, I don't understand why _java_thread_min_stack_allowed has changed from 128 -> 512. I had previously made this comment: > To put it another way, I'd like to see us add extra space to solve > the 64K page issue directly instead of as a side effect of the > red/yellow page addition. And Goetz replied with: > I don't understand. What do you mean by 'directly'? So prior to the fix for 8140520, src/os/solaris/vm/os_solaris.cpp had a block like this: L4468: // For 64kbps there will be a 64kb page size, which makes L4469: // the usable default stack size quite a bit less. Increase the L4470: // stack for 64kb (or any > than 8kb) pages, this increases L4471: // virtual memory fragmentation (since we're not creating the L4472 // stack on a power of 2 boundary. The real fix for this L4473 // should be to fix the guard page mechanism. L4474 L4475 if (vm_page_size() > 8*K) { L4476 threadStackSizeInBytes = (threadStackSizeInBytes != 0) L4477 ? threadStackSizeInBytes + L4478 JavaThread::stack_red_zone_size() + L4479 JavaThread::stack_yellow_zone_size() L4480 : 0; L4481 ThreadStackSize = threadStackSizeInBytes/K; L4482 } The above is an example of what I mean by solving the 64K page issue directly. In the fix for 8140520, that code block was moved to os::Posix::set_minimum_stack_sizes() in src/os/posix/vm/os_posix.cpp and put in a "#ifdef SOLARIS... #endif // SOLARIS" block. Coleen filed a bug to determine whether that code can be deleted: JDK-8161093 Solaris for >8k pagesize adds extra guard pages https://bugs.openjdk.java.net/browse/JDK-8161093 but perhaps this bug shows that the code is needed? OK so this is probably the longest code review comment I have ever written, but the summary is: - I understand bumping _compiler_thread_min_stack_allowed, but should it be solved in a different way? - I don't understand bumping _java_thread_min_stack_allowed src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp No comments. src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp L538: size_t os::Posix::_compiler_thread_min_stack_allowed = 384 * K; L539: size_t os::Posix::_java_thread_min_stack_allowed = 384 * K; Same monster comment as src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp and the same summary: - I understand bumping _compiler_thread_min_stack_allowed, but should it be solved in a different way? - I don't understand bumping _java_thread_min_stack_allowed src/os_cpu/linux_s390/vm/os_linux_s390.cpp L478: size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; L479: size_t os::Posix::_java_thread_min_stack_allowed = 236 * K; Bumping _java_thread_min_stack_allowed but not bumping _compiler_thread_min_stack_allowed. I'm confused here. src/os_cpu/linux_sparc/vm/os_linux_sparc.cpp No comments. src/os_cpu/linux_x86/vm/os_linux_x86.cpp No comments. src/os_cpu/linux_zero/vm/os_linux_zero.cpp No comments. Sorry it took me so long to write this up... Dan > The change affecting the compier threads is in os_linux.cpp, default_guard_size(), > where '|| thr_type == compiler_thread' has been added. The function > was also moved from the os_cpu files, as it's identical on all cpus. > > Best regards, > Goetz. > > >> -----Original Message----- >> From: David Holmes [mailto:david.holmes at oracle.com] >> Sent: Montag, 28. November 2016 00:25 >> To: Lindenmaier, Goetz ; >> 'daniel.daugherty at oracle.com' ; 'hotspot- >> runtime-dev at openjdk.java.net' >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >> >> Hi Goetz, >> >> On 24/11/2016 10:15 PM, Lindenmaier, Goetz wrote: >>> Hi, >>> >>> I now edited the stuff I had proposed below: >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.03/ >>> This includes >>> - the NPTL fix from webrev.02 >> Okay in principle. As discussed this only impacts non-JavaThreads so the >> change should be minimal. >> >>> - merging code on linux >> Went a bit further than I had expected but if this truly isn't CPU >> dependent code then great! >> >>> - not adding OS guard to compiler threads. >> Okay in principle. IIUC we will now save the OS guard page for compiler >> thread stacks. Is that the only impact? The hotspot-compiler-dev folk >> may want to sign off on this part. >> >> >> A few minor comments: >> >> src/os/linux/vm/os_linux.cpp >> >> 2854 return ((thr_type == java_thread || thr_type == >> os::compiler_thread) ... >> >> os:: should be used for both types or none. >> >> 6153 pthread_attr_getguardsize(&attr, &guard_size); >> >> Can you at least verify a zero return code in an assert/assert_status >> please. >> >> --- >> >> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp >> src/os_cpu/linux_s390/vm/os_linux_s390.cpp >> >> Are the changes to min_stack_allowed just fixing an existing bug? >> >> --- >> >> Thanks, >> David >> ----- >> >> >>> I think this should be pushed for this bug ID. For the other changes I'll >>> make another bug. >>> >>> Best regards, >>> Goetz. >>> >>>> -----Original Message----- >>>> From: Lindenmaier, Goetz >>>> Sent: Wednesday, November 23, 2016 8:11 AM >>>> To: David Holmes ; >>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net >>>> Subject: RE: RFR(M): 8169373: Work around linux NPTL stack guard error. >>>> >>>> Hi, >>>> >>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The >>>>> ability for a CompilerThread to execute Java code (can_call_java()) is >>>>> now a dynamic property depending on whether the current compiler is the >>>>> JVMCI compiler. >>>> Ah, then I should also leave space for shadow pages in the minimal stack >> size >>>> of comiler threads. >>>> >>>> Do we agree on the cleanup and on leaving out the OS guard page on >>>> compiler threads? >>>> Then I would edit a change comprising the NPTL workaround and these >>>> additional changes, and split the other issue into a new bug? I think this >>>> will easy the reviewing. >>>> >>>> Best regards, >>>> Goetz. >>>> >>>>> -----Original Message----- >>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>> Sent: Mittwoch, 23. November 2016 02:50 >>>>> To: Lindenmaier, Goetz ; >>>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net >>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>>>> >>>>> On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: >>>>>> Hi Dan, >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>>>>> Sent: Dienstag, 22. November 2016 14:01 >>>>>>> To: Lindenmaier, Goetz ; David Holmes >>>>>>> ; hotspot-runtime-dev at openjdk.java.net >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >>>> error. >>>>>>> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: >>>>>>>> Hi Dan, >>>>>>>> >>>>>>>> I ran into a row of issues, some errors on the platforms. >>>>>>>> >>>>>>>> What I meant with that comment is that >>>>>>>> os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, >>>>>>>> JavaThread::stack_guard_zone_size() + >>>>>>>> JavaThread::stack_shadow_zone_size() + >>>>>>>> (4*BytesPerWord COMPILER2_PRESENT(+2)) * >> 4 >>>> * >>>>> K); >>>>>>>> was executed, and min_stack_allowed used for all stacks. Now, >>>> compiler >>>>>>> and >>>>>>>> vm minimum stack sizes are not increased by these sizes. >>>>>>> Now I see what you mean. Thanks for clearing this up. >>>>>>> >>>>>>> I should have remembered that part of the change because we went >>>> back >>>>>>> and forth about removing the red/yellow zone pages from the other >>>>>>> threads. In particular, we discussed the compiler thread because it >>>>>>> is-a JavaThread. Our conclusion was that a compiler thread doesn't >>>>>>> execute Java bytecode so we could remove the red/yellow pages... >>>>>> Yes, it does not execute java byte code. >>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The >>>>> ability for a CompilerThread to execute Java code (can_call_java()) is >>>>> now a dynamic property depending on whether the current compiler is the >>>>> JVMCI compiler. >>>>> >>>>> David >>>>> ----- >>>>> >>>>>> Therefore you can remove the shadow pages. There is no code that >>>>>> will bang. >>>>>> But red/yellow pages are protected right during thread startup. >>>>>> Therefore you must have enough space for them. >>>>>> On ppc, we try to protect three 64K pages out of the 128K compiler stack. >>>>>> That obviously fails. >>>>>> >>>>>> Therefore I propose: >>>>>> size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // Set >>>>> platform dependent. >>>>>> in os::Posix::set_minimum_stack_sizes(): >>>>>> _java_thread_min_stack_allowed = _java_thread_min_stack_allowed + >>>>>> JavaThread::stack_guard_zone_size() + >>>>>> JavaThread::stack_shadow_zone_size(); >>>>>> >>>>>> (Similar for _compiler_thread_min_stack_allowed). >>>>>> >>>>>> The minimal stack size is made up of three components: >>>>>> 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot >>>>> implementation/platform/os. >>>>>> 2. Sizes of C++ frames: depends on C++ compiler. >>>>>> These are fixed at compile time. >>>>>> 3. Sizes of red/yellow/reserved/shadow pages. Depends on the system >>>>> the >>>>>> VM is used on. This is not fixed at compile time. (Our ppc machines >>>>> differ >>>>>> in page size.) >>>>>> Therefore 3. should not be included in a compile time constant. >>>>>> >>>>>> >>>>>> >>>>>>> And that decision allowed us to be exposed to the 64K page issue >>>>>>> because the "extra" space isn't there anymore. >>>>>>> >>>>>>>> At least the _compiler_thread_min_stack_allowed should be increased >>>>>>>> similarly by red/yellow zone size. The compiler thread is a Java >>>>>>>> thread and must have space for these zones. >>>>>>> I'm not sure that I completely agree (yet). To me, the red/yellow >>>>>>> pages are there for Java thread stack overflow semantics. Yes, the >>>>>>> compiler thread needs extra space when 64K pages are used, but I >>>>>>> would prefer that we add that space via a different calculation. >>>>>> Yes they are. But compiler threads happen tob e a subclass of >>>>>> Java threads and use the same run() method that puts the pages >>>>>> There. >>>>>> I agree that they are not needed for Compiler threads, nor for >>>>>> CodeCacheSweeperThreads. I don't really now about >>>>>> JvmtiAgentThreads and ServiceThreads, but all of the get the guard >>>> pages >>>>>> because they are derived from JavaThread. >>>>>> >>>>>>> To put it another way, I'd like to see us add extra space to solve >>>>>>> the 64K page issue directly instead of as a side effect of the >>>>>>> red/yellow page addition. >>>>>> I don't understand. What do you mean by 'directly'? >>>>>> >>>>>>>> Also, the change added a test that is failing now. >>>>>>> And that's a "good thing" (TM), right? :-) >>>>>> Yes, it showed a bug and thus raised the need to fix it! :) >>>>>> >>>>>> Best regards, >>>>>> Goetz. >>>>>> >>>>>> >>>>>>> Again, thanks for clarifying 8140520's role in this issue. >>>>>>> >>>>>>> Dan >>>>>>> >>>>>>> >>>>>>>> Best regards, >>>>>>>> Goetz. >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>>>>>>> Sent: Montag, 21. November 2016 17:28 >>>>>>>>> To: David Holmes ; Lindenmaier, Goetz >>>>>>>>> ; hotspot-runtime- >>>>> dev at openjdk.java.net >>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >>>>> error. >>>>>>>>> Sorry for the delayed responses to this thread. I've been on vacation... >>>>>>>>> >>>>>>>>> One comment/query embedded below... >>>>>>>>> >>>>>>>>> >>>>>>>>> On 11/10/16 8:40 PM, David Holmes wrote: >>>>>>>>>> Hi Goetz, >>>>>>>>>> >>>>>>>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>> Hi David, >>>>>>>>>>> >>>>>>>>>>> This issue is different to 6675312, see also my comment in the bug. >>>>>>>>>>> >>>>>>>>>>> It appears running jtreg test >>>>> runtime/Thread/TooSmallStackSize.java, >>>>>>>>>>> with my patch below you can reproduce it on linuxx86_64. You can >>>>> not >>>>>>>>>>> do that with 6675312. Also, I would assume there are systems out >>>>> there >>>>>>>>>>> on x86 that uses 64-K pages, did you run the tests on these? I >>>> would >>>>>>>>>>> assume you get hard crashes with stack overflows in the first C2 >>>>>>>>>>> compilation if there is only 64K usable CompilerThreadStack. >>>>>>>>>>> >>>>>>>>>>> My fix does not affect Java threads, which are the largest amount >>>>>>>>>>> of threads used by the VM. It affects only the non-Java threads. >>>>>>>>>>> It adds one page to these threads. The page does not require >>>>> memory, >>>>>>>>>>> as it's protected. The stack will only require more space if the >>>> thread >>>>>>>>>>> ran into a stack overflow before the fix as else the pages are not >>>>>>>>>>> mapped. >>>>>>>>>>> This are stack overflows that cause hard crashes, at least on ppc >>>> the >>>>> VM >>>>>>>>>>> does not properly catch these stack overflows, so any setup >>>> working >>>>>>> now >>>>>>>>>>> will not run into the additional space. Altogether there should be >>>> no >>>>>>>>>>> effect on running systems besides requiring one more entry in the >>>>>>>>>>> page table per non-Java thread. >>>>>>>>>>> >>>>>>>>>>> The problem is caused by a rather recent change (8140520: >>>> segfault >>>>> on >>>>>>>>>>> solaris-amd64 >>>>>>>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed after >>>>>>>>>>> feature-close. As this was a rather recent change, it must be >>>>>>>>>>> possible to >>>>>>>>>>> fix this follow up issue. What else is this period in the project good >>>>>>>>>>> for if not fixing issues? >>>>>>>>>> So I am seeing a number of factors here. >>>>>>>>>> >>>>>>>>>> First, 8140520, set: >>>>>>>>>> >>>>>>>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >>>>>>>>> So I'm confused by the above comment: >>>>>>>>> >>>>>>>>> > The problem is caused by a rather recent change (8140520: >>>> segfault >>>>>>>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) >>>>>>>>> >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- >>>>>>>>> >>>>> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html >>>>>>>>> shows this change: >>>>>>>>> >>>>>>>>> @@ -531,19 +531,17 @@ >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> ////////////////////////////////////////////////////////////////////////////// >>>>>>>>> // >>>>>>>>> // thread stack >>>>>>>>> >>>>>>>>> -size_t os::Linux::min_stack_allowed = 128*K; >>>>>>>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >>>>>>>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; >>>>>>>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * K; >>>>>>>>> >>>>>>>>> so the existing single variable of 'min_stack_allowed' was >>>>>>>>> replaced by three variables: _compiler_thread_min_stack_allowed, >>>>>>>>> _java_thread_min_stack_allowed, and >>>>>>>>> _vm_internal_thread_min_stack_allowed. >>>>>>>>> >>>>>>>>> The old single variable and the three new variables are all >>>>>>>>> initialized to the same value (128K) so the fix for 8140520 >>>>>>>>> did not change stack sizes for this platform. In fact, only >>>>>>>>> one platform had a size change (Solaris X64). >>>>>>>>> >>>>>>>>> So I'm confused about how the fix for 8140520 caused this problem. >>>>>>>>> >>>>>>>>> Based on David's analysis below, it looks to me like this 64K stack >>>>>>>>> guard page problem should also exist prior to the fix for 8140520. >>>>>>>>> >>>>>>>>> Goetz, can you please explain how 8140520 caused this problem? >>>>>>>>> >>>>>>>>> Dan >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Second on linux PPC it is hardwired to use 2 guard pages: >>>>>>>>>> >>>>>>>>>> return 2 * page_size(); >>>>>>>>>> >>>>>>>>>> Third, you had a pagesize of 64K. >>>>>>>>>> >>>>>>>>>> Fourth, NPTL takes the guard space from the stack space - hence >>>> with >>>>> 2 >>>>>>>>>> x 64K guard, and a 128K stack it was all consumed. >>>>>>>>>> >>>>>>>>>> --- >>>>>>>>>> >>>>>>>>>> In the proposed changes you now only use page_size() for the guard, >>>>> so >>>>>>>>>> that alone would have fixed the observed problem. >>>>>>>>>> >>>>>>>>>> But in addition you want to address the NPTL problem by adding >>>> back >>>>>>>>>> the guard space to the stack size requested. That alone would also >>>>>>>>>> have fixed the observed problem. :) >>>>>>>>>> >>>>>>>>>> But in addition you have increased the minimum stack size: >>>>>>>>>> >>>>>>>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; >>>>>>>>>> >>>>>>>>>> which again, on its own would have fixed the original problem. :) >>>>>>>>>> >>>>>>>>>> Did you really intend to increase the real minimum stack from 128K >>>> to >>>>>>>>>> 256K ? (on a 64K page system) >>>>>>>>>> >>>>>>>>>> --- >>>>>>>>>> >>>>>>>>>> Focusing simply on the shared code change to adjust the requested >>>>>>>>>> stacksize by the amount of guard space (if any), this does not seem >>>>>>>>>> unreasonable. As you note it is restricted to non-JavaThreads and >>>> only >>>>>>>>>> adds a page to reserved stack space. >>>>>>>>>> >>>>>>>>>> My only query now is whether the minimum stacksize detection logic >>>>>>>>>> will correctly report the real minimum stack size (taking into account >>>>>>>>>> the need for the guard page) ? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> David >>>>>>>>>> >>>>>>>>>>> So I really think this issue should be fixed. >>>>>>>>>>> >>>>>>>>>>> Best regards, >>>>>>>>>>> Goetz. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>>>>>>>> Sent: Thursday, November 10, 2016 10:02 PM >>>>>>>>>>>> To: Lindenmaier, Goetz ; hotspot- >>>>>>>>> runtime- >>>>>>>>>>>> dev at openjdk.java.net >>>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack >>>> guard >>>>>>> error. >>>>>>>>>>>> Hi Goetz, >>>>>>>>>>>> >>>>>>>>>>>> As per the bug report, this issue was already known (6675312) and >>>>> we >>>>>>>>>>>> chose not to try and address it due to no reported issues at the >>>>> time. >>>>>>>>>>>> While I see that you have encountered an issue (is it real or >>>>>>>>>>>> fabricated?) I think this change is too intrusive to be applied at this >>>>>>>>>>>> stage of the JDK 9 release cycle, as it will change the stack >>>>>>>>>>>> requirements of every application running on Linux. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> David >>>>>>>>>>>> >>>>>>>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Please review this change. I please need a sponsor: >>>>>>>>>>>>> >>>>>>>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >>>>>>>>> stackFix/webrev.01/ >>>>>>>>>>>>> >>>>>>>>>>>>> In the Linux NPTL pthread implementation the guard size >>>>> mechanism >>>>>>>>>>>>> is not >>>>>>>>>>>>> implemented properly. The posix standard requires to add the >>>>> size >>>>>>>>>>>>> of the >>>>>>>>>>>>> guard pages to the stack size, instead Linux takes the space out >>>> of >>>>>>>>>>>>> 'stacksize'. >>>>>>>>>>>>> >>>>>>>>>>>>> The Posix standard >>>>>>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ >>>>>>>>>>>>> says "the implementation allocates extra memory at the >>>>> overflow >>>>>>>>>>>>> end of >>>>>>>>>>>>> the stack". The linux man page >>>>>>>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As >>>>> at >>>>>>>>>>>>> glibc >>>>>>>>>>>>> 2.8, the NPTL threading implementation includes the guard area >>>>>>> within >>>>>>>>>>>>> the stack size allocation, rather than allocating extra space at >>>>>>>>>>>>> the end >>>>>>>>>>>>> of the stack, as POSIX.1 requires". >>>>>>>>>>>>> >>>>>>>>>>>>> I encounter this problem in >>>>> runtime/Thread/TooSmallStackSize.java >>>>>>>>>>>>> on ppc >>>>>>>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K >>>> on >>>>>>> ppc, >>>>>>>>> and >>>>>>>>>>>>> ppc specifies two OS guard pages. The VM crashes in pthread >>>>>>> creation >>>>>>>>>>>>> because there is no usable space in the thread stack after >>>>> allocating >>>>>>>>>>>>> the guard pages. >>>>>>>>>>>>> >>>>>>>>>>>>> But TooSmallStackSize.java requires that the VM comes up with >>>>> the >>>>>>>>>>>>> stack >>>>>>>>>>>>> size mentioned in the error message. >>>>>>>>>>>>> >>>>>>>>>>>>> This fix adapts the requested stack size on Linux by the size of >>>> the >>>>>>>>>>>>> guard pages to mimick proper behaviour, see change to >>>>>>> os_linux.cpp. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> The change also streamlines usage of stack_guard_page on >>>>> linuxppc, >>>>>>>>>>>>> linuxppcle, aixppc and linuxs390. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> To reproduce the error on linux_x86_64, apply below patch and >>>>> call >>>>>>> the >>>>>>>>>>>>> VM with -XX:CompilerThreadStackSize=64. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I'm still exploring why I had to choose such big compiler stacks >>>> on >>>>>>>>>>>>> ppc >>>>>>>>>>>>> to get -version passing, but I wanted to send the RFR now as >>>>> people >>>>>>>>>>>>> obviously looked at the bug I opened (Thanks David!). >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Best regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Goetz. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> diff -r b7ae012c55c3 src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>>>>>>>>>>>> >>>>>>>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 >>>>>>> 12:37:28 >>>>>>>>>>>> 2016 >>>>>>>>>>>>> +0100 >>>>>>>>>>>>> >>>>>>>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 >>>>>>>>> 16:52:17 >>>>>>>>>>>> 2016 >>>>>>>>>>>>> +0100 >>>>>>>>>>>>> >>>>>>>>>>>>> @@ -701,7 +701,7 @@ >>>>>>>>>>>>> >>>>>>>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>>>>>>>>>>>> >>>>>>>>>>>>> // Creating guard page is very expensive. Java thread has >>>>> HotSpot >>>>>>>>>>>>> // guard page, only enable glibc guard page for non-Java >>>>> threads. >>>>>>>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); >>>>>>>>>>>>> >>>>>>>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); >>>>>>>>>>>>> >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> // Java thread: >>>>>>>>>>>>> From jiangli.zhou at oracle.com Tue Nov 29 03:00:29 2016 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 28 Nov 2016 19:00:29 -0800 Subject: RFR: 8170297: runtime/SharedArchiveFile/LargeSharedSpace.java didn't run out of memory In-Reply-To: References: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> <64cebc75-ef4c-16c7-2a01-e1204a792c30@oracle.com> <583CB3EA.2020803@oracle.com> Message-ID: <85F81DC4-7289-4015-BF3B-4B526EB93527@oracle.com> Hi David and Ioi, Here is the updated test: http://cr.openjdk.java.net/~jiangli/8170297/webrev.01/ I?ve split the test cases and reworked the comments. The original bug (8169870) was found on 32-bit platform, so it is worth to test -XX:SharedMiscCodeSize=1600386047 on 32-bit. Tested on linux-64, linux-32 and aarch64. Also running RBT. Thanks, Jiangli > On Nov 28, 2016, at 2:57 PM, Jiangli Zhou wrote: > > Hi David and Ioi, > > Thank you so much for the comments! > > For the compressed klass limit, it is determined by "shared space size + compressed klass space size". I agree, the test case and comments are not well defined for that. I?ll update the test to incorporate your suggestions. > > Thanks, > Jiangli > >> On Nov 28, 2016, at 2:47 PM, Ioi Lam wrote: >> >> >> >> On 11/28/16 2:16 PM, David Holmes wrote: >>> Hi Jiangli, >>> >>> On 29/11/2016 7:52 AM, Jiangli Zhou wrote: >>>> Please review the following fix for test bug JDK-8170297 . >>>> >>>> http://cr.openjdk.java.net/~jiangli/8170297/webrev.00/ >>>> >>>> On some test platforms, the archive might be dumped successfully with the specified large size. Added an extra level of checking in the test. Tested with linux-64, linux-32 and aarch64. >>> >>> I have a problem with this test. :) The test says "the size is too large, exceeding the limit for compressed klass support" - so how can it pass? Is there a limit or not? If this size is too small to fail should we be using a bigger size? Does it need to be dynamic based on the host machine? Do we need to calculate a size based on available memory? >>> >> >> Maybe the comments should spell out the exact same size of the compress klass >> size limit, in both hex and decimal forms? Otherwise it will be hard to know >> what 1066924031 would actually lead to. >> >> I would also suggest splitting the test into two different portions, instead >> of using the loop. It's hard to understand how the 3 output message would related >> to the 2 input numbers.I think a test case should be clear about exactly what it >> intends to test, and not just be "here's bunch of input parameters and it >> should just work". >> >> >> E.g, for the second case (1600386047): >> >> "Loading classes to share" should not happen on (64-bit platforms && compressed >> pointers.) >> >> In this case (1600386047), since the intention is to test the effect on >> the compress klass size limit, you should explicitly pass -XX:+UseCompressedOops >> -XX:+UseCompressedClassPointersto the end of your command-line, so that >> they would override any VM options specified via JTREG. >> >> Also, LargeSharedSpace.java can be run on 32-bit platforms, but the >> message " larger than compressed klass limit" only happens in LP64 code. >> So I think the second test (1600386047) should be excluded from 32-bit platforms >> by using Platform.is32bit(). >> >> #ifdef _LP64 >> if (cds_total + compressed_class_space_size() > UnscaledClassSpaceMax) { >> vm_exit_during_initialization("Unable to dump shared archive.", >> err_msg("Size of archive (" SIZE_FORMAT ") + compressed class space (" >> SIZE_FORMAT ") == total (" SIZE_FORMAT ") is larger than compressed " >> "klass limit: " UINT64_FORMAT, cds_total, compressed_class_space_size(), >> cds_total + compressed_class_space_size(), UnscaledClassSpaceMax)); >> } >> ... >> >> Thanks >> - Ioi >> >> >>> Style nit: you added a few // comments but this test is using /* ... */ comments. The mix looks odd to me especially when comments run into each other. >>> >>> Thanks, >>> David >>> >>>> Thanks, >>>> Jiangli >>>> >> > From ioi.lam at oracle.com Tue Nov 29 04:26:44 2016 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 28 Nov 2016 20:26:44 -0800 Subject: RFR: 8170297: runtime/SharedArchiveFile/LargeSharedSpace.java didn't run out of memory In-Reply-To: <85F81DC4-7289-4015-BF3B-4B526EB93527@oracle.com> References: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> <64cebc75-ef4c-16c7-2a01-e1204a792c30@oracle.com> <583CB3EA.2020803@oracle.com> <85F81DC4-7289-4015-BF3B-4B526EB93527@oracle.com> Message-ID: <583D0384.1030600@oracle.com> Hi Jiangli, The new changes look good. Thanks for taking my suggestion :-) - Ioi On 11/28/16 7:00 PM, Jiangli Zhou wrote: > Hi David and Ioi, > > Here is the updated test: > > http://cr.openjdk.java.net/~jiangli/8170297/webrev.01/ > > > I?ve split the test cases and reworked the comments. The original bug > (8169870) was found on 32-bit platform, so it is worth to test > -XX:SharedMiscCodeSize=1600386047 on 32-bit. Tested on linux-64, > linux-32 and aarch64. Also running RBT. > > Thanks, > Jiangli > >> On Nov 28, 2016, at 2:57 PM, Jiangli Zhou > > wrote: >> >> Hi David and Ioi, >> >> Thank you so much for the comments! >> >> For the compressed klass limit, it is determined by "shared space >> size + compressed klass space size". I agree, the test case and >> comments are not well defined for that. I?ll update the test to >> incorporate your suggestions. >> >> Thanks, >> Jiangli >> >>> On Nov 28, 2016, at 2:47 PM, Ioi Lam >> > wrote: >>> >>> >>> >>> On 11/28/16 2:16 PM, David Holmes wrote: >>>> Hi Jiangli, >>>> >>>> On 29/11/2016 7:52 AM, Jiangli Zhou wrote: >>>>> Please review the following fix for test bug JDK-8170297 >>>>> . >>>>> >>>>> http://cr.openjdk.java.net/~jiangli/8170297/webrev.00/ >>>>> >>>>> >>>> > >>>>> >>>>> On some test platforms, the archive might be dumped successfully >>>>> with the specified large size. Added an extra level of checking in >>>>> the test. Tested with linux-64, linux-32 and aarch64. >>>> >>>> I have a problem with this test. :) The test says "the size is too >>>> large, exceeding the limit for compressed klass support" - so how >>>> can it pass? Is there a limit or not? If this size is too small to >>>> fail should we be using a bigger size? Does it need to be dynamic >>>> based on the host machine? Do we need to calculate a size based on >>>> available memory? >>>> >>> >>> Maybe the comments should spell out the exact same size of the >>> compress klass >>> size limit, in both hex and decimal forms? Otherwise it will be hard >>> to know >>> what 1066924031 would actually lead to. >>> >>> I would also suggest splitting the test into two different portions, >>> instead >>> of using the loop. It's hard to understand how the 3 output message >>> would related >>> to the 2 input numbers.I think a test case should be clear about >>> exactly what it >>> intends to test, and not just be "here's bunch of input parameters >>> and it >>> should just work". >>> >>> >>> E.g, for the second case (1600386047): >>> >>> "Loading classes to share" should not happen on (64-bit platforms >>> && compressed >>> pointers.) >>> >>> In this case (1600386047), since the intention is to test the >>> effect on >>> the compress klass size limit, you should explicitly pass >>> -XX:+UseCompressedOops >>> -XX:+UseCompressedClassPointersto the end of your command-line, so >>> that >>> they would override any VM options specified via JTREG. >>> >>> Also, LargeSharedSpace.java can be run on 32-bit platforms, but the >>> message " larger than compressed klass limit" only happens in LP64 code. >>> So I think the second test (1600386047) should be excluded from >>> 32-bit platforms >>> by using Platform.is32bit(). >>> >>> #ifdef _LP64 >>> if (cds_total + compressed_class_space_size() > >>> UnscaledClassSpaceMax) { >>> vm_exit_during_initialization("Unable to dump shared archive.", >>> err_msg("Size of archive (" SIZE_FORMAT ") + compressed >>> class space (" >>> SIZE_FORMAT ") == total (" SIZE_FORMAT ") is larger >>> than compressed " >>> "klass limit: " UINT64_FORMAT, cds_total, >>> compressed_class_space_size(), >>> cds_total + compressed_class_space_size(), >>> UnscaledClassSpaceMax)); >>> } >>> ... >>> >>> Thanks >>> - Ioi >>> >>> >>>> Style nit: you added a few // comments but this test is using /* >>>> ... */ comments. The mix looks odd to me especially when comments >>>> run into each other. >>>> >>>> Thanks, >>>> David >>>> >>>>> Thanks, >>>>> Jiangli >>>>> >>> >> > From david.holmes at oracle.com Tue Nov 29 04:29:17 2016 From: david.holmes at oracle.com (David Holmes) Date: Tue, 29 Nov 2016 14:29:17 +1000 Subject: RFR: 8170297: runtime/SharedArchiveFile/LargeSharedSpace.java didn't run out of memory In-Reply-To: <85F81DC4-7289-4015-BF3B-4B526EB93527@oracle.com> References: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> <64cebc75-ef4c-16c7-2a01-e1204a792c30@oracle.com> <583CB3EA.2020803@oracle.com> <85F81DC4-7289-4015-BF3B-4B526EB93527@oracle.com> Message-ID: <040cf704-fc12-5c13-c45c-54b8be59d2c6@oracle.com> Thanks Jiangli! That looks a lot clearer to me. David On 29/11/2016 1:00 PM, Jiangli Zhou wrote: > Hi David and Ioi, > > Here is the updated test: > > http://cr.openjdk.java.net/~jiangli/8170297/webrev.01/ > > I?ve split the test cases and reworked the comments. The original bug > (8169870) was found on 32-bit platform, so it is worth to test > -XX:SharedMiscCodeSize=1600386047 on 32-bit. Tested on linux-64, > linux-32 and aarch64. Also running RBT. > > Thanks, > Jiangli > >> On Nov 28, 2016, at 2:57 PM, Jiangli Zhou > > wrote: >> >> Hi David and Ioi, >> >> Thank you so much for the comments! >> >> For the compressed klass limit, it is determined by "shared space size >> + compressed klass space size". I agree, the test case and comments >> are not well defined for that. I?ll update the test to incorporate >> your suggestions. >> >> Thanks, >> Jiangli >> >>> On Nov 28, 2016, at 2:47 PM, Ioi Lam >> > wrote: >>> >>> >>> >>> On 11/28/16 2:16 PM, David Holmes wrote: >>>> Hi Jiangli, >>>> >>>> On 29/11/2016 7:52 AM, Jiangli Zhou wrote: >>>>> Please review the following fix for test bug JDK-8170297 >>>>> . >>>>> >>>>> http://cr.openjdk.java.net/~jiangli/8170297/webrev.00/ >>>>> >>>>> >>>>> On some test platforms, the archive might be dumped successfully >>>>> with the specified large size. Added an extra level of checking in >>>>> the test. Tested with linux-64, linux-32 and aarch64. >>>> >>>> I have a problem with this test. :) The test says "the size is too >>>> large, exceeding the limit for compressed klass support" - so how >>>> can it pass? Is there a limit or not? If this size is too small to >>>> fail should we be using a bigger size? Does it need to be dynamic >>>> based on the host machine? Do we need to calculate a size based on >>>> available memory? >>>> >>> >>> Maybe the comments should spell out the exact same size of the >>> compress klass >>> size limit, in both hex and decimal forms? Otherwise it will be hard >>> to know >>> what 1066924031 would actually lead to. >>> >>> I would also suggest splitting the test into two different portions, >>> instead >>> of using the loop. It's hard to understand how the 3 output message >>> would related >>> to the 2 input numbers.I think a test case should be clear about >>> exactly what it >>> intends to test, and not just be "here's bunch of input parameters and it >>> should just work". >>> >>> >>> E.g, for the second case (1600386047): >>> >>> "Loading classes to share" should not happen on (64-bit platforms >>> && compressed >>> pointers.) >>> >>> In this case (1600386047), since the intention is to test the effect on >>> the compress klass size limit, you should explicitly pass >>> -XX:+UseCompressedOops >>> -XX:+UseCompressedClassPointersto the end of your command-line, so that >>> they would override any VM options specified via JTREG. >>> >>> Also, LargeSharedSpace.java can be run on 32-bit platforms, but the >>> message " larger than compressed klass limit" only happens in LP64 code. >>> So I think the second test (1600386047) should be excluded from >>> 32-bit platforms >>> by using Platform.is32bit(). >>> >>> #ifdef _LP64 >>> if (cds_total + compressed_class_space_size() > >>> UnscaledClassSpaceMax) { >>> vm_exit_during_initialization("Unable to dump shared archive.", >>> err_msg("Size of archive (" SIZE_FORMAT ") + compressed class >>> space (" >>> SIZE_FORMAT ") == total (" SIZE_FORMAT ") is larger >>> than compressed " >>> "klass limit: " UINT64_FORMAT, cds_total, >>> compressed_class_space_size(), >>> cds_total + compressed_class_space_size(), >>> UnscaledClassSpaceMax)); >>> } >>> ... >>> >>> Thanks >>> - Ioi >>> >>> >>>> Style nit: you added a few // comments but this test is using /* ... >>>> */ comments. The mix looks odd to me especially when comments run >>>> into each other. >>>> >>>> Thanks, >>>> David >>>> >>>>> Thanks, >>>>> Jiangli >>>>> >>> >> > From jiangli.zhou at oracle.com Tue Nov 29 05:24:40 2016 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Mon, 28 Nov 2016 21:24:40 -0800 Subject: RFR: 8170297: runtime/SharedArchiveFile/LargeSharedSpace.java didn't run out of memory In-Reply-To: <040cf704-fc12-5c13-c45c-54b8be59d2c6@oracle.com> References: <13706BE6-ACF1-4970-8B32-4DAE8B6AB048@oracle.com> <64cebc75-ef4c-16c7-2a01-e1204a792c30@oracle.com> <583CB3EA.2020803@oracle.com> <85F81DC4-7289-4015-BF3B-4B526EB93527@oracle.com> <040cf704-fc12-5c13-c45c-54b8be59d2c6@oracle.com> Message-ID: Hi David and Ioi, Thanks for the review! Thanks, Jiangli > On Nov 28, 2016, at 8:29 PM, David Holmes wrote: > > Thanks Jiangli! That looks a lot clearer to me. > > David > >> On 29/11/2016 1:00 PM, Jiangli Zhou wrote: >> Hi David and Ioi, >> >> Here is the updated test: >> >> http://cr.openjdk.java.net/~jiangli/8170297/webrev.01/ >> >> I?ve split the test cases and reworked the comments. The original bug >> (8169870) was found on 32-bit platform, so it is worth to test >> -XX:SharedMiscCodeSize=1600386047 on 32-bit. Tested on linux-64, >> linux-32 and aarch64. Also running RBT. >> >> Thanks, >> Jiangli >> >>> On Nov 28, 2016, at 2:57 PM, Jiangli Zhou >> > wrote: >>> >>> Hi David and Ioi, >>> >>> Thank you so much for the comments! >>> >>> For the compressed klass limit, it is determined by "shared space size >>> + compressed klass space size". I agree, the test case and comments >>> are not well defined for that. I?ll update the test to incorporate >>> your suggestions. >>> >>> Thanks, >>> Jiangli >>> >>>> On Nov 28, 2016, at 2:47 PM, Ioi Lam >>> > wrote: >>>> >>>> >>>> >>>>> On 11/28/16 2:16 PM, David Holmes wrote: >>>>> Hi Jiangli, >>>>> >>>>>> On 29/11/2016 7:52 AM, Jiangli Zhou wrote: >>>>>> Please review the following fix for test bug JDK-8170297 >>>>>> . >>>>>> >>>>>> http://cr.openjdk.java.net/~jiangli/8170297/webrev.00/ >>>>>> >>>>>> >>>>>> On some test platforms, the archive might be dumped successfully >>>>>> with the specified large size. Added an extra level of checking in >>>>>> the test. Tested with linux-64, linux-32 and aarch64. >>>>> >>>>> I have a problem with this test. :) The test says "the size is too >>>>> large, exceeding the limit for compressed klass support" - so how >>>>> can it pass? Is there a limit or not? If this size is too small to >>>>> fail should we be using a bigger size? Does it need to be dynamic >>>>> based on the host machine? Do we need to calculate a size based on >>>>> available memory? >>>> >>>> Maybe the comments should spell out the exact same size of the >>>> compress klass >>>> size limit, in both hex and decimal forms? Otherwise it will be hard >>>> to know >>>> what 1066924031 would actually lead to. >>>> >>>> I would also suggest splitting the test into two different portions, >>>> instead >>>> of using the loop. It's hard to understand how the 3 output message >>>> would related >>>> to the 2 input numbers.I think a test case should be clear about >>>> exactly what it >>>> intends to test, and not just be "here's bunch of input parameters and it >>>> should just work". >>>> >>>> >>>> E.g, for the second case (1600386047): >>>> >>>> "Loading classes to share" should not happen on (64-bit platforms >>>> && compressed >>>> pointers.) >>>> >>>> In this case (1600386047), since the intention is to test the effect on >>>> the compress klass size limit, you should explicitly pass >>>> -XX:+UseCompressedOops >>>> -XX:+UseCompressedClassPointersto the end of your command-line, so that >>>> they would override any VM options specified via JTREG. >>>> >>>> Also, LargeSharedSpace.java can be run on 32-bit platforms, but the >>>> message " larger than compressed klass limit" only happens in LP64 code. >>>> So I think the second test (1600386047) should be excluded from >>>> 32-bit platforms >>>> by using Platform.is32bit(). >>>> >>>> #ifdef _LP64 >>>> if (cds_total + compressed_class_space_size() > >>>> UnscaledClassSpaceMax) { >>>> vm_exit_during_initialization("Unable to dump shared archive.", >>>> err_msg("Size of archive (" SIZE_FORMAT ") + compressed class >>>> space (" >>>> SIZE_FORMAT ") == total (" SIZE_FORMAT ") is larger >>>> than compressed " >>>> "klass limit: " UINT64_FORMAT, cds_total, >>>> compressed_class_space_size(), >>>> cds_total + compressed_class_space_size(), >>>> UnscaledClassSpaceMax)); >>>> } >>>> ... >>>> >>>> Thanks >>>> - Ioi >>>> >>>> >>>>> Style nit: you added a few // comments but this test is using /* ... >>>>> */ comments. The mix looks odd to me especially when comments run >>>>> into each other. >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> Thanks, >>>>>> Jiangli >> From david.buck at oracle.com Tue Nov 29 05:42:34 2016 From: david.buck at oracle.com (David Buck) Date: Tue, 29 Nov 2016 14:42:34 +0900 Subject: RFR(XS) 8170431: non-ASCII characters in source code comments (jimage.hpp) Message-ID: <583D154A.9080309@oracle.com> Hi! Please have a look at this small yet urgent fix to comments in a header file. bug report: https://bugs.openjdk.java.net/browse/JDK-8170431 webrev: http://cr.openjdk.java.net/~dbuck/jdk8170431_0/ These changes are exactly the same as the changes made by Naoto to the class library's copy of jimage.hpp: http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/cfa705d1f4e2 This is actually an important fix because VS will choke on this file under certain Windows locales (Chinese and Japanese have been reported so far). So the build is broken for many users without this fix. Cheers, -Buck From david.holmes at oracle.com Tue Nov 29 05:46:53 2016 From: david.holmes at oracle.com (David Holmes) Date: Tue, 29 Nov 2016 15:46:53 +1000 Subject: RFR(XS) 8170431: non-ASCII characters in source code comments (jimage.hpp) In-Reply-To: <583D154A.9080309@oracle.com> References: <583D154A.9080309@oracle.com> Message-ID: <23af55e7-6fce-1f68-62a5-550d08f210e8@oracle.com> Reviewed and approved for immediate push under trivial change rules. Thanks, David On 29/11/2016 3:42 PM, David Buck wrote: > Hi! > > Please have a look at this small yet urgent fix to comments in a header > file. > > bug report: https://bugs.openjdk.java.net/browse/JDK-8170431 > webrev: http://cr.openjdk.java.net/~dbuck/jdk8170431_0/ > > These changes are exactly the same as the changes made by Naoto to the > class library's copy of jimage.hpp: > > http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/cfa705d1f4e2 > > This is actually an important fix because VS will choke on this file > under certain Windows locales (Chinese and Japanese have been reported > so far). So the build is broken for many users without this fix. > > Cheers, > -Buck From goetz.lindenmaier at sap.com Tue Nov 29 09:30:18 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Tue, 29 Nov 2016 09:30:18 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <4dfa4ae6-4fa6-994c-f2cf-8fc309d8c520@oracle.com> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> <4dfa4ae6-4fa6-994c-f2cf-8fc309d8c520@oracle.com> Message-ID: Hi Dan, see my replies inline ... New webrev: http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.05/ > -----Original Message----- > From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > Sent: Dienstag, 29. November 2016 01:38 > To: Lindenmaier, Goetz ; hotspot-compiler- > dev at openjdk.java.net; 'hotspot-runtime-dev at openjdk.java.net' runtime-dev at openjdk.java.net> > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > On 11/28/16 2:08 AM, Lindenmaier, Goetz wrote: > > Hi, > > > > I'm working on a fix for OS guard pages on stacks. I figured there > > are VM guard pages (reserved, yellow, red) on the compiler stacks > > _and_ OS guard pages. For Java threads, the OS guard pages are left > > out. I think this should be done for compiler threads, too. Please > > confirm. > > Webrev: > > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.04/ > > src/os/aix/vm/os_aix.cpp > L888: pthread_attr_setguardsize(&attr, > os::Aix::default_guard_size(thr_type)); > No check or assert on the return status of this call. > Is one needed? I implemented this as the existing code on linux which has no check either. I think a failure is quite theoretical. Because of your comment below I'll leave it as-is. > L3044: // guard pages, so only enable libc guard pages for > non-Java threads. > src/os/linux/vm/os_linux.cpp also has this comment: > // (Remember: compiler thread is a Java thread, too!) Fixed. > L3051: return ((thr_type == java_thread || thr_type == > compiler_thread) ? 0 : 4*K); > nit - please add spaces around the '*' so '4 * K'.' Fixed. > src/os/aix/vm/os_aix.hpp > No comments. > > src/os/linux/vm/os_linux.cpp > L729: // is not implemented properly. The posix standard requires > to add > Typo: 'to add' -> 'adding' Fixed. > L738: pthread_attr_setguardsize(&attr, > os::Linux::default_guard_size(thr_type)); > No check or assert on the return status of this call. > Is one needed? See above. > L2851: // Creating guard page is very expensive. Java thread has > HotSpot > L2852: // guard page, only enable glibc guard page for non-Java > threads. > L2853: // (Remember: compiler thread is a java thread, too!) > Typo: "java thread" -> "Java thread" (consistency) > > This comment block should be common to all the platforms that > define default_guard_size(). Yes, I can see that AIX needs to > add another paragraph, but we should make the core comment common > if possible. I made the first three lines look alike. > L6090: // Java/Compiler thread: > Thanks for making this common in os_linux.cpp. > > L6095: // | glibc guard page | - guard, attached Java > thread usually has > Clarity: "guard," -> "guard page," Fixed. > Typo: "Java thread" -> "JavaThread" (consistency) I changed both to Java thread as at the other locations. > L6099: // | HotSpot Guard Pages | - red and yellow pages > The fairly recently added reserved page should be mentioned > here also? Yes. Fixed. Also fixed it to say JavaThread::stack_reserved_zone_base(). Also fixed comment on bsd. > L6120 // ** P1 (aka bottom) and size ( P2 = P1 - size) are the > address and stack size returned from > Typo: "( P2 = ..." -> "(P2 = ..." Fixed. > L6148: fatal("Can not locate current stack attributes!"); > Typo: "Can not" -> "Cannot" Fixed. > L6175: // stack size includes normal stack and HotSpot guard pages > Perhaps add to the comment: > "for the threads that have HotSpot guard pages." Fixed. I also checked my comments for "OS guard pages" and replaced it by "glibc guard pages" which is used in several places already, same for "VM guard page" --> "HotSpot guard page". I think this is also more consistent. > src/os/posix/vm/os_posix.cpp > L1097: pthread_attr_getstacksize(attr, &stack_size); > L1098: pthread_attr_getguardsize(attr, &guard_size); > Do these two calls need to have their result checked? > > Now I'm starting to wonder if all the uses of these > two APIs need to be checked? Separate bug? It would be more consistent with the specification of the methods, On the other side it's quite unlikely that these fail if attr != NULL. > src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp > L540: size_t os::Posix::_compiler_thread_min_stack_allowed = 512 * K; > L541: size_t os::Posix::_java_thread_min_stack_allowed = 512 * K; > So prior to the fix for 8140520, src/os/aix/vm/os_aix.cpp had > this single min_stack_allowed value: > > L3601: os::Aix::min_stack_allowed = > MAX2(os::Aix::min_stack_allowed, > L3602: JavaThread::stack_guard_zone_size() + > L3603: JavaThread::stack_shadow_zone_size() + > L3604: (4*BytesPerWord > COMPILER2_PRESENT(+2)) * 4 * K); > > and the fix for 8140520 changed that for *NIX platforms to > three mins in src/os/posix/vm/os_posix.cpp: > > L1108: _java_thread_min_stack_allowed = > MAX2(_java_thread_min_stack_allowed, > L1109: JavaThread::stack_guard_zone_size() + > L1110: JavaThread::stack_shadow_zone_size() + > L1111: (4 * > BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K); > > L1150: _compiler_thread_min_stack_allowed = > align_size_up(_compiler_thread_min_stack_allowed, vm_page_size()); > > L1161 _vm_internal_thread_min_stack_allowed = > align_size_up(_vm_internal_thread_min_stack_allowed, vm_page_size()); > > Which means that the compiler_thread no longer benefits from > the extra space for quard and shadow pages. The thinking in > 8140520 was that the compiler_thread and vm_internal_threads > don't need the quard and shadow pages since they don't run > Java code (ignoring JVMCI for now). > > So I can see bumping _compiler_thread_min_stack_allowed from > 128 -> 512 as one solution for getting that extra space back. > However, I don't understand why _java_thread_min_stack_allowed > has changed from 128 -> 512. Because it was never correct before. > I had previously made this comment: > > To put it another way, I'd like to see us add extra space to > > solve the 64K page issue directly instead of as a side effect > > of the red/yellow page addition. > And Goetz replied with: > > I don't understand. What do you mean by 'directly'? > > So prior to the fix for 8140520, src/os/solaris/vm/os_solaris.cpp > had a block like this: > > L4468: // For 64kbps there will be a 64kb page size, which makes > L4469: // the usable default stack size quite a bit less. > Increase the > L4470: // stack for 64kb (or any > than 8kb) pages, this > increases > L4471: // virtual memory fragmentation (since we're not > creating the > L4472 // stack on a power of 2 boundary. The real fix for this > L4473 // should be to fix the guard page mechanism. > L4474 > L4475 if (vm_page_size() > 8*K) { > L4476 threadStackSizeInBytes = (threadStackSizeInBytes != 0) > L4477 ? threadStackSizeInBytes + > L4478 JavaThread::stack_red_zone_size() + > L4479 JavaThread::stack_yellow_zone_size() > L4480 : 0; > L4481 ThreadStackSize = threadStackSizeInBytes/K; > L4482 } > > The above is an example of what I mean by solving the 64K > page issue directly. In the fix for 8140520, that code block > was moved to os::Posix::set_minimum_stack_sizes() in > src/os/posix/vm/os_posix.cpp and put in a "#ifdef SOLARIS... > #endif // SOLARIS" block. Coleen filed a bug to determine > whether that code can be deleted: > > JDK-8161093 Solaris for >8k pagesize adds extra guard pages > https://bugs.openjdk.java.net/browse/JDK-8161093 > > but perhaps this bug shows that the code is needed? > > > OK so this is probably the longest code review comment > I have ever written, but the summary is: > > - I understand bumping _compiler_thread_min_stack_allowed, > but should it be solved in a different way? > - I don't understand bumping _java_thread_min_stack_allowed I plan to do a follow up change to fix this. Let's leave this discussion to that review. Here I just want to fix the NPTL issue and the basic sizing that is needed to pass the new test on ppc/s390. > src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp > No comments. > > src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp > L538: size_t os::Posix::_compiler_thread_min_stack_allowed = 384 * K; > L539: size_t os::Posix::_java_thread_min_stack_allowed = 384 * K; > > Same monster comment as src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp > and the same summary: > > - I understand bumping _compiler_thread_min_stack_allowed, > but should it be solved in a different way? > - I don't understand bumping _java_thread_min_stack_allowed > > src/os_cpu/linux_s390/vm/os_linux_s390.cpp > L478: size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > L479: size_t os::Posix::_java_thread_min_stack_allowed = 236 * K; > Bumping _java_thread_min_stack_allowed but not bumping > _compiler_thread_min_stack_allowed. I'm confused here. The numbers are what I need to startup on the machines. 128 is just fine on the machines we have. (That's the problem of the current setup: you have to tune this compile time constant for the page size of the machine you are running on. But let's discuss this in the follow up change.) > > src/os_cpu/linux_sparc/vm/os_linux_sparc.cpp > No comments. > > src/os_cpu/linux_x86/vm/os_linux_x86.cpp > No comments. > > src/os_cpu/linux_zero/vm/os_linux_zero.cpp > No comments. > > Sorry it took me so long to write this up... No matter, thanks for this thorough review! Best regards, Goetz. > > Dan > > > The change affecting the compier threads is in os_linux.cpp, > default_guard_size(), > > where '|| thr_type == compiler_thread' has been added. The function > > was also moved from the os_cpu files, as it's identical on all cpus. > > > > Best regards, > > Goetz. > > > > > >> -----Original Message----- > >> From: David Holmes [mailto:david.holmes at oracle.com] > >> Sent: Montag, 28. November 2016 00:25 > >> To: Lindenmaier, Goetz ; > >> 'daniel.daugherty at oracle.com' ; 'hotspot- > >> runtime-dev at openjdk.java.net' > >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > >> > >> Hi Goetz, > >> > >> On 24/11/2016 10:15 PM, Lindenmaier, Goetz wrote: > >>> Hi, > >>> > >>> I now edited the stuff I had proposed below: > >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.03/ > >>> This includes > >>> - the NPTL fix from webrev.02 > >> Okay in principle. As discussed this only impacts non-JavaThreads so the > >> change should be minimal. > >> > >>> - merging code on linux > >> Went a bit further than I had expected but if this truly isn't CPU > >> dependent code then great! > >> > >>> - not adding OS guard to compiler threads. > >> Okay in principle. IIUC we will now save the OS guard page for compiler > >> thread stacks. Is that the only impact? The hotspot-compiler-dev folk > >> may want to sign off on this part. > >> > >> > >> A few minor comments: > >> > >> src/os/linux/vm/os_linux.cpp > >> > >> 2854 return ((thr_type == java_thread || thr_type == > >> os::compiler_thread) ... > >> > >> os:: should be used for both types or none. > >> > >> 6153 pthread_attr_getguardsize(&attr, &guard_size); > >> > >> Can you at least verify a zero return code in an assert/assert_status > >> please. > >> > >> --- > >> > >> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp > >> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp > >> src/os_cpu/linux_s390/vm/os_linux_s390.cpp > >> > >> Are the changes to min_stack_allowed just fixing an existing bug? > >> > >> --- > >> > >> Thanks, > >> David > >> ----- > >> > >> > >>> I think this should be pushed for this bug ID. For the other changes I'll > >>> make another bug. > >>> > >>> Best regards, > >>> Goetz. > >>> > >>>> -----Original Message----- > >>>> From: Lindenmaier, Goetz > >>>> Sent: Wednesday, November 23, 2016 8:11 AM > >>>> To: David Holmes ; > >>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > >>>> Subject: RE: RFR(M): 8169373: Work around linux NPTL stack guard error. > >>>> > >>>> Hi, > >>>> > >>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The > >>>>> ability for a CompilerThread to execute Java code (can_call_java()) is > >>>>> now a dynamic property depending on whether the current compiler is > the > >>>>> JVMCI compiler. > >>>> Ah, then I should also leave space for shadow pages in the minimal stack > >> size > >>>> of comiler threads. > >>>> > >>>> Do we agree on the cleanup and on leaving out the OS guard page on > >>>> compiler threads? > >>>> Then I would edit a change comprising the NPTL workaround and these > >>>> additional changes, and split the other issue into a new bug? I think this > >>>> will easy the reviewing. > >>>> > >>>> Best regards, > >>>> Goetz. > >>>> > >>>>> -----Original Message----- > >>>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>>> Sent: Mittwoch, 23. November 2016 02:50 > >>>>> To: Lindenmaier, Goetz ; > >>>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > >>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > error. > >>>>> > >>>>> On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: > >>>>>> Hi Dan, > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >>>>>>> Sent: Dienstag, 22. November 2016 14:01 > >>>>>>> To: Lindenmaier, Goetz ; David Holmes > >>>>>>> ; hotspot-runtime-dev at openjdk.java.net > >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > >>>> error. > >>>>>>> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: > >>>>>>>> Hi Dan, > >>>>>>>> > >>>>>>>> I ran into a row of issues, some errors on the platforms. > >>>>>>>> > >>>>>>>> What I meant with that comment is that > >>>>>>>> os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, > >>>>>>>> JavaThread::stack_guard_zone_size() + > >>>>>>>> JavaThread::stack_shadow_zone_size() + > >>>>>>>> (4*BytesPerWord > COMPILER2_PRESENT(+2)) * > >> 4 > >>>> * > >>>>> K); > >>>>>>>> was executed, and min_stack_allowed used for all stacks. Now, > >>>> compiler > >>>>>>> and > >>>>>>>> vm minimum stack sizes are not increased by these sizes. > >>>>>>> Now I see what you mean. Thanks for clearing this up. > >>>>>>> > >>>>>>> I should have remembered that part of the change because we went > >>>> back > >>>>>>> and forth about removing the red/yellow zone pages from the other > >>>>>>> threads. In particular, we discussed the compiler thread because it > >>>>>>> is-a JavaThread. Our conclusion was that a compiler thread doesn't > >>>>>>> execute Java bytecode so we could remove the red/yellow pages... > >>>>>> Yes, it does not execute java byte code. > >>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The > >>>>> ability for a CompilerThread to execute Java code (can_call_java()) is > >>>>> now a dynamic property depending on whether the current compiler is > the > >>>>> JVMCI compiler. > >>>>> > >>>>> David > >>>>> ----- > >>>>> > >>>>>> Therefore you can remove the shadow pages. There is no code that > >>>>>> will bang. > >>>>>> But red/yellow pages are protected right during thread startup. > >>>>>> Therefore you must have enough space for them. > >>>>>> On ppc, we try to protect three 64K pages out of the 128K compiler > stack. > >>>>>> That obviously fails. > >>>>>> > >>>>>> Therefore I propose: > >>>>>> size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // Set > >>>>> platform dependent. > >>>>>> in os::Posix::set_minimum_stack_sizes(): > >>>>>> _java_thread_min_stack_allowed = _java_thread_min_stack_allowed > + > >>>>>> JavaThread::stack_guard_zone_size() + > >>>>>> JavaThread::stack_shadow_zone_size(); > >>>>>> > >>>>>> (Similar for _compiler_thread_min_stack_allowed). > >>>>>> > >>>>>> The minimal stack size is made up of three components: > >>>>>> 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot > >>>>> implementation/platform/os. > >>>>>> 2. Sizes of C++ frames: depends on C++ compiler. > >>>>>> These are fixed at compile time. > >>>>>> 3. Sizes of red/yellow/reserved/shadow pages. Depends on the > system > >>>>> the > >>>>>> VM is used on. This is not fixed at compile time. (Our ppc > machines > >>>>> differ > >>>>>> in page size.) > >>>>>> Therefore 3. should not be included in a compile time constant. > >>>>>> > >>>>>> > >>>>>> > >>>>>>> And that decision allowed us to be exposed to the 64K page issue > >>>>>>> because the "extra" space isn't there anymore. > >>>>>>> > >>>>>>>> At least the _compiler_thread_min_stack_allowed should be > increased > >>>>>>>> similarly by red/yellow zone size. The compiler thread is a Java > >>>>>>>> thread and must have space for these zones. > >>>>>>> I'm not sure that I completely agree (yet). To me, the red/yellow > >>>>>>> pages are there for Java thread stack overflow semantics. Yes, the > >>>>>>> compiler thread needs extra space when 64K pages are used, but I > >>>>>>> would prefer that we add that space via a different calculation. > >>>>>> Yes they are. But compiler threads happen tob e a subclass of > >>>>>> Java threads and use the same run() method that puts the pages > >>>>>> There. > >>>>>> I agree that they are not needed for Compiler threads, nor for > >>>>>> CodeCacheSweeperThreads. I don't really now about > >>>>>> JvmtiAgentThreads and ServiceThreads, but all of the get the guard > >>>> pages > >>>>>> because they are derived from JavaThread. > >>>>>> > >>>>>>> To put it another way, I'd like to see us add extra space to solve > >>>>>>> the 64K page issue directly instead of as a side effect of the > >>>>>>> red/yellow page addition. > >>>>>> I don't understand. What do you mean by 'directly'? > >>>>>> > >>>>>>>> Also, the change added a test that is failing now. > >>>>>>> And that's a "good thing" (TM), right? :-) > >>>>>> Yes, it showed a bug and thus raised the need to fix it! :) > >>>>>> > >>>>>> Best regards, > >>>>>> Goetz. > >>>>>> > >>>>>> > >>>>>>> Again, thanks for clarifying 8140520's role in this issue. > >>>>>>> > >>>>>>> Dan > >>>>>>> > >>>>>>> > >>>>>>>> Best regards, > >>>>>>>> Goetz. > >>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >>>>>>>>> Sent: Montag, 21. November 2016 17:28 > >>>>>>>>> To: David Holmes ; Lindenmaier, Goetz > >>>>>>>>> ; hotspot-runtime- > >>>>> dev at openjdk.java.net > >>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > >>>>> error. > >>>>>>>>> Sorry for the delayed responses to this thread. I've been on > vacation... > >>>>>>>>> > >>>>>>>>> One comment/query embedded below... > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On 11/10/16 8:40 PM, David Holmes wrote: > >>>>>>>>>> Hi Goetz, > >>>>>>>>>> > >>>>>>>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > >>>>>>>>>>> Hi David, > >>>>>>>>>>> > >>>>>>>>>>> This issue is different to 6675312, see also my comment in the > bug. > >>>>>>>>>>> > >>>>>>>>>>> It appears running jtreg test > >>>>> runtime/Thread/TooSmallStackSize.java, > >>>>>>>>>>> with my patch below you can reproduce it on linuxx86_64. You > can > >>>>> not > >>>>>>>>>>> do that with 6675312. Also, I would assume there are systems out > >>>>> there > >>>>>>>>>>> on x86 that uses 64-K pages, did you run the tests on these? I > >>>> would > >>>>>>>>>>> assume you get hard crashes with stack overflows in the first C2 > >>>>>>>>>>> compilation if there is only 64K usable CompilerThreadStack. > >>>>>>>>>>> > >>>>>>>>>>> My fix does not affect Java threads, which are the largest amount > >>>>>>>>>>> of threads used by the VM. It affects only the non-Java threads. > >>>>>>>>>>> It adds one page to these threads. The page does not require > >>>>> memory, > >>>>>>>>>>> as it's protected. The stack will only require more space if the > >>>> thread > >>>>>>>>>>> ran into a stack overflow before the fix as else the pages are not > >>>>>>>>>>> mapped. > >>>>>>>>>>> This are stack overflows that cause hard crashes, at least on ppc > >>>> the > >>>>> VM > >>>>>>>>>>> does not properly catch these stack overflows, so any setup > >>>> working > >>>>>>> now > >>>>>>>>>>> will not run into the additional space. Altogether there should be > >>>> no > >>>>>>>>>>> effect on running systems besides requiring one more entry in > the > >>>>>>>>>>> page table per non-Java thread. > >>>>>>>>>>> > >>>>>>>>>>> The problem is caused by a rather recent change (8140520: > >>>> segfault > >>>>> on > >>>>>>>>>>> solaris-amd64 > >>>>>>>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed after > >>>>>>>>>>> feature-close. As this was a rather recent change, it must be > >>>>>>>>>>> possible to > >>>>>>>>>>> fix this follow up issue. What else is this period in the project > good > >>>>>>>>>>> for if not fixing issues? > >>>>>>>>>> So I am seeing a number of factors here. > >>>>>>>>>> > >>>>>>>>>> First, 8140520, set: > >>>>>>>>>> > >>>>>>>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >>>>>>>>> So I'm confused by the above comment: > >>>>>>>>> > >>>>>>>>> > The problem is caused by a rather recent change (8140520: > >>>> segfault > >>>>>>>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) > >>>>>>>>> > >>>>>>>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- > >>>>>>>>> > >>>>> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html > >>>>>>>>> shows this change: > >>>>>>>>> > >>>>>>>>> @@ -531,19 +531,17 @@ > >>>>>>>>> } > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> ////////////////////////////////////////////////////////////////////////////// > >>>>>>>>> // > >>>>>>>>> // thread stack > >>>>>>>>> > >>>>>>>>> -size_t os::Linux::min_stack_allowed = 128*K; > >>>>>>>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >>>>>>>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; > >>>>>>>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * > K; > >>>>>>>>> > >>>>>>>>> so the existing single variable of 'min_stack_allowed' was > >>>>>>>>> replaced by three variables: _compiler_thread_min_stack_allowed, > >>>>>>>>> _java_thread_min_stack_allowed, and > >>>>>>>>> _vm_internal_thread_min_stack_allowed. > >>>>>>>>> > >>>>>>>>> The old single variable and the three new variables are all > >>>>>>>>> initialized to the same value (128K) so the fix for 8140520 > >>>>>>>>> did not change stack sizes for this platform. In fact, only > >>>>>>>>> one platform had a size change (Solaris X64). > >>>>>>>>> > >>>>>>>>> So I'm confused about how the fix for 8140520 caused this problem. > >>>>>>>>> > >>>>>>>>> Based on David's analysis below, it looks to me like this 64K stack > >>>>>>>>> guard page problem should also exist prior to the fix for 8140520. > >>>>>>>>> > >>>>>>>>> Goetz, can you please explain how 8140520 caused this problem? > >>>>>>>>> > >>>>>>>>> Dan > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Second on linux PPC it is hardwired to use 2 guard pages: > >>>>>>>>>> > >>>>>>>>>> return 2 * page_size(); > >>>>>>>>>> > >>>>>>>>>> Third, you had a pagesize of 64K. > >>>>>>>>>> > >>>>>>>>>> Fourth, NPTL takes the guard space from the stack space - hence > >>>> with > >>>>> 2 > >>>>>>>>>> x 64K guard, and a 128K stack it was all consumed. > >>>>>>>>>> > >>>>>>>>>> --- > >>>>>>>>>> > >>>>>>>>>> In the proposed changes you now only use page_size() for the > guard, > >>>>> so > >>>>>>>>>> that alone would have fixed the observed problem. > >>>>>>>>>> > >>>>>>>>>> But in addition you want to address the NPTL problem by adding > >>>> back > >>>>>>>>>> the guard space to the stack size requested. That alone would also > >>>>>>>>>> have fixed the observed problem. :) > >>>>>>>>>> > >>>>>>>>>> But in addition you have increased the minimum stack size: > >>>>>>>>>> > >>>>>>>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; > >>>>>>>>>> > >>>>>>>>>> which again, on its own would have fixed the original problem. :) > >>>>>>>>>> > >>>>>>>>>> Did you really intend to increase the real minimum stack from > 128K > >>>> to > >>>>>>>>>> 256K ? (on a 64K page system) > >>>>>>>>>> > >>>>>>>>>> --- > >>>>>>>>>> > >>>>>>>>>> Focusing simply on the shared code change to adjust the requested > >>>>>>>>>> stacksize by the amount of guard space (if any), this does not seem > >>>>>>>>>> unreasonable. As you note it is restricted to non-JavaThreads and > >>>> only > >>>>>>>>>> adds a page to reserved stack space. > >>>>>>>>>> > >>>>>>>>>> My only query now is whether the minimum stacksize detection > logic > >>>>>>>>>> will correctly report the real minimum stack size (taking into > account > >>>>>>>>>> the need for the guard page) ? > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> David > >>>>>>>>>> > >>>>>>>>>>> So I really think this issue should be fixed. > >>>>>>>>>>> > >>>>>>>>>>> Best regards, > >>>>>>>>>>> Goetz. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>>>>>>>>>> Sent: Thursday, November 10, 2016 10:02 PM > >>>>>>>>>>>> To: Lindenmaier, Goetz ; hotspot- > >>>>>>>>> runtime- > >>>>>>>>>>>> dev at openjdk.java.net > >>>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack > >>>> guard > >>>>>>> error. > >>>>>>>>>>>> Hi Goetz, > >>>>>>>>>>>> > >>>>>>>>>>>> As per the bug report, this issue was already known (6675312) > and > >>>>> we > >>>>>>>>>>>> chose not to try and address it due to no reported issues at the > >>>>> time. > >>>>>>>>>>>> While I see that you have encountered an issue (is it real or > >>>>>>>>>>>> fabricated?) I think this change is too intrusive to be applied at > this > >>>>>>>>>>>> stage of the JDK 9 release cycle, as it will change the stack > >>>>>>>>>>>> requirements of every application running on Linux. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> David > >>>>>>>>>>>> > >>>>>>>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > >>>>>>>>>>>>> Hi, > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Please review this change. I please need a sponsor: > >>>>>>>>>>>>> > >>>>>>>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > >>>>>>>>> stackFix/webrev.01/ > >>>>>>>>>>>>> > >>>>>>>>>>>>> In the Linux NPTL pthread implementation the guard size > >>>>> mechanism > >>>>>>>>>>>>> is not > >>>>>>>>>>>>> implemented properly. The posix standard requires to add the > >>>>> size > >>>>>>>>>>>>> of the > >>>>>>>>>>>>> guard pages to the stack size, instead Linux takes the space out > >>>> of > >>>>>>>>>>>>> 'stacksize'. > >>>>>>>>>>>>> > >>>>>>>>>>>>> The Posix standard > >>>>>>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ > >>>>>>>>>>>>> says "the implementation allocates extra memory at the > >>>>> overflow > >>>>>>>>>>>>> end of > >>>>>>>>>>>>> the stack". The linux man page > >>>>>>>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As > >>>>> at > >>>>>>>>>>>>> glibc > >>>>>>>>>>>>> 2.8, the NPTL threading implementation includes the guard > area > >>>>>>> within > >>>>>>>>>>>>> the stack size allocation, rather than allocating extra space at > >>>>>>>>>>>>> the end > >>>>>>>>>>>>> of the stack, as POSIX.1 requires". > >>>>>>>>>>>>> > >>>>>>>>>>>>> I encounter this problem in > >>>>> runtime/Thread/TooSmallStackSize.java > >>>>>>>>>>>>> on ppc > >>>>>>>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K > >>>> on > >>>>>>> ppc, > >>>>>>>>> and > >>>>>>>>>>>>> ppc specifies two OS guard pages. The VM crashes in pthread > >>>>>>> creation > >>>>>>>>>>>>> because there is no usable space in the thread stack after > >>>>> allocating > >>>>>>>>>>>>> the guard pages. > >>>>>>>>>>>>> > >>>>>>>>>>>>> But TooSmallStackSize.java requires that the VM comes up > with > >>>>> the > >>>>>>>>>>>>> stack > >>>>>>>>>>>>> size mentioned in the error message. > >>>>>>>>>>>>> > >>>>>>>>>>>>> This fix adapts the requested stack size on Linux by the size of > >>>> the > >>>>>>>>>>>>> guard pages to mimick proper behaviour, see change to > >>>>>>> os_linux.cpp. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> The change also streamlines usage of stack_guard_page on > >>>>> linuxppc, > >>>>>>>>>>>>> linuxppcle, aixppc and linuxs390. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> To reproduce the error on linux_x86_64, apply below patch > and > >>>>> call > >>>>>>> the > >>>>>>>>>>>>> VM with -XX:CompilerThreadStackSize=64. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> I'm still exploring why I had to choose such big compiler stacks > >>>> on > >>>>>>>>>>>>> ppc > >>>>>>>>>>>>> to get -version passing, but I wanted to send the RFR now as > >>>>> people > >>>>>>>>>>>>> obviously looked at the bug I opened (Thanks David!). > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Goetz. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> diff -r b7ae012c55c3 > src/os_cpu/linux_x86/vm/os_linux_x86.cpp > >>>>>>>>>>>>> > >>>>>>>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 > >>>>>>> 12:37:28 > >>>>>>>>>>>> 2016 > >>>>>>>>>>>>> +0100 > >>>>>>>>>>>>> > >>>>>>>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 > >>>>>>>>> 16:52:17 > >>>>>>>>>>>> 2016 > >>>>>>>>>>>>> +0100 > >>>>>>>>>>>>> > >>>>>>>>>>>>> @@ -701,7 +701,7 @@ > >>>>>>>>>>>>> > >>>>>>>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { > >>>>>>>>>>>>> > >>>>>>>>>>>>> // Creating guard page is very expensive. Java thread has > >>>>> HotSpot > >>>>>>>>>>>>> // guard page, only enable glibc guard page for non-Java > >>>>> threads. > >>>>>>>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); > >>>>>>>>>>>>> > >>>>>>>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); > >>>>>>>>>>>>> > >>>>>>>>>>>>> } > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> // Java thread: > >>>>>>>>>>>>> From boris.molodenkov at oracle.com Tue Nov 29 13:46:56 2016 From: boris.molodenkov at oracle.com (Boris Molodenkov) Date: Tue, 29 Nov 2016 16:46:56 +0300 Subject: RFR: 8166132: Convert TestGlobalDefinitions_test to GTest Message-ID: <583D86D0.4050307@oracle.com> Hi All, Please review the following patch converting TestGlobalDefinitions internal VM tests to GTest. JBS:https://bugs.openjdk.java.net/browse/JDK-8166132 Webrev: http://cr.openjdk.java.net/~bmoloden/8166132/webrev.00/ Thanks, Boris From naoto.sato at oracle.com Tue Nov 29 17:21:20 2016 From: naoto.sato at oracle.com (Naoto Sato) Date: Tue, 29 Nov 2016 09:21:20 -0800 Subject: RFR(XS) 8170431: non-ASCII characters in source code comments (jimage.hpp) In-Reply-To: <583D154A.9080309@oracle.com> References: <583D154A.9080309@oracle.com> Message-ID: Looks fine, David. Naoto On 11/28/16 9:42 PM, David Buck wrote: > Hi! > > Please have a look at this small yet urgent fix to comments in a header > file. > > bug report: https://bugs.openjdk.java.net/browse/JDK-8170431 > webrev: http://cr.openjdk.java.net/~dbuck/jdk8170431_0/ > > These changes are exactly the same as the changes made by Naoto to the > class library's copy of jimage.hpp: > > http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/cfa705d1f4e2 > > This is actually an important fix because VS will choke on this file > under certain Windows locales (Chinese and Japanese have been reported > so far). So the build is broken for many users without this fix. > > Cheers, > -Buck From daniel.daugherty at oracle.com Tue Nov 29 19:04:23 2016 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 29 Nov 2016 12:04:23 -0700 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> <4dfa4ae6-4fa6-994c-f2cf-8fc309d8c520@oracle.com> Message-ID: On 11/29/16 2:30 AM, Lindenmaier, Goetz wrote: > Hi Dan, > > see my replies inline ... > New webrev: > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.05/ src/os/aix/vm/os_aix.cpp L887: // libc guard page nit - You made other existing comments into sentences (leading capital and trailing '.'), but not this new comment. Why? src/os/aix/vm/os_aix.hpp No comments. src/os/linux/vm/os_linux.cpp L6096: // | |/ 1 page glibc guard. nit - "1 page glibc guard" -> "1 glibc guard page." src/os/posix/vm/os_posix.cpp No comments. src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp No comments. src/os_cpu/bsd_x86/vm/os_bsd_x86.cpp L875: // | |/ 1 page glibc guard. nit - "1 page glibc guard" -> "1 glibc guard page." src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp No comments. src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp No comments. src/os_cpu/linux_s390/vm/os_linux_s390.cpp No comments. src/os_cpu/linux_sparc/vm/os_linux_sparc.cpp No comments. src/os_cpu/linux_x86/vm/os_linux_x86.cpp No comments. src/os_cpu/linux_zero/vm/os_linux_zero.cpp No comments. Thumbs up! I don't need to see a new webrev if you choose to fix the minor comments above. Some replies embedded below. > >> -----Original Message----- >> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >> Sent: Dienstag, 29. November 2016 01:38 >> To: Lindenmaier, Goetz ; hotspot-compiler- >> dev at openjdk.java.net; 'hotspot-runtime-dev at openjdk.java.net' > runtime-dev at openjdk.java.net> >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >> >> On 11/28/16 2:08 AM, Lindenmaier, Goetz wrote: >>> Hi, >>> >>> I'm working on a fix for OS guard pages on stacks. I figured there >>> are VM guard pages (reserved, yellow, red) on the compiler stacks >>> _and_ OS guard pages. For Java threads, the OS guard pages are left >>> out. I think this should be done for compiler threads, too. Please >>> confirm. >>> Webrev: >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.04/ >> src/os/aix/vm/os_aix.cpp >> L888: pthread_attr_setguardsize(&attr, >> os::Aix::default_guard_size(thr_type)); >> No check or assert on the return status of this call. >> Is one needed? > I implemented this as the existing code on linux which has > no check either. I think a failure is quite theoretical. > Because of your comment below I'll leave it as-is. OK. > >> L3044: // guard pages, so only enable libc guard pages for >> non-Java threads. >> src/os/linux/vm/os_linux.cpp also has this comment: >> // (Remember: compiler thread is a Java thread, too!) > Fixed. > >> L3051: return ((thr_type == java_thread || thr_type == >> compiler_thread) ? 0 : 4*K); >> nit - please add spaces around the '*' so '4 * K'.' > Fixed. > >> src/os/aix/vm/os_aix.hpp >> No comments. >> >> src/os/linux/vm/os_linux.cpp >> L729: // is not implemented properly. The posix standard requires >> to add >> Typo: 'to add' -> 'adding' > Fixed. > >> L738: pthread_attr_setguardsize(&attr, >> os::Linux::default_guard_size(thr_type)); >> No check or assert on the return status of this call. >> Is one needed? > See above. > >> L2851: // Creating guard page is very expensive. Java thread has >> HotSpot >> L2852: // guard page, only enable glibc guard page for non-Java >> threads. >> L2853: // (Remember: compiler thread is a java thread, too!) >> Typo: "java thread" -> "Java thread" (consistency) >> >> This comment block should be common to all the platforms that >> define default_guard_size(). Yes, I can see that AIX needs to >> add another paragraph, but we should make the core comment common >> if possible. > I made the first three lines look alike. > >> L6090: // Java/Compiler thread: >> Thanks for making this common in os_linux.cpp. >> >> L6095: // | glibc guard page | - guard, attached Java >> thread usually has >> Clarity: "guard," -> "guard page," > Fixed. > >> Typo: "Java thread" -> "JavaThread" (consistency) > I changed both to Java thread as at the other locations. > >> L6099: // | HotSpot Guard Pages | - red and yellow pages >> The fairly recently added reserved page should be mentioned >> here also? > Yes. Fixed. Also fixed it to say JavaThread::stack_reserved_zone_base(). > Also fixed comment on bsd. Thanks for also fixing BSD. >> L6120 // ** P1 (aka bottom) and size ( P2 = P1 - size) are the >> address and stack size returned from >> Typo: "( P2 = ..." -> "(P2 = ..." > Fixed. > >> L6148: fatal("Can not locate current stack attributes!"); >> Typo: "Can not" -> "Cannot" > Fixed. > >> L6175: // stack size includes normal stack and HotSpot guard pages >> Perhaps add to the comment: >> "for the threads that have HotSpot guard pages." > Fixed. I also checked my comments for "OS guard pages" and > replaced it by "glibc guard pages" which is used in several places > already, same for "VM guard page" --> "HotSpot guard page". I > think this is also more consistent. I agree! >> src/os/posix/vm/os_posix.cpp >> L1097: pthread_attr_getstacksize(attr, &stack_size); >> L1098: pthread_attr_getguardsize(attr, &guard_size); >> Do these two calls need to have their result checked? >> >> Now I'm starting to wonder if all the uses of these >> two APIs need to be checked? Separate bug? > It would be more consistent with the specification of the methods, > On the other side it's quite unlikely that these fail if attr != NULL. So should we file a new bug? Or does this fall into the realm of other OS/libc code that we call and assume never fails? :-) > >> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >> L540: size_t os::Posix::_compiler_thread_min_stack_allowed = 512 * K; >> L541: size_t os::Posix::_java_thread_min_stack_allowed = 512 * K; >> So prior to the fix for 8140520, src/os/aix/vm/os_aix.cpp had >> this single min_stack_allowed value: >> >> L3601: os::Aix::min_stack_allowed = >> MAX2(os::Aix::min_stack_allowed, >> L3602: JavaThread::stack_guard_zone_size() + >> L3603: JavaThread::stack_shadow_zone_size() + >> L3604: (4*BytesPerWord >> COMPILER2_PRESENT(+2)) * 4 * K); >> >> and the fix for 8140520 changed that for *NIX platforms to >> three mins in src/os/posix/vm/os_posix.cpp: >> >> L1108: _java_thread_min_stack_allowed = >> MAX2(_java_thread_min_stack_allowed, >> L1109: JavaThread::stack_guard_zone_size() + >> L1110: JavaThread::stack_shadow_zone_size() + >> L1111: (4 * >> BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K); >> >> L1150: _compiler_thread_min_stack_allowed = >> align_size_up(_compiler_thread_min_stack_allowed, vm_page_size()); >> >> L1161 _vm_internal_thread_min_stack_allowed = >> align_size_up(_vm_internal_thread_min_stack_allowed, vm_page_size()); >> >> Which means that the compiler_thread no longer benefits from >> the extra space for quard and shadow pages. The thinking in >> 8140520 was that the compiler_thread and vm_internal_threads >> don't need the quard and shadow pages since they don't run >> Java code (ignoring JVMCI for now). >> >> So I can see bumping _compiler_thread_min_stack_allowed from >> 128 -> 512 as one solution for getting that extra space back. >> However, I don't understand why _java_thread_min_stack_allowed >> has changed from 128 -> 512. > Because it was never correct before. OK. That sounds like the new test that I included with 8140520 would have failed with JavaThread stack sizes even before the product code changes from 8140520 were made. Since the size calculation for JavaThread stack sizes wasn't changed for any platform in 8140520, that tends to indicate that the more limited JDK test (test/tools/launcher/TooSmallStackSize.java) should also have failed before the fix for 8140520. Please clarify the need for the _java_thread_min_stack_allowed change from 128 -> 512. Unless test/tools/launcher/TooSmallStackSize.java is never run in your testing, I'm having troubling seeing why the _java_thread_min_stack_allowed increase is needed. >> I had previously made this comment: >> > To put it another way, I'd like to see us add extra space to >> > solve the 64K page issue directly instead of as a side effect >> > of the red/yellow page addition. >> And Goetz replied with: >> > I don't understand. What do you mean by 'directly'? >> >> So prior to the fix for 8140520, src/os/solaris/vm/os_solaris.cpp >> had a block like this: >> >> L4468: // For 64kbps there will be a 64kb page size, which makes >> L4469: // the usable default stack size quite a bit less. >> Increase the >> L4470: // stack for 64kb (or any > than 8kb) pages, this >> increases >> L4471: // virtual memory fragmentation (since we're not >> creating the >> L4472 // stack on a power of 2 boundary. The real fix for this >> L4473 // should be to fix the guard page mechanism. >> L4474 >> L4475 if (vm_page_size() > 8*K) { >> L4476 threadStackSizeInBytes = (threadStackSizeInBytes != 0) >> L4477 ? threadStackSizeInBytes + >> L4478 JavaThread::stack_red_zone_size() + >> L4479 JavaThread::stack_yellow_zone_size() >> L4480 : 0; >> L4481 ThreadStackSize = threadStackSizeInBytes/K; >> L4482 } >> >> The above is an example of what I mean by solving the 64K >> page issue directly. In the fix for 8140520, that code block >> was moved to os::Posix::set_minimum_stack_sizes() in >> src/os/posix/vm/os_posix.cpp and put in a "#ifdef SOLARIS... >> #endif // SOLARIS" block. Coleen filed a bug to determine >> whether that code can be deleted: >> >> JDK-8161093 Solaris for >8k pagesize adds extra guard pages >> https://bugs.openjdk.java.net/browse/JDK-8161093 >> >> but perhaps this bug shows that the code is needed? >> >> >> OK so this is probably the longest code review comment >> I have ever written, but the summary is: >> >> - I understand bumping _compiler_thread_min_stack_allowed, >> but should it be solved in a different way? >> - I don't understand bumping _java_thread_min_stack_allowed > I plan to do a follow up change to fix this. Let's leave this discussion > to that review. Here I just want to fix the NPTL issue and the basic > sizing that is needed to pass the new test on ppc/s390. Same question here about the simpler JDK version of the test. Does test/tools/launcher/TooSmallStackSize.java get run in your test environments? > >> src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp >> No comments. >> >> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp >> L538: size_t os::Posix::_compiler_thread_min_stack_allowed = 384 * K; >> L539: size_t os::Posix::_java_thread_min_stack_allowed = 384 * K; >> >> Same monster comment as src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >> and the same summary: >> >> - I understand bumping _compiler_thread_min_stack_allowed, >> but should it be solved in a different way? >> - I don't understand bumping _java_thread_min_stack_allowed >> >> src/os_cpu/linux_s390/vm/os_linux_s390.cpp >> L478: size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >> L479: size_t os::Posix::_java_thread_min_stack_allowed = 236 * K; >> Bumping _java_thread_min_stack_allowed but not bumping >> _compiler_thread_min_stack_allowed. I'm confused here. > The numbers are what I need to startup on the machines. 128 is just > fine on the machines we have. (That's the problem of the > current setup: you have to tune this compile time constant for the > page size of the machine you are running on. But let's discuss this > in the follow up change.) OK about discussing this with a follow-up change. I guess I see the compile time initialization as a "minimum setting assuming the smallest page size". If we discover (at runtime) that the page size is bigger, then we adjust the minimum that we need... Again, defer to another bug. Do we have a bug ID yet? >> src/os_cpu/linux_sparc/vm/os_linux_sparc.cpp >> No comments. >> >> src/os_cpu/linux_x86/vm/os_linux_x86.cpp >> No comments. >> >> src/os_cpu/linux_zero/vm/os_linux_zero.cpp >> No comments. >> >> Sorry it took me so long to write this up... > No matter, thanks for this thorough review! You are very welcome. Thanks for being willing to dive into such a complicated area (thread stack sizes)... Dan > > Best regards, > Goetz. > >> Dan >> >>> The change affecting the compier threads is in os_linux.cpp, >> default_guard_size(), >>> where '|| thr_type == compiler_thread' has been added. The function >>> was also moved from the os_cpu files, as it's identical on all cpus. >>> >>> Best regards, >>> Goetz. >>> >>> >>>> -----Original Message----- >>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>> Sent: Montag, 28. November 2016 00:25 >>>> To: Lindenmaier, Goetz ; >>>> 'daniel.daugherty at oracle.com' ; 'hotspot- >>>> runtime-dev at openjdk.java.net' >>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>>> >>>> Hi Goetz, >>>> >>>> On 24/11/2016 10:15 PM, Lindenmaier, Goetz wrote: >>>>> Hi, >>>>> >>>>> I now edited the stuff I had proposed below: >>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.03/ >>>>> This includes >>>>> - the NPTL fix from webrev.02 >>>> Okay in principle. As discussed this only impacts non-JavaThreads so the >>>> change should be minimal. >>>> >>>>> - merging code on linux >>>> Went a bit further than I had expected but if this truly isn't CPU >>>> dependent code then great! >>>> >>>>> - not adding OS guard to compiler threads. >>>> Okay in principle. IIUC we will now save the OS guard page for compiler >>>> thread stacks. Is that the only impact? The hotspot-compiler-dev folk >>>> may want to sign off on this part. >>>> >>>> >>>> A few minor comments: >>>> >>>> src/os/linux/vm/os_linux.cpp >>>> >>>> 2854 return ((thr_type == java_thread || thr_type == >>>> os::compiler_thread) ... >>>> >>>> os:: should be used for both types or none. >>>> >>>> 6153 pthread_attr_getguardsize(&attr, &guard_size); >>>> >>>> Can you at least verify a zero return code in an assert/assert_status >>>> please. >>>> >>>> --- >>>> >>>> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >>>> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp >>>> src/os_cpu/linux_s390/vm/os_linux_s390.cpp >>>> >>>> Are the changes to min_stack_allowed just fixing an existing bug? >>>> >>>> --- >>>> >>>> Thanks, >>>> David >>>> ----- >>>> >>>> >>>>> I think this should be pushed for this bug ID. For the other changes I'll >>>>> make another bug. >>>>> >>>>> Best regards, >>>>> Goetz. >>>>> >>>>>> -----Original Message----- >>>>>> From: Lindenmaier, Goetz >>>>>> Sent: Wednesday, November 23, 2016 8:11 AM >>>>>> To: David Holmes ; >>>>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net >>>>>> Subject: RE: RFR(M): 8169373: Work around linux NPTL stack guard error. >>>>>> >>>>>> Hi, >>>>>> >>>>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The >>>>>>> ability for a CompilerThread to execute Java code (can_call_java()) is >>>>>>> now a dynamic property depending on whether the current compiler is >> the >>>>>>> JVMCI compiler. >>>>>> Ah, then I should also leave space for shadow pages in the minimal stack >>>> size >>>>>> of comiler threads. >>>>>> >>>>>> Do we agree on the cleanup and on leaving out the OS guard page on >>>>>> compiler threads? >>>>>> Then I would edit a change comprising the NPTL workaround and these >>>>>> additional changes, and split the other issue into a new bug? I think this >>>>>> will easy the reviewing. >>>>>> >>>>>> Best regards, >>>>>> Goetz. >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>>> Sent: Mittwoch, 23. November 2016 02:50 >>>>>>> To: Lindenmaier, Goetz ; >>>>>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >> error. >>>>>>> On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: >>>>>>>> Hi Dan, >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>>>>>>> Sent: Dienstag, 22. November 2016 14:01 >>>>>>>>> To: Lindenmaier, Goetz ; David Holmes >>>>>>>>> ; hotspot-runtime-dev at openjdk.java.net >>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >>>>>> error. >>>>>>>>> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: >>>>>>>>>> Hi Dan, >>>>>>>>>> >>>>>>>>>> I ran into a row of issues, some errors on the platforms. >>>>>>>>>> >>>>>>>>>> What I meant with that comment is that >>>>>>>>>> os::Linux::min_stack_allowed = MAX2(os::Linux::min_stack_allowed, >>>>>>>>>> JavaThread::stack_guard_zone_size() + >>>>>>>>>> JavaThread::stack_shadow_zone_size() + >>>>>>>>>> (4*BytesPerWord >> COMPILER2_PRESENT(+2)) * >>>> 4 >>>>>> * >>>>>>> K); >>>>>>>>>> was executed, and min_stack_allowed used for all stacks. Now, >>>>>> compiler >>>>>>>>> and >>>>>>>>>> vm minimum stack sizes are not increased by these sizes. >>>>>>>>> Now I see what you mean. Thanks for clearing this up. >>>>>>>>> >>>>>>>>> I should have remembered that part of the change because we went >>>>>> back >>>>>>>>> and forth about removing the red/yellow zone pages from the other >>>>>>>>> threads. In particular, we discussed the compiler thread because it >>>>>>>>> is-a JavaThread. Our conclusion was that a compiler thread doesn't >>>>>>>>> execute Java bytecode so we could remove the red/yellow pages... >>>>>>>> Yes, it does not execute java byte code. >>>>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. The >>>>>>> ability for a CompilerThread to execute Java code (can_call_java()) is >>>>>>> now a dynamic property depending on whether the current compiler is >> the >>>>>>> JVMCI compiler. >>>>>>> >>>>>>> David >>>>>>> ----- >>>>>>> >>>>>>>> Therefore you can remove the shadow pages. There is no code that >>>>>>>> will bang. >>>>>>>> But red/yellow pages are protected right during thread startup. >>>>>>>> Therefore you must have enough space for them. >>>>>>>> On ppc, we try to protect three 64K pages out of the 128K compiler >> stack. >>>>>>>> That obviously fails. >>>>>>>> >>>>>>>> Therefore I propose: >>>>>>>> size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // Set >>>>>>> platform dependent. >>>>>>>> in os::Posix::set_minimum_stack_sizes(): >>>>>>>> _java_thread_min_stack_allowed = _java_thread_min_stack_allowed >> + >>>>>>>> JavaThread::stack_guard_zone_size() + >>>>>>>> JavaThread::stack_shadow_zone_size(); >>>>>>>> >>>>>>>> (Similar for _compiler_thread_min_stack_allowed). >>>>>>>> >>>>>>>> The minimal stack size is made up of three components: >>>>>>>> 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot >>>>>>> implementation/platform/os. >>>>>>>> 2. Sizes of C++ frames: depends on C++ compiler. >>>>>>>> These are fixed at compile time. >>>>>>>> 3. Sizes of red/yellow/reserved/shadow pages. Depends on the >> system >>>>>>> the >>>>>>>> VM is used on. This is not fixed at compile time. (Our ppc >> machines >>>>>>> differ >>>>>>>> in page size.) >>>>>>>> Therefore 3. should not be included in a compile time constant. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> And that decision allowed us to be exposed to the 64K page issue >>>>>>>>> because the "extra" space isn't there anymore. >>>>>>>>> >>>>>>>>>> At least the _compiler_thread_min_stack_allowed should be >> increased >>>>>>>>>> similarly by red/yellow zone size. The compiler thread is a Java >>>>>>>>>> thread and must have space for these zones. >>>>>>>>> I'm not sure that I completely agree (yet). To me, the red/yellow >>>>>>>>> pages are there for Java thread stack overflow semantics. Yes, the >>>>>>>>> compiler thread needs extra space when 64K pages are used, but I >>>>>>>>> would prefer that we add that space via a different calculation. >>>>>>>> Yes they are. But compiler threads happen tob e a subclass of >>>>>>>> Java threads and use the same run() method that puts the pages >>>>>>>> There. >>>>>>>> I agree that they are not needed for Compiler threads, nor for >>>>>>>> CodeCacheSweeperThreads. I don't really now about >>>>>>>> JvmtiAgentThreads and ServiceThreads, but all of the get the guard >>>>>> pages >>>>>>>> because they are derived from JavaThread. >>>>>>>> >>>>>>>>> To put it another way, I'd like to see us add extra space to solve >>>>>>>>> the 64K page issue directly instead of as a side effect of the >>>>>>>>> red/yellow page addition. >>>>>>>> I don't understand. What do you mean by 'directly'? >>>>>>>> >>>>>>>>>> Also, the change added a test that is failing now. >>>>>>>>> And that's a "good thing" (TM), right? :-) >>>>>>>> Yes, it showed a bug and thus raised the need to fix it! :) >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Goetz. >>>>>>>> >>>>>>>> >>>>>>>>> Again, thanks for clarifying 8140520's role in this issue. >>>>>>>>> >>>>>>>>> Dan >>>>>>>>> >>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> Goetz. >>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>>>>>>>>> Sent: Montag, 21. November 2016 17:28 >>>>>>>>>>> To: David Holmes ; Lindenmaier, Goetz >>>>>>>>>>> ; hotspot-runtime- >>>>>>> dev at openjdk.java.net >>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >>>>>>> error. >>>>>>>>>>> Sorry for the delayed responses to this thread. I've been on >> vacation... >>>>>>>>>>> One comment/query embedded below... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 11/10/16 8:40 PM, David Holmes wrote: >>>>>>>>>>>> Hi Goetz, >>>>>>>>>>>> >>>>>>>>>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>>>> Hi David, >>>>>>>>>>>>> >>>>>>>>>>>>> This issue is different to 6675312, see also my comment in the >> bug. >>>>>>>>>>>>> It appears running jtreg test >>>>>>> runtime/Thread/TooSmallStackSize.java, >>>>>>>>>>>>> with my patch below you can reproduce it on linuxx86_64. You >> can >>>>>>> not >>>>>>>>>>>>> do that with 6675312. Also, I would assume there are systems out >>>>>>> there >>>>>>>>>>>>> on x86 that uses 64-K pages, did you run the tests on these? I >>>>>> would >>>>>>>>>>>>> assume you get hard crashes with stack overflows in the first C2 >>>>>>>>>>>>> compilation if there is only 64K usable CompilerThreadStack. >>>>>>>>>>>>> >>>>>>>>>>>>> My fix does not affect Java threads, which are the largest amount >>>>>>>>>>>>> of threads used by the VM. It affects only the non-Java threads. >>>>>>>>>>>>> It adds one page to these threads. The page does not require >>>>>>> memory, >>>>>>>>>>>>> as it's protected. The stack will only require more space if the >>>>>> thread >>>>>>>>>>>>> ran into a stack overflow before the fix as else the pages are not >>>>>>>>>>>>> mapped. >>>>>>>>>>>>> This are stack overflows that cause hard crashes, at least on ppc >>>>>> the >>>>>>> VM >>>>>>>>>>>>> does not properly catch these stack overflows, so any setup >>>>>> working >>>>>>>>> now >>>>>>>>>>>>> will not run into the additional space. Altogether there should be >>>>>> no >>>>>>>>>>>>> effect on running systems besides requiring one more entry in >> the >>>>>>>>>>>>> page table per non-Java thread. >>>>>>>>>>>>> >>>>>>>>>>>>> The problem is caused by a rather recent change (8140520: >>>>>> segfault >>>>>>> on >>>>>>>>>>>>> solaris-amd64 >>>>>>>>>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed after >>>>>>>>>>>>> feature-close. As this was a rather recent change, it must be >>>>>>>>>>>>> possible to >>>>>>>>>>>>> fix this follow up issue. What else is this period in the project >> good >>>>>>>>>>>>> for if not fixing issues? >>>>>>>>>>>> So I am seeing a number of factors here. >>>>>>>>>>>> >>>>>>>>>>>> First, 8140520, set: >>>>>>>>>>>> >>>>>>>>>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >>>>>>>>>>> So I'm confused by the above comment: >>>>>>>>>>> >>>>>>>>>>> > The problem is caused by a rather recent change (8140520: >>>>>> segfault >>>>>>>>>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) >>>>>>>>>>> >>>>>>>>>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- >>>>>>>>>>> >>>>>>> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html >>>>>>>>>>> shows this change: >>>>>>>>>>> >>>>>>>>>>> @@ -531,19 +531,17 @@ >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>> ////////////////////////////////////////////////////////////////////////////// >>>>>>>>>>> // >>>>>>>>>>> // thread stack >>>>>>>>>>> >>>>>>>>>>> -size_t os::Linux::min_stack_allowed = 128*K; >>>>>>>>>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >>>>>>>>>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; >>>>>>>>>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 * >> K; >>>>>>>>>>> so the existing single variable of 'min_stack_allowed' was >>>>>>>>>>> replaced by three variables: _compiler_thread_min_stack_allowed, >>>>>>>>>>> _java_thread_min_stack_allowed, and >>>>>>>>>>> _vm_internal_thread_min_stack_allowed. >>>>>>>>>>> >>>>>>>>>>> The old single variable and the three new variables are all >>>>>>>>>>> initialized to the same value (128K) so the fix for 8140520 >>>>>>>>>>> did not change stack sizes for this platform. In fact, only >>>>>>>>>>> one platform had a size change (Solaris X64). >>>>>>>>>>> >>>>>>>>>>> So I'm confused about how the fix for 8140520 caused this problem. >>>>>>>>>>> >>>>>>>>>>> Based on David's analysis below, it looks to me like this 64K stack >>>>>>>>>>> guard page problem should also exist prior to the fix for 8140520. >>>>>>>>>>> >>>>>>>>>>> Goetz, can you please explain how 8140520 caused this problem? >>>>>>>>>>> >>>>>>>>>>> Dan >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Second on linux PPC it is hardwired to use 2 guard pages: >>>>>>>>>>>> >>>>>>>>>>>> return 2 * page_size(); >>>>>>>>>>>> >>>>>>>>>>>> Third, you had a pagesize of 64K. >>>>>>>>>>>> >>>>>>>>>>>> Fourth, NPTL takes the guard space from the stack space - hence >>>>>> with >>>>>>> 2 >>>>>>>>>>>> x 64K guard, and a 128K stack it was all consumed. >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> >>>>>>>>>>>> In the proposed changes you now only use page_size() for the >> guard, >>>>>>> so >>>>>>>>>>>> that alone would have fixed the observed problem. >>>>>>>>>>>> >>>>>>>>>>>> But in addition you want to address the NPTL problem by adding >>>>>> back >>>>>>>>>>>> the guard space to the stack size requested. That alone would also >>>>>>>>>>>> have fixed the observed problem. :) >>>>>>>>>>>> >>>>>>>>>>>> But in addition you have increased the minimum stack size: >>>>>>>>>>>> >>>>>>>>>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * K; >>>>>>>>>>>> >>>>>>>>>>>> which again, on its own would have fixed the original problem. :) >>>>>>>>>>>> >>>>>>>>>>>> Did you really intend to increase the real minimum stack from >> 128K >>>>>> to >>>>>>>>>>>> 256K ? (on a 64K page system) >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> >>>>>>>>>>>> Focusing simply on the shared code change to adjust the requested >>>>>>>>>>>> stacksize by the amount of guard space (if any), this does not seem >>>>>>>>>>>> unreasonable. As you note it is restricted to non-JavaThreads and >>>>>> only >>>>>>>>>>>> adds a page to reserved stack space. >>>>>>>>>>>> >>>>>>>>>>>> My only query now is whether the minimum stacksize detection >> logic >>>>>>>>>>>> will correctly report the real minimum stack size (taking into >> account >>>>>>>>>>>> the need for the guard page) ? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> David >>>>>>>>>>>> >>>>>>>>>>>>> So I really think this issue should be fixed. >>>>>>>>>>>>> >>>>>>>>>>>>> Best regards, >>>>>>>>>>>>> Goetz. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>>>>>>>>>> Sent: Thursday, November 10, 2016 10:02 PM >>>>>>>>>>>>>> To: Lindenmaier, Goetz ; hotspot- >>>>>>>>>>> runtime- >>>>>>>>>>>>>> dev at openjdk.java.net >>>>>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack >>>>>> guard >>>>>>>>> error. >>>>>>>>>>>>>> Hi Goetz, >>>>>>>>>>>>>> >>>>>>>>>>>>>> As per the bug report, this issue was already known (6675312) >> and >>>>>>> we >>>>>>>>>>>>>> chose not to try and address it due to no reported issues at the >>>>>>> time. >>>>>>>>>>>>>> While I see that you have encountered an issue (is it real or >>>>>>>>>>>>>> fabricated?) I think this change is too intrusive to be applied at >> this >>>>>>>>>>>>>> stage of the JDK 9 release cycle, as it will change the stack >>>>>>>>>>>>>> requirements of every application running on Linux. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> David >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please review this change. I please need a sponsor: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >>>>>>>>>>> stackFix/webrev.01/ >>>>>>>>>>>>>>> In the Linux NPTL pthread implementation the guard size >>>>>>> mechanism >>>>>>>>>>>>>>> is not >>>>>>>>>>>>>>> implemented properly. The posix standard requires to add the >>>>>>> size >>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>> guard pages to the stack size, instead Linux takes the space out >>>>>> of >>>>>>>>>>>>>>> 'stacksize'. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The Posix standard >>>>>>>>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ >>>>>>>>>>>>>>> says "the implementation allocates extra memory at the >>>>>>> overflow >>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>> the stack". The linux man page >>>>>>>>>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says "As >>>>>>> at >>>>>>>>>>>>>>> glibc >>>>>>>>>>>>>>> 2.8, the NPTL threading implementation includes the guard >> area >>>>>>>>> within >>>>>>>>>>>>>>> the stack size allocation, rather than allocating extra space at >>>>>>>>>>>>>>> the end >>>>>>>>>>>>>>> of the stack, as POSIX.1 requires". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I encounter this problem in >>>>>>> runtime/Thread/TooSmallStackSize.java >>>>>>>>>>>>>>> on ppc >>>>>>>>>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is 128K >>>>>> on >>>>>>>>> ppc, >>>>>>>>>>> and >>>>>>>>>>>>>>> ppc specifies two OS guard pages. The VM crashes in pthread >>>>>>>>> creation >>>>>>>>>>>>>>> because there is no usable space in the thread stack after >>>>>>> allocating >>>>>>>>>>>>>>> the guard pages. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But TooSmallStackSize.java requires that the VM comes up >> with >>>>>>> the >>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>> size mentioned in the error message. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This fix adapts the requested stack size on Linux by the size of >>>>>> the >>>>>>>>>>>>>>> guard pages to mimick proper behaviour, see change to >>>>>>>>> os_linux.cpp. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The change also streamlines usage of stack_guard_page on >>>>>>> linuxppc, >>>>>>>>>>>>>>> linuxppcle, aixppc and linuxs390. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To reproduce the error on linux_x86_64, apply below patch >> and >>>>>>> call >>>>>>>>> the >>>>>>>>>>>>>>> VM with -XX:CompilerThreadStackSize=64. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm still exploring why I had to choose such big compiler stacks >>>>>> on >>>>>>>>>>>>>>> ppc >>>>>>>>>>>>>>> to get -version passing, but I wanted to send the RFR now as >>>>>>> people >>>>>>>>>>>>>>> obviously looked at the bug I opened (Thanks David!). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Goetz. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> diff -r b7ae012c55c3 >> src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>>>>>>>>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov 07 >>>>>>>>> 12:37:28 >>>>>>>>>>>>>> 2016 >>>>>>>>>>>>>>> +0100 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov 10 >>>>>>>>>>> 16:52:17 >>>>>>>>>>>>>> 2016 >>>>>>>>>>>>>>> +0100 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> @@ -701,7 +701,7 @@ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType thr_type) { >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> // Creating guard page is very expensive. Java thread has >>>>>>> HotSpot >>>>>>>>>>>>>>> // guard page, only enable glibc guard page for non-Java >>>>>>> threads. >>>>>>>>>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> // Java thread: >>>>>>>>>>>>>>> From sean.coffey at oracle.com Tue Nov 29 19:59:08 2016 From: sean.coffey at oracle.com (=?UTF-8?Q?Se=c3=a1n_Coffey?=) Date: Tue, 29 Nov 2016 19:59:08 +0000 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> Message-ID: Good to see the exception handling being verbose with context Alan. I've given a quick look at that area and have only some minor comments. Layer.java : 635 if (parentLayers.size() != parentConfigurations.size()) 636 throw new IllegalArgumentException("wrong number of parents"); Could we print the number of parents expected versus detected ? src/jdk.jartool/share/classes/sun/tools/jar/Main.java 856 if (name.startsWith(VERSIONS_DIR)) { 857 throw new InternalError(name); As unlikely as it is, could we hint at what went wrong ? .. ("Can't add package beginning with " + VERSIONS_DIR) some typos : src/jdk.jlink/share/classes/jdk/tools/jlink/internal/plugins/SystemModuleDescriptorPlugin.java 1076 /** 1077 * Loads a Enum field. src/java.base/share/classes/java/lang/module/ModuleDescriptor.java 1121 * @return {@code true} if this is a open module regards, Sean. On 24/11/2016 15:25, Alan Bateman wrote: > Folks on jigsaw-dev will know that we are on a mission to bring the > changes accumulated in the jake forest to jdk9/dev. We can think of > this as a refresh of the module system in JDK 9, the last big refresh > was in May with many small updates since then. > > The focus this time is to bring the changes that are tied to JSR > issues into jdk9/dev, specifically the issues that are tracked on the > JSR issues list [1] as: > > #CompileTimeDependences > #AddExportsInManifest > #ClassFileModuleName > #ClassFileAccPublic > #ServiceLoaderEnhancements > #ResourceEncapsulation/#ClassFilesAsResources > #ReflectiveAccessToNonExportedTypes > #AwkwardStrongEncapsulation > #ReadabilityAddedByLayerCreator > #IndirectQualifiedReflectiveAccess (partial) > #VersionsInModuleNames > #NonHierarchicalLayers > #ModuleAnnotations/#ModuleDeprecation > #ReflectiveAccessByInstrumentationAgents > > Some of these issues are not "Resolved" yet, meaning there is still > ongoing discussion on the EG mailing list. That is okay, there is > nothing final here. If there are changes to these proposals then the > implementation changes will follow. Also, as I said in a mail to > jigsaw-dev yesterday [2], is that we will keep the jake forest open > for ongoing prototyping and iteration, also ongoing implementation > improvements where iteration or bake time is important. > > For the code review then the focus is therefore on sanity checking the > changes that we would like to bring into jdk9/dev. We will not use > this review thread to debate alternative designs or other big > implementation changes that are more appropriate to bake in jake. > > To get going, I've put the webrevs with a snapshot of the changes in > jake here: > http://cr.openjdk.java.net/~alanb/8169069/0/ > > The changes are currently sync'ed against jdk-9+146 and will be > rebased (and re-tested) against jdk9/dev prior to integration. There > are a number of small changes that need to be added to this in the > coming days, I will refresh the webrev every few days to take account > of these updates. > > > A few important points to mention, even if you aren't reviewing the > changes: > > 1. This refresh requires a new version of jtreg to run the tests. The > changes for this new version are in the code-tools/jtreg repository > and the plan is to tag a new build (jtreg4.2-b04) next week. Once the > tag has been added then we'll update the requiredVersion property in > each TEST.ROOT to force everyone to update. > > 2. For developers trying out modules with the main line JDK 9 builds > then be aware that `requires public` changes to `requires transitive` > and the `provides` clause changes to require all providers for a > specific service type to be in the same clause. Also be aware that the > binary form of the module declaration (module-info.class) changes so > you will need to recompile any modules. > > 3. Those running existing code on JDK 9 and ignoring modules will need > to be aware of a disruptive change in this refresh. The disruptive > change is #AwkwardStrongEncapsulation where setAccessible(true) is > changed so that it can't be used to break into non-public > fields/methods of JDK classes. This change is going to expose a lot of > hacks in existing code. We plan to send mail to jdk9-dev in advance of > this integration to create awareness of this change. As per the > original introduction of strong encapsulation then command line > options (and now the manifest of application JAR files) can be used to > keep existing code working. The new option is `--add-opens` to open a > package in a module for deep reflection by other modules. As an > example, if you find yourself with code that hacks into the private > `comparator` field in java.util.TreeMap then running with `--add-opens > java.base/java.util=ALL-UNNAMED` will keep that code working. > > > A few miscellaneous notes for those that are reviewing: > > 1. We have some temporary/transition code in the top-level repo to > deal with the importing of the JavaFX modules. This will be removed > once the changes are in JDK 9 for the OpenJFX project to use. > > 2. In the jdk repo then it's important to understand that the module > system is initialized at startup and there are many places where we > need to keep startup performance in mind. This sometimes means less > elegant code than might be used if startup wasn't such a big concern. > > 3. The changes in the jaxws repo make use of new APIs that means the > code doesn't compile with JDK 7 or JDK 8. Our intention is to work > with the JAXB and JAX-WS maintainers to address the issues in the > upstream project and then bring those changes into jdk9/dev to replace > the patches that we are forced to push for the short term. > > 4. You will see several tests where the value of the @modules tag has > `:open` or `:+open`. This is new jtreg speak. The former means the > test is run with --add-opens to open the package, the latter means the > test is exported at compile-time and exported + open at run-time (the > latter usage will be rare, it's where tests have static references to > JDK internal types and are also doing deep reflection with > setAccessible). > > > In terms of dates then we are aiming to integrate these changes into > jdk9/dev in early December. I will send a follow-up mail next week on > this as we work through the logistics. > > -Alan > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ > [2] > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html From dmitry.samersoff at oracle.com Tue Nov 29 20:12:16 2016 From: dmitry.samersoff at oracle.com (Dmitry Samersoff) Date: Tue, 29 Nov 2016 23:12:16 +0300 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> <583C4019.8040506@oracle.com> Message-ID: <4606fc5a-6f43-02d9-cb00-bb95fb88e432@oracle.com> Karen, Sorry for delay. I was on vacation last week. I plan to review the changes tomorrow. -Dmitry On 2016-11-28 17:47, Karen Kinnear wrote: > Alan, > > I reviewed all the hotspot runtime changes > - except the tests (Christian will review those) > - and jvmti - which Dmitry Samersoff will review. > > They look good. > > thanks, > Karen > >> On Nov 28, 2016, at 9:32 AM, Lois Foltan wrote: >> >> Hi Alan, >> >> I have reviewed the hotspot changes and they look good. Minor nit, src/share/vm/classfile/javaClasses.cpp only differs by the addition of a blank line. >> >> Thanks, >> Lois >> >> On 11/24/2016 10:25 AM, Alan Bateman wrote: >>> Folks on jigsaw-dev will know that we are on a mission to bring the changes accumulated in the jake forest to jdk9/dev. We can think of this as a refresh of the module system in JDK 9, the last big refresh was in May with many small updates since then. >>> >>> The focus this time is to bring the changes that are tied to JSR issues into jdk9/dev, specifically the issues that are tracked on the JSR issues list [1] as: >>> >>> #CompileTimeDependences >>> #AddExportsInManifest >>> #ClassFileModuleName >>> #ClassFileAccPublic >>> #ServiceLoaderEnhancements >>> #ResourceEncapsulation/#ClassFilesAsResources >>> #ReflectiveAccessToNonExportedTypes >>> #AwkwardStrongEncapsulation >>> #ReadabilityAddedByLayerCreator >>> #IndirectQualifiedReflectiveAccess (partial) >>> #VersionsInModuleNames >>> #NonHierarchicalLayers >>> #ModuleAnnotations/#ModuleDeprecation >>> #ReflectiveAccessByInstrumentationAgents >>> >>> Some of these issues are not "Resolved" yet, meaning there is still ongoing discussion on the EG mailing list. That is okay, there is nothing final here. If there are changes to these proposals then the implementation changes will follow. Also, as I said in a mail to jigsaw-dev yesterday [2], is that we will keep the jake forest open for ongoing prototyping and iteration, also ongoing implementation improvements where iteration or bake time is important. >>> >>> For the code review then the focus is therefore on sanity checking the changes that we would like to bring into jdk9/dev. We will not use this review thread to debate alternative designs or other big implementation changes that are more appropriate to bake in jake. >>> >>> To get going, I've put the webrevs with a snapshot of the changes in jake here: >>> http://cr.openjdk.java.net/~alanb/8169069/0/ >>> >>> The changes are currently sync'ed against jdk-9+146 and will be rebased (and re-tested) against jdk9/dev prior to integration. There are a number of small changes that need to be added to this in the coming days, I will refresh the webrev every few days to take account of these updates. >>> >>> >>> A few important points to mention, even if you aren't reviewing the changes: >>> >>> 1. This refresh requires a new version of jtreg to run the tests. The changes for this new version are in the code-tools/jtreg repository and the plan is to tag a new build (jtreg4.2-b04) next week. Once the tag has been added then we'll update the requiredVersion property in each TEST.ROOT to force everyone to update. >>> >>> 2. For developers trying out modules with the main line JDK 9 builds then be aware that `requires public` changes to `requires transitive` and the `provides` clause changes to require all providers for a specific service type to be in the same clause. Also be aware that the binary form of the module declaration (module-info.class) changes so you will need to recompile any modules. >>> >>> 3. Those running existing code on JDK 9 and ignoring modules will need to be aware of a disruptive change in this refresh. The disruptive change is #AwkwardStrongEncapsulation where setAccessible(true) is changed so that it can't be used to break into non-public fields/methods of JDK classes. This change is going to expose a lot of hacks in existing code. We plan to send mail to jdk9-dev in advance of this integration to create awareness of this change. As per the original introduction of strong encapsulation then command line options (and now the manifest of application JAR files) can be used to keep existing code working. The new option is `--add-opens` to open a package in a module for deep reflection by other modules. As an example, if you find yourself with code that hacks into the private `comparator` field in java.util.TreeMap then running with `--add-opens java.base/java.util=ALL-UNNAMED` will keep that code working. >>> >>> >>> A few miscellaneous notes for those that are reviewing: >>> >>> 1. We have some temporary/transition code in the top-level repo to deal with the importing of the JavaFX modules. This will be removed once the changes are in JDK 9 for the OpenJFX project to use. >>> >>> 2. In the jdk repo then it's important to understand that the module system is initialized at startup and there are many places where we need to keep startup performance in mind. This sometimes means less elegant code than might be used if startup wasn't such a big concern. >>> >>> 3. The changes in the jaxws repo make use of new APIs that means the code doesn't compile with JDK 7 or JDK 8. Our intention is to work with the JAXB and JAX-WS maintainers to address the issues in the upstream project and then bring those changes into jdk9/dev to replace the patches that we are forced to push for the short term. >>> >>> 4. You will see several tests where the value of the @modules tag has `:open` or `:+open`. This is new jtreg speak. The former means the test is run with --add-opens to open the package, the latter means the test is exported at compile-time and exported + open at run-time (the latter usage will be rare, it's where tests have static references to JDK internal types and are also doing deep reflection with setAccessible). >>> >>> >>> In terms of dates then we are aiming to integrate these changes into jdk9/dev in early December. I will send a follow-up mail next week on this as we work through the logistics. >>> >>> -Alan >>> >>> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ >>> [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html >> > -- Dmitry Samersoff Oracle Java development team, Saint Petersburg, Russia * I would love to change the world, but they won't give me the sources. From christian.tornqvist at oracle.com Tue Nov 29 21:19:41 2016 From: christian.tornqvist at oracle.com (Christian Tornqvist) Date: Tue, 29 Nov 2016 16:19:41 -0500 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> Message-ID: <908001d24a86$4d05e230$e711a690$@oracle.com> Hi Alan, I've reviewed the test changes for Hotspot and they look good. Thanks, Christian -----Original Message----- From: hotspot-runtime-dev [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On Behalf Of Alan Bateman Sent: Thursday, November 24, 2016 10:25 AM To: jigsaw-dev ; hotspot-runtime-dev at openjdk.java.net runtime ; compiler-dev at openjdk.java.net Subject: Code review for jigsaw/jake -> jdk9/dev sync up Folks on jigsaw-dev will know that we are on a mission to bring the changes accumulated in the jake forest to jdk9/dev. We can think of this as a refresh of the module system in JDK 9, the last big refresh was in May with many small updates since then. The focus this time is to bring the changes that are tied to JSR issues into jdk9/dev, specifically the issues that are tracked on the JSR issues list [1] as: #CompileTimeDependences #AddExportsInManifest #ClassFileModuleName #ClassFileAccPublic #ServiceLoaderEnhancements #ResourceEncapsulation/#ClassFilesAsResources #ReflectiveAccessToNonExportedTypes #AwkwardStrongEncapsulation #ReadabilityAddedByLayerCreator #IndirectQualifiedReflectiveAccess (partial) #VersionsInModuleNames #NonHierarchicalLayers #ModuleAnnotations/#ModuleDeprecation #ReflectiveAccessByInstrumentationAgents Some of these issues are not "Resolved" yet, meaning there is still ongoing discussion on the EG mailing list. That is okay, there is nothing final here. If there are changes to these proposals then the implementation changes will follow. Also, as I said in a mail to jigsaw-dev yesterday [2], is that we will keep the jake forest open for ongoing prototyping and iteration, also ongoing implementation improvements where iteration or bake time is important. For the code review then the focus is therefore on sanity checking the changes that we would like to bring into jdk9/dev. We will not use this review thread to debate alternative designs or other big implementation changes that are more appropriate to bake in jake. To get going, I've put the webrevs with a snapshot of the changes in jake here: http://cr.openjdk.java.net/~alanb/8169069/0/ The changes are currently sync'ed against jdk-9+146 and will be rebased (and re-tested) against jdk9/dev prior to integration. There are a number of small changes that need to be added to this in the coming days, I will refresh the webrev every few days to take account of these updates. A few important points to mention, even if you aren't reviewing the changes: 1. This refresh requires a new version of jtreg to run the tests. The changes for this new version are in the code-tools/jtreg repository and the plan is to tag a new build (jtreg4.2-b04) next week. Once the tag has been added then we'll update the requiredVersion property in each TEST.ROOT to force everyone to update. 2. For developers trying out modules with the main line JDK 9 builds then be aware that `requires public` changes to `requires transitive` and the `provides` clause changes to require all providers for a specific service type to be in the same clause. Also be aware that the binary form of the module declaration (module-info.class) changes so you will need to recompile any modules. 3. Those running existing code on JDK 9 and ignoring modules will need to be aware of a disruptive change in this refresh. The disruptive change is #AwkwardStrongEncapsulation where setAccessible(true) is changed so that it can't be used to break into non-public fields/methods of JDK classes. This change is going to expose a lot of hacks in existing code. We plan to send mail to jdk9-dev in advance of this integration to create awareness of this change. As per the original introduction of strong encapsulation then command line options (and now the manifest of application JAR files) can be used to keep existing code working. The new option is `--add-opens` to open a package in a module for deep reflection by other modules. As an example, if you find yourself with code that hacks into the private `comparator` field in java.util.TreeMap then running with `--add-opens java.base/java.util=ALL-UNNAMED` will keep that code working. A few miscellaneous notes for those that are reviewing: 1. We have some temporary/transition code in the top-level repo to deal with the importing of the JavaFX modules. This will be removed once the changes are in JDK 9 for the OpenJFX project to use. 2. In the jdk repo then it's important to understand that the module system is initialized at startup and there are many places where we need to keep startup performance in mind. This sometimes means less elegant code than might be used if startup wasn't such a big concern. 3. The changes in the jaxws repo make use of new APIs that means the code doesn't compile with JDK 7 or JDK 8. Our intention is to work with the JAXB and JAX-WS maintainers to address the issues in the upstream project and then bring those changes into jdk9/dev to replace the patches that we are forced to push for the short term. 4. You will see several tests where the value of the @modules tag has `:open` or `:+open`. This is new jtreg speak. The former means the test is run with --add-opens to open the package, the latter means the test is exported at compile-time and exported + open at run-time (the latter usage will be rare, it's where tests have static references to JDK internal types and are also doing deep reflection with setAccessible). In terms of dates then we are aiming to integrate these changes into jdk9/dev in early December. I will send a follow-up mail next week on this as we work through the logistics. -Alan [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html From mandy.chung at oracle.com Tue Nov 29 21:37:19 2016 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 29 Nov 2016 13:37:19 -0800 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <583C4019.8040506@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> <583C4019.8040506@oracle.com> Message-ID: Thanks Lois. I removed the blank line. Mandy > On Nov 28, 2016, at 6:32 AM, Lois Foltan wrote: > > Hi Alan, > > I have reviewed the hotspot changes and they look good. Minor nit, src/share/vm/classfile/javaClasses.cpp only differs by the addition of a blank line. > > Thanks, > Lois > > On 11/24/2016 10:25 AM, Alan Bateman wrote: >> Folks on jigsaw-dev will know that we are on a mission to bring the changes accumulated in the jake forest to jdk9/dev. We can think of this as a refresh of the module system in JDK 9, the last big refresh was in May with many small updates since then. >> >> The focus this time is to bring the changes that are tied to JSR issues into jdk9/dev, specifically the issues that are tracked on the JSR issues list [1] as: >> >> #CompileTimeDependences >> #AddExportsInManifest >> #ClassFileModuleName >> #ClassFileAccPublic >> #ServiceLoaderEnhancements >> #ResourceEncapsulation/#ClassFilesAsResources >> #ReflectiveAccessToNonExportedTypes >> #AwkwardStrongEncapsulation >> #ReadabilityAddedByLayerCreator >> #IndirectQualifiedReflectiveAccess (partial) >> #VersionsInModuleNames >> #NonHierarchicalLayers >> #ModuleAnnotations/#ModuleDeprecation >> #ReflectiveAccessByInstrumentationAgents >> >> Some of these issues are not "Resolved" yet, meaning there is still ongoing discussion on the EG mailing list. That is okay, there is nothing final here. If there are changes to these proposals then the implementation changes will follow. Also, as I said in a mail to jigsaw-dev yesterday [2], is that we will keep the jake forest open for ongoing prototyping and iteration, also ongoing implementation improvements where iteration or bake time is important. >> >> For the code review then the focus is therefore on sanity checking the changes that we would like to bring into jdk9/dev. We will not use this review thread to debate alternative designs or other big implementation changes that are more appropriate to bake in jake. >> >> To get going, I've put the webrevs with a snapshot of the changes in jake here: >> http://cr.openjdk.java.net/~alanb/8169069/0/ >> >> The changes are currently sync'ed against jdk-9+146 and will be rebased (and re-tested) against jdk9/dev prior to integration. There are a number of small changes that need to be added to this in the coming days, I will refresh the webrev every few days to take account of these updates. >> >> >> A few important points to mention, even if you aren't reviewing the changes: >> >> 1. This refresh requires a new version of jtreg to run the tests. The changes for this new version are in the code-tools/jtreg repository and the plan is to tag a new build (jtreg4.2-b04) next week. Once the tag has been added then we'll update the requiredVersion property in each TEST.ROOT to force everyone to update. >> >> 2. For developers trying out modules with the main line JDK 9 builds then be aware that `requires public` changes to `requires transitive` and the `provides` clause changes to require all providers for a specific service type to be in the same clause. Also be aware that the binary form of the module declaration (module-info.class) changes so you will need to recompile any modules. >> >> 3. Those running existing code on JDK 9 and ignoring modules will need to be aware of a disruptive change in this refresh. The disruptive change is #AwkwardStrongEncapsulation where setAccessible(true) is changed so that it can't be used to break into non-public fields/methods of JDK classes. This change is going to expose a lot of hacks in existing code. We plan to send mail to jdk9-dev in advance of this integration to create awareness of this change. As per the original introduction of strong encapsulation then command line options (and now the manifest of application JAR files) can be used to keep existing code working. The new option is `--add-opens` to open a package in a module for deep reflection by other modules. As an example, if you find yourself with code that hacks into the private `comparator` field in java.util.TreeMap then running with `--add-opens java.base/java.util=ALL-UNNAMED` will keep that code working. >> >> >> A few miscellaneous notes for those that are reviewing: >> >> 1. We have some temporary/transition code in the top-level repo to deal with the importing of the JavaFX modules. This will be removed once the changes are in JDK 9 for the OpenJFX project to use. >> >> 2. In the jdk repo then it's important to understand that the module system is initialized at startup and there are many places where we need to keep startup performance in mind. This sometimes means less elegant code than might be used if startup wasn't such a big concern. >> >> 3. The changes in the jaxws repo make use of new APIs that means the code doesn't compile with JDK 7 or JDK 8. Our intention is to work with the JAXB and JAX-WS maintainers to address the issues in the upstream project and then bring those changes into jdk9/dev to replace the patches that we are forced to push for the short term. >> >> 4. You will see several tests where the value of the @modules tag has `:open` or `:+open`. This is new jtreg speak. The former means the test is run with --add-opens to open the package, the latter means the test is exported at compile-time and exported + open at run-time (the latter usage will be rare, it's where tests have static references to JDK internal types and are also doing deep reflection with setAccessible). >> >> >> In terms of dates then we are aiming to integrate these changes into jdk9/dev in early December. I will send a follow-up mail next week on this as we work through the logistics. >> >> -Alan >> >> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ >> [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html > From thomas.stuefe at gmail.com Wed Nov 30 07:43:35 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 30 Nov 2016 08:43:35 +0100 Subject: RFR(s): 8170520: Make Metaspace ChunkManager counters non-atomic Message-ID: Hi all, please take a look at this small fix. (This is one of the smaller fixes in preparation for a prototype for JDK-8166690, I try to move the smaller stuff out of the way first). Issue: https://bugs.openjdk.java.net/browse/JDK-8170520 webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8170520-Make- Metaspace-ChunkManager-Counters-NonAtomic/webrev.00/webrev/ Small discussion here: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2016-October/021709.html Background: ChunkManager keeps counters (_free_chunks_count and free_chunks_total). These counters are updated using Atomics. Because this is expensive, code goes thru some lengths to accumulate updates, which in turn makes the code more complicated and error prone as necessary. But updating the counters atomically is not needed, because updates happen under lock protection anyway. The proposed fix makes updates non-atomic and moved the call to inc_free_chunks_total() into ChunkManager::return_chunks() - close to where the chunk is actually returned to the freelist, so the time window where the counters are invalid is very small now. It also fixes an assert and makes inc_free_chunks_total() private because now it can be private. Note that I do not intend to push this into 9, so this is for the upcoming 10 repository. I would like to get some reviews nevertheless, so this small change can be pushed quickly once the 10 repo exists. Thanks! Best Regards, Thomas From goetz.lindenmaier at sap.com Wed Nov 30 07:57:28 2016 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Wed, 30 Nov 2016 07:57:28 +0000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> <4dfa4ae6-4fa6-994c-f2cf-8fc309d8c520@oracle.com> Message-ID: <821371d77c7c4e6d966901b461b3d6a1@DEROTE13DE08.global.corp.sap> Hi Dan, > Thumbs up! I don't need to see a new webrev if you choose > to fix the minor comments above. I anyways did a new one fixing the comments: http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.06/ Would you mind sponsoring this change? I took the minimum stack sizes from my experimental runs where I had removed the automatic resizing to find the really needed space. If I put something smaller there, I could as well put '0' ... as _java_thread_min_stack_allowed = MAX2(_java_thread_min_stack_allowed, JavaThread::stack_guard_zone_size() + JavaThread::stack_shadow_zone_size() + (4 * BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K); will fix it. This code effectively limits the usable stack size to (4 * BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K) which makes the initialization of _java_thread_min_stack_allowed with platform values pointless. I'll open a new bug for the follow-up stack issue, probably tomorrow. I don't feel like addressing testing all the possible error codes, as they probably should be fixed in more places, and there is no issue pending currently. Maybe it should be fixed in jdk10 at some point. Best regards, Goetz > -----Original Message----- > From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > Sent: Dienstag, 29. November 2016 20:04 > To: Lindenmaier, Goetz ; hotspot-compiler- > dev at openjdk.java.net; 'hotspot-runtime-dev at openjdk.java.net' runtime-dev at openjdk.java.net> > Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > > On 11/29/16 2:30 AM, Lindenmaier, Goetz wrote: > > Hi Dan, > > > > see my replies inline ... > > New webrev: > > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.05/ > > src/os/aix/vm/os_aix.cpp > L887: // libc guard page > nit - You made other existing comments into sentences (leading > capital and trailing '.'), but not this new comment. Why? > > src/os/aix/vm/os_aix.hpp > No comments. > > src/os/linux/vm/os_linux.cpp > L6096: // | |/ 1 page glibc guard. > nit - "1 page glibc guard" -> "1 glibc guard page." > > src/os/posix/vm/os_posix.cpp > No comments. > > src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp > No comments. > > src/os_cpu/bsd_x86/vm/os_bsd_x86.cpp > L875: // | |/ 1 page glibc guard. > nit - "1 page glibc guard" -> "1 glibc guard page." > > src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp > No comments. > > src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp > No comments. > > src/os_cpu/linux_s390/vm/os_linux_s390.cpp > No comments. > > src/os_cpu/linux_sparc/vm/os_linux_sparc.cpp > No comments. > > src/os_cpu/linux_x86/vm/os_linux_x86.cpp > No comments. > > src/os_cpu/linux_zero/vm/os_linux_zero.cpp > No comments. > > > Thumbs up! I don't need to see a new webrev if you choose > to fix the minor comments above. > > Some replies embedded below. > > > > > >> -----Original Message----- > >> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >> Sent: Dienstag, 29. November 2016 01:38 > >> To: Lindenmaier, Goetz ; hotspot-compiler- > >> dev at openjdk.java.net; 'hotspot-runtime-dev at openjdk.java.net' >> runtime-dev at openjdk.java.net> > >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > >> > >> On 11/28/16 2:08 AM, Lindenmaier, Goetz wrote: > >>> Hi, > >>> > >>> I'm working on a fix for OS guard pages on stacks. I figured there > >>> are VM guard pages (reserved, yellow, red) on the compiler stacks > >>> _and_ OS guard pages. For Java threads, the OS guard pages are left > >>> out. I think this should be done for compiler threads, too. Please > >>> confirm. > >>> Webrev: > >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.04/ > >> src/os/aix/vm/os_aix.cpp > >> L888: pthread_attr_setguardsize(&attr, > >> os::Aix::default_guard_size(thr_type)); > >> No check or assert on the return status of this call. > >> Is one needed? > > I implemented this as the existing code on linux which has > > no check either. I think a failure is quite theoretical. > > Because of your comment below I'll leave it as-is. > > OK. > > > > > >> L3044: // guard pages, so only enable libc guard pages for > >> non-Java threads. > >> src/os/linux/vm/os_linux.cpp also has this comment: > >> // (Remember: compiler thread is a Java thread, too!) > > Fixed. > > > >> L3051: return ((thr_type == java_thread || thr_type == > >> compiler_thread) ? 0 : 4*K); > >> nit - please add spaces around the '*' so '4 * K'.' > > Fixed. > > > >> src/os/aix/vm/os_aix.hpp > >> No comments. > >> > >> src/os/linux/vm/os_linux.cpp > >> L729: // is not implemented properly. The posix standard requires > >> to add > >> Typo: 'to add' -> 'adding' > > Fixed. > > > >> L738: pthread_attr_setguardsize(&attr, > >> os::Linux::default_guard_size(thr_type)); > >> No check or assert on the return status of this call. > >> Is one needed? > > See above. > > > >> L2851: // Creating guard page is very expensive. Java thread has > >> HotSpot > >> L2852: // guard page, only enable glibc guard page for non-Java > >> threads. > >> L2853: // (Remember: compiler thread is a java thread, too!) > >> Typo: "java thread" -> "Java thread" (consistency) > >> > >> This comment block should be common to all the platforms that > >> define default_guard_size(). Yes, I can see that AIX needs to > >> add another paragraph, but we should make the core comment > common > >> if possible. > > I made the first three lines look alike. > > > >> L6090: // Java/Compiler thread: > >> Thanks for making this common in os_linux.cpp. > >> > >> L6095: // | glibc guard page | - guard, attached Java > >> thread usually has > >> Clarity: "guard," -> "guard page," > > Fixed. > > > >> Typo: "Java thread" -> "JavaThread" (consistency) > > I changed both to Java thread as at the other locations. > > > >> L6099: // | HotSpot Guard Pages | - red and yellow pages > >> The fairly recently added reserved page should be mentioned > >> here also? > > Yes. Fixed. Also fixed it to say JavaThread::stack_reserved_zone_base(). > > Also fixed comment on bsd. > > Thanks for also fixing BSD. > > > >> L6120 // ** P1 (aka bottom) and size ( P2 = P1 - size) are the > >> address and stack size returned from > >> Typo: "( P2 = ..." -> "(P2 = ..." > > Fixed. > > > >> L6148: fatal("Can not locate current stack attributes!"); > >> Typo: "Can not" -> "Cannot" > > Fixed. > > > >> L6175: // stack size includes normal stack and HotSpot guard pages > >> Perhaps add to the comment: > >> "for the threads that have HotSpot guard pages." > > Fixed. I also checked my comments for "OS guard pages" and > > replaced it by "glibc guard pages" which is used in several places > > already, same for "VM guard page" --> "HotSpot guard page". I > > think this is also more consistent. > > I agree! > > > >> src/os/posix/vm/os_posix.cpp > >> L1097: pthread_attr_getstacksize(attr, &stack_size); > >> L1098: pthread_attr_getguardsize(attr, &guard_size); > >> Do these two calls need to have their result checked? > >> > >> Now I'm starting to wonder if all the uses of these > >> two APIs need to be checked? Separate bug? > > It would be more consistent with the specification of the methods, > > On the other side it's quite unlikely that these fail if attr != NULL. > > So should we file a new bug? Or does this fall into the realm of > other OS/libc code that we call and assume never fails? :-) > > > > > >> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp > >> L540: size_t os::Posix::_compiler_thread_min_stack_allowed = 512 * K; > >> L541: size_t os::Posix::_java_thread_min_stack_allowed = 512 * K; > >> So prior to the fix for 8140520, src/os/aix/vm/os_aix.cpp had > >> this single min_stack_allowed value: > >> > >> L3601: os::Aix::min_stack_allowed = > >> MAX2(os::Aix::min_stack_allowed, > >> L3602: JavaThread::stack_guard_zone_size() + > >> L3603: JavaThread::stack_shadow_zone_size() + > >> L3604: (4*BytesPerWord > >> COMPILER2_PRESENT(+2)) * 4 * K); > >> > >> and the fix for 8140520 changed that for *NIX platforms to > >> three mins in src/os/posix/vm/os_posix.cpp: > >> > >> L1108: _java_thread_min_stack_allowed = > >> MAX2(_java_thread_min_stack_allowed, > >> L1109: JavaThread::stack_guard_zone_size() + > >> L1110: JavaThread::stack_shadow_zone_size() + > >> L1111: (4 * > >> BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K); > >> > >> L1150: _compiler_thread_min_stack_allowed = > >> align_size_up(_compiler_thread_min_stack_allowed, vm_page_size()); > >> > >> L1161 _vm_internal_thread_min_stack_allowed = > >> align_size_up(_vm_internal_thread_min_stack_allowed, vm_page_size()); > >> > >> Which means that the compiler_thread no longer benefits from > >> the extra space for quard and shadow pages. The thinking in > >> 8140520 was that the compiler_thread and vm_internal_threads > >> don't need the quard and shadow pages since they don't run > >> Java code (ignoring JVMCI for now). > >> > >> So I can see bumping _compiler_thread_min_stack_allowed from > >> 128 -> 512 as one solution for getting that extra space back. > >> However, I don't understand why _java_thread_min_stack_allowed > >> has changed from 128 -> 512. > > Because it was never correct before. > > OK. That sounds like the new test that I included with 8140520 would > have failed with JavaThread stack sizes even before the product code > changes from 8140520 were made. > > Since the size calculation for JavaThread stack sizes wasn't changed > for any platform in 8140520, that tends to indicate that the more > limited JDK test (test/tools/launcher/TooSmallStackSize.java) should > also have failed before the fix for 8140520. > > Please clarify the need for the _java_thread_min_stack_allowed change > from 128 -> 512. Unless test/tools/launcher/TooSmallStackSize.java > is never run in your testing, I'm having troubling seeing why the > _java_thread_min_stack_allowed increase is needed. > > > >> I had previously made this comment: > >> > To put it another way, I'd like to see us add extra space to > >> > solve the 64K page issue directly instead of as a side effect > >> > of the red/yellow page addition. > >> And Goetz replied with: > >> > I don't understand. What do you mean by 'directly'? > >> > >> So prior to the fix for 8140520, src/os/solaris/vm/os_solaris.cpp > >> had a block like this: > >> > >> L4468: // For 64kbps there will be a 64kb page size, which makes > >> L4469: // the usable default stack size quite a bit less. > >> Increase the > >> L4470: // stack for 64kb (or any > than 8kb) pages, this > >> increases > >> L4471: // virtual memory fragmentation (since we're not > >> creating the > >> L4472 // stack on a power of 2 boundary. The real fix for this > >> L4473 // should be to fix the guard page mechanism. > >> L4474 > >> L4475 if (vm_page_size() > 8*K) { > >> L4476 threadStackSizeInBytes = (threadStackSizeInBytes != 0) > >> L4477 ? threadStackSizeInBytes + > >> L4478 JavaThread::stack_red_zone_size() + > >> L4479 JavaThread::stack_yellow_zone_size() > >> L4480 : 0; > >> L4481 ThreadStackSize = threadStackSizeInBytes/K; > >> L4482 } > >> > >> The above is an example of what I mean by solving the 64K > >> page issue directly. In the fix for 8140520, that code block > >> was moved to os::Posix::set_minimum_stack_sizes() in > >> src/os/posix/vm/os_posix.cpp and put in a "#ifdef SOLARIS... > >> #endif // SOLARIS" block. Coleen filed a bug to determine > >> whether that code can be deleted: > >> > >> JDK-8161093 Solaris for >8k pagesize adds extra guard pages > >> https://bugs.openjdk.java.net/browse/JDK-8161093 > >> > >> but perhaps this bug shows that the code is needed? > >> > >> > >> OK so this is probably the longest code review comment > >> I have ever written, but the summary is: > >> > >> - I understand bumping _compiler_thread_min_stack_allowed, > >> but should it be solved in a different way? > >> - I don't understand bumping _java_thread_min_stack_allowed > > I plan to do a follow up change to fix this. Let's leave this discussion > > to that review. Here I just want to fix the NPTL issue and the basic > > sizing that is needed to pass the new test on ppc/s390. > > Same question here about the simpler JDK version of the test. > > Does test/tools/launcher/TooSmallStackSize.java get run in > your test environments? > > > > > >> src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp > >> No comments. > >> > >> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp > >> L538: size_t os::Posix::_compiler_thread_min_stack_allowed = 384 * K; > >> L539: size_t os::Posix::_java_thread_min_stack_allowed = 384 * K; > >> > >> Same monster comment as src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp > >> and the same summary: > >> > >> - I understand bumping _compiler_thread_min_stack_allowed, > >> but should it be solved in a different way? > >> - I don't understand bumping _java_thread_min_stack_allowed > >> > >> src/os_cpu/linux_s390/vm/os_linux_s390.cpp > >> L478: size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; > >> L479: size_t os::Posix::_java_thread_min_stack_allowed = 236 * K; > >> Bumping _java_thread_min_stack_allowed but not bumping > >> _compiler_thread_min_stack_allowed. I'm confused here. > > The numbers are what I need to startup on the machines. 128 is just > > fine on the machines we have. (That's the problem of the > > current setup: you have to tune this compile time constant for the > > page size of the machine you are running on. But let's discuss this > > in the follow up change.) > > OK about discussing this with a follow-up change. I guess I see > the compile time initialization as a "minimum setting assuming the > smallest page size". If we discover (at runtime) that the page > size is bigger, then we adjust the minimum that we need... > > Again, defer to another bug. Do we have a bug ID yet? > > > >> src/os_cpu/linux_sparc/vm/os_linux_sparc.cpp > >> No comments. > >> > >> src/os_cpu/linux_x86/vm/os_linux_x86.cpp > >> No comments. > >> > >> src/os_cpu/linux_zero/vm/os_linux_zero.cpp > >> No comments. > >> > >> Sorry it took me so long to write this up... > > No matter, thanks for this thorough review! > > You are very welcome. Thanks for being willing to dive into such > a complicated area (thread stack sizes)... > > Dan > > > > > > > Best regards, > > Goetz. > > > >> Dan > >> > >>> The change affecting the compier threads is in os_linux.cpp, > >> default_guard_size(), > >>> where '|| thr_type == compiler_thread' has been added. The function > >>> was also moved from the os_cpu files, as it's identical on all cpus. > >>> > >>> Best regards, > >>> Goetz. > >>> > >>> > >>>> -----Original Message----- > >>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>> Sent: Montag, 28. November 2016 00:25 > >>>> To: Lindenmaier, Goetz ; > >>>> 'daniel.daugherty at oracle.com' ; > 'hotspot- > >>>> runtime-dev at openjdk.java.net' > >>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. > >>>> > >>>> Hi Goetz, > >>>> > >>>> On 24/11/2016 10:15 PM, Lindenmaier, Goetz wrote: > >>>>> Hi, > >>>>> > >>>>> I now edited the stuff I had proposed below: > >>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > stackFix/webrev.03/ > >>>>> This includes > >>>>> - the NPTL fix from webrev.02 > >>>> Okay in principle. As discussed this only impacts non-JavaThreads so the > >>>> change should be minimal. > >>>> > >>>>> - merging code on linux > >>>> Went a bit further than I had expected but if this truly isn't CPU > >>>> dependent code then great! > >>>> > >>>>> - not adding OS guard to compiler threads. > >>>> Okay in principle. IIUC we will now save the OS guard page for compiler > >>>> thread stacks. Is that the only impact? The hotspot-compiler-dev folk > >>>> may want to sign off on this part. > >>>> > >>>> > >>>> A few minor comments: > >>>> > >>>> src/os/linux/vm/os_linux.cpp > >>>> > >>>> 2854 return ((thr_type == java_thread || thr_type == > >>>> os::compiler_thread) ... > >>>> > >>>> os:: should be used for both types or none. > >>>> > >>>> 6153 pthread_attr_getguardsize(&attr, &guard_size); > >>>> > >>>> Can you at least verify a zero return code in an assert/assert_status > >>>> please. > >>>> > >>>> --- > >>>> > >>>> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp > >>>> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp > >>>> src/os_cpu/linux_s390/vm/os_linux_s390.cpp > >>>> > >>>> Are the changes to min_stack_allowed just fixing an existing bug? > >>>> > >>>> --- > >>>> > >>>> Thanks, > >>>> David > >>>> ----- > >>>> > >>>> > >>>>> I think this should be pushed for this bug ID. For the other changes I'll > >>>>> make another bug. > >>>>> > >>>>> Best regards, > >>>>> Goetz. > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Lindenmaier, Goetz > >>>>>> Sent: Wednesday, November 23, 2016 8:11 AM > >>>>>> To: David Holmes ; > >>>>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > >>>>>> Subject: RE: RFR(M): 8169373: Work around linux NPTL stack guard > error. > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. > The > >>>>>>> ability for a CompilerThread to execute Java code (can_call_java()) is > >>>>>>> now a dynamic property depending on whether the current compiler > is > >> the > >>>>>>> JVMCI compiler. > >>>>>> Ah, then I should also leave space for shadow pages in the minimal > stack > >>>> size > >>>>>> of comiler threads. > >>>>>> > >>>>>> Do we agree on the cleanup and on leaving out the OS guard page on > >>>>>> compiler threads? > >>>>>> Then I would edit a change comprising the NPTL workaround and these > >>>>>> additional changes, and split the other issue into a new bug? I think this > >>>>>> will easy the reviewing. > >>>>>> > >>>>>> Best regards, > >>>>>> Goetz. > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>>>>> Sent: Mittwoch, 23. November 2016 02:50 > >>>>>>> To: Lindenmaier, Goetz ; > >>>>>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net > >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > >> error. > >>>>>>> On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: > >>>>>>>> Hi Dan, > >>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] > >>>>>>>>> Sent: Dienstag, 22. November 2016 14:01 > >>>>>>>>> To: Lindenmaier, Goetz ; David > Holmes > >>>>>>>>> ; hotspot-runtime- > dev at openjdk.java.net > >>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard > >>>>>> error. > >>>>>>>>> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: > >>>>>>>>>> Hi Dan, > >>>>>>>>>> > >>>>>>>>>> I ran into a row of issues, some errors on the platforms. > >>>>>>>>>> > >>>>>>>>>> What I meant with that comment is that > >>>>>>>>>> os::Linux::min_stack_allowed = > MAX2(os::Linux::min_stack_allowed, > >>>>>>>>>> JavaThread::stack_guard_zone_size() + > >>>>>>>>>> JavaThread::stack_shadow_zone_size() + > >>>>>>>>>> (4*BytesPerWord > >> COMPILER2_PRESENT(+2)) * > >>>> 4 > >>>>>> * > >>>>>>> K); > >>>>>>>>>> was executed, and min_stack_allowed used for all stacks. Now, > >>>>>> compiler > >>>>>>>>> and > >>>>>>>>>> vm minimum stack sizes are not increased by these sizes. > >>>>>>>>> Now I see what you mean. Thanks for clearing this up. > >>>>>>>>> > >>>>>>>>> I should have remembered that part of the change because we > went > >>>>>> back > >>>>>>>>> and forth about removing the red/yellow zone pages from the other > >>>>>>>>> threads. In particular, we discussed the compiler thread because it > >>>>>>>>> is-a JavaThread. Our conclusion was that a compiler thread doesn't > >>>>>>>>> execute Java bytecode so we could remove the red/yellow pages... > >>>>>>>> Yes, it does not execute java byte code. > >>>>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. > The > >>>>>>> ability for a CompilerThread to execute Java code (can_call_java()) is > >>>>>>> now a dynamic property depending on whether the current compiler > is > >> the > >>>>>>> JVMCI compiler. > >>>>>>> > >>>>>>> David > >>>>>>> ----- > >>>>>>> > >>>>>>>> Therefore you can remove the shadow pages. There is no code that > >>>>>>>> will bang. > >>>>>>>> But red/yellow pages are protected right during thread startup. > >>>>>>>> Therefore you must have enough space for them. > >>>>>>>> On ppc, we try to protect three 64K pages out of the 128K compiler > >> stack. > >>>>>>>> That obviously fails. > >>>>>>>> > >>>>>>>> Therefore I propose: > >>>>>>>> size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // > Set > >>>>>>> platform dependent. > >>>>>>>> in os::Posix::set_minimum_stack_sizes(): > >>>>>>>> _java_thread_min_stack_allowed = > _java_thread_min_stack_allowed > >> + > >>>>>>>> JavaThread::stack_guard_zone_size() + > >>>>>>>> JavaThread::stack_shadow_zone_size(); > >>>>>>>> > >>>>>>>> (Similar for _compiler_thread_min_stack_allowed). > >>>>>>>> > >>>>>>>> The minimal stack size is made up of three components: > >>>>>>>> 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot > >>>>>>> implementation/platform/os. > >>>>>>>> 2. Sizes of C++ frames: depends on C++ compiler. > >>>>>>>> These are fixed at compile time. > >>>>>>>> 3. Sizes of red/yellow/reserved/shadow pages. Depends on the > >> system > >>>>>>> the > >>>>>>>> VM is used on. This is not fixed at compile time. (Our ppc > >> machines > >>>>>>> differ > >>>>>>>> in page size.) > >>>>>>>> Therefore 3. should not be included in a compile time constant. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> And that decision allowed us to be exposed to the 64K page issue > >>>>>>>>> because the "extra" space isn't there anymore. > >>>>>>>>> > >>>>>>>>>> At least the _compiler_thread_min_stack_allowed should be > >> increased > >>>>>>>>>> similarly by red/yellow zone size. The compiler thread is a Java > >>>>>>>>>> thread and must have space for these zones. > >>>>>>>>> I'm not sure that I completely agree (yet). To me, the red/yellow > >>>>>>>>> pages are there for Java thread stack overflow semantics. Yes, the > >>>>>>>>> compiler thread needs extra space when 64K pages are used, but I > >>>>>>>>> would prefer that we add that space via a different calculation. > >>>>>>>> Yes they are. But compiler threads happen tob e a subclass of > >>>>>>>> Java threads and use the same run() method that puts the pages > >>>>>>>> There. > >>>>>>>> I agree that they are not needed for Compiler threads, nor for > >>>>>>>> CodeCacheSweeperThreads. I don't really now about > >>>>>>>> JvmtiAgentThreads and ServiceThreads, but all of the get the guard > >>>>>> pages > >>>>>>>> because they are derived from JavaThread. > >>>>>>>> > >>>>>>>>> To put it another way, I'd like to see us add extra space to solve > >>>>>>>>> the 64K page issue directly instead of as a side effect of the > >>>>>>>>> red/yellow page addition. > >>>>>>>> I don't understand. What do you mean by 'directly'? > >>>>>>>> > >>>>>>>>>> Also, the change added a test that is failing now. > >>>>>>>>> And that's a "good thing" (TM), right? :-) > >>>>>>>> Yes, it showed a bug and thus raised the need to fix it! :) > >>>>>>>> > >>>>>>>> Best regards, > >>>>>>>> Goetz. > >>>>>>>> > >>>>>>>> > >>>>>>>>> Again, thanks for clarifying 8140520's role in this issue. > >>>>>>>>> > >>>>>>>>> Dan > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Best regards, > >>>>>>>>>> Goetz. > >>>>>>>>>> > >>>>>>>>>>> -----Original Message----- > >>>>>>>>>>> From: Daniel D. Daugherty > [mailto:daniel.daugherty at oracle.com] > >>>>>>>>>>> Sent: Montag, 21. November 2016 17:28 > >>>>>>>>>>> To: David Holmes ; Lindenmaier, > Goetz > >>>>>>>>>>> ; hotspot-runtime- > >>>>>>> dev at openjdk.java.net > >>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack > guard > >>>>>>> error. > >>>>>>>>>>> Sorry for the delayed responses to this thread. I've been on > >> vacation... > >>>>>>>>>>> One comment/query embedded below... > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On 11/10/16 8:40 PM, David Holmes wrote: > >>>>>>>>>>>> Hi Goetz, > >>>>>>>>>>>> > >>>>>>>>>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: > >>>>>>>>>>>>> Hi David, > >>>>>>>>>>>>> > >>>>>>>>>>>>> This issue is different to 6675312, see also my comment in the > >> bug. > >>>>>>>>>>>>> It appears running jtreg test > >>>>>>> runtime/Thread/TooSmallStackSize.java, > >>>>>>>>>>>>> with my patch below you can reproduce it on linuxx86_64. You > >> can > >>>>>>> not > >>>>>>>>>>>>> do that with 6675312. Also, I would assume there are systems > out > >>>>>>> there > >>>>>>>>>>>>> on x86 that uses 64-K pages, did you run the tests on these? I > >>>>>> would > >>>>>>>>>>>>> assume you get hard crashes with stack overflows in the first > C2 > >>>>>>>>>>>>> compilation if there is only 64K usable CompilerThreadStack. > >>>>>>>>>>>>> > >>>>>>>>>>>>> My fix does not affect Java threads, which are the largest > amount > >>>>>>>>>>>>> of threads used by the VM. It affects only the non-Java threads. > >>>>>>>>>>>>> It adds one page to these threads. The page does not require > >>>>>>> memory, > >>>>>>>>>>>>> as it's protected. The stack will only require more space if the > >>>>>> thread > >>>>>>>>>>>>> ran into a stack overflow before the fix as else the pages are > not > >>>>>>>>>>>>> mapped. > >>>>>>>>>>>>> This are stack overflows that cause hard crashes, at least on > ppc > >>>>>> the > >>>>>>> VM > >>>>>>>>>>>>> does not properly catch these stack overflows, so any setup > >>>>>> working > >>>>>>>>> now > >>>>>>>>>>>>> will not run into the additional space. Altogether there should > be > >>>>>> no > >>>>>>>>>>>>> effect on running systems besides requiring one more entry in > >> the > >>>>>>>>>>>>> page table per non-Java thread. > >>>>>>>>>>>>> > >>>>>>>>>>>>> The problem is caused by a rather recent change (8140520: > >>>>>> segfault > >>>>>>> on > >>>>>>>>>>>>> solaris-amd64 > >>>>>>>>>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed > after > >>>>>>>>>>>>> feature-close. As this was a rather recent change, it must be > >>>>>>>>>>>>> possible to > >>>>>>>>>>>>> fix this follow up issue. What else is this period in the project > >> good > >>>>>>>>>>>>> for if not fixing issues? > >>>>>>>>>>>> So I am seeing a number of factors here. > >>>>>>>>>>>> > >>>>>>>>>>>> First, 8140520, set: > >>>>>>>>>>>> > >>>>>>>>>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * > K; > >>>>>>>>>>> So I'm confused by the above comment: > >>>>>>>>>>> > >>>>>>>>>>> > The problem is caused by a rather recent change (8140520: > >>>>>> segfault > >>>>>>>>>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) > >>>>>>>>>>> > >>>>>>>>>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- > >>>>>>>>>>> > >>>>>>> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html > >>>>>>>>>>> shows this change: > >>>>>>>>>>> > >>>>>>>>>>> @@ -531,19 +531,17 @@ > >>>>>>>>>>> } > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>> > ////////////////////////////////////////////////////////////////////////////// > >>>>>>>>>>> // > >>>>>>>>>>> // thread stack > >>>>>>>>>>> > >>>>>>>>>>> -size_t os::Linux::min_stack_allowed = 128*K; > >>>>>>>>>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * > K; > >>>>>>>>>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; > >>>>>>>>>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 > * > >> K; > >>>>>>>>>>> so the existing single variable of 'min_stack_allowed' was > >>>>>>>>>>> replaced by three variables: > _compiler_thread_min_stack_allowed, > >>>>>>>>>>> _java_thread_min_stack_allowed, and > >>>>>>>>>>> _vm_internal_thread_min_stack_allowed. > >>>>>>>>>>> > >>>>>>>>>>> The old single variable and the three new variables are all > >>>>>>>>>>> initialized to the same value (128K) so the fix for 8140520 > >>>>>>>>>>> did not change stack sizes for this platform. In fact, only > >>>>>>>>>>> one platform had a size change (Solaris X64). > >>>>>>>>>>> > >>>>>>>>>>> So I'm confused about how the fix for 8140520 caused this > problem. > >>>>>>>>>>> > >>>>>>>>>>> Based on David's analysis below, it looks to me like this 64K stack > >>>>>>>>>>> guard page problem should also exist prior to the fix for > 8140520. > >>>>>>>>>>> > >>>>>>>>>>> Goetz, can you please explain how 8140520 caused this problem? > >>>>>>>>>>> > >>>>>>>>>>> Dan > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> Second on linux PPC it is hardwired to use 2 guard pages: > >>>>>>>>>>>> > >>>>>>>>>>>> return 2 * page_size(); > >>>>>>>>>>>> > >>>>>>>>>>>> Third, you had a pagesize of 64K. > >>>>>>>>>>>> > >>>>>>>>>>>> Fourth, NPTL takes the guard space from the stack space - > hence > >>>>>> with > >>>>>>> 2 > >>>>>>>>>>>> x 64K guard, and a 128K stack it was all consumed. > >>>>>>>>>>>> > >>>>>>>>>>>> --- > >>>>>>>>>>>> > >>>>>>>>>>>> In the proposed changes you now only use page_size() for the > >> guard, > >>>>>>> so > >>>>>>>>>>>> that alone would have fixed the observed problem. > >>>>>>>>>>>> > >>>>>>>>>>>> But in addition you want to address the NPTL problem by adding > >>>>>> back > >>>>>>>>>>>> the guard space to the stack size requested. That alone would > also > >>>>>>>>>>>> have fixed the observed problem. :) > >>>>>>>>>>>> > >>>>>>>>>>>> But in addition you have increased the minimum stack size: > >>>>>>>>>>>> > >>>>>>>>>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * > K; > >>>>>>>>>>>> > >>>>>>>>>>>> which again, on its own would have fixed the original problem. > :) > >>>>>>>>>>>> > >>>>>>>>>>>> Did you really intend to increase the real minimum stack from > >> 128K > >>>>>> to > >>>>>>>>>>>> 256K ? (on a 64K page system) > >>>>>>>>>>>> > >>>>>>>>>>>> --- > >>>>>>>>>>>> > >>>>>>>>>>>> Focusing simply on the shared code change to adjust the > requested > >>>>>>>>>>>> stacksize by the amount of guard space (if any), this does not > seem > >>>>>>>>>>>> unreasonable. As you note it is restricted to non-JavaThreads > and > >>>>>> only > >>>>>>>>>>>> adds a page to reserved stack space. > >>>>>>>>>>>> > >>>>>>>>>>>> My only query now is whether the minimum stacksize detection > >> logic > >>>>>>>>>>>> will correctly report the real minimum stack size (taking into > >> account > >>>>>>>>>>>> the need for the guard page) ? > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> David > >>>>>>>>>>>> > >>>>>>>>>>>>> So I really think this issue should be fixed. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>> Goetz. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] > >>>>>>>>>>>>>> Sent: Thursday, November 10, 2016 10:02 PM > >>>>>>>>>>>>>> To: Lindenmaier, Goetz ; > hotspot- > >>>>>>>>>>> runtime- > >>>>>>>>>>>>>> dev at openjdk.java.net > >>>>>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack > >>>>>> guard > >>>>>>>>> error. > >>>>>>>>>>>>>> Hi Goetz, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> As per the bug report, this issue was already known > (6675312) > >> and > >>>>>>> we > >>>>>>>>>>>>>> chose not to try and address it due to no reported issues at > the > >>>>>>> time. > >>>>>>>>>>>>>> While I see that you have encountered an issue (is it real or > >>>>>>>>>>>>>> fabricated?) I think this change is too intrusive to be applied > at > >> this > >>>>>>>>>>>>>> stage of the JDK 9 release cycle, as it will change the stack > >>>>>>>>>>>>>> requirements of every application running on Linux. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> David > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: > >>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Please review this change. I please need a sponsor: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- > >>>>>>>>>>> stackFix/webrev.01/ > >>>>>>>>>>>>>>> In the Linux NPTL pthread implementation the guard size > >>>>>>> mechanism > >>>>>>>>>>>>>>> is not > >>>>>>>>>>>>>>> implemented properly. The posix standard requires to add > the > >>>>>>> size > >>>>>>>>>>>>>>> of the > >>>>>>>>>>>>>>> guard pages to the stack size, instead Linux takes the space > out > >>>>>> of > >>>>>>>>>>>>>>> 'stacksize'. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The Posix standard > >>>>>>>>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ > >>>>>>>>>>>>>>> says "the implementation allocates extra memory at the > >>>>>>> overflow > >>>>>>>>>>>>>>> end of > >>>>>>>>>>>>>>> the stack". The linux man page > >>>>>>>>>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says > "As > >>>>>>> at > >>>>>>>>>>>>>>> glibc > >>>>>>>>>>>>>>> 2.8, the NPTL threading implementation includes the guard > >> area > >>>>>>>>> within > >>>>>>>>>>>>>>> the stack size allocation, rather than allocating extra space > at > >>>>>>>>>>>>>>> the end > >>>>>>>>>>>>>>> of the stack, as POSIX.1 requires". > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I encounter this problem in > >>>>>>> runtime/Thread/TooSmallStackSize.java > >>>>>>>>>>>>>>> on ppc > >>>>>>>>>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is > 128K > >>>>>> on > >>>>>>>>> ppc, > >>>>>>>>>>> and > >>>>>>>>>>>>>>> ppc specifies two OS guard pages. The VM crashes in > pthread > >>>>>>>>> creation > >>>>>>>>>>>>>>> because there is no usable space in the thread stack after > >>>>>>> allocating > >>>>>>>>>>>>>>> the guard pages. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> But TooSmallStackSize.java requires that the VM comes up > >> with > >>>>>>> the > >>>>>>>>>>>>>>> stack > >>>>>>>>>>>>>>> size mentioned in the error message. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This fix adapts the requested stack size on Linux by the size > of > >>>>>> the > >>>>>>>>>>>>>>> guard pages to mimick proper behaviour, see change to > >>>>>>>>> os_linux.cpp. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The change also streamlines usage of stack_guard_page on > >>>>>>> linuxppc, > >>>>>>>>>>>>>>> linuxppcle, aixppc and linuxs390. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> To reproduce the error on linux_x86_64, apply below patch > >> and > >>>>>>> call > >>>>>>>>> the > >>>>>>>>>>>>>>> VM with -XX:CompilerThreadStackSize=64. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'm still exploring why I had to choose such big compiler > stacks > >>>>>> on > >>>>>>>>>>>>>>> ppc > >>>>>>>>>>>>>>> to get -version passing, but I wanted to send the RFR now as > >>>>>>> people > >>>>>>>>>>>>>>> obviously looked at the bug I opened (Thanks David!). > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Goetz. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> diff -r b7ae012c55c3 > >> src/os_cpu/linux_x86/vm/os_linux_x86.cpp > >>>>>>>>>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov > 07 > >>>>>>>>> 12:37:28 > >>>>>>>>>>>>>> 2016 > >>>>>>>>>>>>>>> +0100 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov > 10 > >>>>>>>>>>> 16:52:17 > >>>>>>>>>>>>>> 2016 > >>>>>>>>>>>>>>> +0100 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> @@ -701,7 +701,7 @@ > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType > thr_type) { > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> // Creating guard page is very expensive. Java thread > has > >>>>>>> HotSpot > >>>>>>>>>>>>>>> // guard page, only enable glibc guard page for non-Java > >>>>>>> threads. > >>>>>>>>>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> // Java thread: > >>>>>>>>>>>>>>> From Alan.Bateman at oracle.com Wed Nov 30 11:13:47 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 30 Nov 2016 11:13:47 +0000 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> Message-ID: Thanks to everyone that have been reviewing these changes. I've pushed an updated set of webrevs here: http://cr.openjdk.java.net/~alanb/8169069/1/ The update includes changes to address most of the reviewer comments so far. There are a few more cleanups that still need to be pushed today but I think we are otherwise in reasonable shape at this point. As I said in the original mail, we will keep the jake forest open as there are still issues to work on after this integration. -Alan From mikael.gerdin at oracle.com Wed Nov 30 12:41:28 2016 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Wed, 30 Nov 2016 13:41:28 +0100 Subject: RFR(s): 8170520: Make Metaspace ChunkManager counters non-atomic In-Reply-To: References: Message-ID: <16d07a40-95a5-9ec0-11a8-def40094e786@oracle.com> Hi Thomas, On 2016-11-30 08:43, Thomas St?fe wrote: > Hi all, > > please take a look at this small fix. (This is one of the smaller fixes in > preparation for a prototype for JDK-8166690, I try to move the smaller > stuff out of the way first). > > Issue: https://bugs.openjdk.java.net/browse/JDK-8170520 > > webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8170520-Make- > Metaspace-ChunkManager-Counters-NonAtomic/webrev.00/webrev/ I threw some testing at your change and it seems like you've forgotten about calling inc_free_chunks_total for the humongous chunks in ~SpaceManager. I'm going to run more tests on a patched build where I just hack in a call to inc_free_chunks_total for reach humongous chunk. It would be great if you could have a look at unifying the ChunkManager::return_chunks API for standard and humongous chunks so that we wouldn't need the special casing for this in ~SpaceManager That way we could also make humongous_dictionary() private to ChunkManager which would improve encapsulation. Thanks /Mikael > > Small discussion here: > http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2016-October/021709.html > > Background: ChunkManager keeps counters (_free_chunks_count and > free_chunks_total). These counters are updated using Atomics. Because this > is expensive, code goes thru some lengths to accumulate updates, which in > turn makes the code more complicated and error prone as necessary. But > updating the counters atomically is not needed, because updates happen > under lock protection anyway. > > The proposed fix makes updates non-atomic and moved the call to > inc_free_chunks_total() into ChunkManager::return_chunks() - close to where > the chunk is actually returned to the freelist, so the time window where > the counters are invalid is very small now. > > It also fixes an assert and makes inc_free_chunks_total() private because > now it can be private. > > Note that I do not intend to push this into 9, so this is for the upcoming > 10 repository. I would like to get some reviews nevertheless, so this small > change can be pushed quickly once the 10 repo exists. > > Thanks! > > Best Regards, Thomas > From thomas.stuefe at gmail.com Wed Nov 30 13:01:07 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 30 Nov 2016 14:01:07 +0100 Subject: RFR(s): 8170520: Make Metaspace ChunkManager counters non-atomic In-Reply-To: <16d07a40-95a5-9ec0-11a8-def40094e786@oracle.com> References: <16d07a40-95a5-9ec0-11a8-def40094e786@oracle.com> Message-ID: Hi Mikael, On Wed, Nov 30, 2016 at 1:41 PM, Mikael Gerdin wrote: > Hi Thomas, > > On 2016-11-30 08:43, Thomas St?fe wrote: > >> Hi all, >> >> please take a look at this small fix. (This is one of the smaller fixes in >> preparation for a prototype for JDK-8166690, I try to move the smaller >> stuff out of the way first). >> >> Issue: https://bugs.openjdk.java.net/browse/JDK-8170520 >> >> webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8170520-Make- >> Metaspace-ChunkManager-Counters-NonAtomic/webrev.00/webrev/ >> > > I threw some testing at your change and it seems like you've forgotten > about calling inc_free_chunks_total for the humongous chunks in > ~SpaceManager. I'm going to run more tests on a patched build where I just > hack in a call to inc_free_chunks_total for reach humongous chunk. > > Ouch, I think you are a right, I forgot about that case :( What tests did you run? > It would be great if you could have a look at unifying the > ChunkManager::return_chunks API for standard and humongous chunks so that > we wouldn't need the special casing for this in ~SpaceManager > That way we could also make humongous_dictionary() private to ChunkManager > which would improve encapsulation. > > Yes, this makes sense, I will do that. Thank you for testing! ..Thomas > Thanks > /Mikael > > > >> Small discussion here: >> http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2 >> 016-October/021709.html >> >> Background: ChunkManager keeps counters (_free_chunks_count and >> free_chunks_total). These counters are updated using Atomics. Because this >> is expensive, code goes thru some lengths to accumulate updates, which in >> turn makes the code more complicated and error prone as necessary. But >> updating the counters atomically is not needed, because updates happen >> under lock protection anyway. >> >> The proposed fix makes updates non-atomic and moved the call to >> inc_free_chunks_total() into ChunkManager::return_chunks() - close to >> where >> the chunk is actually returned to the freelist, so the time window where >> the counters are invalid is very small now. >> >> It also fixes an assert and makes inc_free_chunks_total() private because >> now it can be private. >> >> Note that I do not intend to push this into 9, so this is for the upcoming >> 10 repository. I would like to get some reviews nevertheless, so this >> small >> change can be pushed quickly once the 10 repo exists. >> >> Thanks! >> >> Best Regards, Thomas >> >> From mikael.gerdin at oracle.com Wed Nov 30 13:22:18 2016 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Wed, 30 Nov 2016 14:22:18 +0100 Subject: RFR(s): 8170520: Make Metaspace ChunkManager counters non-atomic In-Reply-To: References: <16d07a40-95a5-9ec0-11a8-def40094e786@oracle.com> Message-ID: Hi Thomas, On 2016-11-30 14:01, Thomas St?fe wrote: > Hi Mikael, > > On Wed, Nov 30, 2016 at 1:41 PM, Mikael Gerdin > wrote: > > Hi Thomas, > > On 2016-11-30 08:43, Thomas St?fe wrote: > > Hi all, > > please take a look at this small fix. (This is one of the > smaller fixes in > preparation for a prototype for JDK-8166690, I try to move the > smaller > stuff out of the way first). > > Issue: https://bugs.openjdk.java.net/browse/JDK-8170520 > > > webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8170520-Make- > > Metaspace-ChunkManager-Counters-NonAtomic/webrev.00/webrev/ > > > I threw some testing at your change and it seems like you've > forgotten about calling inc_free_chunks_total for the humongous > chunks in ~SpaceManager. I'm going to run more tests on a patched > build where I just hack in a call to inc_free_chunks_total for reach > humongous chunk. > > > Ouch, I think you are a right, I forgot about that case :( What tests > did you run? I ran an internal test suite, unfortunately. It basically runs the JCK, loading each test case in a new class loader, thereby triggering insane amounts of class unloading and metaspaces to be created. > > > It would be great if you could have a look at unifying the > ChunkManager::return_chunks API for standard and humongous chunks so > that we wouldn't need the special casing for this in ~SpaceManager > That way we could also make humongous_dictionary() private to > ChunkManager which would improve encapsulation. > > > Yes, this makes sense, I will do that. Thank you for testing! The test passed with my hack to fix the humongous chunks issue but I'll continue running further tests and report back if I see any further failures. /Mikael > > ..Thomas > > > Thanks > /Mikael > > > > Small discussion here: > http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2016-October/021709.html > > > Background: ChunkManager keeps counters (_free_chunks_count and > free_chunks_total). These counters are updated using Atomics. > Because this > is expensive, code goes thru some lengths to accumulate updates, > which in > turn makes the code more complicated and error prone as > necessary. But > updating the counters atomically is not needed, because updates > happen > under lock protection anyway. > > The proposed fix makes updates non-atomic and moved the call to > inc_free_chunks_total() into ChunkManager::return_chunks() - > close to where > the chunk is actually returned to the freelist, so the time > window where > the counters are invalid is very small now. > > It also fixes an assert and makes inc_free_chunks_total() > private because > now it can be private. > > Note that I do not intend to push this into 9, so this is for > the upcoming > 10 repository. I would like to get some reviews nevertheless, so > this small > change can be pushed quickly once the 10 repo exists. > > Thanks! > > Best Regards, Thomas > > From thomas.stuefe at gmail.com Wed Nov 30 13:30:02 2016 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 30 Nov 2016 14:30:02 +0100 Subject: RFR(s): 8170520: Make Metaspace ChunkManager counters non-atomic In-Reply-To: References: <16d07a40-95a5-9ec0-11a8-def40094e786@oracle.com> Message-ID: On Wed, Nov 30, 2016 at 2:22 PM, Mikael Gerdin wrote: > Hi Thomas, > > On 2016-11-30 14:01, Thomas St?fe wrote: > >> Hi Mikael, >> >> On Wed, Nov 30, 2016 at 1:41 PM, Mikael Gerdin > > wrote: >> >> Hi Thomas, >> >> On 2016-11-30 08:43, Thomas St?fe wrote: >> >> Hi all, >> >> please take a look at this small fix. (This is one of the >> smaller fixes in >> preparation for a prototype for JDK-8166690, I try to move the >> smaller >> stuff out of the way first). >> >> Issue: https://bugs.openjdk.java.net/browse/JDK-8170520 >> >> >> webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8170520-Make- >> >> Metaspace-ChunkManager-Counters-NonAtomic/webrev.00/webrev/ >> >> >> I threw some testing at your change and it seems like you've >> forgotten about calling inc_free_chunks_total for the humongous >> chunks in ~SpaceManager. I'm going to run more tests on a patched >> build where I just hack in a call to inc_free_chunks_total for reach >> humongous chunk. >> >> >> Ouch, I think you are a right, I forgot about that case :( What tests >> did you run? >> > > I ran an internal test suite, unfortunately. > It basically runs the JCK, loading each test case in a new class loader, > thereby triggering insane amounts of class unloading and metaspaces to be > created. > > Ah, that explains it. I tried jtreg runtime/Metaspace and -XX:+ExecuteInternalVMTests, both ran fine. But we also run nightly jck tests, so I'd like to check why I did not see an error. What was the exact error you had, did the assert in ChunkManager::locked_verify_free_chunks_total() trigger? About the cleanup for ChunkManager::return_chunks(), should I do this as part of this patch or as an extra patch? > >> >> It would be great if you could have a look at unifying the >> ChunkManager::return_chunks API for standard and humongous chunks so >> that we wouldn't need the special casing for this in ~SpaceManager >> That way we could also make humongous_dictionary() private to >> ChunkManager which would improve encapsulation. >> >> >> Yes, this makes sense, I will do that. Thank you for testing! >> > > The test passed with my hack to fix the humongous chunks issue but I'll > continue running further tests and report back if I see any further > failures. Thanks! ..Thomas > > > /Mikael > > > >> ..Thomas >> >> >> Thanks >> /Mikael >> >> >> >> Small discussion here: >> http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2 >> 016-October/021709.html >> > 2016-October/021709.html> >> >> Background: ChunkManager keeps counters (_free_chunks_count and >> free_chunks_total). These counters are updated using Atomics. >> Because this >> is expensive, code goes thru some lengths to accumulate updates, >> which in >> turn makes the code more complicated and error prone as >> necessary. But >> updating the counters atomically is not needed, because updates >> happen >> under lock protection anyway. >> >> The proposed fix makes updates non-atomic and moved the call to >> inc_free_chunks_total() into ChunkManager::return_chunks() - >> close to where >> the chunk is actually returned to the freelist, so the time >> window where >> the counters are invalid is very small now. >> >> It also fixes an assert and makes inc_free_chunks_total() >> private because >> now it can be private. >> >> Note that I do not intend to push this into 9, so this is for >> the upcoming >> 10 repository. I would like to get some reviews nevertheless, so >> this small >> change can be pushed quickly once the 10 repo exists. >> >> Thanks! >> >> Best Regards, Thomas >> >> >> From mikael.gerdin at oracle.com Wed Nov 30 13:36:40 2016 From: mikael.gerdin at oracle.com (Mikael Gerdin) Date: Wed, 30 Nov 2016 14:36:40 +0100 Subject: RFR(s): 8170520: Make Metaspace ChunkManager counters non-atomic In-Reply-To: References: <16d07a40-95a5-9ec0-11a8-def40094e786@oracle.com> Message-ID: Hi Thomas On 2016-11-30 14:30, Thomas St?fe wrote: > > On Wed, Nov 30, 2016 at 2:22 PM, Mikael Gerdin > > wrote: > > Hi Thomas, > > On 2016-11-30 14:01, Thomas St?fe wrote: > > Hi Mikael, > > On Wed, Nov 30, 2016 at 1:41 PM, Mikael Gerdin > > >> wrote: > > Hi Thomas, > > On 2016-11-30 08:43, Thomas St?fe wrote: > > Hi all, > > please take a look at this small fix. (This is one of the > smaller fixes in > preparation for a prototype for JDK-8166690, I try to > move the > smaller > stuff out of the way first). > > Issue: > https://bugs.openjdk.java.net/browse/JDK-8170520 > > > > > webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/8170520-Make- > > > > > > Metaspace-ChunkManager-Counters-NonAtomic/webrev.00/webrev/ > > > I threw some testing at your change and it seems like you've > forgotten about calling inc_free_chunks_total for the > humongous > chunks in ~SpaceManager. I'm going to run more tests on a > patched > build where I just hack in a call to inc_free_chunks_total > for reach > humongous chunk. > > > Ouch, I think you are a right, I forgot about that case :( > What tests > did you run? > > > I ran an internal test suite, unfortunately. > It basically runs the JCK, loading each test case in a new class > loader, thereby triggering insane amounts of class unloading and > metaspaces to be created. > > > Ah, that explains it. I tried jtreg runtime/Metaspace and > -XX:+ExecuteInternalVMTests, both ran fine. But we also run nightly > jck tests, so I'd like to check why I did not see an error. What was > the exact error you had, did the assert > in ChunkManager::locked_verify_free_chunks_total() trigger? I hit the assert in dec_free_chunks_total # Internal Error (/opt/jprt/T/P1/094027.mgerdin/s/hotspot/src/share/vm/memory/metaspace.cpp:117), pid=7532, tid=7560 # assert(_free_chunks_count > 0 && _free_chunks_total >= v) failed: About to go negative Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x1448fe7] VMError::report_and_die(int, char const*, char const*, char*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned int)+0x137 V [libjvm.so+0x1449ce0] VMError::report_and_die(Thread*, char const*, int, char const*, char const*, char*)+0x30 V [libjvm.so+0x9266a0] report_vm_error(char const*, int, char const*, char const*, ...)+0x60 V [libjvm.so+0x10357ac] ChunkManager::free_chunks_get(unsigned int)+0xcc V [libjvm.so+0x1035953] ChunkManager::chunk_freelist_allocate(unsigned int)+0x23 V [libjvm.so+0x10376cd] SpaceManager::get_new_chunk(unsigned int)+0x2d V [libjvm.so+0x103910b] SpaceManager::grow_and_allocate(unsigned int)+0xeb V [libjvm.so+0x1039439] SpaceManager::allocate_work(unsigned int)+0x1d9 V [libjvm.so+0x10395fc] SpaceManager::allocate(unsigned int)+0x11c V [libjvm.so+0x1039910] Metaspace::allocate(ClassLoaderData*, unsigned int, bool, MetaspaceObj::Type, Thread*)+0x100 V [libjvm.so+0x1063a51] MethodData::allocate(ClassLoaderData*, methodHandle const&, Thread*)+0x41 V [libjvm.so+0x104edf2] Method::build_interpreter_method_data(methodHandle const&, Thread*)+0x82 V [libjvm.so+0x7728d4] ciMethod::ensure_method_data(methodHandle)+0x2b4 V [libjvm.so+0x772a33] ciMethod::ensure_method_data()+0xe3 V [libjvm.so+0x5d958c] Compilation::compile_java_method()+0x73c V [libjvm.so+0x5da10d] Compilation::compile_method()+0x23d V [libjvm.so+0x5dac84] Compilation::Compilation(AbstractCompiler*, ciEnv*, ciMethod*, int, BufferBlob*, DirectiveSet*)+0x3d4 V [libjvm.so+0x5dccfd] Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0x13d V [libjvm.so+0x8b1f93] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x5c3 V [libjvm.so+0x8b309f] CompileBroker::compiler_thread_loop()+0x43f V [libjvm.so+0x139e168] compiler_thread_entry(JavaThread*, Thread*)+0x28 V [libjvm.so+0x13b0cf0] JavaThread::thread_main_inner()+0x220 V [libjvm.so+0x13b0f8a] JavaThread::run()+0x1ca V [libjvm.so+0x1121694] thread_native_entry(Thread*)+0x124 C [libpthread.so.0+0x6b5c] start_thread+0xcc C [libc.so.6+0xf7a5e] clone+0x5e _free_chunks_total was about to go negative but _free_chunks_count was 63. > > About the cleanup for ChunkManager::return_chunks(), should I do this > as part of this patch or as an extra patch? I think it would be better to do that as a separate patch. Thanks /Mikael > > > It would be great if you could have a look at unifying the > ChunkManager::return_chunks API for standard and humongous > chunks so > that we wouldn't need the special casing for this in > ~SpaceManager > That way we could also make humongous_dictionary() private to > ChunkManager which would improve encapsulation. > > > Yes, this makes sense, I will do that. Thank you for testing! > > > The test passed with my hack to fix the humongous chunks issue but > I'll continue running further tests and report back if I see any > further failures. > > > Thanks! > ..Thomas > > > > /Mikael > > > > ..Thomas > > > Thanks > /Mikael > > > > Small discussion here: > http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2016-October/021709.html > > > > > > Background: ChunkManager keeps counters > (_free_chunks_count and > free_chunks_total). These counters are updated using > Atomics. > Because this > is expensive, code goes thru some lengths to > accumulate updates, > which in > turn makes the code more complicated and error prone as > necessary. But > updating the counters atomically is not needed, > because updates > happen > under lock protection anyway. > > The proposed fix makes updates non-atomic and moved > the call to > inc_free_chunks_total() into > ChunkManager::return_chunks() - > close to where > the chunk is actually returned to the freelist, so the > time > window where > the counters are invalid is very small now. > > It also fixes an assert and makes inc_free_chunks_total() > private because > now it can be private. > > Note that I do not intend to push this into 9, so this > is for > the upcoming > 10 repository. I would like to get some reviews > nevertheless, so > this small > change can be pushed quickly once the 10 repo exists. > > Thanks! > > Best Regards, Thomas > > > From daniel.daugherty at oracle.com Wed Nov 30 15:20:14 2016 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 30 Nov 2016 08:20:14 -0700 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: <821371d77c7c4e6d966901b461b3d6a1@DEROTE13DE08.global.corp.sap> References: <2b3d8f5c-95fb-920d-e5c7-1bb1d927979e@oracle.com> <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> <4dfa4ae6-4fa6-994c-f2cf-8fc309d8c520@oracle.com> <821371d77c7c4e6d966901b461b3d6a1@DEROTE13DE08.global.corp.sap> Message-ID: > Would you mind sponsoring this change? Yes, I will sponsor this change. However, I would like to get a thumbs up from David H. on the latest version and I don't see any review from someone on the Compiler Team. Vladimir! We need someone on the Compiler Team to look at these CompilerThread stack size changes... Dan On 11/30/16 12:57 AM, Lindenmaier, Goetz wrote: > Hi Dan, > >> Thumbs up! I don't need to see a new webrev if you choose >> to fix the minor comments above. > I anyways did a new one fixing the comments: > http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.06/ > Would you mind sponsoring this change? > > I took the minimum stack sizes from my experimental runs where > I had removed the automatic resizing to find the really needed space. > If I put something smaller there, I could as well put '0' ... as > _java_thread_min_stack_allowed = MAX2(_java_thread_min_stack_allowed, > JavaThread::stack_guard_zone_size() + > JavaThread::stack_shadow_zone_size() + > (4 * BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K); > will fix it. > This code effectively limits the usable stack size to > (4 * BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K) > which makes the initialization of _java_thread_min_stack_allowed with platform > values pointless. > > I'll open a new bug for the follow-up stack issue, probably tomorrow. > I don't feel like addressing testing all the possible error codes, as > they probably should be fixed in more places, and there is no issue > pending currently. Maybe it should be fixed in jdk10 at some point. > > Best regards, > Goetz > > >> -----Original Message----- >> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >> Sent: Dienstag, 29. November 2016 20:04 >> To: Lindenmaier, Goetz ; hotspot-compiler- >> dev at openjdk.java.net; 'hotspot-runtime-dev at openjdk.java.net' > runtime-dev at openjdk.java.net> >> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >> >> On 11/29/16 2:30 AM, Lindenmaier, Goetz wrote: >>> Hi Dan, >>> >>> see my replies inline ... >>> New webrev: >>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.05/ >> src/os/aix/vm/os_aix.cpp >> L887: // libc guard page >> nit - You made other existing comments into sentences (leading >> capital and trailing '.'), but not this new comment. Why? >> >> src/os/aix/vm/os_aix.hpp >> No comments. >> >> src/os/linux/vm/os_linux.cpp >> L6096: // | |/ 1 page glibc guard. >> nit - "1 page glibc guard" -> "1 glibc guard page." >> >> src/os/posix/vm/os_posix.cpp >> No comments. >> >> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >> No comments. >> >> src/os_cpu/bsd_x86/vm/os_bsd_x86.cpp >> L875: // | |/ 1 page glibc guard. >> nit - "1 page glibc guard" -> "1 glibc guard page." >> >> src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp >> No comments. >> >> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp >> No comments. >> >> src/os_cpu/linux_s390/vm/os_linux_s390.cpp >> No comments. >> >> src/os_cpu/linux_sparc/vm/os_linux_sparc.cpp >> No comments. >> >> src/os_cpu/linux_x86/vm/os_linux_x86.cpp >> No comments. >> >> src/os_cpu/linux_zero/vm/os_linux_zero.cpp >> No comments. >> >> >> Thumbs up! I don't need to see a new webrev if you choose >> to fix the minor comments above. >> >> Some replies embedded below. >> >> >>>> -----Original Message----- >>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>> Sent: Dienstag, 29. November 2016 01:38 >>>> To: Lindenmaier, Goetz ; hotspot-compiler- >>>> dev at openjdk.java.net; 'hotspot-runtime-dev at openjdk.java.net' >>> runtime-dev at openjdk.java.net> >>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>>> >>>> On 11/28/16 2:08 AM, Lindenmaier, Goetz wrote: >>>>> Hi, >>>>> >>>>> I'm working on a fix for OS guard pages on stacks. I figured there >>>>> are VM guard pages (reserved, yellow, red) on the compiler stacks >>>>> _and_ OS guard pages. For Java threads, the OS guard pages are left >>>>> out. I think this should be done for compiler threads, too. Please >>>>> confirm. >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.04/ >>>> src/os/aix/vm/os_aix.cpp >>>> L888: pthread_attr_setguardsize(&attr, >>>> os::Aix::default_guard_size(thr_type)); >>>> No check or assert on the return status of this call. >>>> Is one needed? >>> I implemented this as the existing code on linux which has >>> no check either. I think a failure is quite theoretical. >>> Because of your comment below I'll leave it as-is. >> OK. >> >> >>>> L3044: // guard pages, so only enable libc guard pages for >>>> non-Java threads. >>>> src/os/linux/vm/os_linux.cpp also has this comment: >>>> // (Remember: compiler thread is a Java thread, too!) >>> Fixed. >>> >>>> L3051: return ((thr_type == java_thread || thr_type == >>>> compiler_thread) ? 0 : 4*K); >>>> nit - please add spaces around the '*' so '4 * K'.' >>> Fixed. >>> >>>> src/os/aix/vm/os_aix.hpp >>>> No comments. >>>> >>>> src/os/linux/vm/os_linux.cpp >>>> L729: // is not implemented properly. The posix standard requires >>>> to add >>>> Typo: 'to add' -> 'adding' >>> Fixed. >>> >>>> L738: pthread_attr_setguardsize(&attr, >>>> os::Linux::default_guard_size(thr_type)); >>>> No check or assert on the return status of this call. >>>> Is one needed? >>> See above. >>> >>>> L2851: // Creating guard page is very expensive. Java thread has >>>> HotSpot >>>> L2852: // guard page, only enable glibc guard page for non-Java >>>> threads. >>>> L2853: // (Remember: compiler thread is a java thread, too!) >>>> Typo: "java thread" -> "Java thread" (consistency) >>>> >>>> This comment block should be common to all the platforms that >>>> define default_guard_size(). Yes, I can see that AIX needs to >>>> add another paragraph, but we should make the core comment >> common >>>> if possible. >>> I made the first three lines look alike. >>> >>>> L6090: // Java/Compiler thread: >>>> Thanks for making this common in os_linux.cpp. >>>> >>>> L6095: // | glibc guard page | - guard, attached Java >>>> thread usually has >>>> Clarity: "guard," -> "guard page," >>> Fixed. >>> >>>> Typo: "Java thread" -> "JavaThread" (consistency) >>> I changed both to Java thread as at the other locations. >>> >>>> L6099: // | HotSpot Guard Pages | - red and yellow pages >>>> The fairly recently added reserved page should be mentioned >>>> here also? >>> Yes. Fixed. Also fixed it to say JavaThread::stack_reserved_zone_base(). >>> Also fixed comment on bsd. >> Thanks for also fixing BSD. >> >> >>>> L6120 // ** P1 (aka bottom) and size ( P2 = P1 - size) are the >>>> address and stack size returned from >>>> Typo: "( P2 = ..." -> "(P2 = ..." >>> Fixed. >>> >>>> L6148: fatal("Can not locate current stack attributes!"); >>>> Typo: "Can not" -> "Cannot" >>> Fixed. >>> >>>> L6175: // stack size includes normal stack and HotSpot guard pages >>>> Perhaps add to the comment: >>>> "for the threads that have HotSpot guard pages." >>> Fixed. I also checked my comments for "OS guard pages" and >>> replaced it by "glibc guard pages" which is used in several places >>> already, same for "VM guard page" --> "HotSpot guard page". I >>> think this is also more consistent. >> I agree! >> >> >>>> src/os/posix/vm/os_posix.cpp >>>> L1097: pthread_attr_getstacksize(attr, &stack_size); >>>> L1098: pthread_attr_getguardsize(attr, &guard_size); >>>> Do these two calls need to have their result checked? >>>> >>>> Now I'm starting to wonder if all the uses of these >>>> two APIs need to be checked? Separate bug? >>> It would be more consistent with the specification of the methods, >>> On the other side it's quite unlikely that these fail if attr != NULL. >> So should we file a new bug? Or does this fall into the realm of >> other OS/libc code that we call and assume never fails? :-) >> >> >>>> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >>>> L540: size_t os::Posix::_compiler_thread_min_stack_allowed = 512 * K; >>>> L541: size_t os::Posix::_java_thread_min_stack_allowed = 512 * K; >>>> So prior to the fix for 8140520, src/os/aix/vm/os_aix.cpp had >>>> this single min_stack_allowed value: >>>> >>>> L3601: os::Aix::min_stack_allowed = >>>> MAX2(os::Aix::min_stack_allowed, >>>> L3602: JavaThread::stack_guard_zone_size() + >>>> L3603: JavaThread::stack_shadow_zone_size() + >>>> L3604: (4*BytesPerWord >>>> COMPILER2_PRESENT(+2)) * 4 * K); >>>> >>>> and the fix for 8140520 changed that for *NIX platforms to >>>> three mins in src/os/posix/vm/os_posix.cpp: >>>> >>>> L1108: _java_thread_min_stack_allowed = >>>> MAX2(_java_thread_min_stack_allowed, >>>> L1109: JavaThread::stack_guard_zone_size() + >>>> L1110: JavaThread::stack_shadow_zone_size() + >>>> L1111: (4 * >>>> BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K); >>>> >>>> L1150: _compiler_thread_min_stack_allowed = >>>> align_size_up(_compiler_thread_min_stack_allowed, vm_page_size()); >>>> >>>> L1161 _vm_internal_thread_min_stack_allowed = >>>> align_size_up(_vm_internal_thread_min_stack_allowed, vm_page_size()); >>>> >>>> Which means that the compiler_thread no longer benefits from >>>> the extra space for quard and shadow pages. The thinking in >>>> 8140520 was that the compiler_thread and vm_internal_threads >>>> don't need the quard and shadow pages since they don't run >>>> Java code (ignoring JVMCI for now). >>>> >>>> So I can see bumping _compiler_thread_min_stack_allowed from >>>> 128 -> 512 as one solution for getting that extra space back. >>>> However, I don't understand why _java_thread_min_stack_allowed >>>> has changed from 128 -> 512. >>> Because it was never correct before. >> OK. That sounds like the new test that I included with 8140520 would >> have failed with JavaThread stack sizes even before the product code >> changes from 8140520 were made. >> >> Since the size calculation for JavaThread stack sizes wasn't changed >> for any platform in 8140520, that tends to indicate that the more >> limited JDK test (test/tools/launcher/TooSmallStackSize.java) should >> also have failed before the fix for 8140520. >> >> Please clarify the need for the _java_thread_min_stack_allowed change >> from 128 -> 512. Unless test/tools/launcher/TooSmallStackSize.java >> is never run in your testing, I'm having troubling seeing why the >> _java_thread_min_stack_allowed increase is needed. >> >> >>>> I had previously made this comment: >>>> > To put it another way, I'd like to see us add extra space to >>>> > solve the 64K page issue directly instead of as a side effect >>>> > of the red/yellow page addition. >>>> And Goetz replied with: >>>> > I don't understand. What do you mean by 'directly'? >>>> >>>> So prior to the fix for 8140520, src/os/solaris/vm/os_solaris.cpp >>>> had a block like this: >>>> >>>> L4468: // For 64kbps there will be a 64kb page size, which makes >>>> L4469: // the usable default stack size quite a bit less. >>>> Increase the >>>> L4470: // stack for 64kb (or any > than 8kb) pages, this >>>> increases >>>> L4471: // virtual memory fragmentation (since we're not >>>> creating the >>>> L4472 // stack on a power of 2 boundary. The real fix for this >>>> L4473 // should be to fix the guard page mechanism. >>>> L4474 >>>> L4475 if (vm_page_size() > 8*K) { >>>> L4476 threadStackSizeInBytes = (threadStackSizeInBytes != 0) >>>> L4477 ? threadStackSizeInBytes + >>>> L4478 JavaThread::stack_red_zone_size() + >>>> L4479 JavaThread::stack_yellow_zone_size() >>>> L4480 : 0; >>>> L4481 ThreadStackSize = threadStackSizeInBytes/K; >>>> L4482 } >>>> >>>> The above is an example of what I mean by solving the 64K >>>> page issue directly. In the fix for 8140520, that code block >>>> was moved to os::Posix::set_minimum_stack_sizes() in >>>> src/os/posix/vm/os_posix.cpp and put in a "#ifdef SOLARIS... >>>> #endif // SOLARIS" block. Coleen filed a bug to determine >>>> whether that code can be deleted: >>>> >>>> JDK-8161093 Solaris for >8k pagesize adds extra guard pages >>>> https://bugs.openjdk.java.net/browse/JDK-8161093 >>>> >>>> but perhaps this bug shows that the code is needed? >>>> >>>> >>>> OK so this is probably the longest code review comment >>>> I have ever written, but the summary is: >>>> >>>> - I understand bumping _compiler_thread_min_stack_allowed, >>>> but should it be solved in a different way? >>>> - I don't understand bumping _java_thread_min_stack_allowed >>> I plan to do a follow up change to fix this. Let's leave this discussion >>> to that review. Here I just want to fix the NPTL issue and the basic >>> sizing that is needed to pass the new test on ppc/s390. >> Same question here about the simpler JDK version of the test. >> >> Does test/tools/launcher/TooSmallStackSize.java get run in >> your test environments? >> >> >>>> src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp >>>> No comments. >>>> >>>> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp >>>> L538: size_t os::Posix::_compiler_thread_min_stack_allowed = 384 * K; >>>> L539: size_t os::Posix::_java_thread_min_stack_allowed = 384 * K; >>>> >>>> Same monster comment as src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >>>> and the same summary: >>>> >>>> - I understand bumping _compiler_thread_min_stack_allowed, >>>> but should it be solved in a different way? >>>> - I don't understand bumping _java_thread_min_stack_allowed >>>> >>>> src/os_cpu/linux_s390/vm/os_linux_s390.cpp >>>> L478: size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * K; >>>> L479: size_t os::Posix::_java_thread_min_stack_allowed = 236 * K; >>>> Bumping _java_thread_min_stack_allowed but not bumping >>>> _compiler_thread_min_stack_allowed. I'm confused here. >>> The numbers are what I need to startup on the machines. 128 is just >>> fine on the machines we have. (That's the problem of the >>> current setup: you have to tune this compile time constant for the >>> page size of the machine you are running on. But let's discuss this >>> in the follow up change.) >> OK about discussing this with a follow-up change. I guess I see >> the compile time initialization as a "minimum setting assuming the >> smallest page size". If we discover (at runtime) that the page >> size is bigger, then we adjust the minimum that we need... >> >> Again, defer to another bug. Do we have a bug ID yet? >> >> >>>> src/os_cpu/linux_sparc/vm/os_linux_sparc.cpp >>>> No comments. >>>> >>>> src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>>> No comments. >>>> >>>> src/os_cpu/linux_zero/vm/os_linux_zero.cpp >>>> No comments. >>>> >>>> Sorry it took me so long to write this up... >>> No matter, thanks for this thorough review! >> You are very welcome. Thanks for being willing to dive into such >> a complicated area (thread stack sizes)... >> >> Dan >> >> >> >>> Best regards, >>> Goetz. >>> >>>> Dan >>>> >>>>> The change affecting the compier threads is in os_linux.cpp, >>>> default_guard_size(), >>>>> where '|| thr_type == compiler_thread' has been added. The function >>>>> was also moved from the os_cpu files, as it's identical on all cpus. >>>>> >>>>> Best regards, >>>>> Goetz. >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>> Sent: Montag, 28. November 2016 00:25 >>>>>> To: Lindenmaier, Goetz ; >>>>>> 'daniel.daugherty at oracle.com' ; >> 'hotspot- >>>>>> runtime-dev at openjdk.java.net' >>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>>>>> >>>>>> Hi Goetz, >>>>>> >>>>>> On 24/11/2016 10:15 PM, Lindenmaier, Goetz wrote: >>>>>>> Hi, >>>>>>> >>>>>>> I now edited the stuff I had proposed below: >>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >> stackFix/webrev.03/ >>>>>>> This includes >>>>>>> - the NPTL fix from webrev.02 >>>>>> Okay in principle. As discussed this only impacts non-JavaThreads so the >>>>>> change should be minimal. >>>>>> >>>>>>> - merging code on linux >>>>>> Went a bit further than I had expected but if this truly isn't CPU >>>>>> dependent code then great! >>>>>> >>>>>>> - not adding OS guard to compiler threads. >>>>>> Okay in principle. IIUC we will now save the OS guard page for compiler >>>>>> thread stacks. Is that the only impact? The hotspot-compiler-dev folk >>>>>> may want to sign off on this part. >>>>>> >>>>>> >>>>>> A few minor comments: >>>>>> >>>>>> src/os/linux/vm/os_linux.cpp >>>>>> >>>>>> 2854 return ((thr_type == java_thread || thr_type == >>>>>> os::compiler_thread) ... >>>>>> >>>>>> os:: should be used for both types or none. >>>>>> >>>>>> 6153 pthread_attr_getguardsize(&attr, &guard_size); >>>>>> >>>>>> Can you at least verify a zero return code in an assert/assert_status >>>>>> please. >>>>>> >>>>>> --- >>>>>> >>>>>> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >>>>>> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp >>>>>> src/os_cpu/linux_s390/vm/os_linux_s390.cpp >>>>>> >>>>>> Are the changes to min_stack_allowed just fixing an existing bug? >>>>>> >>>>>> --- >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> ----- >>>>>> >>>>>> >>>>>>> I think this should be pushed for this bug ID. For the other changes I'll >>>>>>> make another bug. >>>>>>> >>>>>>> Best regards, >>>>>>> Goetz. >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Lindenmaier, Goetz >>>>>>>> Sent: Wednesday, November 23, 2016 8:11 AM >>>>>>>> To: David Holmes ; >>>>>>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net >>>>>>>> Subject: RE: RFR(M): 8169373: Work around linux NPTL stack guard >> error. >>>>>>>> Hi, >>>>>>>> >>>>>>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. >> The >>>>>>>>> ability for a CompilerThread to execute Java code (can_call_java()) is >>>>>>>>> now a dynamic property depending on whether the current compiler >> is >>>> the >>>>>>>>> JVMCI compiler. >>>>>>>> Ah, then I should also leave space for shadow pages in the minimal >> stack >>>>>> size >>>>>>>> of comiler threads. >>>>>>>> >>>>>>>> Do we agree on the cleanup and on leaving out the OS guard page on >>>>>>>> compiler threads? >>>>>>>> Then I would edit a change comprising the NPTL workaround and these >>>>>>>> additional changes, and split the other issue into a new bug? I think this >>>>>>>> will easy the reviewing. >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Goetz. >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>>>>> Sent: Mittwoch, 23. November 2016 02:50 >>>>>>>>> To: Lindenmaier, Goetz ; >>>>>>>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net >>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >>>> error. >>>>>>>>> On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: >>>>>>>>>> Hi Dan, >>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>>>>>>>>> Sent: Dienstag, 22. November 2016 14:01 >>>>>>>>>>> To: Lindenmaier, Goetz ; David >> Holmes >>>>>>>>>>> ; hotspot-runtime- >> dev at openjdk.java.net >>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >>>>>>>> error. >>>>>>>>>>> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>>> Hi Dan, >>>>>>>>>>>> >>>>>>>>>>>> I ran into a row of issues, some errors on the platforms. >>>>>>>>>>>> >>>>>>>>>>>> What I meant with that comment is that >>>>>>>>>>>> os::Linux::min_stack_allowed = >> MAX2(os::Linux::min_stack_allowed, >>>>>>>>>>>> JavaThread::stack_guard_zone_size() + >>>>>>>>>>>> JavaThread::stack_shadow_zone_size() + >>>>>>>>>>>> (4*BytesPerWord >>>> COMPILER2_PRESENT(+2)) * >>>>>> 4 >>>>>>>> * >>>>>>>>> K); >>>>>>>>>>>> was executed, and min_stack_allowed used for all stacks. Now, >>>>>>>> compiler >>>>>>>>>>> and >>>>>>>>>>>> vm minimum stack sizes are not increased by these sizes. >>>>>>>>>>> Now I see what you mean. Thanks for clearing this up. >>>>>>>>>>> >>>>>>>>>>> I should have remembered that part of the change because we >> went >>>>>>>> back >>>>>>>>>>> and forth about removing the red/yellow zone pages from the other >>>>>>>>>>> threads. In particular, we discussed the compiler thread because it >>>>>>>>>>> is-a JavaThread. Our conclusion was that a compiler thread doesn't >>>>>>>>>>> execute Java bytecode so we could remove the red/yellow pages... >>>>>>>>>> Yes, it does not execute java byte code. >>>>>>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with JVMCI. >> The >>>>>>>>> ability for a CompilerThread to execute Java code (can_call_java()) is >>>>>>>>> now a dynamic property depending on whether the current compiler >> is >>>> the >>>>>>>>> JVMCI compiler. >>>>>>>>> >>>>>>>>> David >>>>>>>>> ----- >>>>>>>>> >>>>>>>>>> Therefore you can remove the shadow pages. There is no code that >>>>>>>>>> will bang. >>>>>>>>>> But red/yellow pages are protected right during thread startup. >>>>>>>>>> Therefore you must have enough space for them. >>>>>>>>>> On ppc, we try to protect three 64K pages out of the 128K compiler >>>> stack. >>>>>>>>>> That obviously fails. >>>>>>>>>> >>>>>>>>>> Therefore I propose: >>>>>>>>>> size_t os::Posix::_java_thread_min_stack_allowed = 48 * K; // >> Set >>>>>>>>> platform dependent. >>>>>>>>>> in os::Posix::set_minimum_stack_sizes(): >>>>>>>>>> _java_thread_min_stack_allowed = >> _java_thread_min_stack_allowed >>>> + >>>>>>>>>> JavaThread::stack_guard_zone_size() + >>>>>>>>>> JavaThread::stack_shadow_zone_size(); >>>>>>>>>> >>>>>>>>>> (Similar for _compiler_thread_min_stack_allowed). >>>>>>>>>> >>>>>>>>>> The minimal stack size is made up of three components: >>>>>>>>>> 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot >>>>>>>>> implementation/platform/os. >>>>>>>>>> 2. Sizes of C++ frames: depends on C++ compiler. >>>>>>>>>> These are fixed at compile time. >>>>>>>>>> 3. Sizes of red/yellow/reserved/shadow pages. Depends on the >>>> system >>>>>>>>> the >>>>>>>>>> VM is used on. This is not fixed at compile time. (Our ppc >>>> machines >>>>>>>>> differ >>>>>>>>>> in page size.) >>>>>>>>>> Therefore 3. should not be included in a compile time constant. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> And that decision allowed us to be exposed to the 64K page issue >>>>>>>>>>> because the "extra" space isn't there anymore. >>>>>>>>>>> >>>>>>>>>>>> At least the _compiler_thread_min_stack_allowed should be >>>> increased >>>>>>>>>>>> similarly by red/yellow zone size. The compiler thread is a Java >>>>>>>>>>>> thread and must have space for these zones. >>>>>>>>>>> I'm not sure that I completely agree (yet). To me, the red/yellow >>>>>>>>>>> pages are there for Java thread stack overflow semantics. Yes, the >>>>>>>>>>> compiler thread needs extra space when 64K pages are used, but I >>>>>>>>>>> would prefer that we add that space via a different calculation. >>>>>>>>>> Yes they are. But compiler threads happen tob e a subclass of >>>>>>>>>> Java threads and use the same run() method that puts the pages >>>>>>>>>> There. >>>>>>>>>> I agree that they are not needed for Compiler threads, nor for >>>>>>>>>> CodeCacheSweeperThreads. I don't really now about >>>>>>>>>> JvmtiAgentThreads and ServiceThreads, but all of the get the guard >>>>>>>> pages >>>>>>>>>> because they are derived from JavaThread. >>>>>>>>>> >>>>>>>>>>> To put it another way, I'd like to see us add extra space to solve >>>>>>>>>>> the 64K page issue directly instead of as a side effect of the >>>>>>>>>>> red/yellow page addition. >>>>>>>>>> I don't understand. What do you mean by 'directly'? >>>>>>>>>> >>>>>>>>>>>> Also, the change added a test that is failing now. >>>>>>>>>>> And that's a "good thing" (TM), right? :-) >>>>>>>>>> Yes, it showed a bug and thus raised the need to fix it! :) >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> Goetz. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Again, thanks for clarifying 8140520's role in this issue. >>>>>>>>>>> >>>>>>>>>>> Dan >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Best regards, >>>>>>>>>>>> Goetz. >>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Daniel D. Daugherty >> [mailto:daniel.daugherty at oracle.com] >>>>>>>>>>>>> Sent: Montag, 21. November 2016 17:28 >>>>>>>>>>>>> To: David Holmes ; Lindenmaier, >> Goetz >>>>>>>>>>>>> ; hotspot-runtime- >>>>>>>>> dev at openjdk.java.net >>>>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack >> guard >>>>>>>>> error. >>>>>>>>>>>>> Sorry for the delayed responses to this thread. I've been on >>>> vacation... >>>>>>>>>>>>> One comment/query embedded below... >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 11/10/16 8:40 PM, David Holmes wrote: >>>>>>>>>>>>>> Hi Goetz, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>>>>>> Hi David, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This issue is different to 6675312, see also my comment in the >>>> bug. >>>>>>>>>>>>>>> It appears running jtreg test >>>>>>>>> runtime/Thread/TooSmallStackSize.java, >>>>>>>>>>>>>>> with my patch below you can reproduce it on linuxx86_64. You >>>> can >>>>>>>>> not >>>>>>>>>>>>>>> do that with 6675312. Also, I would assume there are systems >> out >>>>>>>>> there >>>>>>>>>>>>>>> on x86 that uses 64-K pages, did you run the tests on these? I >>>>>>>> would >>>>>>>>>>>>>>> assume you get hard crashes with stack overflows in the first >> C2 >>>>>>>>>>>>>>> compilation if there is only 64K usable CompilerThreadStack. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My fix does not affect Java threads, which are the largest >> amount >>>>>>>>>>>>>>> of threads used by the VM. It affects only the non-Java threads. >>>>>>>>>>>>>>> It adds one page to these threads. The page does not require >>>>>>>>> memory, >>>>>>>>>>>>>>> as it's protected. The stack will only require more space if the >>>>>>>> thread >>>>>>>>>>>>>>> ran into a stack overflow before the fix as else the pages are >> not >>>>>>>>>>>>>>> mapped. >>>>>>>>>>>>>>> This are stack overflows that cause hard crashes, at least on >> ppc >>>>>>>> the >>>>>>>>> VM >>>>>>>>>>>>>>> does not properly catch these stack overflows, so any setup >>>>>>>> working >>>>>>>>>>> now >>>>>>>>>>>>>>> will not run into the additional space. Altogether there should >> be >>>>>>>> no >>>>>>>>>>>>>>> effect on running systems besides requiring one more entry in >>>> the >>>>>>>>>>>>>>> page table per non-Java thread. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The problem is caused by a rather recent change (8140520: >>>>>>>> segfault >>>>>>>>> on >>>>>>>>>>>>>>> solaris-amd64 >>>>>>>>>>>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed >> after >>>>>>>>>>>>>>> feature-close. As this was a rather recent change, it must be >>>>>>>>>>>>>>> possible to >>>>>>>>>>>>>>> fix this follow up issue. What else is this period in the project >>>> good >>>>>>>>>>>>>>> for if not fixing issues? >>>>>>>>>>>>>> So I am seeing a number of factors here. >>>>>>>>>>>>>> >>>>>>>>>>>>>> First, 8140520, set: >>>>>>>>>>>>>> >>>>>>>>>>>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * >> K; >>>>>>>>>>>>> So I'm confused by the above comment: >>>>>>>>>>>>> >>>>>>>>>>>>> > The problem is caused by a rather recent change (8140520: >>>>>>>> segfault >>>>>>>>>>>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" option) >>>>>>>>>>>>> >>>>>>>>>>>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- >>>>>>>>>>>>> >>>>>>>>> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html >>>>>>>>>>>>> shows this change: >>>>>>>>>>>>> >>>>>>>>>>>>> @@ -531,19 +531,17 @@ >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >> ////////////////////////////////////////////////////////////////////////////// >>>>>>>>>>>>> // >>>>>>>>>>>>> // thread stack >>>>>>>>>>>>> >>>>>>>>>>>>> -size_t os::Linux::min_stack_allowed = 128*K; >>>>>>>>>>>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * >> K; >>>>>>>>>>>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; >>>>>>>>>>>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 128 >> * >>>> K; >>>>>>>>>>>>> so the existing single variable of 'min_stack_allowed' was >>>>>>>>>>>>> replaced by three variables: >> _compiler_thread_min_stack_allowed, >>>>>>>>>>>>> _java_thread_min_stack_allowed, and >>>>>>>>>>>>> _vm_internal_thread_min_stack_allowed. >>>>>>>>>>>>> >>>>>>>>>>>>> The old single variable and the three new variables are all >>>>>>>>>>>>> initialized to the same value (128K) so the fix for 8140520 >>>>>>>>>>>>> did not change stack sizes for this platform. In fact, only >>>>>>>>>>>>> one platform had a size change (Solaris X64). >>>>>>>>>>>>> >>>>>>>>>>>>> So I'm confused about how the fix for 8140520 caused this >> problem. >>>>>>>>>>>>> Based on David's analysis below, it looks to me like this 64K stack >>>>>>>>>>>>> guard page problem should also exist prior to the fix for >> 8140520. >>>>>>>>>>>>> Goetz, can you please explain how 8140520 caused this problem? >>>>>>>>>>>>> >>>>>>>>>>>>> Dan >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Second on linux PPC it is hardwired to use 2 guard pages: >>>>>>>>>>>>>> >>>>>>>>>>>>>> return 2 * page_size(); >>>>>>>>>>>>>> >>>>>>>>>>>>>> Third, you had a pagesize of 64K. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Fourth, NPTL takes the guard space from the stack space - >> hence >>>>>>>> with >>>>>>>>> 2 >>>>>>>>>>>>>> x 64K guard, and a 128K stack it was all consumed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> --- >>>>>>>>>>>>>> >>>>>>>>>>>>>> In the proposed changes you now only use page_size() for the >>>> guard, >>>>>>>>> so >>>>>>>>>>>>>> that alone would have fixed the observed problem. >>>>>>>>>>>>>> >>>>>>>>>>>>>> But in addition you want to address the NPTL problem by adding >>>>>>>> back >>>>>>>>>>>>>> the guard space to the stack size requested. That alone would >> also >>>>>>>>>>>>>> have fixed the observed problem. :) >>>>>>>>>>>>>> >>>>>>>>>>>>>> But in addition you have increased the minimum stack size: >>>>>>>>>>>>>> >>>>>>>>>>>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = 192 * >> K; >>>>>>>>>>>>>> which again, on its own would have fixed the original problem. >> :) >>>>>>>>>>>>>> Did you really intend to increase the real minimum stack from >>>> 128K >>>>>>>> to >>>>>>>>>>>>>> 256K ? (on a 64K page system) >>>>>>>>>>>>>> >>>>>>>>>>>>>> --- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Focusing simply on the shared code change to adjust the >> requested >>>>>>>>>>>>>> stacksize by the amount of guard space (if any), this does not >> seem >>>>>>>>>>>>>> unreasonable. As you note it is restricted to non-JavaThreads >> and >>>>>>>> only >>>>>>>>>>>>>> adds a page to reserved stack space. >>>>>>>>>>>>>> >>>>>>>>>>>>>> My only query now is whether the minimum stacksize detection >>>> logic >>>>>>>>>>>>>> will correctly report the real minimum stack size (taking into >>>> account >>>>>>>>>>>>>> the need for the guard page) ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> David >>>>>>>>>>>>>> >>>>>>>>>>>>>>> So I really think this issue should be fixed. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>> Goetz. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>>>>>>>>>>>> Sent: Thursday, November 10, 2016 10:02 PM >>>>>>>>>>>>>>>> To: Lindenmaier, Goetz ; >> hotspot- >>>>>>>>>>>>> runtime- >>>>>>>>>>>>>>>> dev at openjdk.java.net >>>>>>>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack >>>>>>>> guard >>>>>>>>>>> error. >>>>>>>>>>>>>>>> Hi Goetz, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As per the bug report, this issue was already known >> (6675312) >>>> and >>>>>>>>> we >>>>>>>>>>>>>>>> chose not to try and address it due to no reported issues at >> the >>>>>>>>> time. >>>>>>>>>>>>>>>> While I see that you have encountered an issue (is it real or >>>>>>>>>>>>>>>> fabricated?) I think this change is too intrusive to be applied >> at >>>> this >>>>>>>>>>>>>>>> stage of the JDK 9 release cycle, as it will change the stack >>>>>>>>>>>>>>>> requirements of every application running on Linux. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> David >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Please review this change. I please need a sponsor: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >>>>>>>>>>>>> stackFix/webrev.01/ >>>>>>>>>>>>>>>>> In the Linux NPTL pthread implementation the guard size >>>>>>>>> mechanism >>>>>>>>>>>>>>>>> is not >>>>>>>>>>>>>>>>> implemented properly. The posix standard requires to add >> the >>>>>>>>> size >>>>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>>>> guard pages to the stack size, instead Linux takes the space >> out >>>>>>>> of >>>>>>>>>>>>>>>>> 'stacksize'. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The Posix standard >>>>>>>>>>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ >>>>>>>>>>>>>>>>> says "the implementation allocates extra memory at the >>>>>>>>> overflow >>>>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>>>> the stack". The linux man page >>>>>>>>>>>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize says >> "As >>>>>>>>> at >>>>>>>>>>>>>>>>> glibc >>>>>>>>>>>>>>>>> 2.8, the NPTL threading implementation includes the guard >>>> area >>>>>>>>>>> within >>>>>>>>>>>>>>>>> the stack size allocation, rather than allocating extra space >> at >>>>>>>>>>>>>>>>> the end >>>>>>>>>>>>>>>>> of the stack, as POSIX.1 requires". >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I encounter this problem in >>>>>>>>> runtime/Thread/TooSmallStackSize.java >>>>>>>>>>>>>>>>> on ppc >>>>>>>>>>>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is >> 128K >>>>>>>> on >>>>>>>>>>> ppc, >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> ppc specifies two OS guard pages. The VM crashes in >> pthread >>>>>>>>>>> creation >>>>>>>>>>>>>>>>> because there is no usable space in the thread stack after >>>>>>>>> allocating >>>>>>>>>>>>>>>>> the guard pages. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> But TooSmallStackSize.java requires that the VM comes up >>>> with >>>>>>>>> the >>>>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>>>> size mentioned in the error message. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This fix adapts the requested stack size on Linux by the size >> of >>>>>>>> the >>>>>>>>>>>>>>>>> guard pages to mimick proper behaviour, see change to >>>>>>>>>>> os_linux.cpp. >>>>>>>>>>>>>>>>> The change also streamlines usage of stack_guard_page on >>>>>>>>> linuxppc, >>>>>>>>>>>>>>>>> linuxppcle, aixppc and linuxs390. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> To reproduce the error on linux_x86_64, apply below patch >>>> and >>>>>>>>> call >>>>>>>>>>> the >>>>>>>>>>>>>>>>> VM with -XX:CompilerThreadStackSize=64. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'm still exploring why I had to choose such big compiler >> stacks >>>>>>>> on >>>>>>>>>>>>>>>>> ppc >>>>>>>>>>>>>>>>> to get -version passing, but I wanted to send the RFR now as >>>>>>>>> people >>>>>>>>>>>>>>>>> obviously looked at the bug I opened (Thanks David!). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Goetz. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> diff -r b7ae012c55c3 >>>> src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>>>>>>>>>>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov >> 07 >>>>>>>>>>> 12:37:28 >>>>>>>>>>>>>>>> 2016 >>>>>>>>>>>>>>>>> +0100 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov >> 10 >>>>>>>>>>>>> 16:52:17 >>>>>>>>>>>>>>>> 2016 >>>>>>>>>>>>>>>>> +0100 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> @@ -701,7 +701,7 @@ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType >> thr_type) { >>>>>>>>>>>>>>>>> // Creating guard page is very expensive. Java thread >> has >>>>>>>>> HotSpot >>>>>>>>>>>>>>>>> // guard page, only enable glibc guard page for non-Java >>>>>>>>> threads. >>>>>>>>>>>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> // Java thread: >>>>>>>>>>>>>>>>> From andrey.x.nazarov at oracle.com Tue Nov 29 17:26:33 2016 From: andrey.x.nazarov at oracle.com (Andrey Nazarov) Date: Tue, 29 Nov 2016 20:26:33 +0300 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> Message-ID: <6296BDD5-8FC3-411C-9860-54BAE3E202B5@oracle.com> Hi, I?ve reviewed Langtools code. There are various comment ?//TODO?, ?//FIXME?, ?//XXX?. I think they should be revised. May be issues should be filed to track them. Unused import at 37 import java.io.IOException; in langtools/test/tools/javac/modules/ModuleInfoTest.java ASCII graphics issue at 64 line in test/tools/javac/modules/GraphsTest.java Unused builder method in langtools/test/tools/lib/toolbox/ModuleBuilder.java 159 public ModuleBuilder exportsDynamicPrivate(String pkg) { 141 public ModuleBuilder exportsDynamic(String pkg) { 179 public ModuleBuilder exportsDynamicTo(String pkg, String module) { 199 public ModuleBuilder exportsDynamicPrivateTo(String pkg, String module) { Javadoc is used instead of comment langtools/test/tools/javac/modules/AnnotationsOnModules.java /** 25 * @test ?Andrey > On 24 Nov 2016, at 18:25, Alan Bateman wrote: > > Folks on jigsaw-dev will know that we are on a mission to bring the changes accumulated in the jake forest to jdk9/dev. We can think of this as a refresh of the module system in JDK 9, the last big refresh was in May with many small updates since then. > > The focus this time is to bring the changes that are tied to JSR issues into jdk9/dev, specifically the issues that are tracked on the JSR issues list [1] as: > > #CompileTimeDependences > #AddExportsInManifest > #ClassFileModuleName > #ClassFileAccPublic > #ServiceLoaderEnhancements > #ResourceEncapsulation/#ClassFilesAsResources > #ReflectiveAccessToNonExportedTypes > #AwkwardStrongEncapsulation > #ReadabilityAddedByLayerCreator > #IndirectQualifiedReflectiveAccess (partial) > #VersionsInModuleNames > #NonHierarchicalLayers > #ModuleAnnotations/#ModuleDeprecation > #ReflectiveAccessByInstrumentationAgents > > Some of these issues are not "Resolved" yet, meaning there is still ongoing discussion on the EG mailing list. That is okay, there is nothing final here. If there are changes to these proposals then the implementation changes will follow. Also, as I said in a mail to jigsaw-dev yesterday [2], is that we will keep the jake forest open for ongoing prototyping and iteration, also ongoing implementation improvements where iteration or bake time is important. > > For the code review then the focus is therefore on sanity checking the changes that we would like to bring into jdk9/dev. We will not use this review thread to debate alternative designs or other big implementation changes that are more appropriate to bake in jake. > > To get going, I've put the webrevs with a snapshot of the changes in jake here: > http://cr.openjdk.java.net/~alanb/8169069/0/ > > The changes are currently sync'ed against jdk-9+146 and will be rebased (and re-tested) against jdk9/dev prior to integration. There are a number of small changes that need to be added to this in the coming days, I will refresh the webrev every few days to take account of these updates. > > > A few important points to mention, even if you aren't reviewing the changes: > > 1. This refresh requires a new version of jtreg to run the tests. The changes for this new version are in the code-tools/jtreg repository and the plan is to tag a new build (jtreg4.2-b04) next week. Once the tag has been added then we'll update the requiredVersion property in each TEST.ROOT to force everyone to update. > > 2. For developers trying out modules with the main line JDK 9 builds then be aware that `requires public` changes to `requires transitive` and the `provides` clause changes to require all providers for a specific service type to be in the same clause. Also be aware that the binary form of the module declaration (module-info.class) changes so you will need to recompile any modules. > > 3. Those running existing code on JDK 9 and ignoring modules will need to be aware of a disruptive change in this refresh. The disruptive change is #AwkwardStrongEncapsulation where setAccessible(true) is changed so that it can't be used to break into non-public fields/methods of JDK classes. This change is going to expose a lot of hacks in existing code. We plan to send mail to jdk9-dev in advance of this integration to create awareness of this change. As per the original introduction of strong encapsulation then command line options (and now the manifest of application JAR files) can be used to keep existing code working. The new option is `--add-opens` to open a package in a module for deep reflection by other modules. As an example, if you find yourself with code that hacks into the private `comparator` field in java.util.TreeMap then running with `--add-opens java.base/java.util=ALL-UNNAMED` will keep that code working. > > > A few miscellaneous notes for those that are reviewing: > > 1. We have some temporary/transition code in the top-level repo to deal with the importing of the JavaFX modules. This will be removed once the changes are in JDK 9 for the OpenJFX project to use. > > 2. In the jdk repo then it's important to understand that the module system is initialized at startup and there are many places where we need to keep startup performance in mind. This sometimes means less elegant code than might be used if startup wasn't such a big concern. > > 3. The changes in the jaxws repo make use of new APIs that means the code doesn't compile with JDK 7 or JDK 8. Our intention is to work with the JAXB and JAX-WS maintainers to address the issues in the upstream project and then bring those changes into jdk9/dev to replace the patches that we are forced to push for the short term. > > 4. You will see several tests where the value of the @modules tag has `:open` or `:+open`. This is new jtreg speak. The former means the test is run with --add-opens to open the package, the latter means the test is exported at compile-time and exported + open at run-time (the latter usage will be rare, it's where tests have static references to JDK internal types and are also doing deep reflection with setAccessible). > > > In terms of dates then we are aiming to integrate these changes into jdk9/dev in early December. I will send a follow-up mail next week on this as we work through the logistics. > > -Alan > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ > [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/010219.html From weijun.wang at oracle.com Wed Nov 30 11:21:29 2016 From: weijun.wang at oracle.com (Wang Weijun) Date: Wed, 30 Nov 2016 19:21:29 +0800 Subject: Code review for jigsaw/jake -> jdk9/dev sync up In-Reply-To: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> References: <6da3ac71-ca2b-3284-576d-055c19a65521@oracle.com> Message-ID: <53095bcf-426b-ad18-d4c3-f86a5f009244@oracle.com> test/java/security/testlibrary/Proc.java: if (hasModules) { Stream.of(jdk.internal.misc.VM.getRuntimeArguments()) - .filter(arg -> arg.startsWith("--add-exports=")) + .filter(arg -> arg.startsWith("--add-exports=") || + arg.startsWith("--add-opens")) .forEach(cmd::add); } Maybe "--add-opens=" to be more precise? Thanks Max From david.holmes at oracle.com Wed Nov 30 21:50:39 2016 From: david.holmes at oracle.com (David Holmes) Date: Thu, 1 Dec 2016 07:50:39 +1000 Subject: RFR(M): 8169373: Work around linux NPTL stack guard error. In-Reply-To: References: <1267cfcb-fa62-fb7b-76d5-a9d61c899ce5@oracle.com> <793ff470bbf14241a1d627bff25ed627@DEROTE13DE08.global.corp.sap> <45f66348d6d54f8d826c707aa70f2352@DEROTE13DE08.global.corp.sap> <24078321-9ca8-28e8-3895-59b09ab398f5@oracle.com> <4dfa4ae6-4fa6-994c-f2cf-8fc309d8c520@oracle.com> <821371d77c7c4e6d966901b461b3d6a1@DEROTE13DE08.global.corp.sap> Message-ID: <49acd657-aefa-b935-1b2c-195fc2d125ba@oracle.com> On 1/12/2016 1:20 AM, Daniel D. Daugherty wrote: >> Would you mind sponsoring this change? > > Yes, I will sponsor this change. However, I would like to get a > thumbs up from David H. on the latest version and I don't see > any review from someone on the Compiler Team. I'm okay with proposed changes - but also want to hear from compiler team and I'd want to see this put through some advance testing before it gets pushed (full rbt run). I have one minor nit in the wording of the fatal error messages "failed with errno" - these methods don't touch errno so I'd prefer it if it said error. Thanks, David ----- > Vladimir! We need someone on the Compiler Team to look at these > CompilerThread stack size changes... > > Dan > > > On 11/30/16 12:57 AM, Lindenmaier, Goetz wrote: >> Hi Dan, >> >>> Thumbs up! I don't need to see a new webrev if you choose >>> to fix the minor comments above. >> I anyways did a new one fixing the comments: >> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.06/ >> Would you mind sponsoring this change? >> >> I took the minimum stack sizes from my experimental runs where >> I had removed the automatic resizing to find the really needed space. >> If I put something smaller there, I could as well put '0' ... as >> _java_thread_min_stack_allowed = MAX2(_java_thread_min_stack_allowed, >> >> JavaThread::stack_guard_zone_size() + >> >> JavaThread::stack_shadow_zone_size() + >> (4 * BytesPerWord >> COMPILER2_PRESENT(+ 2)) * 4 * K); >> will fix it. >> This code effectively limits the usable stack size to >> (4 * BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K) >> which makes the initialization of _java_thread_min_stack_allowed with >> platform >> values pointless. >> >> I'll open a new bug for the follow-up stack issue, probably tomorrow. >> I don't feel like addressing testing all the possible error codes, as >> they probably should be fixed in more places, and there is no issue >> pending currently. Maybe it should be fixed in jdk10 at some point. >> >> Best regards, >> Goetz >> >> >>> -----Original Message----- >>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>> Sent: Dienstag, 29. November 2016 20:04 >>> To: Lindenmaier, Goetz ; hotspot-compiler- >>> dev at openjdk.java.net; 'hotspot-runtime-dev at openjdk.java.net' >> runtime-dev at openjdk.java.net> >>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard error. >>> >>> On 11/29/16 2:30 AM, Lindenmaier, Goetz wrote: >>>> Hi Dan, >>>> >>>> see my replies inline ... >>>> New webrev: >>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.05/ >>> src/os/aix/vm/os_aix.cpp >>> L887: // libc guard page >>> nit - You made other existing comments into sentences (leading >>> capital and trailing '.'), but not this new comment. >>> Why? >>> >>> src/os/aix/vm/os_aix.hpp >>> No comments. >>> >>> src/os/linux/vm/os_linux.cpp >>> L6096: // | |/ 1 page glibc guard. >>> nit - "1 page glibc guard" -> "1 glibc guard page." >>> >>> src/os/posix/vm/os_posix.cpp >>> No comments. >>> >>> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >>> No comments. >>> >>> src/os_cpu/bsd_x86/vm/os_bsd_x86.cpp >>> L875: // | |/ 1 page glibc guard. >>> nit - "1 page glibc guard" -> "1 glibc guard page." >>> >>> src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp >>> No comments. >>> >>> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp >>> No comments. >>> >>> src/os_cpu/linux_s390/vm/os_linux_s390.cpp >>> No comments. >>> >>> src/os_cpu/linux_sparc/vm/os_linux_sparc.cpp >>> No comments. >>> >>> src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>> No comments. >>> >>> src/os_cpu/linux_zero/vm/os_linux_zero.cpp >>> No comments. >>> >>> >>> Thumbs up! I don't need to see a new webrev if you choose >>> to fix the minor comments above. >>> >>> Some replies embedded below. >>> >>> >>>>> -----Original Message----- >>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>>> Sent: Dienstag, 29. November 2016 01:38 >>>>> To: Lindenmaier, Goetz ; hotspot-compiler- >>>>> dev at openjdk.java.net; 'hotspot-runtime-dev at openjdk.java.net' >>>> runtime-dev at openjdk.java.net> >>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >>>>> error. >>>>> >>>>> On 11/28/16 2:08 AM, Lindenmaier, Goetz wrote: >>>>>> Hi, >>>>>> >>>>>> I'm working on a fix for OS guard pages on stacks. I figured there >>>>>> are VM guard pages (reserved, yellow, red) on the compiler stacks >>>>>> _and_ OS guard pages. For Java threads, the OS guard pages are left >>>>>> out. I think this should be done for compiler threads, too. Please >>>>>> confirm. >>>>>> Webrev: >>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc-stackFix/webrev.04/ >>>>>> >>>>> src/os/aix/vm/os_aix.cpp >>>>> L888: pthread_attr_setguardsize(&attr, >>>>> os::Aix::default_guard_size(thr_type)); >>>>> No check or assert on the return status of this call. >>>>> Is one needed? >>>> I implemented this as the existing code on linux which has >>>> no check either. I think a failure is quite theoretical. >>>> Because of your comment below I'll leave it as-is. >>> OK. >>> >>> >>>>> L3044: // guard pages, so only enable libc guard pages for >>>>> non-Java threads. >>>>> src/os/linux/vm/os_linux.cpp also has this comment: >>>>> // (Remember: compiler thread is a Java thread, too!) >>>> Fixed. >>>> >>>>> L3051: return ((thr_type == java_thread || thr_type == >>>>> compiler_thread) ? 0 : 4*K); >>>>> nit - please add spaces around the '*' so '4 * K'.' >>>> Fixed. >>>> >>>>> src/os/aix/vm/os_aix.hpp >>>>> No comments. >>>>> >>>>> src/os/linux/vm/os_linux.cpp >>>>> L729: // is not implemented properly. The posix standard >>>>> requires >>>>> to add >>>>> Typo: 'to add' -> 'adding' >>>> Fixed. >>>> >>>>> L738: pthread_attr_setguardsize(&attr, >>>>> os::Linux::default_guard_size(thr_type)); >>>>> No check or assert on the return status of this call. >>>>> Is one needed? >>>> See above. >>>> >>>>> L2851: // Creating guard page is very expensive. Java >>>>> thread has >>>>> HotSpot >>>>> L2852: // guard page, only enable glibc guard page for >>>>> non-Java >>>>> threads. >>>>> L2853: // (Remember: compiler thread is a java thread, too!) >>>>> Typo: "java thread" -> "Java thread" (consistency) >>>>> >>>>> This comment block should be common to all the platforms >>>>> that >>>>> define default_guard_size(). Yes, I can see that AIX >>>>> needs to >>>>> add another paragraph, but we should make the core comment >>> common >>>>> if possible. >>>> I made the first three lines look alike. >>>> >>>>> L6090: // Java/Compiler thread: >>>>> Thanks for making this common in os_linux.cpp. >>>>> >>>>> L6095: // | glibc guard page | - guard, attached Java >>>>> thread usually has >>>>> Clarity: "guard," -> "guard page," >>>> Fixed. >>>> >>>>> Typo: "Java thread" -> "JavaThread" (consistency) >>>> I changed both to Java thread as at the other locations. >>>> >>>>> L6099: // | HotSpot Guard Pages | - red and yellow pages >>>>> The fairly recently added reserved page should be mentioned >>>>> here also? >>>> Yes. Fixed. Also fixed it to say >>>> JavaThread::stack_reserved_zone_base(). >>>> Also fixed comment on bsd. >>> Thanks for also fixing BSD. >>> >>> >>>>> L6120 // ** P1 (aka bottom) and size ( P2 = P1 - size) are the >>>>> address and stack size returned from >>>>> Typo: "( P2 = ..." -> "(P2 = ..." >>>> Fixed. >>>> >>>>> L6148: fatal("Can not locate current stack attributes!"); >>>>> Typo: "Can not" -> "Cannot" >>>> Fixed. >>>> >>>>> L6175: // stack size includes normal stack and HotSpot >>>>> guard pages >>>>> Perhaps add to the comment: >>>>> "for the threads that have HotSpot guard pages." >>>> Fixed. I also checked my comments for "OS guard pages" and >>>> replaced it by "glibc guard pages" which is used in several places >>>> already, same for "VM guard page" --> "HotSpot guard page". I >>>> think this is also more consistent. >>> I agree! >>> >>> >>>>> src/os/posix/vm/os_posix.cpp >>>>> L1097: pthread_attr_getstacksize(attr, &stack_size); >>>>> L1098: pthread_attr_getguardsize(attr, &guard_size); >>>>> Do these two calls need to have their result checked? >>>>> >>>>> Now I'm starting to wonder if all the uses of these >>>>> two APIs need to be checked? Separate bug? >>>> It would be more consistent with the specification of the methods, >>>> On the other side it's quite unlikely that these fail if attr != NULL. >>> So should we file a new bug? Or does this fall into the realm of >>> other OS/libc code that we call and assume never fails? :-) >>> >>> >>>>> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >>>>> L540: size_t os::Posix::_compiler_thread_min_stack_allowed = >>>>> 512 * K; >>>>> L541: size_t os::Posix::_java_thread_min_stack_allowed = 512 >>>>> * K; >>>>> So prior to the fix for 8140520, >>>>> src/os/aix/vm/os_aix.cpp had >>>>> this single min_stack_allowed value: >>>>> >>>>> L3601: os::Aix::min_stack_allowed = >>>>> MAX2(os::Aix::min_stack_allowed, >>>>> L3602: JavaThread::stack_guard_zone_size() + >>>>> L3603: JavaThread::stack_shadow_zone_size() + >>>>> L3604: (4*BytesPerWord >>>>> COMPILER2_PRESENT(+2)) * 4 * K); >>>>> >>>>> and the fix for 8140520 changed that for *NIX platforms to >>>>> three mins in src/os/posix/vm/os_posix.cpp: >>>>> >>>>> L1108: _java_thread_min_stack_allowed = >>>>> MAX2(_java_thread_min_stack_allowed, >>>>> L1109: JavaThread::stack_guard_zone_size() + >>>>> L1110: JavaThread::stack_shadow_zone_size() + >>>>> L1111: (4 * >>>>> BytesPerWord COMPILER2_PRESENT(+ 2)) * 4 * K); >>>>> >>>>> L1150: _compiler_thread_min_stack_allowed = >>>>> align_size_up(_compiler_thread_min_stack_allowed, vm_page_size()); >>>>> >>>>> L1161 _vm_internal_thread_min_stack_allowed = >>>>> align_size_up(_vm_internal_thread_min_stack_allowed, vm_page_size()); >>>>> >>>>> Which means that the compiler_thread no longer benefits >>>>> from >>>>> the extra space for quard and shadow pages. The thinking in >>>>> 8140520 was that the compiler_thread and >>>>> vm_internal_threads >>>>> don't need the quard and shadow pages since they don't run >>>>> Java code (ignoring JVMCI for now). >>>>> >>>>> So I can see bumping _compiler_thread_min_stack_allowed >>>>> from >>>>> 128 -> 512 as one solution for getting that extra space >>>>> back. >>>>> However, I don't understand why >>>>> _java_thread_min_stack_allowed >>>>> has changed from 128 -> 512. >>>> Because it was never correct before. >>> OK. That sounds like the new test that I included with 8140520 would >>> have failed with JavaThread stack sizes even before the product code >>> changes from 8140520 were made. >>> >>> Since the size calculation for JavaThread stack sizes wasn't changed >>> for any platform in 8140520, that tends to indicate that the more >>> limited JDK test (test/tools/launcher/TooSmallStackSize.java) should >>> also have failed before the fix for 8140520. >>> >>> Please clarify the need for the _java_thread_min_stack_allowed change >>> from 128 -> 512. Unless test/tools/launcher/TooSmallStackSize.java >>> is never run in your testing, I'm having troubling seeing why the >>> _java_thread_min_stack_allowed increase is needed. >>> >>> >>>>> I had previously made this comment: >>>>> > To put it another way, I'd like to see us add extra >>>>> space to >>>>> > solve the 64K page issue directly instead of as a side >>>>> effect >>>>> > of the red/yellow page addition. >>>>> And Goetz replied with: >>>>> > I don't understand. What do you mean by 'directly'? >>>>> >>>>> So prior to the fix for 8140520, >>>>> src/os/solaris/vm/os_solaris.cpp >>>>> had a block like this: >>>>> >>>>> L4468: // For 64kbps there will be a 64kb page size, >>>>> which makes >>>>> L4469: // the usable default stack size quite a bit less. >>>>> Increase the >>>>> L4470: // stack for 64kb (or any > than 8kb) pages, this >>>>> increases >>>>> L4471: // virtual memory fragmentation (since we're not >>>>> creating the >>>>> L4472 // stack on a power of 2 boundary. The real fix >>>>> for this >>>>> L4473 // should be to fix the guard page mechanism. >>>>> L4474 >>>>> L4475 if (vm_page_size() > 8*K) { >>>>> L4476 threadStackSizeInBytes = >>>>> (threadStackSizeInBytes != 0) >>>>> L4477 ? threadStackSizeInBytes + >>>>> L4478 JavaThread::stack_red_zone_size() + >>>>> L4479 JavaThread::stack_yellow_zone_size() >>>>> L4480 : 0; >>>>> L4481 ThreadStackSize = threadStackSizeInBytes/K; >>>>> L4482 } >>>>> >>>>> The above is an example of what I mean by solving the 64K >>>>> page issue directly. In the fix for 8140520, that code >>>>> block >>>>> was moved to os::Posix::set_minimum_stack_sizes() in >>>>> src/os/posix/vm/os_posix.cpp and put in a "#ifdef >>>>> SOLARIS... >>>>> #endif // SOLARIS" block. Coleen filed a bug to determine >>>>> whether that code can be deleted: >>>>> >>>>> JDK-8161093 Solaris for >8k pagesize adds extra guard pages >>>>> https://bugs.openjdk.java.net/browse/JDK-8161093 >>>>> >>>>> but perhaps this bug shows that the code is needed? >>>>> >>>>> >>>>> OK so this is probably the longest code review comment >>>>> I have ever written, but the summary is: >>>>> >>>>> - I understand bumping _compiler_thread_min_stack_allowed, >>>>> but should it be solved in a different way? >>>>> - I don't understand bumping _java_thread_min_stack_allowed >>>> I plan to do a follow up change to fix this. Let's leave this >>>> discussion >>>> to that review. Here I just want to fix the NPTL issue and the basic >>>> sizing that is needed to pass the new test on ppc/s390. >>> Same question here about the simpler JDK version of the test. >>> >>> Does test/tools/launcher/TooSmallStackSize.java get run in >>> your test environments? >>> >>> >>>>> src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp >>>>> No comments. >>>>> >>>>> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp >>>>> L538: size_t os::Posix::_compiler_thread_min_stack_allowed = >>>>> 384 * K; >>>>> L539: size_t os::Posix::_java_thread_min_stack_allowed = 384 >>>>> * K; >>>>> >>>>> Same monster comment as >>>>> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >>>>> and the same summary: >>>>> >>>>> - I understand bumping _compiler_thread_min_stack_allowed, >>>>> but should it be solved in a different way? >>>>> - I don't understand bumping _java_thread_min_stack_allowed >>>>> >>>>> src/os_cpu/linux_s390/vm/os_linux_s390.cpp >>>>> L478: size_t os::Posix::_compiler_thread_min_stack_allowed = >>>>> 128 * K; >>>>> L479: size_t os::Posix::_java_thread_min_stack_allowed = 236 >>>>> * K; >>>>> Bumping _java_thread_min_stack_allowed but not bumping >>>>> _compiler_thread_min_stack_allowed. I'm confused here. >>>> The numbers are what I need to startup on the machines. 128 is just >>>> fine on the machines we have. (That's the problem of the >>>> current setup: you have to tune this compile time constant for the >>>> page size of the machine you are running on. But let's discuss this >>>> in the follow up change.) >>> OK about discussing this with a follow-up change. I guess I see >>> the compile time initialization as a "minimum setting assuming the >>> smallest page size". If we discover (at runtime) that the page >>> size is bigger, then we adjust the minimum that we need... >>> >>> Again, defer to another bug. Do we have a bug ID yet? >>> >>> >>>>> src/os_cpu/linux_sparc/vm/os_linux_sparc.cpp >>>>> No comments. >>>>> >>>>> src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>>>> No comments. >>>>> >>>>> src/os_cpu/linux_zero/vm/os_linux_zero.cpp >>>>> No comments. >>>>> >>>>> Sorry it took me so long to write this up... >>>> No matter, thanks for this thorough review! >>> You are very welcome. Thanks for being willing to dive into such >>> a complicated area (thread stack sizes)... >>> >>> Dan >>> >>> >>> >>>> Best regards, >>>> Goetz. >>>> >>>>> Dan >>>>> >>>>>> The change affecting the compier threads is in os_linux.cpp, >>>>> default_guard_size(), >>>>>> where '|| thr_type == compiler_thread' has been added. The function >>>>>> was also moved from the os_cpu files, as it's identical on all cpus. >>>>>> >>>>>> Best regards, >>>>>> Goetz. >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>>> Sent: Montag, 28. November 2016 00:25 >>>>>>> To: Lindenmaier, Goetz ; >>>>>>> 'daniel.daugherty at oracle.com' ; >>> 'hotspot- >>>>>>> runtime-dev at openjdk.java.net' >>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >>>>>>> error. >>>>>>> >>>>>>> Hi Goetz, >>>>>>> >>>>>>> On 24/11/2016 10:15 PM, Lindenmaier, Goetz wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I now edited the stuff I had proposed below: >>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >>> stackFix/webrev.03/ >>>>>>>> This includes >>>>>>>> - the NPTL fix from webrev.02 >>>>>>> Okay in principle. As discussed this only impacts non-JavaThreads >>>>>>> so the >>>>>>> change should be minimal. >>>>>>> >>>>>>>> - merging code on linux >>>>>>> Went a bit further than I had expected but if this truly isn't CPU >>>>>>> dependent code then great! >>>>>>> >>>>>>>> - not adding OS guard to compiler threads. >>>>>>> Okay in principle. IIUC we will now save the OS guard page for >>>>>>> compiler >>>>>>> thread stacks. Is that the only impact? The hotspot-compiler-dev >>>>>>> folk >>>>>>> may want to sign off on this part. >>>>>>> >>>>>>> >>>>>>> A few minor comments: >>>>>>> >>>>>>> src/os/linux/vm/os_linux.cpp >>>>>>> >>>>>>> 2854 return ((thr_type == java_thread || thr_type == >>>>>>> os::compiler_thread) ... >>>>>>> >>>>>>> os:: should be used for both types or none. >>>>>>> >>>>>>> 6153 pthread_attr_getguardsize(&attr, &guard_size); >>>>>>> >>>>>>> Can you at least verify a zero return code in an >>>>>>> assert/assert_status >>>>>>> please. >>>>>>> >>>>>>> --- >>>>>>> >>>>>>> src/os_cpu/aix_ppc/vm/os_aix_ppc.cpp >>>>>>> src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp >>>>>>> src/os_cpu/linux_s390/vm/os_linux_s390.cpp >>>>>>> >>>>>>> Are the changes to min_stack_allowed just fixing an existing bug? >>>>>>> >>>>>>> --- >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> ----- >>>>>>> >>>>>>> >>>>>>>> I think this should be pushed for this bug ID. For the other >>>>>>>> changes I'll >>>>>>>> make another bug. >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Goetz. >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Lindenmaier, Goetz >>>>>>>>> Sent: Wednesday, November 23, 2016 8:11 AM >>>>>>>>> To: David Holmes ; >>>>>>>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net >>>>>>>>> Subject: RE: RFR(M): 8169373: Work around linux NPTL stack guard >>> error. >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with >>>>>>>>>> JVMCI. >>> The >>>>>>>>>> ability for a CompilerThread to execute Java code >>>>>>>>>> (can_call_java()) is >>>>>>>>>> now a dynamic property depending on whether the current compiler >>> is >>>>> the >>>>>>>>>> JVMCI compiler. >>>>>>>>> Ah, then I should also leave space for shadow pages in the minimal >>> stack >>>>>>> size >>>>>>>>> of comiler threads. >>>>>>>>> >>>>>>>>> Do we agree on the cleanup and on leaving out the OS guard page on >>>>>>>>> compiler threads? >>>>>>>>> Then I would edit a change comprising the NPTL workaround and >>>>>>>>> these >>>>>>>>> additional changes, and split the other issue into a new bug? >>>>>>>>> I think this >>>>>>>>> will easy the reviewing. >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> Goetz. >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>>>>>> Sent: Mittwoch, 23. November 2016 02:50 >>>>>>>>>> To: Lindenmaier, Goetz ; >>>>>>>>>> daniel.daugherty at oracle.com; hotspot-runtime-dev at openjdk.java.net >>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack guard >>>>> error. >>>>>>>>>> On 22/11/2016 11:19 PM, Lindenmaier, Goetz wrote: >>>>>>>>>>> Hi Dan, >>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Daniel D. Daugherty [mailto:daniel.daugherty at oracle.com] >>>>>>>>>>>> Sent: Dienstag, 22. November 2016 14:01 >>>>>>>>>>>> To: Lindenmaier, Goetz ; David >>> Holmes >>>>>>>>>>>> ; hotspot-runtime- >>> dev at openjdk.java.net >>>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack >>>>>>>>>>>> guard >>>>>>>>> error. >>>>>>>>>>>> On 11/22/16 3:55 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>>>> Hi Dan, >>>>>>>>>>>>> >>>>>>>>>>>>> I ran into a row of issues, some errors on the platforms. >>>>>>>>>>>>> >>>>>>>>>>>>> What I meant with that comment is that >>>>>>>>>>>>> os::Linux::min_stack_allowed = >>> MAX2(os::Linux::min_stack_allowed, >>>>>>>>>>>>> >>>>>>>>>>>>> JavaThread::stack_guard_zone_size() + >>>>>>>>>>>>> >>>>>>>>>>>>> JavaThread::stack_shadow_zone_size() + >>>>>>>>>>>>> (4*BytesPerWord >>>>> COMPILER2_PRESENT(+2)) * >>>>>>> 4 >>>>>>>>> * >>>>>>>>>> K); >>>>>>>>>>>>> was executed, and min_stack_allowed used for all stacks. Now, >>>>>>>>> compiler >>>>>>>>>>>> and >>>>>>>>>>>>> vm minimum stack sizes are not increased by these sizes. >>>>>>>>>>>> Now I see what you mean. Thanks for clearing this up. >>>>>>>>>>>> >>>>>>>>>>>> I should have remembered that part of the change because we >>> went >>>>>>>>> back >>>>>>>>>>>> and forth about removing the red/yellow zone pages from the >>>>>>>>>>>> other >>>>>>>>>>>> threads. In particular, we discussed the compiler thread >>>>>>>>>>>> because it >>>>>>>>>>>> is-a JavaThread. Our conclusion was that a compiler thread >>>>>>>>>>>> doesn't >>>>>>>>>>>> execute Java bytecode so we could remove the red/yellow >>>>>>>>>>>> pages... >>>>>>>>>>> Yes, it does not execute java byte code. >>>>>>>>>> Bzzzt! Sorry Goetz and Dan but that is no longer correct with >>>>>>>>>> JVMCI. >>> The >>>>>>>>>> ability for a CompilerThread to execute Java code >>>>>>>>>> (can_call_java()) is >>>>>>>>>> now a dynamic property depending on whether the current compiler >>> is >>>>> the >>>>>>>>>> JVMCI compiler. >>>>>>>>>> >>>>>>>>>> David >>>>>>>>>> ----- >>>>>>>>>> >>>>>>>>>>> Therefore you can remove the shadow pages. There is no code >>>>>>>>>>> that >>>>>>>>>>> will bang. >>>>>>>>>>> But red/yellow pages are protected right during thread startup. >>>>>>>>>>> Therefore you must have enough space for them. >>>>>>>>>>> On ppc, we try to protect three 64K pages out of the 128K >>>>>>>>>>> compiler >>>>> stack. >>>>>>>>>>> That obviously fails. >>>>>>>>>>> >>>>>>>>>>> Therefore I propose: >>>>>>>>>>> size_t os::Posix::_java_thread_min_stack_allowed = 48 >>>>>>>>>>> * K; // >>> Set >>>>>>>>>> platform dependent. >>>>>>>>>>> in os::Posix::set_minimum_stack_sizes(): >>>>>>>>>>> _java_thread_min_stack_allowed = >>> _java_thread_min_stack_allowed >>>>> + >>>>>>>>>>> >>>>>>>>>>> JavaThread::stack_guard_zone_size() + >>>>>>>>>>> >>>>>>>>>>> JavaThread::stack_shadow_zone_size(); >>>>>>>>>>> >>>>>>>>>>> (Similar for _compiler_thread_min_stack_allowed). >>>>>>>>>>> >>>>>>>>>>> The minimal stack size is made up of three components: >>>>>>>>>>> 1. Sizes of interpreter/C1/C2 frames. Depends on HotSpot >>>>>>>>>> implementation/platform/os. >>>>>>>>>>> 2. Sizes of C++ frames: depends on C++ compiler. >>>>>>>>>>> These are fixed at compile time. >>>>>>>>>>> 3. Sizes of red/yellow/reserved/shadow pages. Depends >>>>>>>>>>> on the >>>>> system >>>>>>>>>> the >>>>>>>>>>> VM is used on. This is not fixed at compile time. >>>>>>>>>>> (Our ppc >>>>> machines >>>>>>>>>> differ >>>>>>>>>>> in page size.) >>>>>>>>>>> Therefore 3. should not be included in a compile time constant. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> And that decision allowed us to be exposed to the 64K page >>>>>>>>>>>> issue >>>>>>>>>>>> because the "extra" space isn't there anymore. >>>>>>>>>>>> >>>>>>>>>>>>> At least the _compiler_thread_min_stack_allowed should be >>>>> increased >>>>>>>>>>>>> similarly by red/yellow zone size. The compiler thread is >>>>>>>>>>>>> a Java >>>>>>>>>>>>> thread and must have space for these zones. >>>>>>>>>>>> I'm not sure that I completely agree (yet). To me, the >>>>>>>>>>>> red/yellow >>>>>>>>>>>> pages are there for Java thread stack overflow semantics. >>>>>>>>>>>> Yes, the >>>>>>>>>>>> compiler thread needs extra space when 64K pages are used, >>>>>>>>>>>> but I >>>>>>>>>>>> would prefer that we add that space via a different >>>>>>>>>>>> calculation. >>>>>>>>>>> Yes they are. But compiler threads happen tob e a subclass of >>>>>>>>>>> Java threads and use the same run() method that puts the pages >>>>>>>>>>> There. >>>>>>>>>>> I agree that they are not needed for Compiler threads, nor for >>>>>>>>>>> CodeCacheSweeperThreads. I don't really now about >>>>>>>>>>> JvmtiAgentThreads and ServiceThreads, but all of the get >>>>>>>>>>> the guard >>>>>>>>> pages >>>>>>>>>>> because they are derived from JavaThread. >>>>>>>>>>> >>>>>>>>>>>> To put it another way, I'd like to see us add extra space to >>>>>>>>>>>> solve >>>>>>>>>>>> the 64K page issue directly instead of as a side effect of the >>>>>>>>>>>> red/yellow page addition. >>>>>>>>>>> I don't understand. What do you mean by 'directly'? >>>>>>>>>>> >>>>>>>>>>>>> Also, the change added a test that is failing now. >>>>>>>>>>>> And that's a "good thing" (TM), right? :-) >>>>>>>>>>> Yes, it showed a bug and thus raised the need to fix it! :) >>>>>>>>>>> >>>>>>>>>>> Best regards, >>>>>>>>>>> Goetz. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Again, thanks for clarifying 8140520's role in this issue. >>>>>>>>>>>> >>>>>>>>>>>> Dan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Best regards, >>>>>>>>>>>>> Goetz. >>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: Daniel D. Daugherty >>> [mailto:daniel.daugherty at oracle.com] >>>>>>>>>>>>>> Sent: Montag, 21. November 2016 17:28 >>>>>>>>>>>>>> To: David Holmes ; Lindenmaier, >>> Goetz >>>>>>>>>>>>>> ; hotspot-runtime- >>>>>>>>>> dev at openjdk.java.net >>>>>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack >>> guard >>>>>>>>>> error. >>>>>>>>>>>>>> Sorry for the delayed responses to this thread. I've been on >>>>> vacation... >>>>>>>>>>>>>> One comment/query embedded below... >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 11/10/16 8:40 PM, David Holmes wrote: >>>>>>>>>>>>>>> Hi Goetz, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 11/11/2016 8:00 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>>>>>>> Hi David, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This issue is different to 6675312, see also my comment >>>>>>>>>>>>>>>> in the >>>>> bug. >>>>>>>>>>>>>>>> It appears running jtreg test >>>>>>>>>> runtime/Thread/TooSmallStackSize.java, >>>>>>>>>>>>>>>> with my patch below you can reproduce it on linuxx86_64. >>>>>>>>>>>>>>>> You >>>>> can >>>>>>>>>> not >>>>>>>>>>>>>>>> do that with 6675312. Also, I would assume there are >>>>>>>>>>>>>>>> systems >>> out >>>>>>>>>> there >>>>>>>>>>>>>>>> on x86 that uses 64-K pages, did you run the tests on >>>>>>>>>>>>>>>> these? I >>>>>>>>> would >>>>>>>>>>>>>>>> assume you get hard crashes with stack overflows in the >>>>>>>>>>>>>>>> first >>> C2 >>>>>>>>>>>>>>>> compilation if there is only 64K usable >>>>>>>>>>>>>>>> CompilerThreadStack. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> My fix does not affect Java threads, which are the largest >>> amount >>>>>>>>>>>>>>>> of threads used by the VM. It affects only the non-Java >>>>>>>>>>>>>>>> threads. >>>>>>>>>>>>>>>> It adds one page to these threads. The page does not >>>>>>>>>>>>>>>> require >>>>>>>>>> memory, >>>>>>>>>>>>>>>> as it's protected. The stack will only require more >>>>>>>>>>>>>>>> space if the >>>>>>>>> thread >>>>>>>>>>>>>>>> ran into a stack overflow before the fix as else the >>>>>>>>>>>>>>>> pages are >>> not >>>>>>>>>>>>>>>> mapped. >>>>>>>>>>>>>>>> This are stack overflows that cause hard crashes, at >>>>>>>>>>>>>>>> least on >>> ppc >>>>>>>>> the >>>>>>>>>> VM >>>>>>>>>>>>>>>> does not properly catch these stack overflows, so any setup >>>>>>>>> working >>>>>>>>>>>> now >>>>>>>>>>>>>>>> will not run into the additional space. Altogether there >>>>>>>>>>>>>>>> should >>> be >>>>>>>>> no >>>>>>>>>>>>>>>> effect on running systems besides requiring one more >>>>>>>>>>>>>>>> entry in >>>>> the >>>>>>>>>>>>>>>> page table per non-Java thread. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The problem is caused by a rather recent change (8140520: >>>>>>>>> segfault >>>>>>>>>> on >>>>>>>>>>>>>>>> solaris-amd64 >>>>>>>>>>>>>>>> with "-XX:VMThreadStackSize=1" option) which was pushed >>> after >>>>>>>>>>>>>>>> feature-close. As this was a rather recent change, it >>>>>>>>>>>>>>>> must be >>>>>>>>>>>>>>>> possible to >>>>>>>>>>>>>>>> fix this follow up issue. What else is this period in >>>>>>>>>>>>>>>> the project >>>>> good >>>>>>>>>>>>>>>> for if not fixing issues? >>>>>>>>>>>>>>> So I am seeing a number of factors here. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> First, 8140520, set: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * >>> K; >>>>>>>>>>>>>> So I'm confused by the above comment: >>>>>>>>>>>>>> >>>>>>>>>>>>>> > The problem is caused by a rather recent change >>>>>>>>>>>>>> (8140520: >>>>>>>>> segfault >>>>>>>>>>>>>> > on solaris-amd64 with "-XX:VMThreadStackSize=1" >>>>>>>>>>>>>> option) >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://cr.openjdk.java.net/~dcubed/8140520-webrev/5-jdk9-hs- >>>>>>>>>>>>>> >>>>>>>>>> open/hotspot/src/os_cpu/linux_ppc/vm/os_linux_ppc.cpp.frames.html >>>>>>>>>>>>>> shows this change: >>>>>>>>>>>>>> >>>>>>>>>>>>>> @@ -531,19 +531,17 @@ >>>>>>>>>>>>>> } >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>> ////////////////////////////////////////////////////////////////////////////// >>> >>>>>>>>>>>>>> // >>>>>>>>>>>>>> // thread stack >>>>>>>>>>>>>> >>>>>>>>>>>>>> -size_t os::Linux::min_stack_allowed = 128*K; >>>>>>>>>>>>>> +size_t os::Posix::_compiler_thread_min_stack_allowed = 128 * >>> K; >>>>>>>>>>>>>> +size_t os::Posix::_java_thread_min_stack_allowed = 128 * K; >>>>>>>>>>>>>> +size_t os::Posix::_vm_internal_thread_min_stack_allowed = >>>>>>>>>>>>>> 128 >>> * >>>>> K; >>>>>>>>>>>>>> so the existing single variable of 'min_stack_allowed' was >>>>>>>>>>>>>> replaced by three variables: >>> _compiler_thread_min_stack_allowed, >>>>>>>>>>>>>> _java_thread_min_stack_allowed, and >>>>>>>>>>>>>> _vm_internal_thread_min_stack_allowed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The old single variable and the three new variables are all >>>>>>>>>>>>>> initialized to the same value (128K) so the fix for 8140520 >>>>>>>>>>>>>> did not change stack sizes for this platform. In fact, only >>>>>>>>>>>>>> one platform had a size change (Solaris X64). >>>>>>>>>>>>>> >>>>>>>>>>>>>> So I'm confused about how the fix for 8140520 caused this >>> problem. >>>>>>>>>>>>>> Based on David's analysis below, it looks to me like this >>>>>>>>>>>>>> 64K stack >>>>>>>>>>>>>> guard page problem should also exist prior to the fix for >>> 8140520. >>>>>>>>>>>>>> Goetz, can you please explain how 8140520 caused this >>>>>>>>>>>>>> problem? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dan >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Second on linux PPC it is hardwired to use 2 guard pages: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> return 2 * page_size(); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Third, you had a pagesize of 64K. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Fourth, NPTL takes the guard space from the stack space - >>> hence >>>>>>>>> with >>>>>>>>>> 2 >>>>>>>>>>>>>>> x 64K guard, and a 128K stack it was all consumed. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In the proposed changes you now only use page_size() for the >>>>> guard, >>>>>>>>>> so >>>>>>>>>>>>>>> that alone would have fixed the observed problem. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But in addition you want to address the NPTL problem by >>>>>>>>>>>>>>> adding >>>>>>>>> back >>>>>>>>>>>>>>> the guard space to the stack size requested. That alone >>>>>>>>>>>>>>> would >>> also >>>>>>>>>>>>>>> have fixed the observed problem. :) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But in addition you have increased the minimum stack size: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ! size_t os::Posix::_compiler_thread_min_stack_allowed = >>>>>>>>>>>>>>> 192 * >>> K; >>>>>>>>>>>>>>> which again, on its own would have fixed the original >>>>>>>>>>>>>>> problem. >>> :) >>>>>>>>>>>>>>> Did you really intend to increase the real minimum stack >>>>>>>>>>>>>>> from >>>>> 128K >>>>>>>>> to >>>>>>>>>>>>>>> 256K ? (on a 64K page system) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Focusing simply on the shared code change to adjust the >>> requested >>>>>>>>>>>>>>> stacksize by the amount of guard space (if any), this >>>>>>>>>>>>>>> does not >>> seem >>>>>>>>>>>>>>> unreasonable. As you note it is restricted to >>>>>>>>>>>>>>> non-JavaThreads >>> and >>>>>>>>> only >>>>>>>>>>>>>>> adds a page to reserved stack space. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My only query now is whether the minimum stacksize detection >>>>> logic >>>>>>>>>>>>>>> will correctly report the real minimum stack size (taking >>>>>>>>>>>>>>> into >>>>> account >>>>>>>>>>>>>>> the need for the guard page) ? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> David >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So I really think this issue should be fixed. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>> Goetz. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>>> From: David Holmes [mailto:david.holmes at oracle.com] >>>>>>>>>>>>>>>>> Sent: Thursday, November 10, 2016 10:02 PM >>>>>>>>>>>>>>>>> To: Lindenmaier, Goetz ; >>> hotspot- >>>>>>>>>>>>>> runtime- >>>>>>>>>>>>>>>>> dev at openjdk.java.net >>>>>>>>>>>>>>>>> Subject: Re: RFR(M): 8169373: Work around linux NPTL stack >>>>>>>>> guard >>>>>>>>>>>> error. >>>>>>>>>>>>>>>>> Hi Goetz, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> As per the bug report, this issue was already known >>> (6675312) >>>>> and >>>>>>>>>> we >>>>>>>>>>>>>>>>> chose not to try and address it due to no reported >>>>>>>>>>>>>>>>> issues at >>> the >>>>>>>>>> time. >>>>>>>>>>>>>>>>> While I see that you have encountered an issue (is it >>>>>>>>>>>>>>>>> real or >>>>>>>>>>>>>>>>> fabricated?) I think this change is too intrusive to be >>>>>>>>>>>>>>>>> applied >>> at >>>>> this >>>>>>>>>>>>>>>>> stage of the JDK 9 release cycle, as it will change the >>>>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>>>> requirements of every application running on Linux. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> David >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 11/11/2016 1:58 AM, Lindenmaier, Goetz wrote: >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Please review this change. I please need a sponsor: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> http://cr.openjdk.java.net/~goetz/wr16/8169373-ppc- >>>>>>>>>>>>>> stackFix/webrev.01/ >>>>>>>>>>>>>>>>>> In the Linux NPTL pthread implementation the guard size >>>>>>>>>> mechanism >>>>>>>>>>>>>>>>>> is not >>>>>>>>>>>>>>>>>> implemented properly. The posix standard requires to add >>> the >>>>>>>>>> size >>>>>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>>>>> guard pages to the stack size, instead Linux takes the >>>>>>>>>>>>>>>>>> space >>> out >>>>>>>>> of >>>>>>>>>>>>>>>>>> 'stacksize'. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The Posix standard >>>>>>>>>>>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ >>>>>>>>>>>>>>>>>> says "the implementation allocates extra memory >>>>>>>>>>>>>>>>>> at the >>>>>>>>>> overflow >>>>>>>>>>>>>>>>>> end of >>>>>>>>>>>>>>>>>> the stack". The linux man page >>>>>>>>>>>>>>>>>> https://linux.die.net/man/3/pthread_attr_setguardsize >>>>>>>>>>>>>>>>>> says >>> "As >>>>>>>>>> at >>>>>>>>>>>>>>>>>> glibc >>>>>>>>>>>>>>>>>> 2.8, the NPTL threading implementation includes the guard >>>>> area >>>>>>>>>>>> within >>>>>>>>>>>>>>>>>> the stack size allocation, rather than allocating >>>>>>>>>>>>>>>>>> extra space >>> at >>>>>>>>>>>>>>>>>> the end >>>>>>>>>>>>>>>>>> of the stack, as POSIX.1 requires". >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I encounter this problem in >>>>>>>>>> runtime/Thread/TooSmallStackSize.java >>>>>>>>>>>>>>>>>> on ppc >>>>>>>>>>>>>>>>>> with 64K pages. _compiler_thread_min_stack_allowed is >>> 128K >>>>>>>>> on >>>>>>>>>>>> ppc, >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> ppc specifies two OS guard pages. The VM crashes in >>> pthread >>>>>>>>>>>> creation >>>>>>>>>>>>>>>>>> because there is no usable space in the thread stack >>>>>>>>>>>>>>>>>> after >>>>>>>>>> allocating >>>>>>>>>>>>>>>>>> the guard pages. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> But TooSmallStackSize.java requires that the VM comes up >>>>> with >>>>>>>>>> the >>>>>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>>>>> size mentioned in the error message. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This fix adapts the requested stack size on Linux by >>>>>>>>>>>>>>>>>> the size >>> of >>>>>>>>> the >>>>>>>>>>>>>>>>>> guard pages to mimick proper behaviour, see change to >>>>>>>>>>>> os_linux.cpp. >>>>>>>>>>>>>>>>>> The change also streamlines usage of stack_guard_page on >>>>>>>>>> linuxppc, >>>>>>>>>>>>>>>>>> linuxppcle, aixppc and linuxs390. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> To reproduce the error on linux_x86_64, apply below patch >>>>> and >>>>>>>>>> call >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> VM with -XX:CompilerThreadStackSize=64. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm still exploring why I had to choose such big compiler >>> stacks >>>>>>>>> on >>>>>>>>>>>>>>>>>> ppc >>>>>>>>>>>>>>>>>> to get -version passing, but I wanted to send the RFR >>>>>>>>>>>>>>>>>> now as >>>>>>>>>> people >>>>>>>>>>>>>>>>>> obviously looked at the bug I opened (Thanks David!). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Goetz. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> diff -r b7ae012c55c3 >>>>> src/os_cpu/linux_x86/vm/os_linux_x86.cpp >>>>>>>>>>>>>>>>>> --- a/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Mon Nov >>> 07 >>>>>>>>>>>> 12:37:28 >>>>>>>>>>>>>>>>> 2016 >>>>>>>>>>>>>>>>>> +0100 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> +++ b/src/os_cpu/linux_x86/vm/os_linux_x86.cpp Thu Nov >>> 10 >>>>>>>>>>>>>> 16:52:17 >>>>>>>>>>>>>>>>> 2016 >>>>>>>>>>>>>>>>>> +0100 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> @@ -701,7 +701,7 @@ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> size_t os::Linux::default_guard_size(os::ThreadType >>> thr_type) { >>>>>>>>>>>>>>>>>> // Creating guard page is very expensive. Java >>>>>>>>>>>>>>>>>> thread >>> has >>>>>>>>>> HotSpot >>>>>>>>>>>>>>>>>> // guard page, only enable glibc guard page for >>>>>>>>>>>>>>>>>> non-Java >>>>>>>>>> threads. >>>>>>>>>>>>>>>>>> - return (thr_type == java_thread ? 0 : page_size()); >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> + return (thr_type == java_thread ? 0 : 64*K); >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> // Java thread: >>>>>>>>>>>>>>>>>> >