From david.holmes at oracle.com Fri May 1 02:34:19 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 1 May 2020 12:34:19 +1000 Subject: RFR(S) : 8243429 : use reproducible random in :vmTestbase_nsk_stress In-Reply-To: <82A98A70-3E9B-4A63-BE6D-293EF64A5C8B@oracle.com> References: <82A98A70-3E9B-4A63-BE6D-293EF64A5C8B@oracle.com> Message-ID: <639de8f9-81f6-dc1b-1bc6-260fbeba68dc@oracle.com> Hi Igor, On 1/05/2020 3:15 am, Igor Ignatyev wrote: > http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00 >> 82 lines changed: 41 ins; 4 del; 37 mod; The patch files in your webrevs are not generating properly due to the use of mq. can you fix that? > Hi all, > > could you please review this small patch? > from JBS: >> this subtask is to use j.t.l.Utils.getRandomInstance() as a random number generator, where applicable, in :vmTestbase_nsk_stress test group and marking the tests which make use of "randomness" with a proper k/w. I'm a bit confused about the strategy here: vmTestbase/nsk/stress/except/except012.java - Random random = new Random(0); + Random random = new Random(42); but: vmTestbase/nsk/stress/jni/jnistress002.java - static Random myRandom = new Random(); + Random myRandom = new Random(Utils.getRandomInstance().nextLong()); why are these random instances seeded completely differently? And looking at: vmTestbase/nsk/stress/jni/gclocker/gcl001.java you changed it to use the LocalRandom utility rather than creating its own random instance. Why was that applicable here but not in other tests? The changes from Math.random to a local RNG from getRandomInstance() seem okay- but again why that versus LocalRandom? Do we even need LocalRandom? Thanks, David > testing: : vmTestbase_nsk_stress test group > JBS: https://bugs.openjdk.java.net/browse/JDK-8243429 > webrevs: > - code changes: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.code >> 58 lines changed: 41 ins; 4 del; 13 mod; > > - adding k/w: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.kw >> 11 lines changed: 0 ins; 0 del; 11 mod; > > - full: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00 >> 82 lines changed: 41 ins; 4 del; 37 mod; > > > Thanks, > -- Igor > From igor.ignatyev at oracle.com Fri May 1 03:27:57 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Thu, 30 Apr 2020 20:27:57 -0700 Subject: RFR(S) : 8243429 : use reproducible random in :vmTestbase_nsk_stress In-Reply-To: <639de8f9-81f6-dc1b-1bc6-260fbeba68dc@oracle.com> References: <82A98A70-3E9B-4A63-BE6D-293EF64A5C8B@oracle.com> <639de8f9-81f6-dc1b-1bc6-260fbeba68dc@oracle.com> Message-ID: <2D6B045B-1694-48B1-900D-ADBA9BD1876B@oracle.com> Hi David, > On Apr 30, 2020, at 7:34 PM, David Holmes wrote: > > Hi Igor, > > On 1/05/2020 3:15 am, Igor Ignatyev wrote: >> http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00 >>> 82 lines changed: 41 ins; 4 del; 37 mod; > > The patch files in your webrevs are not generating properly due to the use of mq. can you fix that? unfortunately, I've already squashed all the patches together, so I was able to upload only the final version -- http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00/8243429.patch > >> Hi all, >> could you please review this small patch? >> from JBS: >>> this subtask is to use j.t.l.Utils.getRandomInstance() as a random number generator, where applicable, in :vmTestbase_nsk_stress test group and marking the tests which make use of "randomness" with a proper k/w. > > I'm a bit confused about the strategy here: > > vmTestbase/nsk/stress/except/except012.java > > - Random random = new Random(0); > + Random random = new Random(42); > > but: > > vmTestbase/nsk/stress/jni/jnistress002.java > > - static Random myRandom = new Random(); > + Random myRandom = new Random(Utils.getRandomInstance().nextLong()); > > why are these random instances seeded completely differently? because they were different before my patch, except012 used a fixed seed (so it doesn't qualify to be marked w/ randomness k/w), and jnistress002 used a "random" seed; after the patch except012 still uses a fixed (but a "well-known" and easier to grep) seed value, and jnistress002 uses random, yet reproducible/configurable seed. I've explained that in my previous RFRs for hotspot tests outside of vmTestbase, e.g. https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037828.html also, in jnistress002, we can't use Utils.getRandomInstance() as there would multiple threads (JNIter002) using the same RNG. we could change it to use LocalRandom, I must admit I was doing different parts of these patches on different days, and given their tediousness I might have became less consistent in my choices b/w using LocalRandom and using a "threadlocal" instance of Random. I, personally, think we need to switch to LocalRandom in all such cases, or even better to replace Utils.getRandomInstance() w/ something akin to LocalRandom, but such change is somewhat orthogonal to the series of 'use reproducible random' patches, so I'd prefer to do it separately. > > And looking at: > > vmTestbase/nsk/stress/jni/gclocker/gcl001.java > > you changed it to use the LocalRandom utility rather than creating its own random instance. Why was that applicable here but not in other tests? > > The changes from Math.random to a local RNG from getRandomInstance() seem okay- but again why that versus LocalRandom? Do we even need LocalRandom? (although I guess you've already figured that out from other patches, I'll answer here for completeness) vmTestbase/nsk/stress/jni/gclocker/gcl001.java used the same instance of RNG in multiple threads, hence it would be really reproducible if we just switched to getRandomInstance(), LocalRandom is needed exactly for that -- provide reproducible random in multi-thread use cases. -- Igor > > Thanks, > David > >> testing: : vmTestbase_nsk_stress test group >> JBS: https://bugs.openjdk.java.net/browse/JDK-8243429 >> webrevs: >> - code changes: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.code >>> 58 lines changed: 41 ins; 4 del; 13 mod; >> - adding k/w: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.kw >>> 11 lines changed: 0 ins; 0 del; 11 mod; >> - full: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00 >>> 82 lines changed: 41 ins; 4 del; 37 mod; >> Thanks, >> -- Igor From david.holmes at oracle.com Fri May 1 04:14:52 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 1 May 2020 04:14:52 +0000 (UTC) Subject: RFR(S) : 8243429 : use reproducible random in :vmTestbase_nsk_stress In-Reply-To: <2D6B045B-1694-48B1-900D-ADBA9BD1876B@oracle.com> References: <82A98A70-3E9B-4A63-BE6D-293EF64A5C8B@oracle.com> <639de8f9-81f6-dc1b-1bc6-260fbeba68dc@oracle.com> <2D6B045B-1694-48B1-900D-ADBA9BD1876B@oracle.com> Message-ID: <37330d42-0b50-a6f0-2072-6802ed690f3a@oracle.com> On 1/05/2020 1:27 pm, Igor Ignatyev wrote: > Hi David, > >> On Apr 30, 2020, at 7:34 PM, David Holmes > > wrote: >> >> Hi Igor, >> >> On 1/05/2020 3:15 am, Igor Ignatyev wrote: >>> http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00 >>>> 82 lines changed: 41 ins; 4 del; 37 mod; >> >> The patch files in your webrevs are not generating properly due to the >> use of mq. can you fix that? > unfortunately, I've already squashed all the patches together, so I was > able to upload only the final version -- > http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00/8243429.patch >> >>> Hi all, >>> could you please review this small patch? >>> from JBS: >>>> this subtask is to use j.t.l.Utils.getRandomInstance() as a random >>>> number generator, where applicable, in :vmTestbase_nsk_stress test >>>> group and marking the tests which make use of "randomness" with a >>>> proper k/w. >> >> I'm a bit confused about the strategy here: >> >> vmTestbase/nsk/stress/except/except012.java >> >> - ???????????Random random = new Random(0); >> + ???????????Random random = new Random(42); >> >> but: >> >> vmTestbase/nsk/stress/jni/jnistress002.java >> >> - ???static Random myRandom = new Random(); >> + ???Random myRandom = new Random(Utils.getRandomInstance().nextLong()); >> >> why are these random instances seeded completely differently? > > because they were different before my patch, except012 used a fixed seed > (so it doesn't qualify to be marked w/ randomness k/w), and jnistress002 > used a "random" seed; after the patch except012 still uses a fixed (but > a "well-known" and easier to grep) seed value, and jnistress002 uses > random, yet reproducible/configurable seed. I've explained that in my > previous RFRs for hotspot tests outside of vmTestbase, e.g. > https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037828.html Not everyone on hotspot-dev is also on hotspot-compiler-dev. I'm trying to understand why we still have different usage patterns. IIUC there's no reason that except012 couldn't be seeded the same way as jnistress002 - right? Then it would qualify for "randomness" keyword. What is it about any given test that determines whether we seed the RNG differently on each test run versus fixing it at 42 (for example)? > > also, in jnistress002, we can't use Utils.getRandomInstance() as there > would multiple threads (JNIter002) using the same RNG. we could change > it to use LocalRandom, I must admit I was doing different parts of these > patches on different days, and given their tediousness I might have > became less consistent in my choices b/w using LocalRandom and using a > "threadlocal" instance of Random. I, personally, think we need to switch > to LocalRandom in all such cases, or even better to replace > Utils.getRandomInstance() w/ something akin to LocalRandom, but such > change is somewhat orthogonal to the series of 'use reproducible random' > patches, so I'd prefer to do it separately. Sure but it remains confusing to see such a variety of cases. Utils.getRandomInstance() is a shared RNG so you never want to use that directly in different threads if you truly want reproducability - the most you can use it for is the source of seeds for other RNGs - but even that is non-deterministic is different threads are involved. So your changes from Math.random are not going to be reproducible due to different threads being involved. >> >> And looking at: >> >> vmTestbase/nsk/stress/jni/gclocker/gcl001.java >> >> you changed it to use the LocalRandom utility rather than creating its >> own random instance. Why was that applicable here but not in other tests? >> >> The changes from Math.random to a local RNG from getRandomInstance() >> seem okay- but again why that versus LocalRandom? Do we even need >> LocalRandom? > > (although I guess you've already figured that out from other patches, > I'll answer here for completeness) > > vmTestbase/nsk/stress/jni/gclocker/gcl001.java used the same instance of > RNG in multiple threads, hence it would be really reproducible if we > just switched to getRandomInstance(), LocalRandom is needed exactly for > that -- provide reproducible random in multi-thread use cases. I'm not sure if you wrote that exactly as intended. Utils.getRandomInstance() does not produce a RNG that gives reproducible results across threads, so we should, it seems, never use it directly in any multi-threaded stress test. Secondly, LocalRandom is a wrapper for ThreadLocalRandom.current() _but_ you have no control over the seed so seems not so useful for achieving reproducability! David ----- > > -- Igor > >> >> Thanks, >> David >> >>> testing: : vmTestbase_nsk_stress test group >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8243429 >>> webrevs: >>> - code changes: >>> http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.code >>>> 58 lines changed: 41 ins; 4 del; 13 mod; >>> - adding k/w: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.kw >>>> 11 lines changed: 0 ins; 0 del; 11 mod; >>> - full: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00 >>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>> Thanks, >>> -- Igor > From igor.ignatyev at oracle.com Fri May 1 04:30:27 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Thu, 30 Apr 2020 21:30:27 -0700 Subject: RFR(S) : 8243429 : use reproducible random in :vmTestbase_nsk_stress In-Reply-To: <37330d42-0b50-a6f0-2072-6802ed690f3a@oracle.com> References: <82A98A70-3E9B-4A63-BE6D-293EF64A5C8B@oracle.com> <639de8f9-81f6-dc1b-1bc6-260fbeba68dc@oracle.com> <2D6B045B-1694-48B1-900D-ADBA9BD1876B@oracle.com> <37330d42-0b50-a6f0-2072-6802ed690f3a@oracle.com> Message-ID: <63DACE82-1258-4F30-8889-3EAF7848FC05@oracle.com> > On Apr 30, 2020, at 9:14 PM, David Holmes wrote: > > On 1/05/2020 1:27 pm, Igor Ignatyev wrote: >> Hi David, >>> On Apr 30, 2020, at 7:34 PM, David Holmes >> wrote: >>> >>> Hi Igor, >>> >>> On 1/05/2020 3:15 am, Igor Ignatyev wrote: >>>> http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00 >>>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>> >>> The patch files in your webrevs are not generating properly due to the use of mq. can you fix that? >> unfortunately, I've already squashed all the patches together, so I was able to upload only the final version -- http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00/8243429.patch >>> >>>> Hi all, >>>> could you please review this small patch? >>>> from JBS: >>>>> this subtask is to use j.t.l.Utils.getRandomInstance() as a random number generator, where applicable, in :vmTestbase_nsk_stress test group and marking the tests which make use of "randomness" with a proper k/w. >>> >>> I'm a bit confused about the strategy here: >>> >>> vmTestbase/nsk/stress/except/except012.java >>> >>> - Random random = new Random(0); >>> + Random random = new Random(42); >>> >>> but: >>> >>> vmTestbase/nsk/stress/jni/jnistress002.java >>> >>> - static Random myRandom = new Random(); >>> + Random myRandom = new Random(Utils.getRandomInstance().nextLong()); >>> >>> why are these random instances seeded completely differently? >> because they were different before my patch, except012 used a fixed seed (so it doesn't qualify to be marked w/ randomness k/w), and jnistress002 used a "random" seed; after the patch except012 still uses a fixed (but a "well-known" and easier to grep) seed value, and jnistress002 uses random, yet reproducible/configurable seed. I've explained that in my previous RFRs for hotspot tests outside of vmTestbase, e.g. https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037828.html > > Not everyone on hotspot-dev is also on hotspot-compiler-dev. true, yet almost every hotspot-X-dev alias received their porting of 'use reproducible random in hotspot X tests' a couple weeks ago, which all have references to the same email thread ;) . in any case, I apologize I should have provided enough context in each of new RFR. > > I'm trying to understand why we still have different usage patterns. IIUC there's no reason that except012 couldn't be seeded the same way as jnistress002 - right? Then it would qualify for "randomness" keyword. What is it about any given test that determines whether we seed the RNG differently on each test run versus fixing it at 42 (for example)? it's more the question for the author of particular test. I didn't plan to (significantly) change behavior of the tests, so I tried not to convert any non "randomness" tests to "randomness" tests. > >> also, in jnistress002, we can't use Utils.getRandomInstance() as there would multiple threads (JNIter002) using the same RNG. we could change it to use LocalRandom, I must admit I was doing different parts of these patches on different days, and given their tediousness I might have became less consistent in my choices b/w using LocalRandom and using a "threadlocal" instance of Random. I, personally, think we need to switch to LocalRandom in all such cases, or even better to replace Utils.getRandomInstance() w/ something akin to LocalRandom, but such change is somewhat orthogonal to the series of 'use reproducible random' patches, so I'd prefer to do it separately. > > Sure but it remains confusing to see such a variety of cases. Utils.getRandomInstance() is a shared RNG so you never want to use that directly in different threads if you truly want reproducability - the most you can use it for is the source of seeds for other RNGs - but even that is non-deterministic is different threads are involved. right, that's why I think it's better/safer to use something similar to LocalRandom in all tests w/o thinking if it's multi-threaded or not. > So your changes from Math.random are not going to be reproducible due to different threads being involved. really? could you please point me to the test/tests which used Math.random and now have multiple threads using the same instance of Random? > > >>> >>> And looking at: >>> >>> vmTestbase/nsk/stress/jni/gclocker/gcl001.java >>> >>> you changed it to use the LocalRandom utility rather than creating its own random instance. Why was that applicable here but not in other tests? >>> >>> The changes from Math.random to a local RNG from getRandomInstance() seem okay- but again why that versus LocalRandom? Do we even need LocalRandom? >> (although I guess you've already figured that out from other patches, I'll answer here for completeness) >> vmTestbase/nsk/stress/jni/gclocker/gcl001.java used the same instance of RNG in multiple threads, hence it would be really reproducible if we just switched to getRandomInstance(), LocalRandom is needed exactly for that -- provide reproducible random in multi-thread use cases. > > I'm not sure if you wrote that exactly as intended. Utils.getRandomInstance() does not produce a RNG that gives reproducible results across threads, so we should, it seems, never use it directly in any multi-threaded stress test. sorry, I missed 'not', it should have been 'hence it would not be really reproducible' > Secondly, LocalRandom is a wrapper for ThreadLocalRandom.current() _but_ you have no control over the seed so seems not so useful for achieving reproducability! I guess you haven't updated your workspace, LocalRandom is now (after 8242314) using ThreadLocal: > private final static ThreadLocal random = new ThreadLocal<>() { > protected Random initialValue() { > // each thread gets its own seed independendly on the order they > // use LocalRandom, yet it depends on the order threads are created. > // main thread uses original seed > return new Random(Utils.SEED ^ (Thread.currentThread().getId() - 1)); > } > }; -- Igor > > David > ----- > >> -- Igor >>> >>> Thanks, >>> David >>> >>>> testing: : vmTestbase_nsk_stress test group >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8243429 >>>> webrevs: >>>> - code changes: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.code >>>>> 58 lines changed: 41 ins; 4 del; 13 mod; >>>> - adding k/w: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.kw >>>>> 11 lines changed: 0 ins; 0 del; 11 mod; >>>> - full: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00 >>>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>>> Thanks, >>>> -- Igor From david.holmes at oracle.com Fri May 1 04:52:20 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 1 May 2020 14:52:20 +1000 Subject: RFR(S) : 8243429 : use reproducible random in :vmTestbase_nsk_stress In-Reply-To: <63DACE82-1258-4F30-8889-3EAF7848FC05@oracle.com> References: <82A98A70-3E9B-4A63-BE6D-293EF64A5C8B@oracle.com> <639de8f9-81f6-dc1b-1bc6-260fbeba68dc@oracle.com> <2D6B045B-1694-48B1-900D-ADBA9BD1876B@oracle.com> <37330d42-0b50-a6f0-2072-6802ed690f3a@oracle.com> <63DACE82-1258-4F30-8889-3EAF7848FC05@oracle.com> Message-ID: <381b3793-07cd-0c72-f946-87061eab20e4@oracle.com> On 1/05/2020 2:30 pm, Igor Ignatyev wrote: >> On Apr 30, 2020, at 9:14 PM, David Holmes > > wrote: >> >> On 1/05/2020 1:27 pm, Igor Ignatyev wrote: >>> Hi David, >>>> On Apr 30, 2020, at 7:34 PM, David Holmes >>> > wrote: >>>> >>>> Hi Igor, >>>> >>>> On 1/05/2020 3:15 am, Igor Ignatyev wrote: >>>>> http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00 >>>>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>>> >>>> The patch files in your webrevs are not generating properly due to >>>> the use of mq. can you fix that? >>> unfortunately, I've already squashed all the patches together, so I >>> was able to upload only the final version -- >>> http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00/8243429.patch >>>> >>>>> Hi all, >>>>> could you please review this small patch? >>>>> from JBS: >>>>>> this subtask is to use j.t.l.Utils.getRandomInstance() as a random >>>>>> number generator, where applicable, in :vmTestbase_nsk_stress test >>>>>> group and marking the tests which make use of "randomness" with a >>>>>> proper k/w. >>>> >>>> I'm a bit confused about the strategy here: >>>> >>>> vmTestbase/nsk/stress/except/except012.java >>>> >>>> - ???????????Random random = new Random(0); >>>> + ???????????Random random = new Random(42); >>>> >>>> but: >>>> >>>> vmTestbase/nsk/stress/jni/jnistress002.java >>>> >>>> - ???static Random myRandom = new Random(); >>>> + ???Random myRandom = new Random(Utils.getRandomInstance().nextLong()); >>>> >>>> why are these random instances seeded completely differently? >>> because they were different before my patch, except012 used a fixed >>> seed (so it doesn't qualify to be marked w/ randomness k/w), and >>> jnistress002 used a "random" seed; after the patch except012 still >>> uses a fixed (but a "well-known" and easier to grep) seed value, and >>> jnistress002 uses random, yet reproducible/configurable seed. I've >>> explained that in my previous RFRs for hotspot tests outside of >>> vmTestbase, e.g. >>> https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037828.html >> >> Not everyone on hotspot-dev is also on hotspot-compiler-dev. > true, yet almost every hotspot-X-dev alias received their porting of > 'use reproducible random in hotspot X tests' a couple weeks ago, which > all have references to the same email thread ;) . in any case, I > apologize I should have provided enough context in each of new RFR. > >> >> I'm trying to understand why we still have different usage patterns. >> IIUC there's no reason that except012 couldn't be seeded the same way >> as jnistress002 - right? Then it would qualify for "randomness" >> keyword. What is it about any given test that determines whether we >> seed the RNG differently on each test run versus fixing it at 42 (for >> example)? > it's more the question for the author of particular test. I didn't plan > to (significantly) change behavior of the tests, so I tried not to > convert any non "randomness" tests to "randomness" tests. Okay. But making changes in this area it just draws attention to the different way we do things, and it is n't obvious why there are differnet ways. >> >>> also, in jnistress002, we can't use Utils.getRandomInstance() as >>> there would multiple threads (JNIter002) using the same RNG. we could >>> change it to use LocalRandom, I must admit I was doing different >>> parts of these patches on different days, and given their tediousness >>> I might have became less consistent in my choices b/w using >>> LocalRandom and using a "threadlocal" instance of Random. I, >>> personally, think we need to switch to LocalRandom in all such cases, >>> or even better to replace Utils.getRandomInstance() w/ something akin >>> to LocalRandom, but such change is somewhat orthogonal to the series >>> of 'use reproducible random' patches, so I'd prefer to do it separately. >> >> Sure but it remains confusing to see such a variety of cases. >> Utils.getRandomInstance() is a shared RNG so you never want to use >> that directly in different threads if you truly want reproducability - >> the most you can use it for is the source of seeds for other RNGs - >> but even that is non-deterministic is different threads are involved. > right, that's why I think it's better/safer to use something similar to > LocalRandom in all tests w/o thinking if it's multi-threaded or not. > >> So your changes from Math.random are not going to be reproducible due >> to different threads being involved. > really? could you please point me to the test/tests which used > Math.random and now have multiple threads using the same instance of > Random? Apologies. On examining the numeric* tests in detail it seems the RNG is only used during the setup phase. >> >>>> >>>> And looking at: >>>> >>>> vmTestbase/nsk/stress/jni/gclocker/gcl001.java >>>> >>>> you changed it to use the LocalRandom utility rather than creating >>>> its own random instance. Why was that applicable here but not in >>>> other tests? >>>> >>>> The changes from Math.random to a local RNG from getRandomInstance() >>>> seem okay- but again why that versus LocalRandom? Do we even need >>>> LocalRandom? >>> (although I guess you've already figured that out from other patches, >>> I'll answer here for completeness) >>> vmTestbase/nsk/stress/jni/gclocker/gcl001.java used the same instance >>> of RNG in multiple threads, hence it would be really reproducible if >>> we just switched to getRandomInstance(), LocalRandom is needed >>> exactly for that -- provide reproducible random in multi-thread use >>> cases. >> >> I'm not sure if you wrote that exactly as intended. >> Utils.getRandomInstance() does not produce a RNG that gives >> reproducible results across threads, so we should, it seems, never use >> it directly in any multi-threaded stress test. > sorry, I missed 'not', it should have been 'hence it would not be really > reproducible' > >> Secondly, LocalRandom is a wrapper for ThreadLocalRandom.current() >> _but_ you have no control over the seed so seems not so useful for >> achieving reproducability! > I guess you haven't updated your workspace, LocalRandom is now (after > 8242314) using ThreadLocal: > >> private final static ThreadLocal random = new ThreadLocal<>() { >> ? ? ? ??protected Random initialValue() { >> ? ? ? ? ? ??// each thread gets its own seed independendly on the >> order they >> ? ? ? ? ? ??// use LocalRandom, yet it depends on the order threads >> are created. >> ? ? ? ? ? ??// main thread uses original seed >> ? ? ? ? ? ??return new Random(Utils.SEED ^ >> (Thread.currentThread().getId() - 1)); >> ? ? ? ??} >> ? ??}; Ah yes - sorry - my local repo on my laptop (only used for code browsing) was missing that change. Okay that makes a big difference :) However I will note that using the thread ID as part of the seed may not lead to reproducible results if the thread creation itself is not deterministic/reproducible. Thanks, David > -- Igor > >> >> David >> ----- >> >>> -- Igor >>>> >>>> Thanks, >>>> David >>>> >>>>> testing: : vmTestbase_nsk_stress test group >>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8243429 >>>>> webrevs: >>>>> - code changes: >>>>> http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.code >>>>>> 58 lines changed: 41 ins; 4 del; 13 mod; >>>>> - adding k/w: >>>>> http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.kw >>>>>> 11 lines changed: 0 ins; 0 del; 11 mod; >>>>> - full: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00 >>>>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>>>> Thanks, >>>>> -- Igor > From igor.ignatyev at oracle.com Fri May 1 05:20:52 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Thu, 30 Apr 2020 22:20:52 -0700 Subject: RFR(S) : 8243429 : use reproducible random in :vmTestbase_nsk_stress In-Reply-To: <381b3793-07cd-0c72-f946-87061eab20e4@oracle.com> References: <82A98A70-3E9B-4A63-BE6D-293EF64A5C8B@oracle.com> <639de8f9-81f6-dc1b-1bc6-260fbeba68dc@oracle.com> <2D6B045B-1694-48B1-900D-ADBA9BD1876B@oracle.com> <37330d42-0b50-a6f0-2072-6802ed690f3a@oracle.com> <63DACE82-1258-4F30-8889-3EAF7848FC05@oracle.com> <381b3793-07cd-0c72-f946-87061eab20e4@oracle.com> Message-ID: <5C13C637-62A4-418C-92A5-D1936CB9AF74@oracle.com> > On Apr 30, 2020, at 9:52 PM, David Holmes wrote: > > On 1/05/2020 2:30 pm, Igor Ignatyev wrote: >>> On Apr 30, 2020, at 9:14 PM, David Holmes >> wrote: >>> >>> On 1/05/2020 1:27 pm, Igor Ignatyev wrote: >>>> Hi David, >>>>> On Apr 30, 2020, at 7:34 PM, David Holmes > wrote: >>>>> >>>>> Hi Igor, >>>>> >>>>> On 1/05/2020 3:15 am, Igor Ignatyev wrote: >>>>>> http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00 >>>>>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>>>> >>>>> The patch files in your webrevs are not generating properly due to the use of mq. can you fix that? >>>> unfortunately, I've already squashed all the patches together, so I was able to upload only the final version -- http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00/8243429.patch >>>>> >>>>>> Hi all, >>>>>> could you please review this small patch? >>>>>> from JBS: >>>>>>> this subtask is to use j.t.l.Utils.getRandomInstance() as a random number generator, where applicable, in :vmTestbase_nsk_stress test group and marking the tests which make use of "randomness" with a proper k/w. >>>>> >>>>> I'm a bit confused about the strategy here: >>>>> >>>>> vmTestbase/nsk/stress/except/except012.java >>>>> >>>>> - Random random = new Random(0); >>>>> + Random random = new Random(42); >>>>> >>>>> but: >>>>> >>>>> vmTestbase/nsk/stress/jni/jnistress002.java >>>>> >>>>> - static Random myRandom = new Random(); >>>>> + Random myRandom = new Random(Utils.getRandomInstance().nextLong()); >>>>> >>>>> why are these random instances seeded completely differently? >>>> because they were different before my patch, except012 used a fixed seed (so it doesn't qualify to be marked w/ randomness k/w), and jnistress002 used a "random" seed; after the patch except012 still uses a fixed (but a "well-known" and easier to grep) seed value, and jnistress002 uses random, yet reproducible/configurable seed. I've explained that in my previous RFRs for hotspot tests outside of vmTestbase, e.g. https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037828.html >>> >>> Not everyone on hotspot-dev is also on hotspot-compiler-dev. >> true, yet almost every hotspot-X-dev alias received their porting of 'use reproducible random in hotspot X tests' a couple weeks ago, which all have references to the same email thread ;) . in any case, I apologize I should have provided enough context in each of new RFR. >>> >>> I'm trying to understand why we still have different usage patterns. IIUC there's no reason that except012 couldn't be seeded the same way as jnistress002 - right? Then it would qualify for "randomness" keyword. What is it about any given test that determines whether we seed the RNG differently on each test run versus fixing it at 42 (for example)? >> it's more the question for the author of particular test. I didn't plan to (significantly) change behavior of the tests, so I tried not to convert any non "randomness" tests to "randomness" tests. > > Okay. But making changes in this area it just draws attention to the different way we do things, and it is n't obvious why there are differnet ways. I guess that some people decided to use fixed seed so their tests are more-or-less stable to themselves and results from different runs can be compared; yet others decided that they would like their tests to cover different paths in different runs. and I really don't know which solution is better, but I believe that solution which allows you to get the best of both is the best. that's to say, I'll definitely file a task to look into all places we use fixed seed in hotspot tests and see if it makes sense to switch to a "random" seed value, and will work on it as part of switching to CI-run/build-based seed or, as you called it in another thread, "one blessed seed". thank you for all these discussions! > >>> >>>> also, in jnistress002, we can't use Utils.getRandomInstance() as there would multiple threads (JNIter002) using the same RNG. we could change it to use LocalRandom, I must admit I was doing different parts of these patches on different days, and given their tediousness I might have became less consistent in my choices b/w using LocalRandom and using a "threadlocal" instance of Random. I, personally, think we need to switch to LocalRandom in all such cases, or even better to replace Utils.getRandomInstance() w/ something akin to LocalRandom, but such change is somewhat orthogonal to the series of 'use reproducible random' patches, so I'd prefer to do it separately. >>> >>> Sure but it remains confusing to see such a variety of cases. Utils.getRandomInstance() is a shared RNG so you never want to use that directly in different threads if you truly want reproducability - the most you can use it for is the source of seeds for other RNGs - but even that is non-deterministic is different threads are involved. >> right, that's why I think it's better/safer to use something similar to LocalRandom in all tests w/o thinking if it's multi-threaded or not. >>> So your changes from Math.random are not going to be reproducible due to different threads being involved. >> really? could you please point me to the test/tests which used Math.random and now have multiple threads using the same instance of Random? > > Apologies. On examining the numeric* tests in detail it seems the RNG is only used during the setup phase. > >>> >>>>> >>>>> And looking at: >>>>> >>>>> vmTestbase/nsk/stress/jni/gclocker/gcl001.java >>>>> >>>>> you changed it to use the LocalRandom utility rather than creating its own random instance. Why was that applicable here but not in other tests? >>>>> >>>>> The changes from Math.random to a local RNG from getRandomInstance() seem okay- but again why that versus LocalRandom? Do we even need LocalRandom? >>>> (although I guess you've already figured that out from other patches, I'll answer here for completeness) >>>> vmTestbase/nsk/stress/jni/gclocker/gcl001.java used the same instance of RNG in multiple threads, hence it would be really reproducible if we just switched to getRandomInstance(), LocalRandom is needed exactly for that -- provide reproducible random in multi-thread use cases. >>> >>> I'm not sure if you wrote that exactly as intended. Utils.getRandomInstance() does not produce a RNG that gives reproducible results across threads, so we should, it seems, never use it directly in any multi-threaded stress test. >> sorry, I missed 'not', it should have been 'hence it would not be really reproducible' >>> Secondly, LocalRandom is a wrapper for ThreadLocalRandom.current() _but_ you have no control over the seed so seems not so useful for achieving reproducability! >> I guess you haven't updated your workspace, LocalRandom is now (after 8242314) using ThreadLocal: >>> private final static ThreadLocal random = new ThreadLocal<>() { >>> protected Random initialValue() { >>> // each thread gets its own seed independendly on the order they >>> // use LocalRandom, yet it depends on the order threads are created. >>> // main thread uses original seed >>> return new Random(Utils.SEED ^ (Thread.currentThread().getId() - 1)); >>> } >>> }; > > Ah yes - sorry - my local repo on my laptop (only used for code browsing) was missing that change. Okay that makes a big difference :) > > However I will note that using the thread ID as part of the seed may not lead to reproducible results if the thread creation itself is not deterministic/reproducible. true, yet in most cases it's still better than prev. solution, and it's never worse. and in all of my runs w/ instrumented builds, thread creation and hence the seeds were deterministic. so,.. can I count you as a reviewer? -- Igor > > Thanks, > David > >> -- Igor >>> >>> David >>> ----- >>> >>>> -- Igor >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>>> testing: : vmTestbase_nsk_stress test group >>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8243429 >>>>>> webrevs: >>>>>> - code changes: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.code >>>>>>> 58 lines changed: 41 ins; 4 del; 13 mod; >>>>>> - adding k/w: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.kw >>>>>>> 11 lines changed: 0 ins; 0 del; 11 mod; >>>>>> - full: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00 >>>>>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>>>>> Thanks, >>>>>> -- Igor From david.holmes at oracle.com Fri May 1 05:52:21 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 1 May 2020 15:52:21 +1000 Subject: RFR(S) : 8243429 : use reproducible random in :vmTestbase_nsk_stress In-Reply-To: <5C13C637-62A4-418C-92A5-D1936CB9AF74@oracle.com> References: <82A98A70-3E9B-4A63-BE6D-293EF64A5C8B@oracle.com> <639de8f9-81f6-dc1b-1bc6-260fbeba68dc@oracle.com> <2D6B045B-1694-48B1-900D-ADBA9BD1876B@oracle.com> <37330d42-0b50-a6f0-2072-6802ed690f3a@oracle.com> <63DACE82-1258-4F30-8889-3EAF7848FC05@oracle.com> <381b3793-07cd-0c72-f946-87061eab20e4@oracle.com> <5C13C637-62A4-418C-92A5-D1936CB9AF74@oracle.com> Message-ID: <855fbb7d-fe6e-383a-2fa8-a8d05de0cd29@oracle.com> On 1/05/2020 3:20 pm, Igor Ignatyev wrote: >> On Apr 30, 2020, at 9:52 PM, David Holmes > > wrote: >> >> On 1/05/2020 2:30 pm, Igor Ignatyev wrote: >>>> On Apr 30, 2020, at 9:14 PM, David Holmes >>> > wrote: >>>> >>>> On 1/05/2020 1:27 pm, Igor Ignatyev wrote: >>>>> Hi David, >>>>>> On Apr 30, 2020, at 7:34 PM, David Holmes >>>>> >>>>>> > >>>>>> wrote: >>>>>> >>>>>> Hi Igor, >>>>>> >>>>>> On 1/05/2020 3:15 am, Igor Ignatyev wrote: >>>>>>> http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00 >>>>>>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>>>>> >>>>>> The patch files in your webrevs are not generating properly due to >>>>>> the use of mq. can you fix that? >>>>> unfortunately, I've already squashed all the patches together, so I >>>>> was able to upload only the final version -- >>>>> http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00/8243429.patch >>>>>> >>>>>>> Hi all, >>>>>>> could you please review this small patch? >>>>>>> from JBS: >>>>>>>> this subtask is to use j.t.l.Utils.getRandomInstance() as a >>>>>>>> random number generator, where applicable, in >>>>>>>> :vmTestbase_nsk_stress test group and marking the tests which >>>>>>>> make use of "randomness" with a proper k/w. >>>>>> >>>>>> I'm a bit confused about the strategy here: >>>>>> >>>>>> vmTestbase/nsk/stress/except/except012.java >>>>>> >>>>>> - ???????????Random random = new Random(0); >>>>>> + ???????????Random random = new Random(42); >>>>>> >>>>>> but: >>>>>> >>>>>> vmTestbase/nsk/stress/jni/jnistress002.java >>>>>> >>>>>> - ???static Random myRandom = new Random(); >>>>>> + ???Random myRandom = new >>>>>> Random(Utils.getRandomInstance().nextLong()); >>>>>> >>>>>> why are these random instances seeded completely differently? >>>>> because they were different before my patch, except012 used a fixed >>>>> seed (so it doesn't qualify to be marked w/ randomness k/w), and >>>>> jnistress002 used a "random" seed; after the patch except012 still >>>>> uses a fixed (but a "well-known" and easier to grep) seed value, >>>>> and jnistress002 uses random, yet reproducible/configurable seed. >>>>> I've explained that in my previous RFRs for hotspot tests outside >>>>> of vmTestbase, e.g. >>>>> https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037828.html >>>> >>>> Not everyone on hotspot-dev is also on hotspot-compiler-dev. >>> true, yet almost every hotspot-X-dev alias received their porting of >>> 'use reproducible random in hotspot X tests' a couple weeks ago, >>> which all have references to the same email thread ;) . in any case, >>> I apologize I should have provided enough context in each of new RFR. >>>> >>>> I'm trying to understand why we still have different usage patterns. >>>> IIUC there's no reason that except012 couldn't be seeded the same >>>> way as jnistress002 - right? Then it would qualify for "randomness" >>>> keyword. What is it about any given test that determines whether we >>>> seed the RNG differently on each test run versus fixing it at 42 >>>> (for example)? >>> it's more the question for the author of particular test. I didn't >>> plan to (significantly) change behavior of the tests, so I tried not >>> to convert any non "randomness" tests to "randomness" tests. >> >> Okay. But making changes in this area it just draws attention to the >> different way we do things, and it is n't obvious why there are >> differnet ways. > I guess that some people decided to use fixed seed so their tests are > more-or-less stable to themselves and results from different runs can be > compared; yet others decided that they would like their tests to cover > different paths in different runs. and I really don't know which > solution is better, but I believe that solution which allows you to get > the best of both is the best. that's to say, I'll definitely file a task > to look into all places we use fixed seed in hotspot tests and see if it > makes sense to switch to a "random" seed value, and will work on it as > part of switching to CI-run/build-based seed or, as you called it in > another thread, "one blessed seed". thank you for all these discussions! Sounds good. Sorry for drawing this out so much. >> >>>> >>>>> also, in jnistress002, we can't use Utils.getRandomInstance() as >>>>> there would multiple threads (JNIter002) using the same RNG. we >>>>> could change it to use LocalRandom, I must admit I was doing >>>>> different parts of these patches on different days, and given their >>>>> tediousness I might have became less consistent in my choices b/w >>>>> using LocalRandom and using a "threadlocal" instance of Random. I, >>>>> personally, think we need to switch to LocalRandom in all such >>>>> cases, or even better to replace Utils.getRandomInstance() w/ >>>>> something akin to LocalRandom, but such change is somewhat >>>>> orthogonal to the series of 'use reproducible random' patches, so >>>>> I'd prefer to do it separately. >>>> >>>> Sure but it remains confusing to see such a variety of cases. >>>> Utils.getRandomInstance() is a shared RNG so you never want to use >>>> that directly in different threads if you truly want reproducability >>>> - the most you can use it for is the source of seeds for other RNGs >>>> - but even that is non-deterministic is different threads are involved. >>> right, that's why I think it's better/safer to use something similar >>> to LocalRandom in all tests w/o thinking if it's multi-threaded or not. >>>> So your changes from Math.random are not going to be reproducible >>>> due to different threads being involved. >>> really? could you please point me to the test/tests which used >>> Math.random and now have multiple threads using the same instance of >>> Random? >> >> Apologies. On examining the numeric* tests in detail it seems the RNG >> is only used during the setup phase. >> >>>> >>>>>> >>>>>> And looking at: >>>>>> >>>>>> vmTestbase/nsk/stress/jni/gclocker/gcl001.java >>>>>> >>>>>> you changed it to use the LocalRandom utility rather than creating >>>>>> its own random instance. Why was that applicable here but not in >>>>>> other tests? >>>>>> >>>>>> The changes from Math.random to a local RNG from >>>>>> getRandomInstance() seem okay- but again why that versus >>>>>> LocalRandom? Do we even need LocalRandom? >>>>> (although I guess you've already figured that out from other >>>>> patches, I'll answer here for completeness) >>>>> vmTestbase/nsk/stress/jni/gclocker/gcl001.java used the same >>>>> instance of RNG in multiple threads, hence it would be really >>>>> reproducible if we just switched to getRandomInstance(), >>>>> LocalRandom is needed exactly for that -- provide reproducible >>>>> random in multi-thread use cases. >>>> >>>> I'm not sure if you wrote that exactly as intended. >>>> Utils.getRandomInstance() does not produce a RNG that gives >>>> reproducible results across threads, so we should, it seems, never >>>> use it directly in any multi-threaded stress test. >>> sorry, I missed 'not', it should have been 'hence it would not be >>> really reproducible' >>>> Secondly, LocalRandom is a wrapper for ThreadLocalRandom.current() >>>> _but_ you have no control over the seed so seems not so useful for >>>> achieving reproducability! >>> I guess you haven't updated your workspace, LocalRandom is now (after >>> 8242314) using ThreadLocal: >>>> private final static ThreadLocal random = new ThreadLocal<>() { >>>> ? ? ? ??protected Random initialValue() { >>>> ? ? ? ? ? ??// each thread gets its own seed independendly on the >>>> order they >>>> ? ? ? ? ? ??// use LocalRandom, yet it depends on the order threads >>>> are created. >>>> ? ? ? ? ? ??// main thread uses original seed >>>> ? ? ? ? ? ??return new Random(Utils.SEED ^ >>>> (Thread.currentThread().getId() - 1)); >>>> ? ? ? ??} >>>> ? ??}; >> >> Ah yes - sorry - my local repo on my laptop (only used for code >> browsing) was missing that change. Okay that makes a big difference :) >> >> However I will note that using the thread ID as part of the seed may >> not lead to reproducible results if the thread creation itself is not >> deterministic/reproducible. > true, yet in most cases it's still better than prev. solution, and it's > never worse. and in all of my runs w/ instrumented builds, thread > creation and hence the seeds were deterministic. Definitely better than before. :) I would have to wonder whether dynamic compiler threads would be the mostly likely source of disruption. > so,.. can I count you as a reviewer? Of course :) Thanks, David > -- Igor >> >> Thanks, >> David >> >>> -- Igor >>>> >>>> David >>>> ----- >>>> >>>>> -- Igor >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>>> testing: : vmTestbase_nsk_stress test group >>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8243429 >>>>>>> webrevs: >>>>>>> - code changes: >>>>>>> http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.code >>>>>>>> 58 lines changed: 41 ins; 4 del; 13 mod; >>>>>>> - adding k/w: >>>>>>> http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.kw >>>>>>>> 11 lines changed: 0 ins; 0 del; 11 mod; >>>>>>> - full: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00 >>>>>>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>>>>>> Thanks, >>>>>>> -- Igor > From erik.joelsson at oracle.com Fri May 1 13:04:59 2020 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Fri, 1 May 2020 06:04:59 -0700 Subject: [aarch64-port-dev ] FW: [Mach5] mach5-one-phh-JDK-8151779-20200427-2151-10554367: FAILED In-Reply-To: <9725176C-5E5A-499B-8093-A5865C4AC443@amazon.com> References: <858sifgbt3.fsf@arm.com> <57BB18E5-1F35-45AC-88BF-8D8C6A0767E3@amazon.com> <56d85fb9-6ce8-7c00-ff4f-bf3fe40718a4@redhat.com> <28A344AC-8FF4-49FB-96B9-6AC886C05930@amazon.com> <9725176C-5E5A-499B-8093-A5865C4AC443@amazon.com> Message-ID: <23d2cf6e-da04-4375-b777-0d29c8d813f3@oracle.com> Hello, My OracleJDK 14 displays: java version "14" 2020-03-17 Not sure where you found a version output without the word "version" in it. From what I understand, any distributor is free to change the "openjdk" prefix of this line, so relying on there only being 2 cases is not a good idea. If we assume the boot jdk must be an OpenJDK derivative, then a regular expression that tries to capture the line in more detail would be preferred. The aspects I would try to capture would be the word version and a bunch of numbers and dots (and possibly underscore if we want to keep it compatible with even older java versions for easier backports) inside double quotes. Something like this perhaps: grep "version \"[0-9\._]*\"" I tried that expression manually on Mac, Linux and Solaris so should be portable enough. /Erik On 2020-04-30 16:48, Liu, Xin wrote: > Hi, Andrew, > > How about this? I can use awk to capture java -version. There're 2 cases. > > I) openjdk > openjdk version "14.0.1" 2020-04-14 > 2) oraclejdk > java 14.0.1 2020-04-14 > > if somehow java displays some error/warning messages, awk can filter them out and capture the version line. > Eg. > $ ~/builds/jdk-14.0.1+7/bin/java -version > [0.009s][error][cds] Unable to map CDS archive -- os::vm_allocation_granularity() expected: 65536 actual: 4096 > openjdk version "14.0.1" 2020-04-14 > OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7) > OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode) > $ ~/builds/jdk-14.0.1+7/bin/java -version 2>&1 | awk '/^(openjdk version|java)/ {print $0}' > openjdk version "14.0.1" 2020-04-14 > > I think this awk stmt is portable, but it's always good to ask experts to review it, so I cc build-dev. > > Hers is the change. > > diff --git a/make/autoconf/boot-jdk.m4 b/make/autoconf/boot-jdk.m4 > --- a/make/autoconf/boot-jdk.m4 > +++ b/make/autoconf/boot-jdk.m4 > @@ -74,7 +74,7 @@ > BOOT_JDK_FOUND=no > else > # Oh, this is looking good! We probably have found a proper JDK. Is it the correct version? > - BOOT_JDK_VERSION=`"$BOOT_JDK/bin/java$EXE_SUFFIX" $USER_BOOT_JDK_OPTIONS -version 2>&1 | $HEAD -n 1` > + BOOT_JDK_VERSION=`"$BOOT_JDK/bin/java$EXE_SUFFIX" $USER_BOOT_JDK_OPTIONS -version 2>&1 | $AWK '/^(openjdk version|java)/ {print [$]0}'` > if [ [[ "$BOOT_JDK_VERSION" =~ "Picked up" ]] ]; then > AC_MSG_NOTICE([You have _JAVA_OPTIONS or JAVA_TOOL_OPTIONS set. This can mess up the build. Please use --with-boot-jdk-jvmargs instead.]) > AC_MSG_NOTICE([Java reports: "$BOOT_JDK_VERSION".]) > @@ -529,7 +529,7 @@ > BUILD_JDK_FOUND=no > else > # Oh, this is looking good! We probably have found a proper JDK. Is it the correct version? > - BUILD_JDK_VERSION=`"$BUILD_JDK/bin/java" -version 2>&1 | $HEAD -n 1` > + BUILD_JDK_VERSION=`"$BUILD_JDK/bin/java" -version 2>&1 | $AWK '/^(openjdk version|java)/ {print [$]0}'` > > # Extra M4 quote needed to protect [] in grep expression. > [FOUND_CORRECT_VERSION=`echo $BUILD_JDK_VERSION | $EGREP "\"$VERSION_FEATURE([\.+-].*)?\""`] > > > > ?On 4/30/20, 2:52 PM, "aarch64-port-dev on behalf of Liu, Xin" wrote: > > Hi, Andrew, > > That's a hack. A general way should use grep or sed to capture the needed line instead of hardcoding first or second line. > Okay, Let me try to do that. > > Thanks, > --lx > > > On 4/30/20, 1:19 AM, "aph at redhat.com" wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > On 4/30/20 12:43 AM, Liu, Xin wrote: > > One trick here. It's very easy to cheat configure by hacking the boot-jdk.m4 to "$HEAD -n 2". Everything looks fine then. > > The fix should be submitted to build-dev. > > -- > Andrew Haley (he/him) > Java Platform Lead Engineer > Red Hat UK Ltd. > https://keybase.io/andrewhaley > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 > > > From igor.ignatyev at oracle.com Fri May 1 16:28:56 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Fri, 1 May 2020 09:28:56 -0700 Subject: RFR(S) : 8243429 : use reproducible random in :vmTestbase_nsk_stress In-Reply-To: <855fbb7d-fe6e-383a-2fa8-a8d05de0cd29@oracle.com> References: <82A98A70-3E9B-4A63-BE6D-293EF64A5C8B@oracle.com> <639de8f9-81f6-dc1b-1bc6-260fbeba68dc@oracle.com> <2D6B045B-1694-48B1-900D-ADBA9BD1876B@oracle.com> <37330d42-0b50-a6f0-2072-6802ed690f3a@oracle.com> <63DACE82-1258-4F30-8889-3EAF7848FC05@oracle.com> <381b3793-07cd-0c72-f946-87061eab20e4@oracle.com> <5C13C637-62A4-418C-92A5-D1936CB9AF74@oracle.com> <855fbb7d-fe6e-383a-2fa8-a8d05de0cd29@oracle.com> Message-ID: <2E6FF710-8566-42CF-88BE-3B6705470B8C@oracle.com> David, Katya, thanks for the reviews. pushed. -- Igor > On Apr 30, 2020, at 10:52 PM, David Holmes wrote: > > On 1/05/2020 3:20 pm, Igor Ignatyev wrote: >>> On Apr 30, 2020, at 9:52 PM, David Holmes >> wrote: >>> >>> On 1/05/2020 2:30 pm, Igor Ignatyev wrote: >>>>> On Apr 30, 2020, at 9:14 PM, David Holmes >>> wrote: >>>>> >>>>> On 1/05/2020 1:27 pm, Igor Ignatyev wrote: >>>>>> Hi David, >>>>>>> On Apr 30, 2020, at 7:34 PM, David Holmes > wrote: >>>>>>> >>>>>>> Hi Igor, >>>>>>> >>>>>>> On 1/05/2020 3:15 am, Igor Ignatyev wrote: >>>>>>>> http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00 >>>>>>>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>>>>>> >>>>>>> The patch files in your webrevs are not generating properly due to the use of mq. can you fix that? >>>>>> unfortunately, I've already squashed all the patches together, so I was able to upload only the final version -- http://cr.openjdk.java.net/~iignatyev/8243429/webrev.00/8243429.patch >>>>>>> >>>>>>>> Hi all, >>>>>>>> could you please review this small patch? >>>>>>>> from JBS: >>>>>>>>> this subtask is to use j.t.l.Utils.getRandomInstance() as a random number generator, where applicable, in :vmTestbase_nsk_stress test group and marking the tests which make use of "randomness" with a proper k/w. >>>>>>> >>>>>>> I'm a bit confused about the strategy here: >>>>>>> >>>>>>> vmTestbase/nsk/stress/except/except012.java >>>>>>> >>>>>>> - Random random = new Random(0); >>>>>>> + Random random = new Random(42); >>>>>>> >>>>>>> but: >>>>>>> >>>>>>> vmTestbase/nsk/stress/jni/jnistress002.java >>>>>>> >>>>>>> - static Random myRandom = new Random(); >>>>>>> + Random myRandom = new Random(Utils.getRandomInstance().nextLong()); >>>>>>> >>>>>>> why are these random instances seeded completely differently? >>>>>> because they were different before my patch, except012 used a fixed seed (so it doesn't qualify to be marked w/ randomness k/w), and jnistress002 used a "random" seed; after the patch except012 still uses a fixed (but a "well-known" and easier to grep) seed value, and jnistress002 uses random, yet reproducible/configurable seed. I've explained that in my previous RFRs for hotspot tests outside of vmTestbase, e.g. https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037828.html >>>>> >>>>> Not everyone on hotspot-dev is also on hotspot-compiler-dev. >>>> true, yet almost every hotspot-X-dev alias received their porting of 'use reproducible random in hotspot X tests' a couple weeks ago, which all have references to the same email thread ;) . in any case, I apologize I should have provided enough context in each of new RFR. >>>>> >>>>> I'm trying to understand why we still have different usage patterns. IIUC there's no reason that except012 couldn't be seeded the same way as jnistress002 - right? Then it would qualify for "randomness" keyword. What is it about any given test that determines whether we seed the RNG differently on each test run versus fixing it at 42 (for example)? >>>> it's more the question for the author of particular test. I didn't plan to (significantly) change behavior of the tests, so I tried not to convert any non "randomness" tests to "randomness" tests. >>> >>> Okay. But making changes in this area it just draws attention to the different way we do things, and it is n't obvious why there are differnet ways. >> I guess that some people decided to use fixed seed so their tests are more-or-less stable to themselves and results from different runs can be compared; yet others decided that they would like their tests to cover different paths in different runs. and I really don't know which solution is better, but I believe that solution which allows you to get the best of both is the best. that's to say, I'll definitely file a task to look into all places we use fixed seed in hotspot tests and see if it makes sense to switch to a "random" seed value, and will work on it as part of switching to CI-run/build-based seed or, as you called it in another thread, "one blessed seed". thank you for all these discussions! > > Sounds good. Sorry for drawing this out so much. > >>> >>>>> >>>>>> also, in jnistress002, we can't use Utils.getRandomInstance() as there would multiple threads (JNIter002) using the same RNG. we could change it to use LocalRandom, I must admit I was doing different parts of these patches on different days, and given their tediousness I might have became less consistent in my choices b/w using LocalRandom and using a "threadlocal" instance of Random. I, personally, think we need to switch to LocalRandom in all such cases, or even better to replace Utils.getRandomInstance() w/ something akin to LocalRandom, but such change is somewhat orthogonal to the series of 'use reproducible random' patches, so I'd prefer to do it separately. >>>>> >>>>> Sure but it remains confusing to see such a variety of cases. Utils.getRandomInstance() is a shared RNG so you never want to use that directly in different threads if you truly want reproducability - the most you can use it for is the source of seeds for other RNGs - but even that is non-deterministic is different threads are involved. >>>> right, that's why I think it's better/safer to use something similar to LocalRandom in all tests w/o thinking if it's multi-threaded or not. >>>>> So your changes from Math.random are not going to be reproducible due to different threads being involved. >>>> really? could you please point me to the test/tests which used Math.random and now have multiple threads using the same instance of Random? >>> >>> Apologies. On examining the numeric* tests in detail it seems the RNG is only used during the setup phase. >>> >>>>> >>>>>>> >>>>>>> And looking at: >>>>>>> >>>>>>> vmTestbase/nsk/stress/jni/gclocker/gcl001.java >>>>>>> >>>>>>> you changed it to use the LocalRandom utility rather than creating its own random instance. Why was that applicable here but not in other tests? >>>>>>> >>>>>>> The changes from Math.random to a local RNG from getRandomInstance() seem okay- but again why that versus LocalRandom? Do we even need LocalRandom? >>>>>> (although I guess you've already figured that out from other patches, I'll answer here for completeness) >>>>>> vmTestbase/nsk/stress/jni/gclocker/gcl001.java used the same instance of RNG in multiple threads, hence it would be really reproducible if we just switched to getRandomInstance(), LocalRandom is needed exactly for that -- provide reproducible random in multi-thread use cases. >>>>> >>>>> I'm not sure if you wrote that exactly as intended. Utils.getRandomInstance() does not produce a RNG that gives reproducible results across threads, so we should, it seems, never use it directly in any multi-threaded stress test. >>>> sorry, I missed 'not', it should have been 'hence it would not be really reproducible' >>>>> Secondly, LocalRandom is a wrapper for ThreadLocalRandom.current() _but_ you have no control over the seed so seems not so useful for achieving reproducability! >>>> I guess you haven't updated your workspace, LocalRandom is now (after 8242314) using ThreadLocal: >>>>> private final static ThreadLocal random = new ThreadLocal<>() { >>>>> protected Random initialValue() { >>>>> // each thread gets its own seed independendly on the order they >>>>> // use LocalRandom, yet it depends on the order threads are created. >>>>> // main thread uses original seed >>>>> return new Random(Utils.SEED ^ (Thread.currentThread().getId() - 1)); >>>>> } >>>>> }; >>> >>> Ah yes - sorry - my local repo on my laptop (only used for code browsing) was missing that change. Okay that makes a big difference :) >>> >>> However I will note that using the thread ID as part of the seed may not lead to reproducible results if the thread creation itself is not deterministic/reproducible. >> true, yet in most cases it's still better than prev. solution, and it's never worse. and in all of my runs w/ instrumented builds, thread creation and hence the seeds were deterministic. > > Definitely better than before. :) > > I would have to wonder whether dynamic compiler threads would be the mostly likely source of disruption. > >> so,.. can I count you as a reviewer? > > Of course :) > > Thanks, > David > >> -- Igor >>> >>> Thanks, >>> David >>> >>>> -- Igor >>>>> >>>>> David >>>>> ----- >>>>> >>>>>> -- Igor >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>>> testing: : vmTestbase_nsk_stress test group >>>>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8243429 >>>>>>>> webrevs: >>>>>>>> - code changes: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.code >>>>>>>>> 58 lines changed: 41 ins; 4 del; 13 mod; >>>>>>>> - adding k/w: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00.kw >>>>>>>>> 11 lines changed: 0 ins; 0 del; 11 mod; >>>>>>>> - full: http://cr.openjdk.java.net/~iignatyev//8243429/webrev.00 >>>>>>>>> 82 lines changed: 41 ins; 4 del; 37 mod; >>>>>>>> Thanks, >>>>>>>> -- Igor From chris.plummer at oracle.com Fri May 1 19:34:25 2020 From: chris.plummer at oracle.com (Chris Plummer) Date: Fri, 1 May 2020 12:34:25 -0700 Subject: RFR: 8244078: ProcessTools executeTestJvm and createJavaProcessBuilder have inconsistent handling of test.*.opts In-Reply-To: <44e3e1e7-7135-1892-12d3-cbf3707cfce0@oracle.com> References: <44e3e1e7-7135-1892-12d3-cbf3707cfce0@oracle.com> Message-ID: On 4/30/20 2:07 AM, Stefan Karlsson wrote: > Hi all, > > Please review this patch to make it less likely that we accidentally > add or fail to add test.java.opts and test.vm.opts to our spawned test > JVMs. > > https://cr.openjdk.java.net/~stefank/8244078/webrev.01/ > https://bugs.openjdk.java.net/browse/JDK-8244078 > > ProcessTools.createJavaProcessBuilder(cmd) creates a ProcessBuilder > *without* test.java.opts and test.vm.opts. There is a > (addTestVmAndJavaOptions, cmd) overload that allows the caller to > opt-in to the addition of these flags. The created ProcessBuilder is > then used to start the JVM, and almost always fed into an OutputAnalyzer. > > There's another function executeTestJvm, that both creates a > ProcessBuilder and then feeds it into an OutputAnalyzer (plus a bit > more). This function uses createJavaProcessBuilder(true, cmd), and > thereby adds the test.java.opts and test.vm.opts flags. > > This means that one has to know about this difference when reading > code using createJavaProcessBuilder(cmd) and executeTestJvm(cmd), and > when creating (copying) code using these functions. > > It has been suggested that createJavaProcessBuilder is intended to be > a lower-level, building block API and that it's not confusing that > they have different behavior. I don't really agree, but I'm buying > into the notion of lower-level vs higher-level APIs here. So, my > proposal is to remove the addTestVmAndJavaOptions feature from > createJavaProcessBuilder, and instead create a new function called > createTestJvm that adds test.java.opts and test.vm.opts. The name is > intentionally similar to executeTestJvm, and in fact, the > executeTestJvm implementation will use createTestJvm: > > ???? public static OutputAnalyzer executeTestJvm(String... cmds) > throws Exception { > - ProcessBuilder pb = createJavaProcessBuilder(true, cmds); > + ProcessBuilder pb = createTestJvm(cmds); > ???????? return executeProcess(pb); > ???? } > > The rest of the patch is mainly: > - leaving createJavaProcessBuilder(cmd) as is > - replacing createJavaProcessBuilder(false, cmd) with > createJavaProcessBuilder(cmd) > - replacing createJavaProcessBuilder(true, cmd) with createTestJvm(cmd) > > There was one odd thing in jdi that requires extra scrutiny: > https://cr.openjdk.java.net/~stefank/8244078/webrev.01/test/jdk/com/sun/jdi/lib/jdb/Debuggee.java.udiff.html > Yes, that did look a odd at first glance, but I think that fact that your removed addTestVmAndJavaOptions() and everything still built is proof enough that it was just bit rotted code and not needed. Chris > > I've run this through tier1-3, and are currently running this through > higher tiers. > > Thanks, > StefanK From paul.sandoz at oracle.com Fri May 1 20:36:10 2020 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 1 May 2020 13:36:10 -0700 Subject: RFR JDK-8223347 Integration of Vector API (Incubator): Java API, implementation, and tests In-Reply-To: <1886691513.405349.1587389515699.JavaMail.zimbra@u-pem.fr> References: <49F81016-55DF-4E3C-940A-A8DF8C712259@oracle.com> <1886691513.405349.1587389515699.JavaMail.zimbra@u-pem.fr> Message-ID: Hi Remi, Thanks for the feedback. I am gonna need to go over this with John. Some comments inline. > On Apr 20, 2020, at 6:31 AM, Remi Forax wrote: > > Hi Paul, > about the API part (not the implementation), > there are location where the same concept is described with a different names which doesn't help to understand how thing work > - vector length <=> vector lane count > - vector shape <=> vector bits size > - element size <=> lane size > > "size" should never be used (VectorSpecies.elementSize() by example ) because it's never clear if it's the byte size or the bits size, using byteSize and bitsSize as used several times in the API should be used everywhere. I agree with being explicit here e.g. s/Vector.elementSize/Vector.elementBitSize and other usages. I would prefer to consistently use length in methods to refer to the lane count, it matches more closely to an array and for thinking about bounds checking. How about on Vector: /** * Returns number of vector lanes ({@code VLENGTH}), commonly referred to as the lane count. * * @return the number of vector lanes */ public abstract int length(); And then change VectorShape.laneCount to VectorShape.laneLength. ? > > Initially, having VectorSpecies and VectorShape confuse me, it's not the shape of the vector but only it's bits size, > renaming VectorShape to VectorBitsSize will help, i think. > Shape is quite a fundamental concept specified and used in the API, representing the information capacity of a vector. I think it would be misleading to refer to it only as VectorBitSize and unwieldy to change the specification accordingly. In this sense it serves as a useful abstraction to talk about vectors of the same bit size, and operations that change that size or not (such as shape-changing or shape-invariant). > You have the same values defined at several places in the API (sometimes with different names), usually it's aliases but it makes the API bigger that it should. > I think all the reference to the element type, vector byte size, vector bit size, element byte size, element byte size, vector lane count should not appear on Vector because they are accessible from Species, > and to go from a vector to the corresponding species, species() is enough. > But it also makes it less usable, esp. for length(). > You have also the same methods defined at several places, on VectorSpecies and as a static method taking a species as parameter, > all the methods of VectorSpecies that takes or return a vector/mask/shuffle should be static in vector/mask/shuffle. > It?s a little more nuanced, but I agree there is some duplication e.g. VectorSpecies.fromByteArray compared to the richer set of methods defined on [E]Vector. I think we can remove that method from VectorSpecies. Then I think we are dealing with the more ?erased/reflective? and long carrier accepting methods for general operation without appealing to E. > I think Binary and Associative should be merged to one VectorOperation, given that the difference is subtle and the whole point of this API is that if the hardware do not provide a way to reduce a binary op, it can be done in Java. Not quite true, ARM/x86 implementations do provide intrinsic implementations (leveraging a sequence of instructions) for some reduction operations e.g. min/max. I cannot recall there current state of the ARM/x86 implementations for other reductive operations but the plan is to optimize all of them at some point. Further, hardware may evolve in the future. > > I have no idea what a Conversion.rangeType() is ? > It?s the element type of the resulting output vector, Doc on VectorOperators.Operator: /** * Reports the special return type of this operator. * If this operator is a boolean, returns {@code boolean.class}. * If this operator is a {@code Conversion}, * returns its {@linkplain Conversion#rangeType range type}. * * Otherwise, the operator's return value always has * whatever type was given as an input, and this method * returns {@code Object.class} to denote that fact. * @return the special return type, or {@code Object.class} if none */ Class rangeType(); > More specially: > - VectorSpecies.loopbound() is not used in the example ? In the package-info? Yes, we should update that. > - VectorSpecies.arrayType()/genericElementType() are more for implementers than users , no ? at least arrayType should be arrayElementType. I don?t think we need to expose genericElementType in the API. I suspect arrayType may have been added before it was added to Class in 12 for the constable support. I think we can remove it. > - VectorSpecies.withLanes() => withElementType() > > - VectorMask.check* are implementer methods ? The intent is to provide abstractions so the developer can perform ?reflective" checks on vector, mask, or shuffle. > - VectorMask.equal(VectorMask) => equiv > Because it?s too close to the name ?equals?? > - VectorShuffle.check* are implementer methods ? > - VectorShuffle.vectorSpecies() => VectorShuffle.species() (as in Vector and VectorMask) Ok. > - VectorShuffle.toVector() should return a IntVector No, it returns a vector according to its species. Note the constraint that shuffle source indexes are always in the range of -VLENGTH to VLENGTH - 1. > - VectorShuffle.laneIsValid(), it seems to be an operation on Vector, not VectorShuffle. > No, its specific to shuffle for masking out exceptional indexes (those < 0). > - IntVector.max, why there is no binary version that takes a mask ? It's documented but not why it's not available. > It's to reduce the number of methods. On Vector: * This is not a full-service named operation like * {@link #add(Vector) add()}. A masked version of * this operation is not directly available * but may be obtained via the masked version of * {@code lanewise}. Subclasses define an additional * scalar-broadcast overloading of this method. > > and i'm also not a big fans of the method that returns a long instead of a vector and only works on 64 bits values. > Can you provide more details? Thanks, Paul. From vladimir.kozlov at oracle.com Sat May 2 00:23:29 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Fri, 1 May 2020 17:23:29 -0700 Subject: RFR (XXL): 8223347: Integration of Vector API (Incubator): General HotSpot changes In-Reply-To: <25a564a1-7f40-6988-060f-86b06e02ad21@oracle.com> References: <25a564a1-7f40-6988-060f-86b06e02ad21@oracle.com> Message-ID: <08421e2e-985d-5901-eb20-0ae96a48d8a0@oracle.com> I looked on these changes and compiler changes seems fine. Thanks, Vladimir On 4/16/20 5:32 AM, Vladimir Ivanov wrote: > Hi, > > Any more reviews, please? Especially, compiler and runtime-related changes. > > Thanks in advance! > > Best regards, > Vladimir Ivanov > > On 04.04.2020 02:12, Vladimir Ivanov wrote: >> Hi, >> >> Following up on review requests of API [0] and Java implementation [1] for Vector API (JEP 338 [2]), here's a request >> for review of general HotSpot changes (in shared code) required for supporting the API: >> >> >> http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/all.00-03/ >> >> (First of all, to set proper expectations: since the JEP is still in Candidate state, the intention is to initiate >> preliminary round(s) of review to inform the community and gather feedback before sending out final/official RFRs once >> the JEP is Targeted to a release.) >> >> Vector API (being developed in Project Panama [3]) relies on JVM support to utilize optimal vector hardware >> instructions at runtime. It interacts with JVM through intrinsics (declared in jdk.internal.vm.vector.VectorSupport >> [4]) which expose vector operations support in C2 JIT-compiler. >> >> As Paul wrote earlier: "A vector intrinsic is an internal low-level vector operation. The last argument to the >> intrinsic is fall back behavior in Java, implementing the scalar operation over the number of elements held by the >> vector.? Thus, If the intrinsic is not supported in C2 for the other arguments then the Java implementation is >> executed (the Java implementation is always executed when running in the interpreter or for C1)." >> >> The rest of JVM support is about aggressively optimizing vector boxes to minimize (ideally eliminate) the overhead of >> boxing for vector values. >> It's a stop-the-gap solution for vector box elimination problem until inline classes arrive. Vector classes are >> value-based and in the longer term will be migrated to inline classes once the support becomes available. >> >> Vector API talk from JVMLS'18 [5] contains brief overview of JVM implementation and some details. >> >> Complete implementation resides in vector-unstable branch of panama/dev repository [6]. >> >> Now to gory details (the patch is split in multiple "sub-webrevs"): >> >> =========================================================== >> >> (1) http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/00.backend.shared/ >> >> Ideal vector nodes for new operations introduced by Vector API. >> >> (Platform-specific back end support will be posted for review separately). >> >> =========================================================== >> >> (2) http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/01.intrinsics/ >> >> JVM Java interface (VectorSupport) and intrinsic support in C2. >> >> Vector instances are initially represented as VectorBox macro nodes and "unboxing" is represented by VectorUnbox node. >> It simplifies vector box elimination analysis and the nodes are expanded later right before EA pass. >> >> Vectors have 2-level on-heap representation: for the vector value primitive array is used as a backing storage and it >> is encapsulated in a typed wrapper (e.g., Int256Vector - vector of 8 ints - contains a int[8] instance which is used >> to store vector value). >> >> Unless VectorBox node goes away, it needs to be expanded into an allocation eventually, but it is a pure node and >> doesn't have any JVM state associated with it. The problem is solved by keeping JVM state separately in a >> VectorBoxAllocate node associated with VectorBox node and use it during expansion. >> >> Also, to simplify vector box elimination, inlining of vector reboxing calls (VectorSupport::maybeRebox) is delayed >> until the analysis is over. >> >> =========================================================== >> >> (3) http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/02.vbox_elimination/ >> >> Vector box elimination analysis implementation. (Brief overview: slides #36-42 [5].) >> >> The main part is devoted to scalarization across safepoints and rematerialization support during deoptimization. In >> C2-generated code vector operations work with raw vector values which live in registers or spilled on the stack and it >> allows to avoid boxing/unboxing when a vector value is alive across a safepoint. As with other values, there's just a >> location of the vector value at the safepoint and vector type information recorded in the relevant nmethod metadata >> and all the heavy-lifting happens only when rematerialization takes place. >> >> The analysis preserves object identity invariants except during aggressive reboxing (guarded by >> -XX:+EnableAggressiveReboxing). >> >> (Aggressive reboxing is crucial for cases when vectors "escape": it allocates a fresh instance at every escape point >> thus enabling original instance to go away.) >> >> =========================================================== >> >> (4) http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/03.module.hotspot/ >> >> HotSpot changes for jdk.incubator.vector module. Vector support is makred experimental and turned off by default. JEP >> 338 proposes the API to be released as an incubator module, so a user has to specify "--add-module >> jdk.incubator.vector" on the command line to be able to use it. >> When user does that, JVM automatically enables Vector API support. >> It improves usability (user doesn't need to separately "open" the API and enable JVM support) while minimizing risks >> of destabilitzation from new code when the API is not used. >> >> >> That's it! Will be happy to answer any questions. >> >> And thanks in advance for any feedback! >> >> Best regards, >> Vladimir Ivanov >> >> [0] https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html >> >> [1] https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-April/041228.html >> >> [2] https://openjdk.java.net/jeps/338 >> >> [3] https://openjdk.java.net/projects/panama/ >> >> [4] >> http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/01.intrinsics/src/java.base/share/classes/jdk/internal/vm/vector/VectorSupport.java.html >> >> >> [5] http://cr.openjdk.java.net/~vlivanov/talks/2018_JVMLS_VectorAPI.pdf >> >> [6] http://hg.openjdk.java.net/panama/dev/shortlog/92bbd44386e9 >> >> ???? $ hg clone http://hg.openjdk.java.net/panama/dev/ -b vector-unstable From vladimir.kozlov at oracle.com Sat May 2 00:31:37 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Fri, 1 May 2020 17:31:37 -0700 Subject: RFR (XXL): 8223347: Integration of Vector API (Incubator): x86 backend changes In-Reply-To: References: Message-ID: Changes seems fine. Nice work. Why it is called "vector-unstable branch"? Thanks, Vladimir K On 4/3/20 5:16 PM, Viswanathan, Sandhya wrote: > Hi, > > > Following up on review requests of API [0], Java implementation [1] and > > General Hotspot changes[3] for Vector API, here's a request for review > > of x86 backend changes required for supporting the API: > > > > JEP: https://openjdk.java.net/jeps/338 > > JBS: https://bugs.openjdk.java.net/browse/JDK-8223347 > > Webrev:http://cr.openjdk.java.net/~sviswanathan/VAPI_RFR/x86_webrev/webrev.00/ > > > > Complete implementation resides in vector-unstable branch of > > panama/dev repository [3]. > > Looking forward to your feedback. > > Best Regards, > Sandhya > > > [0] https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html > > > > [1] https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-April/065587.html > > > > [2] https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037798.html > > > > [3] https://openjdk.java.net/projects/panama/ > > $ hg clone http://hg.openjdk.java.net/panama/dev/ -b vector-unstable > > > > > From sandhya.viswanathan at intel.com Sat May 2 00:55:57 2020 From: sandhya.viswanathan at intel.com (Viswanathan, Sandhya) Date: Sat, 2 May 2020 00:55:57 +0000 Subject: RFR (XXL): 8223347: Integration of Vector API (Incubator): x86 backend changes In-Reply-To: References: Message-ID: Hi Vladimir, Thanks a lot for the feedback. We used an old existing separate branch to share the code for review and to track changes. We didn?t know how to change the name of the branch from vector-unstable to vector-stable. Best Regards, Sandhya -----Original Message----- From: Vladimir Kozlov Sent: Friday, May 01, 2020 5:32 PM To: Viswanathan, Sandhya ; hotspot-compiler-dev at openjdk.java.net; core-libs-dev at openjdk.java.net; hotspot-dev Subject: Re: RFR (XXL): 8223347: Integration of Vector API (Incubator): x86 backend changes Changes seems fine. Nice work. Why it is called "vector-unstable branch"? Thanks, Vladimir K On 4/3/20 5:16 PM, Viswanathan, Sandhya wrote: > Hi, > > > Following up on review requests of API [0], Java implementation [1] and > > General Hotspot changes[3] for Vector API, here's a request for review > > of x86 backend changes required for supporting the API: > > > > JEP: https://openjdk.java.net/jeps/338 > > JBS: https://bugs.openjdk.java.net/browse/JDK-8223347 > > Webrev:http://cr.openjdk.java.net/~sviswanathan/VAPI_RFR/x86_webrev/webrev.00/ > > > > Complete implementation resides in vector-unstable branch of > > panama/dev repository [3]. > > Looking forward to your feedback. > > Best Regards, > Sandhya > > > [0] https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html > > > > [1] https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-April/065587.html > > > > [2] https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037798.html > > > > [3] https://openjdk.java.net/projects/panama/ > > $ hg clone http://hg.openjdk.java.net/panama/dev/ -b vector-unstable > > > > > From vladimir.kozlov at oracle.com Sat May 2 01:05:19 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Fri, 1 May 2020 18:05:19 -0700 Subject: RFR (XXL): 8223347: Integration of Vector API (Incubator): x86 backend changes In-Reply-To: References: Message-ID: <5b4f8d0f-d355-8914-193a-3c87eccf2d34@oracle.com> On 5/1/20 5:55 PM, Viswanathan, Sandhya wrote: > Hi Vladimir, > > Thanks a lot for the feedback. > > We used an old existing separate branch to share the code for review and to track changes. > We didn?t know how to change the name of the branch from vector-unstable to vector-stable. Good to know that it does not mean that code is "unstable" ;) Katya filed today new bug [1]. Please look. Regards, Vladimir [1] https://bugs.openjdk.java.net/browse/JDK-8244269 > > Best Regards, > Sandhya > > -----Original Message----- > From: Vladimir Kozlov > Sent: Friday, May 01, 2020 5:32 PM > To: Viswanathan, Sandhya ; hotspot-compiler-dev at openjdk.java.net; core-libs-dev at openjdk.java.net; hotspot-dev > Subject: Re: RFR (XXL): 8223347: Integration of Vector API (Incubator): x86 backend changes > > Changes seems fine. Nice work. > > Why it is called "vector-unstable branch"? > > Thanks, > Vladimir K > > On 4/3/20 5:16 PM, Viswanathan, Sandhya wrote: >> Hi, >> >> >> Following up on review requests of API [0], Java implementation [1] and >> >> General Hotspot changes[3] for Vector API, here's a request for review >> >> of x86 backend changes required for supporting the API: >> >> >> >> JEP: https://openjdk.java.net/jeps/338 >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8223347 >> >> Webrev:http://cr.openjdk.java.net/~sviswanathan/VAPI_RFR/x86_webrev/webrev.00/ >> >> >> >> Complete implementation resides in vector-unstable branch of >> >> panama/dev repository [3]. >> >> Looking forward to your feedback. >> >> Best Regards, >> Sandhya >> >> >> [0] https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html >> >> >> >> [1] https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-April/065587.html >> >> >> >> [2] https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037798.html >> >> >> >> [3] https://openjdk.java.net/projects/panama/ >> >> $ hg clone http://hg.openjdk.java.net/panama/dev/ -b vector-unstable >> >> >> >> >> From palacsint at palacsint.hu Sat May 2 23:22:04 2020 From: palacsint at palacsint.hu (=?UTF-8?B?S2FyYWvDsyBNaWtsw7Nz?=) Date: Sun, 3 May 2020 01:22:04 +0200 Subject: Thread-Local Handshakes slowdown on single-cpu machines In-Reply-To: <5fb7355d-52ab-c8bb-1fdd-fd0b13a9de85@oracle.com> References: <5e65ba58-b599-e83c-b4ea-3d766ed3c457@palacsint.hu> <66059982-49bf-1a15-39fb-22008b069573@oracle.com> <7216aa59-ad6f-dd3c-4f37-4723d3b7e9a7@palacsint.hu> <857ffb5e-d9e5-3976-3968-124e3b47effb@oracle.com> <46fd88cd-3e68-ebbf-869f-355ebb196146@oracle.com> <5fb7355d-52ab-c8bb-1fdd-fd0b13a9de85@oracle.com> Message-ID: Hi Robbin, On 4/29/20 9:45 AM, Robbin Ehn wrote: > I have tested at least three different approaches trying to ease off the > CPU, all off them suffered from performance regression on MP machines. > We have internally seen this on very rare occasions on MP (we do not > have UPs). Since this have been so rare, not trivial to fix without > performance regression and I was under the assumption that UP were > basically dead :), it have not been prioritized. We have found that on VirtualBox machines used for testing where a single CPU was enough till the JDK was upgraded to JDK12+. Maybe the cheapest cloud instances are affected too (I'm not familiar with that). Best, Miklos From mikael.vidstedt at oracle.com Mon May 4 05:12:20 2020 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Sun, 3 May 2020 22:12:20 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) Message-ID: Please review this change which implements part of JEP 381: JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! Background: Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. Testing: A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. Cheers, Mikael [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ From stefan.karlsson at oracle.com Mon May 4 06:30:19 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Mon, 4 May 2020 08:30:19 +0200 Subject: RFR: 8244078: ProcessTools executeTestJvm and createJavaProcessBuilder have inconsistent handling of test.*.opts In-Reply-To: References: <44e3e1e7-7135-1892-12d3-cbf3707cfce0@oracle.com> Message-ID: <13c1f541-ac07-c925-26ee-574d5bba65df@oracle.com> On 2020-05-01 21:34, Chris Plummer wrote: > On 4/30/20 2:07 AM, Stefan Karlsson wrote: ... >> There was one odd thing in jdi that requires extra scrutiny: >> https://cr.openjdk.java.net/~stefank/8244078/webrev.01/test/jdk/com/sun/jdi/lib/jdb/Debuggee.java.udiff.html >> > Yes, that did look a odd at first glance, but I think that fact that > your removed addTestVmAndJavaOptions() and everything still built is > proof enough that it was just bit rotted code and not needed. Thanks for checking on this. StefanK > > Chris >> >> I've run this through tier1-3, and are currently running this through >> higher tiers. >> >> Thanks, >> StefanK > > From stefan.karlsson at oracle.com Mon May 4 08:28:27 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Mon, 4 May 2020 10:28:27 +0200 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: Hi Mikael, On 2020-05-04 07:12, Mikael Vidstedt wrote: > > Please review this change which implements part of JEP 381: > > JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 > webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ I went over this patch and collected some comments: src/hotspot/share/adlc/output_c.cpp src/hotspot/share/adlc/output_h.cpp Awkward code layout after change to. src/hotspot/share/c1/c1_Runtime1.cpp src/hotspot/share/classfile/classListParser.cpp src/hotspot/share/memory/arena.hpp src/hotspot/share/opto/chaitin.cpp test/hotspot/jtreg/gc/TestCardTablePageCommits.java Surrounding comments still refers to Sparc and/or Solaris. There are even more places if you search in the rest of the HotSpot source. Are we leaving those for a separate cleanup pass? src/hotspot/share/gc/g1/g1HeapRegionAttr.hpp Remove comment: // We use different types to represent the state value depending on platform as // some have issues loading parts of words. src/hotspot/share/gc/shared/memset_with_concurrent_readers.hpp Fuse the declaration and definition, now that we only have one implementation. Maybe even remove function/file at some point. src/hotspot/share/utilities/globalDefinitions.hpp Now that STACK_BIAS is always 0, should we remove its usages? Follow-up RFE? src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot/src/org/graalvm/compiler/hotspot/meta/HotSpotGraphBuilderPlugins.java Maybe remove decryptSuffix? src/utils/hsdis/Makefile Is this really correct? Shouldn't: ARCH1=$(CPU:x86_64=amd64) ARCH2=$(ARCH1:i686=i386) ARCH=$(ARCH2:sparc64=sparcv9) be changed to: ARCH1=$(CPU:x86_64=amd64) ARCH=$(ARCH1:i686=i386) so that we have ARCH defined? Other than that this looks good to me. StefanK > JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 > > > Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! > > > Background: > > Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. > > For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. > > In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. > > A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! > > Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. > > Testing: > > A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. > > Cheers, > Mikael > > [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ > From kim.barrett at oracle.com Mon May 4 08:54:25 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Mon, 4 May 2020 04:54:25 -0400 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: > On May 4, 2020, at 4:28 AM, Stefan Karlsson wrote: > > Hi Mikael, > > On 2020-05-04 07:12, Mikael Vidstedt wrote: >> Please review this change which implements part of JEP 381: >> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ > > [?] > > src/hotspot/share/gc/shared/memset_with_concurrent_readers.hpp > > Fuse the declaration and definition, now that we only have one implementation. Maybe even remove function/file at some point. I think that cleanup is covered by JDK-8142349. From thomas.schatzl at oracle.com Mon May 4 09:11:56 2020 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 4 May 2020 11:11:56 +0200 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: Hi, On 04.05.20 10:28, Stefan Karlsson wrote: > Hi Mikael, > > On 2020-05-04 07:12, Mikael Vidstedt wrote: >> >> Please review this change which implements part of JEP 381: >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >> webrev: >> http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ >> > > I went over this patch and collected some comments: > > src/hotspot/share/adlc/output_c.cpp > src/hotspot/share/adlc/output_h.cpp > > Awkward code layout after change to. > > > src/hotspot/share/c1/c1_Runtime1.cpp > src/hotspot/share/classfile/classListParser.cpp > src/hotspot/share/memory/arena.hpp > src/hotspot/share/opto/chaitin.cpp > test/hotspot/jtreg/gc/TestCardTablePageCommits.jav > > Surrounding comments still refers to Sparc and/or Solaris. > > There are even more places if you search in the rest of the HotSpot > source. Are we leaving those for a separate cleanup pass? In addition to "sparc", "solaris", also "solstudio"/"Sun Studio"/"SS compiler bug"/"niagara" yield some search (=grep) results. Some of these locations look like additional RFEs. Thanks, Thomas From thomas.stuefe at gmail.com Mon May 4 09:29:14 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 4 May 2020 11:29:14 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: Message-ID: Hi, I updated the webrev and rebased it to jdk head (now atop of 4198213fc371: 8230940: Obsolete MonitorBound). New webrev: http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.01/webrev/ Delta to last: http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.01/webrev/index.html @Ioi Lam : > // User may have specified an invalid base address. Should we ignore >it or assert? >guarantee(CompressedKlassPointers::is_valid_base((address)shared_base), > "SharedBaseAddress: " PTR_FORMAT " is not a valid base.", >p2i(shared_base)); > >This will cause the VM to crash. I think it's better (1) exit the VM >properly with an error code, or (2) override the user's input. I reworked this coding. I opted for ignoring this - if the user specifies a SharedBaseAddress which is invalid on the given platform the VM will print a warning and choose the default value. This mirrors (roughly) how we handle mapping errors when we cannot attach at SharedBaseAddress due to ASLR, which is also not treated as a fatal error. >For testing the CDS relocation code, I would suggest running: > >cd test/hotspot/jtreg >jtreg -javaoption:-XX:+UnlockDiagnosticVMOptions \ > -javaoption:-XX:ArchiveRelocationMode=1 \ > -javaoption:-XX:NativeMemoryTracking=detail > :hotspot_cds_relocation > >This will place the CCS at random locations picked by the OS. Tested on my linux x64 box, all went well. >metaspace.cpp: > >If your intention is to "shake things up a little", it's not a good idea >to include it in a complex change set. If things indeed go wrong, we >don't know who caused it (your CCS changes, or old bugs triggered by >this debug code), and we will end up backing out the entire changeset. I see your point and removed that code. I am not completely happy with it because this code is supposed to test whether this patch works as intended (removing all dependencies to Metaspace::reserve_alignment()). But since I have enough test runs with an increased alignment, I can disable this and will move this into a separate RFR. >I would suggest putting this in a different RFE, and even push it now. Cannot push it now, since it depends on this rework patch. In isolation it will cause crashes in CDS, which is one of the reason for this rework. @Nick Gasson : >On line 42 I'd prefer to use (4 << LogKlassAlignmentInBytes)*G as it >currently is in metaspace.cpp instead of the literal 32. Done. As for the discussions about optimizing the compressed class pointer encoding on aarch64, unless you have specific wishes about this patch I am ignoring it for now; I understand you want to do any optimizations for that in a follow up patch. Thank you, Thomas From aph at redhat.com Mon May 4 10:21:21 2020 From: aph at redhat.com (Andrew Haley) Date: Mon, 4 May 2020 11:21:21 +0100 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: Message-ID: <5cf3654a-e5d0-6405-6568-15639353b778@redhat.com> On 5/4/20 10:29 AM, Thomas St?fe wrote: > As for the discussions about optimizing the compressed class pointer > encoding on aarch64, unless you have specific wishes about this patch I am > ignoring it for now; I understand you want to do any optimizations for that > in a follow up patch. OK. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From richard.reingruber at sap.com Mon May 4 10:33:08 2020 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Mon, 4 May 2020 10:33:08 +0000 Subject: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant In-Reply-To: <9c49ea2d-e3b8-b576-1d17-d18ad87cd6ed@oracle.com> References: <3c59b9f9-ec38-18c9-8f24-e1186a08a04a@oracle.com> <410eed04-e2ef-0f4f-1c56-19e6734a10f6@oracle.com> <81d7caa8-4244-85f3-4d4e-78117fe5e25b@oss.nttdata.com> <550b95ac-8b29-1eb8-a507-533e81d02322@oracle.com> <9c49ea2d-e3b8-b576-1d17-d18ad87cd6ed@oracle.com> Message-ID: // Trimmed the list of recipients. If the list gets too long then the message needs to be approved // by a moderator. Hi David, > On 28/04/2020 12:09 am, Reingruber, Richard wrote: > > Hi David, > > > >> Not a review but some general commentary ... > > > > That's welcome. > Having had to take an even closer look now I have a review comment too :) > src/hotspot/share/prims/jvmtiThreadState.cpp > void JvmtiThreadState::invalidate_cur_stack_depth() { > ! assert(SafepointSynchronize::is_at_safepoint() || > ! (Thread::current()->is_VM_thread() && > get_thread()->is_vmthread_processing_handshake()) || > (JavaThread *)Thread::current() == get_thread(), > "must be current thread or at safepoint"); You're looking at an outdated webrev, I'm afraid. This would be the post with the current webrev.1 http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-April/031245.html Thanks, Richard. -----Original Message----- From: David Holmes Sent: Montag, 4. Mai 2020 08:51 To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant Hi Richard, On 28/04/2020 12:09 am, Reingruber, Richard wrote: > Hi David, > >> Not a review but some general commentary ... > > That's welcome. Having had to take an even closer look now I have a review comment too :) src/hotspot/share/prims/jvmtiThreadState.cpp void JvmtiThreadState::invalidate_cur_stack_depth() { ! assert(SafepointSynchronize::is_at_safepoint() || ! (Thread::current()->is_VM_thread() && get_thread()->is_vmthread_processing_handshake()) || (JavaThread *)Thread::current() == get_thread(), "must be current thread or at safepoint"); The message needs updating to include handshakes. More below ... >> On 25/04/2020 2:08 am, Reingruber, Richard wrote: >>> Hi Yasumasa, Patricio, >>> >>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>> Does it help you? I think it gives you to remove workaround. >>>>> >>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>> >>>> Thanks for your information. >>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>> I will modify and will test it after yours. >>> >>> Thanks :) >>> >>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>> to me, how this has to be handled. >>> >>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>> >>> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >>> also I'm unsure if a thread should do safepoint checks while executing a handshake. > >> I'm growing increasingly concerned that use of direct handshakes to >> replace VM operations needs a much greater examination for correctness >> than might initially be thought. I see a number of issues: > > I agree. I'll address your concerns in the context of this review thread for JDK-8238585 below. > > In addition I would suggest to take the general part of the discussion to a dedicated thread or to > the review thread for JDK-8242427. I would like to keep this thread closer to its subject. I will focus on the issues in the context of this particular change then, though the issues themselves are applicable to all handshake situations (and more so with direct handshakes). This is mostly just discussion. >> First, the VMThread executes (most) VM operations with a clean stack in >> a clean state, so it has lots of room to work. If we now execute the >> same logic in a JavaThread then we risk hitting stackoverflows if >> nothing else. But we are also now executing code in a JavaThread and so >> we have to be sure that code is not going to act differently (in a bad >> way) if executed by a JavaThread rather than the VMThread. For example, >> may it be possible that if executing in the VMThread we defer some >> activity that might require execution of Java code, or else hand it off >> to one of the service threads? If we execute that code directly in the >> current JavaThread instead we may not be in a valid state (e.g. consider >> re-entrancy to various subsystems that is not allowed). > > It is not too complex, what EnterInterpOnlyModeClosure::do_thread() is doing. I already added a > paragraph to the JBS-Item [1] explaining why the direct handshake is sufficient from a > synchronization point of view. Just to be clear, your proposed change is not using a direct handshake. > Furthermore the stack is walked and the return pc of compiled frames is replaced with the address of > the deopt handler. > > I can't see why this cannot be done with a direct handshake. Something very similar is already done > in JavaThread::deoptimize_marked_methods() which is executed as part of an ordinary handshake. Note that existing non-direct handshakes may also have issues that not have been fully investigated. > The demand on stack-space should be very modest. I would not expect a higher risk for stackoverflow. For the target thread if you use more stack than would be used stopping at a safepoint then you are at risk. For the thread initiating the direct handshake if you use more stack than would be used enqueuing a VM operation, then you are at risk. As we have not quantified these numbers, nor have any easy way to establish the stack use of the actual code to be executed, we're really just hoping for the best. This is a general problem with handshakes that needs to be investigated more deeply. As a simple, general, example just imagine if the code involves logging that might utilise an on-stack buffer. >> Second, we have this question mark over what happens if the operation >> hits further safepoint or handshake polls/checks? Are there constraints >> on what is allowed here? How can we recognise this problem may exist and >> so deal with it? > > The thread in EnterInterpOnlyModeClosure::do_thread() can't become safepoint/handshake safe. I > tested locally test/hotspot/jtreg:vmTestbase_nsk_jvmti with a NoSafepointVerifier. That's good to hear but such tests are not exhaustive, they will detect if you do reach a safepoint/handshake but they can't prove that you cannot reach one. What you have done is necessary but may not be sufficient. Plus you didn't actually add the NSV to the code - is there a reason we can't actually keep it in do_thread? (I'm not sure if the NSV also acts as a NoHandshakeVerifier?) >> Third, while we are generally considering what appear to be >> single-thread operations, which should be amenable to a direct >> handshake, we also have to be careful that some of the code involved >> doesn't already expect/assume we are at a safepoint - e.g. a VM op may >> not need to take a lock where a direct handshake might! > > See again my arguments in the JBS item [1]. Yes I see the reasoning and that is good. My point is a general one as it may not be obvious when such assumptions exist in the current code. Thanks, David > Thanks, > Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8238585 > > -----Original Message----- > From: David Holmes > Sent: Montag, 27. April 2020 07:16 > To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net > Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant > > Hi all, > > Not a review but some general commentary ... > > On 25/04/2020 2:08 am, Reingruber, Richard wrote: >> Hi Yasumasa, Patricio, >> >>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>> Does it help you? I think it gives you to remove workaround. >>>> >>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >> >>> Thanks for your information. >>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>> I will modify and will test it after yours. >> >> Thanks :) >> >>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>> to me, how this has to be handled. >> >>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >> >> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >> also I'm unsure if a thread should do safepoint checks while executing a handshake. > > I'm growing increasingly concerned that use of direct handshakes to > replace VM operations needs a much greater examination for correctness > than might initially be thought. I see a number of issues: > > First, the VMThread executes (most) VM operations with a clean stack in > a clean state, so it has lots of room to work. If we now execute the > same logic in a JavaThread then we risk hitting stackoverflows if > nothing else. But we are also now executing code in a JavaThread and so > we have to be sure that code is not going to act differently (in a bad > way) if executed by a JavaThread rather than the VMThread. For example, > may it be possible that if executing in the VMThread we defer some > activity that might require execution of Java code, or else hand it off > to one of the service threads? If we execute that code directly in the > current JavaThread instead we may not be in a valid state (e.g. consider > re-entrancy to various subsystems that is not allowed). > > Second, we have this question mark over what happens if the operation > hits further safepoint or handshake polls/checks? Are there constraints > on what is allowed here? How can we recognise this problem may exist and > so deal with it? > > Third, while we are generally considering what appear to be > single-thread operations, which should be amenable to a direct > handshake, we also have to be careful that some of the code involved > doesn't already expect/assume we are at a safepoint - e.g. a VM op may > not need to take a lock where a direct handshake might! > > Cheers, > David > ----- > >> @Patricio, coming back to my question [1]: >> >> In the example you gave in your answer [2]: the java thread would execute a vm operation during a >> direct handshake operation, while the VMThread is actually in the middle of a VM_HandshakeAllThreads >> operation, waiting to handshake the same handshakee: why can't the VMThread just proceed? The >> handshakee would be safepoint safe, wouldn't it? >> >> Thanks, Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301677&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301677 >> >> [2] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301763&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301763 >> >> -----Original Message----- >> From: Yasumasa Suenaga >> Sent: Freitag, 24. April 2020 17:23 >> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >> >> Hi Richard, >> >> On 2020/04/24 23:44, Reingruber, Richard wrote: >>> Hi Yasumasa, >>> >>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>> Does it help you? I think it gives you to remove workaround. >>> >>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >> >> Thanks for your information. >> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >> I will modify and will test it after yours. >> >> >>> Also my first impression was that it won't be that easy from a synchronization point of view to >>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>> to me, how this has to be handled. >> >> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >> >> >> Thanks, >> >> Yasumasa >> >> >>> So it appears to me that it would be easier to push JDK-8242427 after this (JDK-8238585). >>> >>>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>> >>> Would be interesting to see how you handled the issues above :) >>> >>> Thanks, Richard. >>> >>> [1] See question in comment https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14302030&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14302030 >>> >>> -----Original Message----- >>> From: Yasumasa Suenaga >>> Sent: Freitag, 24. April 2020 13:34 >>> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>> >>> Hi Richard, >>> >>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>> Does it help you? I think it gives you to remove workaround. >>> >>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> On 2020/04/24 17:18, Reingruber, Richard wrote: >>>> Hi Patricio, Vladimir, and Serguei, >>>> >>>> now that direct handshakes are available, I've updated the patch to make use of them. >>>> >>>> In addition I have done some clean-up changes I missed in the first webrev. >>>> >>>> Finally I have implemented the workaround suggested by Patricio to avoid nesting the handshake >>>> into the vm operation VM_SetFramePop [1] >>>> >>>> Kindly review again: >>>> >>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1/ >>>> Webrev(delta): http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1.inc/ >>>> >>>> I updated the JBS item explaining why the vm operation VM_EnterInterpOnlyMode can be replaced with a >>>> direct handshake: >>>> >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>> >>>> Testing: >>>> >>>> * JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>> >>>> * Submit-repo: mach5-one-rrich-JDK-8238585-20200423-1436-10441737 >>>> >>>> Thanks, >>>> Richard. >>>> >>>> [1] An assertion in Handshake::execute_direct() fails, if called be VMThread, because it is no JavaThread. >>>> >>>> -----Original Message----- >>>> From: hotspot-dev On Behalf Of Reingruber, Richard >>>> Sent: Freitag, 14. Februar 2020 19:47 >>>> To: Patricio Chilano ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>> Subject: RE: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>> >>>> Hi Patricio, >>>> >>>> > > I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>> > > handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>> > > Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>> > > >>>> > > > Alternatively I think you could do something similar to what we do in >>>> > > > Deoptimization::deoptimize_all_marked(): >>>> > > > >>>> > > > EnterInterpOnlyModeClosure hs; >>>> > > > if (SafepointSynchronize::is_at_safepoint()) { >>>> > > > hs.do_thread(state->get_thread()); >>>> > > > } else { >>>> > > > Handshake::execute(&hs, state->get_thread()); >>>> > > > } >>>> > > > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>> > > > HandshakeClosure() constructor) >>>> > > >>>> > > Maybe this could be used also in the Handshake::execute() methods as general solution? >>>> > Right, we could also do that. Avoiding to clear the polling page in >>>> > HandshakeState::clear_handshake() should be enough to fix this issue and >>>> > execute a handshake inside a safepoint, but adding that "if" statement >>>> > in Hanshake::execute() sounds good to avoid all the extra code that we >>>> > go through when executing a handshake. I filed 8239084 to make that change. >>>> >>>> Thanks for taking care of this and creating the RFE. >>>> >>>> > >>>> > > > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>> > > > always called in a nested operation or just sometimes. >>>> > > >>>> > > At least one execution path without vm operation exists: >>>> > > >>>> > > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>> > > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>> > > JvmtiEventControllerPrivate::recompute_enabled() : void >>>> > > JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>> > > JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>> > > JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>> > > jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>> > > >>>> > > I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>> > > handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>> > > encouraged to do it with a handshake :) >>>> > Ah! I think you can still do it with a handshake with the >>>> > Deoptimization::deoptimize_all_marked() like solution. I can change the >>>> > if-else statement with just the Handshake::execute() call in 8239084. >>>> > But up to you. : ) >>>> >>>> Well, I think that's enough encouragement :) >>>> I'll wait for 8239084 and try then again. >>>> (no urgency and all) >>>> >>>> Thanks, >>>> Richard. >>>> >>>> -----Original Message----- >>>> From: Patricio Chilano >>>> Sent: Freitag, 14. Februar 2020 15:54 >>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>> >>>> Hi Richard, >>>> >>>> On 2/14/20 9:58 AM, Reingruber, Richard wrote: >>>>> Hi Patricio, >>>>> >>>>> thanks for having a look. >>>>> >>>>> > I?m only commenting on the handshake changes. >>>>> > I see that operation VM_EnterInterpOnlyMode can be called inside >>>>> > operation VM_SetFramePop which also allows nested operations. Here is a >>>>> > comment in VM_SetFramePop definition: >>>>> > >>>>> > // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>> > // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>> > >>>>> > So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>> > could have a handshake inside a safepoint operation. The issue I see >>>>> > there is that at the end of the handshake the polling page of the target >>>>> > thread could be disarmed. So if the target thread happens to be in a >>>>> > blocked state just transiently and wakes up then it will not stop for >>>>> > the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>> > polling page is armed at the beginning of disarm_safepoint(). >>>>> >>>>> I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>>> handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>>> Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>>> >>>>> > Alternatively I think you could do something similar to what we do in >>>>> > Deoptimization::deoptimize_all_marked(): >>>>> > >>>>> > EnterInterpOnlyModeClosure hs; >>>>> > if (SafepointSynchronize::is_at_safepoint()) { >>>>> > hs.do_thread(state->get_thread()); >>>>> > } else { >>>>> > Handshake::execute(&hs, state->get_thread()); >>>>> > } >>>>> > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>> > HandshakeClosure() constructor) >>>>> >>>>> Maybe this could be used also in the Handshake::execute() methods as general solution? >>>> Right, we could also do that. Avoiding to clear the polling page in >>>> HandshakeState::clear_handshake() should be enough to fix this issue and >>>> execute a handshake inside a safepoint, but adding that "if" statement >>>> in Hanshake::execute() sounds good to avoid all the extra code that we >>>> go through when executing a handshake. I filed 8239084 to make that change. >>>> >>>>> > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>> > always called in a nested operation or just sometimes. >>>>> >>>>> At least one execution path without vm operation exists: >>>>> >>>>> JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>>> JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>>> JvmtiEventControllerPrivate::recompute_enabled() : void >>>>> JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>>> JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>>> JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>>> jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>>> >>>>> I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>>> handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>>> encouraged to do it with a handshake :) >>>> Ah! I think you can still do it with a handshake with the >>>> Deoptimization::deoptimize_all_marked() like solution. I can change the >>>> if-else statement with just the Handshake::execute() call in 8239084. >>>> But up to you.? : ) >>>> >>>> Thanks, >>>> Patricio >>>>> Thanks again, >>>>> Richard. >>>>> >>>>> -----Original Message----- >>>>> From: Patricio Chilano >>>>> Sent: Donnerstag, 13. Februar 2020 18:47 >>>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>> >>>>> Hi Richard, >>>>> >>>>> I?m only commenting on the handshake changes. >>>>> I see that operation VM_EnterInterpOnlyMode can be called inside >>>>> operation VM_SetFramePop which also allows nested operations. Here is a >>>>> comment in VM_SetFramePop definition: >>>>> >>>>> // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>> // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>> >>>>> So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>> could have a handshake inside a safepoint operation. The issue I see >>>>> there is that at the end of the handshake the polling page of the target >>>>> thread could be disarmed. So if the target thread happens to be in a >>>>> blocked state just transiently and wakes up then it will not stop for >>>>> the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>> polling page is armed at the beginning of disarm_safepoint(). >>>>> >>>>> I think one option could be to remove >>>>> SafepointMechanism::disarm_if_needed() in >>>>> HandshakeState::clear_handshake() and let each JavaThread disarm itself >>>>> for the handshake case. >>>>> >>>>> Alternatively I think you could do something similar to what we do in >>>>> Deoptimization::deoptimize_all_marked(): >>>>> >>>>> ? EnterInterpOnlyModeClosure hs; >>>>> ? if (SafepointSynchronize::is_at_safepoint()) { >>>>> ??? hs.do_thread(state->get_thread()); >>>>> ? } else { >>>>> ??? Handshake::execute(&hs, state->get_thread()); >>>>> ? } >>>>> (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>> HandshakeClosure() constructor) >>>>> >>>>> I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>> always called in a nested operation or just sometimes. >>>>> >>>>> Thanks, >>>>> Patricio >>>>> >>>>> On 2/12/20 7:23 AM, Reingruber, Richard wrote: >>>>>> // Repost including hotspot runtime and gc lists. >>>>>> // Dean Long suggested to do so, because the enhancement replaces a vm operation >>>>>> // with a handshake. >>>>>> // Original thread: http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-February/030359.html >>>>>> >>>>>> Hi, >>>>>> >>>>>> could I please get reviews for this small enhancement in hotspot's jvmti implementation: >>>>>> >>>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.0/ >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>>>> >>>>>> The change avoids making all compiled methods on stack not_entrant when switching a java thread to >>>>>> interpreter only execution for jvmti purposes. It is sufficient to deoptimize the compiled frames on stack. >>>>>> >>>>>> Additionally a handshake is used instead of a vm operation to walk the stack and do the deoptimizations. >>>>>> >>>>>> Testing: JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>>>> >>>>>> Thanks, Richard. >>>>>> >>>>>> See also my question if anyone knows a reason for making the compiled methods not_entrant: >>>>>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-January/030339.html >>>> From kim.barrett at oracle.com Mon May 4 10:47:11 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Mon, 4 May 2020 06:47:11 -0400 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: <553D0344-188E-455B-A03E-D080C1484B41@oracle.com> > On May 4, 2020, at 1:12 AM, Mikael Vidstedt wrote: > > > Please review this change which implements part of JEP 381: > > JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 > webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ > JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 I've only looked at the src/hotspot changes so far. I've not duplicated comments already made by Stefan. Looks good, other than a few very minor issues, some of which might already be covered by planned followup RFEs. ------------------------------------------------------------------------------ I think with sparc removal, c1's pack64/unpack64 stuff is no longer used. So I think that can be removed from c1_LIR.[ch]pp too. ------------------------------------------------------------------------------ src/hotspot/share/opto/generateOptoStub.cpp 225 // Clear last_Java_pc and (optionally)_flags The sparc-specific clearing of "flags" is gone. ------------------------------------------------------------------------------ src/hotspot/share/runtime/deoptimization.cpp 1086 *((jlong *) check_alignment_get_addr(obj, index, 8)) = (jlong) *((jlong *) &val); [pre-existing] The rhs cast to jlong is unnecessary, since it's dereferencing a jlong*. ------------------------------------------------------------------------------ src/hotspot/share/runtime/flags/jvmFlagConstraintsCompiler.cpp 236 JVMFlag::Error CompilerThreadPriorityConstraintFunc(intx value, bool verbose) { 237 return JVMFlag::SUCCESS; 238 } After SOLARIS code removal we no longer need this constraint function. ------------------------------------------------------------------------------ src/hotspot/share/runtime/globals.hpp 2392 experimental(size_t, ArrayAllocatorMallocLimit, \ 2393 (size_t)-1, \ Combine these lines. ------------------------------------------------------------------------------ src/hotspot/share/utilities/dtrace.hpp Shuold just eliminate all traces of HS_DTRACE_WORKAROUND_TAIL_CALL_BUG. ------------------------------------------------------------------------------ From matthias.baesken at sap.com Mon May 4 10:55:30 2020 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Mon, 4 May 2020 10:55:30 +0000 Subject: RFR [XS]: 8244196: adjust output in os_linux In-Reply-To: <9f7e3d4c-d6e0-6a6e-16ae-51fba319fcfa@oracle.com> References: <9f7e3d4c-d6e0-6a6e-16ae-51fba319fcfa@oracle.com> Message-ID: Hi David , the newlines were removed by me because when looking at the hs_err files it seemed to me, that we have too much of them. ( but of course this might be a matter of taste , and not so much of correctness ) The output part looks like (from one of our Linux x86_64 machines) this now : /sys/kernel/mm/transparent_hugepage/enabled:[always] madvise never /sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):[always] madvise never /proc/sys/kernel/threads-max (system-wide limit on the number of threads):1032335 /proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):2147483647 /proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):81920 Best regards, Matthias -----Original Message----- From: David Holmes Sent: Freitag, 1. Mai 2020 01:31 To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' Subject: Re: RFR [XS]: 8244196: adjust output in os_linux Hi Matthias, On 1/05/2020 1:46 am, Baesken, Matthias wrote: > Hello, please review this small change to os_linux.cpp . > > It uses the recently introduced _print_ascii_file_h > Function t more places (makes the code a bit shorter ) . I like the intent here but I think we have lost a few newlines in the process. It isn't even clear to me that the result of print_ascii_file will end with a newline being printed. Can you check the before and after layout for readability please (and add to bug report). Thanks, David > > Bug/webrev : > > https://bugs.openjdk.java.net/browse/JDK-8244196 > > http://cr.openjdk.java.net/~mbaesken/webrevs/8244196.0/ > > Thanks, Matthias > From vladimir.kozlov at oracle.com Mon May 4 19:01:17 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Mon, 4 May 2020 12:01:17 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: <25d5e2fa-8909-0b94-e0b2-6b5aaa224492@oracle.com> JIT, AOT, JVMCI and Graal changes seem fine to me. It would be interesting to see shared code execution coverage change. There are places where we use flags and setting instead of #ifdef SPARC which may not be executed now or executed partially. We may simplify such code too. Thanks, Vladimir On 5/3/20 10:12 PM, Mikael Vidstedt wrote: > > Please review this change which implements part of JEP 381: > > JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 > webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ > JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 > > > Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! > > > Background: > > Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. > > For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. > > In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. > > A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! > > Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. > > Testing: > > A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. > > Cheers, > Mikael > > [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ > From paul.sandoz at oracle.com Mon May 4 20:09:54 2020 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 4 May 2020 13:09:54 -0700 Subject: RFR (XXL): 8223347: Integration of Vector API (Incubator): General HotSpot changes In-Reply-To: References: <25a564a1-7f40-6988-060f-86b06e02ad21@oracle.com> Message-ID: <07723ED2-E6B5-4D7E-A32D-B33CC5C0B682@oracle.com> Hi Colleen, Thanks for the review. IIRC at one point the number of vector-based modifications was far higher, you would have hated it even more :-) thankfully Vladimir I. managed to reduce it down. Your general idea of separation is a good one, Paul. > On Apr 17, 2020, at 5:21 AM, coleen.phillimore at oracle.com wrote: > > > Hi, I looked at the runtime changes and they seem fine. > > http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/all.00-03/src/hotspot/share/classfile/vmSymbols.hpp.udiff.html > > I really hate that intrinsics are in this header file and this change has to add so much in this file. I filed https://bugs.openjdk.java.net/browse/JDK-8243066 for after this change. Feel free to add details to the RFE. At least, I think all of these symbols are shared in CDS. > > Coleen > > On 4/16/20 8:32 AM, Vladimir Ivanov wrote: >> Hi, >> >> Any more reviews, please? Especially, compiler and runtime-related changes. >> >> Thanks in advance! >> >> Best regards, >> Vladimir Ivanov >> >> On 04.04.2020 02:12, Vladimir Ivanov wrote: >>> Hi, >>> >>> Following up on review requests of API [0] and Java implementation [1] for Vector API (JEP 338 [2]), here's a request for review of general HotSpot changes (in shared code) required for supporting the API: >>> >>> >>> http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/all.00-03/ >>> >>> (First of all, to set proper expectations: since the JEP is still in Candidate state, the intention is to initiate preliminary round(s) of review to inform the community and gather feedback before sending out final/official RFRs once the JEP is Targeted to a release.) >>> >>> Vector API (being developed in Project Panama [3]) relies on JVM support to utilize optimal vector hardware instructions at runtime. It interacts with JVM through intrinsics (declared in jdk.internal.vm.vector.VectorSupport [4]) which expose vector operations support in C2 JIT-compiler. >>> >>> As Paul wrote earlier: "A vector intrinsic is an internal low-level vector operation. The last argument to the intrinsic is fall back behavior in Java, implementing the scalar operation over the number of elements held by the vector. Thus, If the intrinsic is not supported in C2 for the other arguments then the Java implementation is executed (the Java implementation is always executed when running in the interpreter or for C1)." >>> >>> The rest of JVM support is about aggressively optimizing vector boxes to minimize (ideally eliminate) the overhead of boxing for vector values. >>> It's a stop-the-gap solution for vector box elimination problem until inline classes arrive. Vector classes are value-based and in the longer term will be migrated to inline classes once the support becomes available. >>> >>> Vector API talk from JVMLS'18 [5] contains brief overview of JVM implementation and some details. >>> >>> Complete implementation resides in vector-unstable branch of panama/dev repository [6]. >>> >>> Now to gory details (the patch is split in multiple "sub-webrevs"): >>> >>> =========================================================== >>> >>> (1) http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/00.backend.shared/ >>> >>> Ideal vector nodes for new operations introduced by Vector API. >>> >>> (Platform-specific back end support will be posted for review separately). >>> >>> =========================================================== >>> >>> (2) http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/01.intrinsics/ >>> >>> JVM Java interface (VectorSupport) and intrinsic support in C2. >>> >>> Vector instances are initially represented as VectorBox macro nodes and "unboxing" is represented by VectorUnbox node. It simplifies vector box elimination analysis and the nodes are expanded later right before EA pass. >>> >>> Vectors have 2-level on-heap representation: for the vector value primitive array is used as a backing storage and it is encapsulated in a typed wrapper (e.g., Int256Vector - vector of 8 ints - contains a int[8] instance which is used to store vector value). >>> >>> Unless VectorBox node goes away, it needs to be expanded into an allocation eventually, but it is a pure node and doesn't have any JVM state associated with it. The problem is solved by keeping JVM state separately in a VectorBoxAllocate node associated with VectorBox node and use it during expansion. >>> >>> Also, to simplify vector box elimination, inlining of vector reboxing calls (VectorSupport::maybeRebox) is delayed until the analysis is over. >>> >>> =========================================================== >>> >>> (3) http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/02.vbox_elimination/ >>> >>> Vector box elimination analysis implementation. (Brief overview: slides #36-42 [5].) >>> >>> The main part is devoted to scalarization across safepoints and rematerialization support during deoptimization. In C2-generated code vector operations work with raw vector values which live in registers or spilled on the stack and it allows to avoid boxing/unboxing when a vector value is alive across a safepoint. As with other values, there's just a location of the vector value at the safepoint and vector type information recorded in the relevant nmethod metadata and all the heavy-lifting happens only when rematerialization takes place. >>> >>> The analysis preserves object identity invariants except during aggressive reboxing (guarded by -XX:+EnableAggressiveReboxing). >>> >>> (Aggressive reboxing is crucial for cases when vectors "escape": it allocates a fresh instance at every escape point thus enabling original instance to go away.) >>> >>> =========================================================== >>> >>> (4) http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/03.module.hotspot/ >>> >>> HotSpot changes for jdk.incubator.vector module. Vector support is makred experimental and turned off by default. JEP 338 proposes the API to be released as an incubator module, so a user has to specify "--add-module jdk.incubator.vector" on the command line to be able to use it. >>> When user does that, JVM automatically enables Vector API support. >>> It improves usability (user doesn't need to separately "open" the API and enable JVM support) while minimizing risks of destabilitzation from new code when the API is not used. >>> >>> >>> That's it! Will be happy to answer any questions. >>> >>> And thanks in advance for any feedback! >>> >>> Best regards, >>> Vladimir Ivanov >>> >>> [0] https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html >>> >>> [1] https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-April/041228.html >>> >>> [2] https://openjdk.java.net/jeps/338 >>> >>> [3] https://openjdk.java.net/projects/panama/ >>> >>> [4] http://cr.openjdk.java.net/~vlivanov/panama/vector/jep338/hotspot.shared/webrev.00/01.intrinsics/src/java.base/share/classes/jdk/internal/vm/vector/VectorSupport.java.html >>> >>> [5] http://cr.openjdk.java.net/~vlivanov/talks/2018_JVMLS_VectorAPI.pdf >>> >>> [6] http://hg.openjdk.java.net/panama/dev/shortlog/92bbd44386e9 >>> >>> $ hg clone http://hg.openjdk.java.net/panama/dev/ -b vector-unstable > From igor.ignatyev at oracle.com Mon May 4 21:29:39 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Mon, 4 May 2020 14:29:39 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: Hi Mikael, the changes in /test/ look good to me. I have a question regarding src/jdk.internal.vm.compiler/*, aren't these files part of graal-compiler and hence will be brought back by the next graal update? Thanks, -- Igor > On May 3, 2020, at 10:12 PM, Mikael Vidstedt wrote: > > > Please review this change which implements part of JEP 381: > > JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 > webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ > JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 > > > Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! > > > Background: > > Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. > > For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. > > In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. > > A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! > > Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. > > Testing: > > A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. > > Cheers, > Mikael > > [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ > From vladimir.kozlov at oracle.com Mon May 4 21:49:47 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Mon, 4 May 2020 14:49:47 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: <60d3a22f-d4b4-280d-a8a7-1a306b7ad483@oracle.com> I filed Graal issue to change mx script to filter out SPARC code when we do sync Graal changes into JDK. For Graal shared code we may need to have versioning for latest JDK as we do in other cases. Regards, Vladimir On 5/4/20 2:29 PM, Igor Ignatyev wrote: > Hi Mikael, > > the changes in /test/ look good to me. > > I have a question regarding src/jdk.internal.vm.compiler/*, aren't these files part of graal-compiler and hence will be brought back by the next graal update? > > Thanks, > -- Igor > >> On May 3, 2020, at 10:12 PM, Mikael Vidstedt wrote: >> >> >> Please review this change which implements part of JEP 381: >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ >> JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 >> >> >> Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! >> >> >> Background: >> >> Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. >> >> For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. >> >> In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. >> >> A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! >> >> Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. >> >> Testing: >> >> A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. >> >> Cheers, >> Mikael >> >> [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ >> > From david.holmes at oracle.com Tue May 5 07:30:53 2020 From: david.holmes at oracle.com (David Holmes) Date: Tue, 5 May 2020 17:30:53 +1000 Subject: RFR [XS]: 8244196: adjust output in os_linux In-Reply-To: References: <9f7e3d4c-d6e0-6a6e-16ae-51fba319fcfa@oracle.com> Message-ID: <9397d7b3-08ed-383f-7e8b-f7ac189220ab@oracle.com> On 4/05/2020 8:55 pm, Baesken, Matthias wrote: > Hi David , the newlines were removed by me because when looking at the hs_err files it seemed to me, that we have too much of them. > ( but of course this might be a matter of taste , and not so much of correctness ) Very much a matter of taste. :) Presently we print: header1: value1 header2: value2 whereas you are now putting all that on one line and deleting empty lines: header1:value1 header2:value2 While I personally prefer things on one line and no blank lines I am concerned that this make unnecessarily break scripts that try to mine hs_err logs. Also you need as space after the : so this static void _print_ascii_file_h(const char* header, const char* filename, outputStream* st) { st->print("%s", header); should use "%s " (and arguably the : should be part of the format string rather than in the header itself). > The output part looks like (from one of our Linux x86_64 machines) this now : > > /sys/kernel/mm/transparent_hugepage/enabled:[always] madvise never > /sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):[always] madvise never > > /proc/sys/kernel/threads-max (system-wide limit on the number of threads):1032335 > /proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):2147483647 > /proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):81920 Please add before/after examples t the bug report. Thanks, David > Best regards, Matthias > > -----Original Message----- > From: David Holmes > Sent: Freitag, 1. Mai 2020 01:31 > To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' > Subject: Re: RFR [XS]: 8244196: adjust output in os_linux > > Hi Matthias, > > On 1/05/2020 1:46 am, Baesken, Matthias wrote: >> Hello, please review this small change to os_linux.cpp . >> >> It uses the recently introduced _print_ascii_file_h >> Function t more places (makes the code a bit shorter ) . > > I like the intent here but I think we have lost a few newlines in the > process. It isn't even clear to me that the result of print_ascii_file > will end with a newline being printed. Can you check the before and > after layout for readability please (and add to bug report). > > Thanks, > David > >> >> Bug/webrev : >> >> https://bugs.openjdk.java.net/browse/JDK-8244196 >> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8244196.0/ >> >> Thanks, Matthias >> From aph at redhat.com Tue May 5 11:32:35 2020 From: aph at redhat.com (Andrew Haley) Date: Tue, 5 May 2020 12:32:35 +0100 Subject: AArch64: Neoverse N1 atomic operations and -moutline-atomics Message-ID: <3a5e43e7-2039-8574-7470-5827267fb88d@redhat.com> Seen here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956418 It looks like there's a Neoverse N1 processor "feature" that leads to severe performance differences between atomic operations using the atomic memory instructions (e.g.CAS) and those using Exclusive Loads and Stores (e.g. LDAXR and STLXR.) The latter can be much slower in some cases. Therefore, if we want to avoid severe performance degradation we have to use atomic memory instructions on Neoverse N1 processors. This has caused some interesting problems for the authors of libgcc and glibc, which are statically compiled and do not generate code depending on the processor in use, as we do in generated code in HotSpot. The solution, which no-one likes very much, is to get GCC to generate subroutine calls for all atomic memory operations. The called routines look like this: __aarch64_cas4_relax: <+0>: adrp x16, 0x420000 <__aarch64_have_lse_atomics> <+4>: ldrb w16, [x16, :lo12:__aarch64_have_lse_atomics] <+8>: cbz w16, 0x400584 <__aarch64_cas4_relax+20> <+12>: cas w0, w1, [x2] <+16>: ret <+20>: mov w16, w0 <+24>: ldxr w0, [x2] <+28>: cmp w0, w16 <+32>: b.ne 0x40059c <__aarch64_cas4_relax+44> <+36>: stxr w17, w1, [x2] <+40>: cbnz w17, 0x400588 <__aarch64_cas4_relax+24> <+44>: ret This is, as you can see, horrible, but but thankfully we don't have to change AArch64 HotSpot because we use GCC's atomic builtins everywhere in C++ code. All we'll have to do is compile HotSpot with a new GCC and we'll be done. However, we also generate atomic operations. Happily, for some time we have generated and used atomic memory instructions whenever possible, so we don't have to change that either. Yay! Go us. https://gcc.gnu.org/legacy-ml/gcc-patches/2019-09/msg01036.html -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From aph at redhat.com Tue May 5 11:40:54 2020 From: aph at redhat.com (Andrew Haley) Date: Tue, 5 May 2020 12:40:54 +0100 Subject: [aarch64-port-dev ] AArch64: Neoverse N1 atomic operations and -moutline-atomics In-Reply-To: <3a5e43e7-2039-8574-7470-5827267fb88d@redhat.com> References: <3a5e43e7-2039-8574-7470-5827267fb88d@redhat.com> Message-ID: On 5/5/20 12:32 PM, Andrew Haley wrote: > This is, as you can see, horrible, but but thankfully we don't have to > change AArch64 HotSpot because we use GCC's atomic builtins everywhere > in C++ code. All we'll have to do is compile HotSpot with a new GCC > and we'll be done. Having said that, we should fix the AArch64 HotSpot build so that we use the -moutline-atomics option whenever it's available. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From matthias.baesken at sap.com Tue May 5 14:54:23 2020 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Tue, 5 May 2020 14:54:23 +0000 Subject: RFR [XS]: 8244196: adjust output in os_linux In-Reply-To: <9397d7b3-08ed-383f-7e8b-f7ac189220ab@oracle.com> References: <9f7e3d4c-d6e0-6a6e-16ae-51fba319fcfa@oracle.com> <9397d7b3-08ed-383f-7e8b-f7ac189220ab@oracle.com> Message-ID: Hi David , new webrev : http://cr.openjdk.java.net/~mbaesken/webrevs/8244196.1/ >While I personally prefer things on one line and no blank lines I am >concerned that this make unnecessarily break scripts that try to mine >hs_err logs. Yes that might indeed be true . I changed the output back and use print_cr for the header output (so we continue to have separate lines for header and content). I added " :" as suggested. Also added some example output to the JBS issue (however now the before/after is not that different any more ) . Best regards, Matthias -----Original Message----- From: David Holmes Sent: Dienstag, 5. Mai 2020 09:31 To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' Subject: Re: RFR [XS]: 8244196: adjust output in os_linux On 4/05/2020 8:55 pm, Baesken, Matthias wrote: > Hi David , the newlines were removed by me because when looking at the hs_err files it seemed to me, that we have too much of them. > ( but of course this might be a matter of taste , and not so much of correctness ) Very much a matter of taste. :) Presently we print: header1: value1 header2: value2 whereas you are now putting all that on one line and deleting empty lines: header1:value1 header2:value2 While I personally prefer things on one line and no blank lines I am concerned that this make unnecessarily break scripts that try to mine hs_err logs. Also you need as space after the : so this static void _print_ascii_file_h(const char* header, const char* filename, outputStream* st) { st->print("%s", header); should use "%s " (and arguably the : should be part of the format string rather than in the header itself). > The output part looks like (from one of our Linux x86_64 machines) this now : > > /sys/kernel/mm/transparent_hugepage/enabled:[always] madvise never > /sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):[always] madvise never > > /proc/sys/kernel/threads-max (system-wide limit on the number of threads):1032335 > /proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):2147483647 > /proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):81920 Please add before/after examples t the bug report. Thanks, David From xxinliu at amazon.com Tue May 5 16:38:27 2020 From: xxinliu at amazon.com (Liu, Xin) Date: Tue, 5 May 2020 16:38:27 +0000 Subject: [aarch64-port-dev ] FW: [Mach5] mach5-one-phh-JDK-8151779-20200427-2151-10554367: FAILED In-Reply-To: <56d85fb9-6ce8-7c00-ff4f-bf3fe40718a4@redhat.com> References: <858sifgbt3.fsf@arm.com> <57BB18E5-1F35-45AC-88BF-8D8C6A0767E3@amazon.com> <56d85fb9-6ce8-7c00-ff4f-bf3fe40718a4@redhat.com> Message-ID: <8BFC17D2-4DF6-44A3-8335-BE8EDD23E77E@amazon.com> Hello, Andrew, May I ask you to sponsor this change? I got it reviewed by Erik and Magnus. https://mail.openjdk.java.net/pipermail/build-dev/2020-May/027356.html Here is the latest webrev: https://cr.openjdk.java.net/~xliu/8244248/02/webrev/ thanks, --lx ?On 4/30/20, 1:19 AM, "aph at redhat.com" wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On 4/30/20 12:43 AM, Liu, Xin wrote: > One trick here. It's very easy to cheat configure by hacking the boot-jdk.m4 to "$HEAD -n 2". Everything looks fine then. The fix should be submitted to build-dev. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From david.holmes at oracle.com Tue May 5 23:34:09 2020 From: david.holmes at oracle.com (David Holmes) Date: Wed, 6 May 2020 09:34:09 +1000 Subject: RFR [XS]: 8244196: adjust output in os_linux In-Reply-To: References: <9f7e3d4c-d6e0-6a6e-16ae-51fba319fcfa@oracle.com> <9397d7b3-08ed-383f-7e8b-f7ac189220ab@oracle.com> Message-ID: <76066991-7145-4198-0a4b-b6f878eef422@oracle.com> Hi Matthias, On 6/05/2020 12:54 am, Baesken, Matthias wrote: > Hi David , new webrev : > > http://cr.openjdk.java.net/~mbaesken/webrevs/8244196.1/ > > >> While I personally prefer things on one line and no blank lines I am >> concerned that this make unnecessarily break scripts that try to mine >> hs_err logs. > > Yes that might indeed be true . I changed the output back and use print_cr for the header output (so we continue to have separate lines for header and content). > I added " :" as suggested. You don't want the space before the : (the space that was missing in the previous version was after the :) > Also added some example output to the JBS issue (however now the before/after is not that different any more ) . Thanks. I'm okay with these changes once the space is removed. Thanks, David > > Best regards, Matthias > > -----Original Message----- > From: David Holmes > Sent: Dienstag, 5. Mai 2020 09:31 > To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' > Subject: Re: RFR [XS]: 8244196: adjust output in os_linux > > On 4/05/2020 8:55 pm, Baesken, Matthias wrote: >> Hi David , the newlines were removed by me because when looking at the hs_err files it seemed to me, that we have too much of them. >> ( but of course this might be a matter of taste , and not so much of correctness ) > > Very much a matter of taste. :) Presently we print: > > header1: > value1 > > > header2: > value2 > > > > whereas you are now putting all that on one line and deleting empty lines: > > header1:value1 > header2:value2 > > While I personally prefer things on one line and no blank lines I am > concerned that this make unnecessarily break scripts that try to mine > hs_err logs. > > Also you need as space after the : so this > > static void _print_ascii_file_h(const char* header, const char* > filename, outputStream* st) { > st->print("%s", header); > > should use "%s " (and arguably the : should be part of the format string > rather than in the header itself). > >> The output part looks like (from one of our Linux x86_64 machines) this now : >> >> /sys/kernel/mm/transparent_hugepage/enabled:[always] madvise never >> /sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):[always] madvise never >> >> /proc/sys/kernel/threads-max (system-wide limit on the number of threads):1032335 >> /proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):2147483647 >> /proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):81920 > > Please add before/after examples t the bug report. > > Thanks, > David > > From john.r.rose at oracle.com Wed May 6 03:01:50 2020 From: john.r.rose at oracle.com (John Rose) Date: Tue, 5 May 2020 20:01:50 -0700 Subject: RFR JDK-8223347 Integration of Vector API (Incubator): Java API, implementation, and tests In-Reply-To: References: <49F81016-55DF-4E3C-940A-A8DF8C712259@oracle.com> <1886691513.405349.1587389515699.JavaMail.zimbra@u-pem.fr> Message-ID: On May 1, 2020, at 1:36 PM, Paul Sandoz wrote: > > Hi Remi, > > Thanks for the feedback. I am gonna need to go over this with John. Some comments inline. > > >> On Apr 20, 2020, at 6:31 AM, Remi Forax wrote: >> >> Hi Paul, >> about the API part (not the implementation), >> there are location where the same concept is described with a different names which doesn't help to understand how thing work >> - vector length <=> vector lane count >> - vector shape <=> vector bits size >> - element size <=> lane size >> >> "size" should never be used (VectorSpecies.elementSize() by example ) because it's never clear if it's the byte size or the bits size, using byteSize and bitsSize as used several times in the API should be used everywhere. > > I agree with being explicit here e.g. s/Vector.elementSize/Vector.elementBitSize and other usages. The name ?size? follows the precedent of the API point Integer.SIZE, which is the size of an `int` in bits. Although bit-size and byte-size is an important thing to keep track of, in the case of the Vector type, it lives primarily in registers, like `int`, and only secondarily in memory (as `int[]` lives primarily in memory). When you need the size of a datum in a register, you appeal to its bit count, not its byte count. When you store in memory, then you might want its byte size but for vectors that is an edge case. So I don?t think we need bitSize, as a more explicit term for size, for vectors and their lanes. It is enough to say ?size?. > > I would prefer to consistently use length in methods to refer to the lane count, it matches more closely to an array and for thinking about bounds checking. Yes, in this way (contrary to what I just said), a Vector is like an array, and so it just has a length. It?s a small array, and it lives in a register, and it doesn?t support good random access, but it has a length like an array. > > How about on Vector: > > /** > * Returns number of vector lanes ({@code VLENGTH}), commonly referred to as the lane count. > * > * @return the number of vector lanes > */ > public abstract int length(); Yes. Although laneCount would be a runner-up for naming this API point. But a Vector is enough like an array or String that giving it a length is the least confusing choice. With documentation that says somewhere that a vector looks like a little array formed from its lanes, so its length as a vector is the count of its lanes. Also, the semi-formal symbol {@code VLENGTH} is employed throughout the javadoc, and would have to change to something even uglier like {@code VLANECOUNT} if we got rid of Vector::length. > And then change VectorShape.laneCount to VectorShape.laneLength. > > ? That sounds wrong, as if it were reporting the length of a lane somewhere. Lane-count is correct. As the javadoc explains clearly, a vector has a certain number of lanes. > > >> >> Initially, having VectorSpecies and VectorShape confuse me, it's not the shape of the vector but only it's bits size, >> renaming VectorShape to VectorBitsSize will help, i think. >> > > Shape is quite a fundamental concept specified and used in the API, representing the information capacity of a vector. > > I think it would be misleading to refer to it only as VectorBitSize and unwieldy to change the specification accordingly. In this sense it serves as a useful abstraction to talk about vectors of the same bit size, and operations that change that size or not (such as shape-changing or shape-invariant). More comments: Shape also plays a critical role in the miscibility of vectors. Two vectors can line up to perform a lane-wise operation if and only if they have the same shape. This is fundamental. If we were to make these rules more ?clever?, allowing cross-shape lanewise operations, we?d give up on our current performance levels, because the necessary dynamic checks would put sand in the gears of the optimizer. There are present and future hardware platforms, and software abstractions, for which shape is more than just ?the number of my bits?. The max-bits shape is a dynamically determined shape which is not the same as any statically determined shape. In the future, shapes may be distinguished according to whether the involved vectors have intrinsic masks or not. In the near future, we might like to take the option of making synthetic shapes, such as ?a pair of 2-lane doubles of VLENGTH=4?, which are often seen in other vector developer toolkits. > >> You have the same values defined at several places in the API (sometimes with different names), usually it's aliases but it makes the API bigger that it should. >> I think all the reference to the element type, vector byte size, vector bit size, element byte size, element byte size, vector lane count should not appear on Vector because they are accessible from Species, >> and to go from a vector to the corresponding species, species() is enough. >> > > But it also makes it less usable, esp. for length(). The following is not an obviously or universally correct principle: ?There should only be one way to do a given task.? It is surely not universally true for the Vector API. When we point out that ?API point X is a duplicate of API point Y?, we may or may not be willing to follow that up with ?so get rid of API point X since the programmer can reach for Y?. If it?s our favorite utility method X, we will defend it vigorously. If it?s one we can?t imagine using, we will suggest that it be voted off the island. Only the most pure of purists will try to prove that some API has the minimum number of points, and rest happily only when that is the case. When Remi objects to Vector::length he is perhaps being such a purist, but very few programmers like to live on such spartan foundations, in practice. So utility methods like Vector::length, Vector::broadcast, Vector::maskAll, Vector::toDoubleArray, etc. help make common coding tasks easier. The most important thing to make easy is not reducing keystrokes (code golf) but rather making code as clear as possible. SKI combinators are a minimal API for Turing computations, but even if I loved to code with them, my code would not be reviewable for correctness, and even I couldn?t debug it as well, without some ?convenience combinators? available. Likewise, for vectors, it will be so common to loop over the lanes of a vector, that adding just the tokens ?.species()? to every loop head will make those loops harder to read and prove correct. In short: Code as you like, but if you expect to review or debug it, your code had better not have much extra ?noise? in it. This motivates convenience methods. Most of those in the Vector API have been placed there because they made somebody?s job easier writing clear Vector API code. A second reason for the convenience methods, in the Vector API, is discoverability. We put add, sub, mul, div, min, max, eq, le on Vector not because that?s the only possible place for their unique instantiation, but because they are friendly signs, on the front door of the API (Vector) that it is open for business. When you noodle around with a Vector in an API, you instantly see a familiar looking set of operations. You are not overwhelmed by hyperbolic arc-cosine, and on the other hand you are not left with an room empty of everything but abstract sculptures (the architectural version of having only higher-order functions). When you explore Vector::add, you find that it does about what you think it should, and how it should. You also discover the special shed (VectorOperations) where all the other tools are kept, and observe its connection to those previously-scary abstract sculptures (the lanewise and reduction higher order methods). If the purist succeeded in putting everything in one place, the initial experience of working with Vector would be very uninformative, more like playing a role-playing exploration game than getting work done on a ?batteries included? platform. I think the above reasons are also why IntStream::sum is such a good idea. Anyway, simple stuff should be easy to read, and complicated stuff should be discoverable from simple stuff. That why we repeat ourselves, because we think it will help users. > > >> You have also the same methods defined at several places, on VectorSpecies and as a static method taking a species as parameter, >> all the methods of VectorSpecies that takes or return a vector/mask/shuffle should be static in vector/mask/shuffle. >> > > It?s a little more nuanced, but I agree there is some duplication e.g. VectorSpecies.fromByteArray compared to the richer set of methods defined on [E]Vector. I think we can remove that method from VectorSpecies. Then I think we are dealing with the more ?erased/reflective? and long carrier accepting methods for general operation without appealing to E. Yes, I?m OK with removing that; it?s more like a leftover from previous rounds. OTOH, the static factories for methods annoy me for various reasons. I hope we can eventually find a better notation. Putting methods on VectorSpecies or on Vector itself sometimes is a viable alternative, sometimes not. Vector::broadcast is a static factory in disguise, which justifies its redundant existence in terms of usability and discoverability. But to do a better job on more API points, we might have to wait for specialized generics to make better moves. I?m content to wait? > > >> I think Binary and Associative should be merged to one VectorOperation, given that the difference is subtle and the whole point of this API is that if the hardware do not provide a way to reduce a binary op, it can be done in Java. > > Not quite true, ARM/x86 implementations do provide intrinsic implementations (leveraging a sequence of instructions) for some reduction operations e.g. min/max. I cannot recall there current state of the ARM/x86 implementations for other reductive operations but the plan is to optimize all of them at some point. Further, hardware may evolve in the future. It?s also useful, in an IDE, to have a static type check on the VectorOperation token passed to a reduction operation. You don?t want to accidentally reduce by SUB or DIV, and Associative allows the type-checker to stop it. > >> >> I have no idea what a Conversion.rangeType() is ? >> > > It?s the element type of the resulting output vector, > > Doc on VectorOperators.Operator: > > /** > * Reports the special return type of this operator. > * If this operator is a boolean, returns {@code boolean.class}. > * If this operator is a {@code Conversion}, > * returns its {@linkplain Conversion#rangeType range type}. > * > * Otherwise, the operator's return value always has > * whatever type was given as an input, and this method > * returns {@code Object.class} to denote that fact. > * @return the special return type, or {@code Object.class} if none > */ > Class rangeType(); Vector operators are static operation symbols, some of which have little accessor methods to describe themselves. The range type of a conversion is like `int` in C++ `operator int()`. It?s spelled as a type, not with characters. > > >> More specially: >> - VectorSpecies.loopbound() is not used in the example ? > > In the package-info? Yes, we should update that. +1 We should also make sure it optimizes. > > >> - VectorSpecies.arrayType()/genericElementType() are more for implementers than users , no ? at least arrayType should be arrayElementType. > > I don?t think we need to expose genericElementType in the API. +1 I wish we had Class.wrapperType, Class.primitiveType also. > I suspect arrayType may have been added before it was added to Class in 12 for the constable support. I think we can remove it. Yep. Class.arrayType is a nice $0.25 payoff of long technical debt. (I wonder how much that would compound with interest over 25 years?) > >> - VectorSpecies.withLanes() => withElementType() >> >> - VectorMask.check* are implementer methods ? > > The intent is to provide abstractions so the developer can perform ?reflective" checks on vector, mask, or shuffle. Yes. Also the Vector API intentionally allows some flexibility in coding with strong typing (IntVector, etc.) and weak typing (Vector, etc.). We think we can narrow that gap with specialized generics, but for now it?s important to allow both sides to be inhabited. When bridging the gap, you need casts, and there are lots of operations in the API which help with this?make it easier to read when a transition is being made. In particular, the check methods allow an intermediate value which may have lost static type information to be re-equipped with that static type information. Maybe they will be less important in the future, but I think even in the future it will be useful to be able to assert that two vectors are of the same shape. (Remember that vector shape is always a constraint on correct mixing of vectors, so it?s something the programmer needs help with sometimes.) > > >> - VectorMask.equal(VectorMask) => equiv >> > > Because it?s too close to the name ?equals?? Should have been eq, I guess. Perhaps it?s a leftover from before we had eq/ne/le/lt/gt/ge. > > >> - VectorShuffle.check* are implementer methods ? >> - VectorShuffle.vectorSpecies() => VectorShuffle.species() (as in Vector and VectorMask) > > Ok. It?s hair-splitting, but a VectorShuffle, because it isn?t a vector, doesn?t have a species. It operates on vectors of a common species. So, while we have Vector::species, we have (at a larger conceptual distance) VectorShuffle::vectorSpecies and VectorMask::vectorSpecies. The shuffles and masks know which species of vector they apply to, but that?s ?their species? indirectly. Who was Babe Ruth?s team? His kids didn?t have the same team, but they had their father?s team, only indirectly. > > >> - VectorShuffle.toVector() should return a IntVector > > No, it returns a vector according to its species. Note the constraint that shuffle source indexes are always in the range of -VLENGTH to VLENGTH - 1. This is a sore point in the API. A VectorShuffle isn?t a vector in part because its lanes contain source indexes (lane pointers) instead of vector data. What?s the implementation type? We don?t know exactly, could be byte, could be int, could be some clever union type superimposed on the vectors being operated on. It gets worse when the vector being operated on is too long and too narrow (at the same time) for its lanes to be able to encode all the necessary values. A vector of 512 lanes each of 8 bits is out of luck in this sense. In fact, it needs 10 bits (not 9) per source index in order to represent the ?negative conjugate? (invalid) source index values, which are required for various common operations. What to do? I?m not sure, but the answer probably has to do with treating VectorShuffle::toVector as a contracting and/or expanding operation, complete with part numbers. Or it could be treated as producing an unstated companion type, which must be queried reflectively. For now we?ve papered over the problems, but the paper won?t hold forever. > > >> - VectorShuffle.laneIsValid(), it seems to be an operation on Vector, not VectorShuffle. >> > > No, its specific to shuffle for masking out exceptional indexes (those < 0). (?negative conjugates?. A term I just made up today. Not tired of it yet.) See the javadoc for what they do and how they work. In short, we always add exactly one bit to the necessary set of bits for representing a lane index. The scheme works for VLENGTH which is not a power of two. And it gives various commonly seen idioms for masking, merging, and error checking. > > >> - IntVector.max, why there is no binary version that takes a mask ? It's documented but not why it's not available. >> > > It's to reduce the number of methods. On Vector: > > * This is not a full-service named operation like > * {@link #add(Vector) add()}. A masked version of > * this operation is not directly available > * but may be obtained via the masked version of > * {@code lanewise}. Subclasses define an additional > * scalar-broadcast overloading of this method. > We should probably document ?full service?. (Briefly!) Joe Darcy also mentioned this. > >> >> and i'm also not a big fans of the method that returns a long instead of a vector and only works on 64 bits values. >> > > Can you provide more details? The API points which work with double and long, in the weakly typed API, use those types as ?top for primitive?. Any primitive will fit without loss in either a long or double (sometimes both), just as any Vector will fit in a Vector, without loss. If you just need to broadcast zero, or get back a count, then putting it into a long is a really cheap form of boxing, and easy to read and prove correct. (And the redundant API point that helps you tells you where to find the ?real stuff? that?s going on, so you can learn about it if you need to.) I think these API points might need a little more documentation, to make it clear that they are doing *nothing* more than the corresponding strongly-typed API points, plus (to make up the primitive type differences) a cheap type conversion and/or error check. (Type conversion on output, error check on input, seems to be the most useful choice here. It?s an oddly reversed occurrence of Postel?s Law.) In particular, if I ask an IntVector to sum up its lanes, I don?t expect that the long version of the result will carry any more bits of information than the strongly typed int version (which truncates overflow). HTH ? John From vkempik at azul.com Wed May 6 09:43:33 2020 From: vkempik at azul.com (Vladimir Kempik) Date: Wed, 6 May 2020 09:43:33 +0000 Subject: RFR: 8243470: [macos] bring back O2 opt level for unsafe.cpp In-Reply-To: References: Message-ID: <7EA2B268-4267-4219-B257-1EC46808C82C@azul.com> Adding hotspot-runtime-dev > 23 ???. 2020 ?., ? 18:26, Vladimir Kempik ???????(?): > > > Hello > Please review a fix for JDK-8243470 > > Long time ago as part of JEP284: New HotSpot Build System this fix was applied to jdk9 https://bugs.openjdk.java.net/browse/JDK-8152666 > At that time it was decided to lower optimisation level for unsafe.cpp from O2 to O1 only for clang on Macosx. > > I suppose it was done due to issues in Set/Get helper functions where too optimistic optimisations were eliminating some null pointer checks. it was probably a clang bug. > That issue could be checked with test jdk/test/sun/misc/CopyMemory.java. > > I believe that workaround (going from O2 to O1) produced this issue - https://bugs.openjdk.java.net/browse/JDK-8234963 (Thread.getStackTrace is slow with clang). > JDK-8234963 can only be seen on mac with libjvm compiled by clang. > > Here I propose the patch which eliminates that workaround for clang 8+. > > I have tested clang versions 8/9/9.1/10, all of them showed good results: > 1) CopyMemory test passes fine on 11/14/15. > 2) jdk11/jdk14 passed tck. Regression testing were good as well. jdk15: no new failures in tck. > 3) The testRandom "benchmark" from 8234963 shows great improvements on my machine, going down from ~200 seconds to ~150 seconds (the newer clang the better result). For comparision, gcc built libjvm for jdk11 shows ~130 seconds on my machine. > > The webrev: http://cr.openjdk.java.net/~vkempik/8243470/webrev.00/ > > getStackTrace benchmark: http://cr.openjdk.java.net/~vkempik/8243470/TestRandom.java > > The bug: https://bugs.openjdk.java.net/browse/JDK-8243470 > > Thanks, Vladimir From matthias.baesken at sap.com Wed May 6 11:18:13 2020 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Wed, 6 May 2020 11:18:13 +0000 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java Message-ID: Hello, I noticed issues on one of our Linux test machines with jtreg test test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java After 8231111: Cgroups v2: Rework Metrics in java.base so as to recognize unified hierarchy . we see errors on one of our Linux test machines in jtreg test test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java . > Error: > test result: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: 'OperatingSystemMXBean.getTotalSwapSpaceSize: 0' missing from stdout/stderr > Test output reports: > OperatingSystemMXBean.getTotalSwapSpaceSize: 16106061824 > OperatingSystemMXBean.getFreeSwapSpaceSize: 15958736896 > We also see this output: > WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. > A check for the warning message "Your kernel does not support swap limit capabilities" could be added to the test. Bug/webrev : https://bugs.openjdk.java.net/browse/JDK-8244500 http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.0/ Thanks, Matthias From martin.doerr at sap.com Wed May 6 13:30:09 2020 From: martin.doerr at sap.com (Doerr, Martin) Date: Wed, 6 May 2020 13:30:09 +0000 Subject: RFR [XS]: 8244196: adjust output in os_linux In-Reply-To: References: <9f7e3d4c-d6e0-6a6e-16ae-51fba319fcfa@oracle.com> <9397d7b3-08ed-383f-7e8b-f7ac189220ab@oracle.com> <76066991-7145-4198-0a4b-b6f878eef422@oracle.com> Message-ID: Hi Matthias, > I'm okay with these changes once the space is removed. +1 I don't need to see another webrev for that. You even have spaces before '.' and ':' in the JIRA issue ?? Anyway, thanks for adding an output example. Best regards, Martin > -----Original Message----- > From: David Holmes > Sent: Mittwoch, 6. Mai 2020 01:34 > To: Baesken, Matthias ; 'hotspot- > dev at openjdk.java.net' > Subject: Re: RFR [XS]: 8244196: adjust output in os_linux > > Hi Matthias, > > On 6/05/2020 12:54 am, Baesken, Matthias wrote: > > Hi David , new webrev : > > > > http://cr.openjdk.java.net/~mbaesken/webrevs/8244196.1/ > > > > > >> While I personally prefer things on one line and no blank lines I am > >> concerned that this make unnecessarily break scripts that try to mine > >> hs_err logs. > > > > Yes that might indeed be true . I changed the output back and use > print_cr for the header output (so we continue to have separate lines for > header and content). > > I added " :" as suggested. > > You don't want the space before the : (the space that was missing in the > previous version was after the :) > > > Also added some example output to the JBS issue (however now the > before/after is not that different any more ) . > > Thanks. > > I'm okay with these changes once the space is removed. > > Thanks, > David > > > > > Best regards, Matthias > > > > -----Original Message----- > > From: David Holmes > > Sent: Dienstag, 5. Mai 2020 09:31 > > To: Baesken, Matthias ; 'hotspot- > dev at openjdk.java.net' > > Subject: Re: RFR [XS]: 8244196: adjust output in os_linux > > > > On 4/05/2020 8:55 pm, Baesken, Matthias wrote: > >> Hi David , the newlines were removed by me because when looking at > the hs_err files it seemed to me, that we have too much of them. > >> ( but of course this might be a matter of taste , and not so much of > correctness ) > > > > Very much a matter of taste. :) Presently we print: > > > > header1: > > value1 > > > > > > header2: > > value2 > > > > > > > > whereas you are now putting all that on one line and deleting empty lines: > > > > header1:value1 > > header2:value2 > > > > While I personally prefer things on one line and no blank lines I am > > concerned that this make unnecessarily break scripts that try to mine > > hs_err logs. > > > > Also you need as space after the : so this > > > > static void _print_ascii_file_h(const char* header, const char* > > filename, outputStream* st) { > > st->print("%s", header); > > > > should use "%s " (and arguably the : should be part of the format string > > rather than in the header itself). > > > >> The output part looks like (from one of our Linux x86_64 machines) this > now : > >> > >> /sys/kernel/mm/transparent_hugepage/enabled:[always] madvise never > >> /sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction > efforts parameter):[always] madvise never > >> > >> /proc/sys/kernel/threads-max (system-wide limit on the number of > threads):1032335 > >> /proc/sys/vm/max_map_count (maximum number of memory map areas > a process may have):2147483647 > >> /proc/sys/kernel/pid_max (system-wide limit on number of process > identifiers):81920 > > > > Please add before/after examples t the bug report. > > > > Thanks, > > David > > > > From paul.sandoz at oracle.com Wed May 6 16:02:12 2020 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 6 May 2020 09:02:12 -0700 (PDT) Subject: RFR JDK-8223347 Integration of Vector API (Incubator): Java API, implementation, and tests In-Reply-To: References: <49F81016-55DF-4E3C-940A-A8DF8C712259@oracle.com> <1886691513.405349.1587389515699.JavaMail.zimbra@u-pem.fr> Message-ID: <523F2CB3-AAA3-4343-80B9-7542652804CE@oracle.com> Hi John, Thanks. For the benefit of others, John and I had a long chat about this and Joe?s CSR comments. I have a better appreciation of your approach to the design and some of the more subjective aspects to guide developers to API points, and to make code more readable (that?s creative API design :-) ). I agree with your assessment on size, lane count, and Mask/Shuffle.vectorSpecies. Re: VectorSpecies.fromByteArray, I now see the method Vector.reinterpretShape appeals to VectorSpecies.fromByteArray for its specification. Removal would result in a less elegant specification of the behavior (making harder to understand). In that sense I think it?s worth its weight. However, I would suggest keeping in sync with a proposed change (on panama-dev) to the related load/store byte[]/ByteBuffer methods, requiring they all accept a ByteOrder. I think this does bring up the wider point you raised about where factory methods reside, and I agree about waiting for specialized generics, as that might allow us to make better moves. Paul. > On May 5, 2020, at 8:01 PM, John Rose wrote: > > On May 1, 2020, at 1:36 PM, Paul Sandoz > wrote: >> >> Hi Remi, >> >> Thanks for the feedback. I am gonna need to go over this with John. Some comments inline. >> >> >>> On Apr 20, 2020, at 6:31 AM, Remi Forax wrote: >>> >>> Hi Paul, >>> about the API part (not the implementation), >>> there are location where the same concept is described with a different names which doesn't help to understand how thing work >>> - vector length <=> vector lane count >>> - vector shape <=> vector bits size >>> - element size <=> lane size >>> >>> "size" should never be used (VectorSpecies.elementSize() by example ) because it's never clear if it's the byte size or the bits size, using byteSize and bitsSize as used several times in the API should be used everywhere. >> >> I agree with being explicit here e.g. s/Vector.elementSize/Vector.elementBitSize and other usages. > > The name ?size? follows the precedent of the API point > Integer.SIZE, which is the size of an `int` in bits. Although > bit-size and byte-size is an important thing to keep track of, > in the case of the Vector type, it lives primarily in registers, > like `int`, and only secondarily in memory (as `int[]` lives > primarily in memory). When you need the size of a datum > in a register, you appeal to its bit count, not its byte count. > > When you store in memory, then you might want its byte > size but for vectors that is an edge case. > > So I don?t think we need bitSize, as a more explicit term for > size, for vectors and their lanes. It is enough to say ?size?. > >> >> I would prefer to consistently use length in methods to refer to the lane count, it matches more closely to an array and for thinking about bounds checking. > > Yes, in this way (contrary to what I just said), a Vector is like > an array, and so it just has a length. It?s a small array, and it > lives in a register, and it doesn?t support good random access, > but it has a length like an array. > >> >> How about on Vector: >> >> /** >> * Returns number of vector lanes ({@code VLENGTH}), commonly referred to as the lane count. >> * >> * @return the number of vector lanes >> */ >> public abstract int length(); > > Yes. Although laneCount would be a runner-up for naming this API point. > But a Vector is enough like an array or String that giving it a length > is the least confusing choice. With documentation that says somewhere > that a vector looks like a little array formed from its lanes, so its length > as a vector is the count of its lanes. > > Also, the semi-formal symbol {@code VLENGTH} is employed throughout > the javadoc, and would have to change to something even uglier like > {@code VLANECOUNT} if we got rid of Vector::length. > >> And then change VectorShape.laneCount to VectorShape.laneLength. >> >> ? > > That sounds wrong, as if it were reporting the length of a lane somewhere. > > Lane-count is correct. As the javadoc explains clearly, a vector > has a certain number of lanes. > >> >> >>> >>> Initially, having VectorSpecies and VectorShape confuse me, it's not the shape of the vector but only it's bits size, >>> renaming VectorShape to VectorBitsSize will help, i think. >>> >> >> Shape is quite a fundamental concept specified and used in the API, representing the information capacity of a vector. >> >> I think it would be misleading to refer to it only as VectorBitSize and unwieldy to change the specification accordingly. In this sense it serves as a useful abstraction to talk about vectors of the same bit size, and operations that change that size or not (such as shape-changing or shape-invariant). > > More comments: > > Shape also plays a critical role in the miscibility of vectors. > Two vectors can line up to perform a lane-wise operation if > and only if they have the same shape. This is fundamental. > If we were to make these rules more ?clever?, allowing > cross-shape lanewise operations, we?d give up on our current > performance levels, because the necessary dynamic checks > would put sand in the gears of the optimizer. > > There are present and future hardware platforms, and software > abstractions, for which shape is more than just ?the number of > my bits?. The max-bits shape is a dynamically determined > shape which is not the same as any statically determined shape. > In the future, shapes may be distinguished according to whether > the involved vectors have intrinsic masks or not. In the near > future, we might like to take the option of making synthetic > shapes, such as ?a pair of 2-lane doubles of VLENGTH=4?, > which are often seen in other vector developer toolkits. > >> >>> You have the same values defined at several places in the API (sometimes with different names), usually it's aliases but it makes the API bigger that it should. >>> I think all the reference to the element type, vector byte size, vector bit size, element byte size, element byte size, vector lane count should not appear on Vector because they are accessible from Species, >>> and to go from a vector to the corresponding species, species() is enough. >>> >> >> But it also makes it less usable, esp. for length(). > > The following is not an obviously or universally correct principle: > ?There should only be one way to do a given task.? It is surely > not universally true for the Vector API. When we point out that > ?API point X is a duplicate of API point Y?, we may or may not > be willing to follow that up with ?so get rid of API point X since > the programmer can reach for Y?. If it?s our favorite utility > method X, we will defend it vigorously. If it?s one we can?t > imagine using, we will suggest that it be voted off the island. > Only the most pure of purists will try to prove that some API > has the minimum number of points, and rest happily only > when that is the case. When Remi objects to Vector::length > he is perhaps being such a purist, but very few programmers > like to live on such spartan foundations, in practice. > > So utility methods like Vector::length, Vector::broadcast, > Vector::maskAll, Vector::toDoubleArray, etc. help make > common coding tasks easier. The most important thing > to make easy is not reducing keystrokes (code golf) but > rather making code as clear as possible. SKI combinators > are a minimal API for Turing computations, but even if > I loved to code with them, my code would not be reviewable > for correctness, and even I couldn?t debug it as well, > without some ?convenience combinators? available. > Likewise, for vectors, it will be so common to loop over > the lanes of a vector, that adding just the tokens > ?.species()? to every loop head will make those loops > harder to read and prove correct. In short: Code > as you like, but if you expect to review or debug it, > your code had better not have much extra ?noise? > in it. This motivates convenience methods. Most > of those in the Vector API have been placed there > because they made somebody?s job easier writing > clear Vector API code. > > A second reason for the convenience methods, in > the Vector API, is discoverability. We put add, sub, > mul, div, min, max, eq, le on Vector not because > that?s the only possible place for their unique > instantiation, but because they are friendly signs, > on the front door of the API (Vector) that it is > open for business. When you noodle around with > a Vector in an API, you instantly see a familiar > looking set of operations. You are not overwhelmed > by hyperbolic arc-cosine, and on the other hand > you are not left with an room empty of everything > but abstract sculptures (the architectural version > of having only higher-order functions). > > When you explore Vector::add, you find that it does > about what you think it should, and how it should. > You also discover the special shed (VectorOperations) > where all the other tools are kept, and observe its > connection to those previously-scary abstract > sculptures (the lanewise and reduction higher order > methods). > > If the purist succeeded in putting everything in one > place, the initial experience of working with Vector > would be very uninformative, more like playing a > role-playing exploration game than getting work > done on a ?batteries included? platform. > > I think the above reasons are also why IntStream::sum > is such a good idea. Anyway, simple stuff should be easy to > read, and complicated stuff should be discoverable from > simple stuff. That why we repeat ourselves, because we > think it will help users. > >> >> >>> You have also the same methods defined at several places, on VectorSpecies and as a static method taking a species as parameter, >>> all the methods of VectorSpecies that takes or return a vector/mask/shuffle should be static in vector/mask/shuffle. >>> >> >> It?s a little more nuanced, but I agree there is some duplication e.g. VectorSpecies.fromByteArray compared to the richer set of methods defined on [E]Vector. I think we can remove that method from VectorSpecies. Then I think we are dealing with the more ?erased/reflective? and long carrier accepting methods for general operation without appealing to E. > > Yes, I?m OK with removing that; it?s more like a leftover from > previous rounds. > > OTOH, the static factories for methods annoy me for various > reasons. I hope we can eventually find a better notation. > Putting methods on VectorSpecies or on Vector itself sometimes > is a viable alternative, sometimes not. Vector::broadcast is > a static factory in disguise, which justifies its redundant > existence in terms of usability and discoverability. But to > do a better job on more API points, we might have to wait > for specialized generics to make better moves. I?m content > to wait? > >> >> >>> I think Binary and Associative should be merged to one VectorOperation, given that the difference is subtle and the whole point of this API is that if the hardware do not provide a way to reduce a binary op, it can be done in Java. >> >> Not quite true, ARM/x86 implementations do provide intrinsic implementations (leveraging a sequence of instructions) for some reduction operations e.g. min/max. I cannot recall there current state of the ARM/x86 implementations for other reductive operations but the plan is to optimize all of them at some point. Further, hardware may evolve in the future. > > It?s also useful, in an IDE, to have a static type check on the > VectorOperation token passed to a reduction operation. > You don?t want to accidentally reduce by SUB or DIV, > and Associative allows the type-checker to stop it. > >> >>> >>> I have no idea what a Conversion.rangeType() is ? >>> >> >> It?s the element type of the resulting output vector, >> >> Doc on VectorOperators.Operator: >> >> /** >> * Reports the special return type of this operator. >> * If this operator is a boolean, returns {@code boolean.class}. >> * If this operator is a {@code Conversion}, >> * returns its {@linkplain Conversion#rangeType range type}. >> * >> * Otherwise, the operator's return value always has >> * whatever type was given as an input, and this method >> * returns {@code Object.class} to denote that fact. >> * @return the special return type, or {@code Object.class} if none >> */ >> Class rangeType(); > > Vector operators are static operation symbols, some of which > have little accessor methods to describe themselves. The range > type of a conversion is like `int` in C++ `operator int()`. > It?s spelled as a type, not with characters. > >> >> >>> More specially: >>> - VectorSpecies.loopbound() is not used in the example ? >> >> In the package-info? Yes, we should update that. > > +1 We should also make sure it optimizes. > >> >> >>> - VectorSpecies.arrayType()/genericElementType() are more for implementers than users , no ? at least arrayType should be arrayElementType. >> >> I don?t think we need to expose genericElementType in the API. > > +1 I wish we had Class.wrapperType, Class.primitiveType also. > >> I suspect arrayType may have been added before it was added to Class in 12 for the constable support. I think we can remove it. > > Yep. Class.arrayType is a nice $0.25 payoff of long technical > debt. (I wonder how much that would compound with interest > over 25 years?) > >> >>> - VectorSpecies.withLanes() => withElementType() >>> >>> - VectorMask.check* are implementer methods ? >> >> The intent is to provide abstractions so the developer can perform ?reflective" checks on vector, mask, or shuffle. > > Yes. Also the Vector API intentionally allows some flexibility > in coding with strong typing (IntVector, etc.) and weak typing > (Vector, etc.). We think we can narrow that gap > with specialized generics, but for now it?s important to allow > both sides to be inhabited. When bridging the gap, you need > casts, and there are lots of operations in the API which help > with this?make it easier to read when a transition is being > made. In particular, the check methods allow an intermediate > value which may have lost static type information to be > re-equipped with that static type information. Maybe they > will be less important in the future, but I think even in the > future it will be useful to be able to assert that two vectors > are of the same shape. (Remember that vector shape is > always a constraint on correct mixing of vectors, so it?s > something the programmer needs help with sometimes.) > >> >> >>> - VectorMask.equal(VectorMask) => equiv >>> >> >> Because it?s too close to the name ?equals?? > > Should have been eq, I guess. Perhaps it?s a leftover from > before we had eq/ne/le/lt/gt/ge. > >> >> >>> - VectorShuffle.check* are implementer methods ? >>> - VectorShuffle.vectorSpecies() => VectorShuffle.species() (as in Vector and VectorMask) >> >> Ok. > > It?s hair-splitting, but a VectorShuffle, because it isn?t a vector, > doesn?t have a species. It operates on vectors of a common species. > So, while we have Vector::species, we have (at a larger conceptual > distance) VectorShuffle::vectorSpecies and VectorMask::vectorSpecies. > The shuffles and masks know which species of vector they apply > to, but that?s ?their species? indirectly. Who was Babe Ruth?s team? > His kids didn?t have the same team, but they had their father?s team, > only indirectly. > >> >> >>> - VectorShuffle.toVector() should return a IntVector >> >> No, it returns a vector according to its species. Note the constraint that shuffle source indexes are always in the range of -VLENGTH to VLENGTH - 1. > > This is a sore point in the API. A VectorShuffle isn?t a vector > in part because its lanes contain source indexes (lane pointers) > instead of vector data. What?s the implementation type? We > don?t know exactly, could be byte, could be int, could be some > clever union type superimposed on the vectors being operated on. > > It gets worse when the vector being operated on is too long > and too narrow (at the same time) for its lanes to be able to > encode all the necessary values. A vector of 512 lanes each > of 8 bits is out of luck in this sense. In fact, it needs 10 bits > (not 9) per source index in order to represent the ?negative > conjugate? (invalid) source index values, which are required > for various common operations. > > What to do? I?m not sure, but the answer probably has to do > with treating VectorShuffle::toVector as a contracting and/or > expanding operation, complete with part numbers. Or it could > be treated as producing an unstated companion type, which > must be queried reflectively. For now we?ve papered over > the problems, but the paper won?t hold forever. > >> >> >>> - VectorShuffle.laneIsValid(), it seems to be an operation on Vector, not VectorShuffle. >>> >> >> No, its specific to shuffle for masking out exceptional indexes (those < 0). > > (?negative conjugates?. A term I just made up today. Not tired of it yet.) > > See the javadoc for what they do and how they work. In short, > we always add exactly one bit to the necessary set of bits for > representing a lane index. The scheme works for VLENGTH > which is not a power of two. And it gives various commonly > seen idioms for masking, merging, and error checking. > >> >> >>> - IntVector.max, why there is no binary version that takes a mask ? It's documented but not why it's not available. >>> >> >> It's to reduce the number of methods. On Vector: >> >> * This is not a full-service named operation like >> * {@link #add(Vector) add()}. A masked version of >> * this operation is not directly available >> * but may be obtained via the masked version of >> * {@code lanewise}. Subclasses define an additional >> * scalar-broadcast overloading of this method. >> > > We should probably document ?full service?. (Briefly!) > Joe Darcy also mentioned this. > >> >>> >>> and i'm also not a big fans of the method that returns a long instead of a vector and only works on 64 bits values. >>> >> >> Can you provide more details? > > The API points which work with double and long, in the weakly > typed API, use those types as ?top for primitive?. Any primitive > will fit without loss in either a long or double (sometimes both), > just as any Vector will fit in a Vector, without loss. If you > just need to broadcast zero, or get back a count, then putting it > into a long is a really cheap form of boxing, and easy to read > and prove correct. (And the redundant API point that helps > you tells you where to find the ?real stuff? that?s going on, > so you can learn about it if you need to.) > > I think these API points might need a little more documentation, > to make it clear that they are doing *nothing* more than the > corresponding strongly-typed API points, plus (to make up > the primitive type differences) a cheap type conversion and/or > error check. (Type conversion on output, error check on input, > seems to be the most useful choice here. It?s an oddly reversed > occurrence of Postel?s Law.) In particular, if I ask an IntVector > to sum up its lanes, I don?t expect that the long version of the > result will carry any more bits of information than the strongly > typed int version (which truncates overflow). > > HTH > > ? John From john.r.rose at oracle.com Wed May 6 17:22:37 2020 From: john.r.rose at oracle.com (John Rose) Date: Wed, 6 May 2020 10:22:37 -0700 Subject: RFR JDK-8223347 Integration of Vector API (Incubator): Java API, implementation, and tests In-Reply-To: <523F2CB3-AAA3-4343-80B9-7542652804CE@oracle.com> References: <49F81016-55DF-4E3C-940A-A8DF8C712259@oracle.com> <1886691513.405349.1587389515699.JavaMail.zimbra@u-pem.fr> <523F2CB3-AAA3-4343-80B9-7542652804CE@oracle.com> Message-ID: Thanks, Paul! Talking with you about it helped me formulate my thoughts better. > On May 6, 2020, at 9:02 AM, Paul Sandoz wrote: > > Hi John, > > Thanks. For the benefit of others, John and I had a long chat about this and Joe?s CSR comments. > > I have a better appreciation of your approach to the design and some of the more subjective aspects to guide developers to API points, and to make code more readable (that?s creative API design :-) ). > > I agree with your assessment on size, lane count, and Mask/Shuffle.vectorSpecies. > > Re: VectorSpecies.fromByteArray, I now see the method Vector.reinterpretShape appeals to VectorSpecies.fromByteArray for its specification. Removal would result in a less elegant specification of the behavior (making harder to understand). In that sense I think it?s worth its weight. However, I would suggest keeping in sync with a proposed change (on panama-dev) to the related load/store byte[]/ByteBuffer methods, requiring they all accept a ByteOrder. > I think this does bring up the wider point you raised about where factory methods reside, and I agree about waiting for specialized generics, as that might allow us to make better moves. > > Paul. From xxinliu at amazon.com Wed May 6 18:58:13 2020 From: xxinliu at amazon.com (Liu, Xin) Date: Wed, 6 May 2020 18:58:13 +0000 Subject: [aarch64-port-dev ] FW: [Mach5] mach5-one-phh-JDK-8151779-20200427-2151-10554367: FAILED In-Reply-To: <8BFC17D2-4DF6-44A3-8335-BE8EDD23E77E@amazon.com> References: <858sifgbt3.fsf@arm.com> <57BB18E5-1F35-45AC-88BF-8D8C6A0767E3@amazon.com> <56d85fb9-6ce8-7c00-ff4f-bf3fe40718a4@redhat.com> <8BFC17D2-4DF6-44A3-8335-BE8EDD23E77E@amazon.com> Message-ID: <8FF2776D-E831-481C-9A8B-374617E914B6@amazon.com> Hi, Erik, Thank you for helping me on autoconf. I've learned so much from you. May I say you have reviewed this change? Any suggestion or objection? Thanks, --lx ?On 5/5/20, 9:38 AM, "Liu, Xin" wrote: Hello, Andrew, May I ask you to sponsor this change? I got it reviewed by Erik and Magnus. https://mail.openjdk.java.net/pipermail/build-dev/2020-May/027356.html Here is the latest webrev: https://cr.openjdk.java.net/~xliu/8244248/02/webrev/ thanks, --lx On 4/30/20, 1:19 AM, "aph at redhat.com" wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On 4/30/20 12:43 AM, Liu, Xin wrote: > One trick here. It's very easy to cheat configure by hacking the boot-jdk.m4 to "$HEAD -n 2". Everything looks fine then. The fix should be submitted to build-dev. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From erik.joelsson at oracle.com Wed May 6 21:06:28 2020 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Wed, 6 May 2020 14:06:28 -0700 Subject: [aarch64-port-dev ] FW: [Mach5] mach5-one-phh-JDK-8151779-20200427-2151-10554367: FAILED In-Reply-To: <8FF2776D-E831-481C-9A8B-374617E914B6@amazon.com> References: <858sifgbt3.fsf@arm.com> <57BB18E5-1F35-45AC-88BF-8D8C6A0767E3@amazon.com> <56d85fb9-6ce8-7c00-ff4f-bf3fe40718a4@redhat.com> <8BFC17D2-4DF6-44A3-8335-BE8EDD23E77E@amazon.com> <8FF2776D-E831-481C-9A8B-374617E914B6@amazon.com> Message-ID: Yes, of course. /Erik On 2020-05-06 11:58, Liu, Xin wrote: > Hi, Erik, > > Thank you for helping me on autoconf. I've learned so much from you. > May I say you have reviewed this change? Any suggestion or objection? > > Thanks, > --lx > > > ?On 5/5/20, 9:38 AM, "Liu, Xin" wrote: > > Hello, Andrew, > > May I ask you to sponsor this change? I got it reviewed by Erik and Magnus. > https://mail.openjdk.java.net/pipermail/build-dev/2020-May/027356.html > > Here is the latest webrev: > https://cr.openjdk.java.net/~xliu/8244248/02/webrev/ > > thanks, > --lx > > On 4/30/20, 1:19 AM, "aph at redhat.com" wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > On 4/30/20 12:43 AM, Liu, Xin wrote: > > One trick here. It's very easy to cheat configure by hacking the boot-jdk.m4 to "$HEAD -n 2". Everything looks fine then. > > The fix should be submitted to build-dev. > > -- > Andrew Haley (he/him) > Java Platform Lead Engineer > Red Hat UK Ltd. > https://keybase.io/andrewhaley > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 > > > From xxinliu at amazon.com Wed May 6 23:38:30 2020 From: xxinliu at amazon.com (Liu, Xin) Date: Wed, 6 May 2020 23:38:30 +0000 Subject: [aarch64-port-dev ] FW: [Mach5] mach5-one-phh-JDK-8151779-20200427-2151-10554367: FAILED In-Reply-To: References: <858sifgbt3.fsf@arm.com> <57BB18E5-1F35-45AC-88BF-8D8C6A0767E3@amazon.com> <56d85fb9-6ce8-7c00-ff4f-bf3fe40718a4@redhat.com> <8BFC17D2-4DF6-44A3-8335-BE8EDD23E77E@amazon.com> <8FF2776D-E831-481C-9A8B-374617E914B6@amazon.com> Message-ID: Thank you! --lx ?On 5/6/20, 2:09 PM, "Erik Joelsson" wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Yes, of course. /Erik On 2020-05-06 11:58, Liu, Xin wrote: > Hi, Erik, > > Thank you for helping me on autoconf. I've learned so much from you. > May I say you have reviewed this change? Any suggestion or objection? > > Thanks, > --lx > > > On 5/5/20, 9:38 AM, "Liu, Xin" wrote: > > Hello, Andrew, > > May I ask you to sponsor this change? I got it reviewed by Erik and Magnus. > https://mail.openjdk.java.net/pipermail/build-dev/2020-May/027356.html > > Here is the latest webrev: > https://cr.openjdk.java.net/~xliu/8244248/02/webrev/ > > thanks, > --lx > > On 4/30/20, 1:19 AM, "aph at redhat.com" wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > On 4/30/20 12:43 AM, Liu, Xin wrote: > > One trick here. It's very easy to cheat configure by hacking the boot-jdk.m4 to "$HEAD -n 2". Everything looks fine then. > > The fix should be submitted to build-dev. > > -- > Andrew Haley (he/him) > Java Platform Lead Engineer > Red Hat UK Ltd. > https://keybase.io/andrewhaley > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 > > > From mikael.vidstedt at oracle.com Thu May 7 05:16:27 2020 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Wed, 6 May 2020 22:16:27 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: Thank you for the review! Comments inline.. > On May 4, 2020, at 1:28 AM, Stefan Karlsson wrote: > > Hi Mikael, > > On 2020-05-04 07:12, Mikael Vidstedt wrote: >> Please review this change which implements part of JEP 381: >> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ > > I went over this patch and collected some comments: > > src/hotspot/share/adlc/output_c.cpp > src/hotspot/share/adlc/output_h.cpp > > Awkward code layout after change to. Indeed - fixed! > src/hotspot/share/c1/c1_Runtime1.cpp > src/hotspot/share/classfile/classListParser.cpp > src/hotspot/share/memory/arena.hpp > src/hotspot/share/opto/chaitin.cpp > test/hotspot/jtreg/gc/TestCardTablePageCommits.java > > Surrounding comments still refers to Sparc and/or Solaris. > > There are even more places if you search in the rest of the HotSpot source. Are we leaving those for a separate cleanup pass? Correct - I deliberately avoided changing comments that were not immediately ?obvious? how to address and/or that were pre-existing issues, since it?s not necessarily wrong for a comment to refer to Solaris or SPARC even after these changes. I would prefer to do that as follow-ups. Fair? > src/hotspot/share/gc/g1/g1HeapRegionAttr.hpp > > Remove comment: > // We use different types to represent the state value depending on platform as > // some have issues loading parts of words. Fixed. > src/hotspot/share/gc/shared/memset_with_concurrent_readers.hpp > > Fuse the declaration and definition, now that we only have one implementation. Maybe even remove function/file at some point. Fixed (fused). > src/hotspot/share/utilities/globalDefinitions.hpp > > Now that STACK_BIAS is always 0, should we remove its usages? Follow-up RFE? Yes, this is one of the things I have on my list to file a follow-up for. > src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot/src/org/graalvm/compiler/hotspot/meta/HotSpotGraphBuilderPlugins.java > > Maybe remove decryptSuffix? Fixed. > src/utils/hsdis/Makefile > > Is this really correct? > > Shouldn't: > ARCH1=$(CPU:x86_64=amd64) > ARCH2=$(ARCH1:i686=i386) > ARCH=$(ARCH2:sparc64=sparcv9) > > be changed to: > ARCH1=$(CPU:x86_64=amd64) > ARCH=$(ARCH1:i686=i386) > > so that we have ARCH defined? Very good catch! This Makefile could use some indentation love or just a plain rewrite.. In either case I fixed the ARCH definition and tested it to make sure the end result seemed to do the right thing (and AFAICT it does). > Other than that this looks good to me. Thank you! Cheers, Mikael > >> JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 >> Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! >> Background: >> Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. >> For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. >> In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. >> A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! >> Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. >> Testing: >> A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. >> Cheers, >> Mikael >> [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ From mikael.vidstedt at oracle.com Thu May 7 05:19:07 2020 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Wed, 6 May 2020 22:19:07 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: <2DDEE6DB-577A-4E23-B78D-0D19A15083F2@oracle.com> > On May 4, 2020, at 2:11 AM, Thomas Schatzl wrote: > > Hi, > > On 04.05.20 10:28, Stefan Karlsson wrote: >> Hi Mikael, >> On 2020-05-04 07:12, Mikael Vidstedt wrote: >>> >>> Please review this change which implements part of JEP 381: >>> >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >>> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ >> I went over this patch and collected some comments: >> src/hotspot/share/adlc/output_c.cpp >> src/hotspot/share/adlc/output_h.cpp >> Awkward code layout after change to. >> src/hotspot/share/c1/c1_Runtime1.cpp >> src/hotspot/share/classfile/classListParser.cpp >> src/hotspot/share/memory/arena.hpp >> src/hotspot/share/opto/chaitin.cpp >> test/hotspot/jtreg/gc/TestCardTablePageCommits.jav > > > Surrounding comments still refers to Sparc and/or Solaris. > > > > There are even more places if you search in the rest of the HotSpot > > source. Are we leaving those for a separate cleanup pass? > > In addition to "sparc", "solaris", also "solstudio"/"Sun Studio"/"SS compiler bug"/"niagara" yield some search (=grep) results. > > Some of these locations look like additional RFEs. Ah good! I found and fixed a few additional places based on those strings, but would like to handling the remaining comment updates as RFEs. Thank you for having a look! Cheers, Mikael From mikael.vidstedt at oracle.com Thu May 7 05:23:13 2020 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Wed, 6 May 2020 22:23:13 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: <553D0344-188E-455B-A03E-D080C1484B41@oracle.com> References: <553D0344-188E-455B-A03E-D080C1484B41@oracle.com> Message-ID: <40FCCD0E-5C84-482D-9231-428898E1E0AD@oracle.com> Kim, thank you for the review! Comments inline.. > On May 4, 2020, at 3:47 AM, Kim Barrett wrote: > >> On May 4, 2020, at 1:12 AM, Mikael Vidstedt wrote: >> >> >> Please review this change which implements part of JEP 381: >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ >> JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 > > I've only looked at the src/hotspot changes so far. I've not > duplicated comments already made by Stefan. > > Looks good, other than a few very minor issues, some of which might > already be covered by planned followup RFEs. > > ------------------------------------------------------------------------------ > > I think with sparc removal, c1's pack64/unpack64 stuff is no longer > used. So I think that can be removed from c1_LIR.[ch]pp too. Good catch. Fixed. > ------------------------------------------------------------------------------ > src/hotspot/share/opto/generateOptoStub.cpp > 225 // Clear last_Java_pc and (optionally)_flags > > The sparc-specific clearing of "flags" is gone. Fixed. > ------------------------------------------------------------------------------ > src/hotspot/share/runtime/deoptimization.cpp > 1086 *((jlong *) check_alignment_get_addr(obj, index, 8)) = (jlong) *((jlong *) &val); > > [pre-existing] > The rhs cast to jlong is unnecessary, since it's dereferencing a jlong*. When I first updated the code I actually remove the cast, but it just ended up looking asymmetrical so I chose to leave it there. Let me know if you feel strongly that it should go. (I don?t like these casts in general). > ------------------------------------------------------------------------------ > src/hotspot/share/runtime/flags/jvmFlagConstraintsCompiler.cpp > 236 JVMFlag::Error CompilerThreadPriorityConstraintFunc(intx value, bool verbose) { > 237 return JVMFlag::SUCCESS; > 238 } > > After SOLARIS code removal we no longer need this constraint function. Fixed. (I had that on my follow-up list, but included it in the upcoming webrev.) > ------------------------------------------------------------------------------ > src/hotspot/share/runtime/globals.hpp > 2392 experimental(size_t, ArrayAllocatorMallocLimit, \ > 2393 (size_t)-1, \ > > Combine these lines. Fixed. > ------------------------------------------------------------------------------ > src/hotspot/share/utilities/dtrace.hpp > > Shuold just eliminate all traces of HS_DTRACE_WORKAROUND_TAIL_CALL_BUG. Fixed - more code removed! Cheers, Mikael From mikael.vidstedt at oracle.com Thu May 7 05:25:22 2020 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Wed, 6 May 2020 22:25:22 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: <25d5e2fa-8909-0b94-e0b2-6b5aaa224492@oracle.com> References: <25d5e2fa-8909-0b94-e0b2-6b5aaa224492@oracle.com> Message-ID: <4B24DD51-8AA4-4896-B1F2-59FA5233F125@oracle.com> Vladimir, thank you for the review! Note that based on Stefan?s comments I have removed the decryptSuffix variable in src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot/src/org/graalvm/compiler/hotspot/meta/HotSpotGraphBuilderPlugins.java in the upcoming webrev. Cheers, Mikael > On May 4, 2020, at 12:01 PM, Vladimir Kozlov wrote: > > JIT, AOT, JVMCI and Graal changes seem fine to me. > > It would be interesting to see shared code execution coverage change. There are places where we use flags and setting instead of #ifdef SPARC which may not be executed now or executed partially. We may simplify such code too. > > Thanks, > Vladimir > > On 5/3/20 10:12 PM, Mikael Vidstedt wrote: >> Please review this change which implements part of JEP 381: >> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ >> JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 >> Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! >> Background: >> Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. >> For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. >> In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. >> A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! >> Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. >> Testing: >> A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. >> Cheers, >> Mikael >> [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ From mikael.vidstedt at oracle.com Thu May 7 05:27:04 2020 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Wed, 6 May 2020 22:27:04 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: <7F17A7EF-C9B3-40A0-816B-53614A56B7CA@oracle.com> Igor, thank you for the review, and again for helping make the test changes in the first place! :) I hope Vladimir?s reply clarifies how we?re planning on handling the Graal related changes. Cheers, Mikael > On May 4, 2020, at 2:29 PM, Igor Ignatyev wrote: > > Hi Mikael, > > the changes in /test/ look good to me. > > I have a question regarding src/jdk.internal.vm.compiler/*, aren't these files part of graal-compiler and hence will be brought back by the next graal update? > > Thanks, > -- Igor > >> On May 3, 2020, at 10:12 PM, Mikael Vidstedt wrote: >> >> >> Please review this change which implements part of JEP 381: >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ >> JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 >> >> >> Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! >> >> >> Background: >> >> Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. >> >> For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. >> >> In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. >> >> A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! >> >> Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. >> >> Testing: >> >> A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. >> >> Cheers, >> Mikael >> >> [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ >> > From mikael.vidstedt at oracle.com Thu May 7 05:35:30 2020 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Wed, 6 May 2020 22:35:30 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: <394BD86E-EC91-440E-9936-696B2B453093@oracle.com> New webrev here: webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.01/hotspot/open/webrev/ incremental: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.01/hotspot.incr/open/webrev/ Remaining items: * File follow-up to remove STACK_BIAS * File follow-ups to change/update/remove flags and/or flag documentation: UseLWPSynchronization, BranchOnRegister, LIRFillDelaySlots, ArrayAllocatorMallocLimit, ThreadPriorityPolicy * File follow-up(s) to update comments ("solaris", ?sparc?, ?solstudio?, ?sunos?, ?sun studio?, ?s compiler bug?, ?niagara?, ?) Please let me know if there?s something I have missed! Cheers, Mikael > On May 3, 2020, at 10:12 PM, Mikael Vidstedt wrote: > > > Please review this change which implements part of JEP 381: > > JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 > webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ > JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 > > > Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! > > > Background: > > Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. > > For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. > > In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. > > A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! > > Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. > > Testing: > > A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. > > Cheers, > Mikael > > [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ > From kim.barrett at oracle.com Thu May 7 06:35:22 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Thu, 7 May 2020 02:35:22 -0400 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: <394BD86E-EC91-440E-9936-696B2B453093@oracle.com> References: <394BD86E-EC91-440E-9936-696B2B453093@oracle.com> Message-ID: > On May 7, 2020, at 1:35 AM, Mikael Vidstedt wrote: > > > New webrev here: > > webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.01/hotspot/open/webrev/ > incremental: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.01/hotspot.incr/open/webrev/ > > Remaining items: > > * File follow-up to remove STACK_BIAS > > * File follow-ups to change/update/remove flags and/or flag documentation: UseLWPSynchronization, BranchOnRegister, LIRFillDelaySlots, ArrayAllocatorMallocLimit, ThreadPriorityPolicy > > * File follow-up(s) to update comments ("solaris", ?sparc?, ?solstudio?, ?sunos?, ?sun studio?, ?s compiler bug?, ?niagara?, ?) > > > Please let me know if there?s something I have missed! Looks good. From thomas.stuefe at gmail.com Thu May 7 14:21:19 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 7 May 2020 16:21:19 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: Message-ID: Hi all, please take a look at the third iteration of this change: http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ Changes in this version: - at the request of Coleen (sorry I did not see your post earlier) I removed all platform dependent files and put the platform dependent coding back to metaspace.cpp and compressedOops.cpp to ease the review pain. I used plain platform defines though (#ifdef AARCH64) instead of hiding them behind another macro to make things clearer. Note that I still intent to put this code away into the platform corners but will do so in a follow up RFE. - I reinstated, in a fashion, the special handling of reservations on Windows. On all platforms we reserve address space to map the archive files in with a subsequent mapping operation. However, on Windows, we cannot use MapViewOfFile() into an existing mapping. So I remove the mapping again before mapping the archives - see comment in code for details. All CI tests at SAP run through without problems, including on Win64 and aarch64, but I would be happy if others were to run test too. Thank you, Thomas On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe wrote: > Hi all, > > Could I have reviews for the following proposal of reworking cds/class > space reservation? > > Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 > > Webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ > > (Many thanks to Ioi Lam for so patiently explaining CDS internals to me, > and to Andrew Haley and Nick Gasson for help with aarch64!) > > Reservation of the compressed class space is needlessly complicated and > has some minor issues. It can be simplified and made clearer. > > The complexity stems from the fact that this area lives at the > intersection of two to three sub systems, depending on how one counts. > Metaspace, CDS, and the platform which may or may not its own view of how > to reserve class space. And this code has been growing organically over > time. > > One small example: > > ReservedSpace Metaspace::reserve_preferred_space(size_t size, size_t > alignment, > bool large_pages, char > *requested_addr, > bool use_requested_addr) > > which I spent hours decoding, resulting in a very confused mail to > hs-runtime and aarch64-port-dev [2]. > > This patch attempts to simplify cds and metaspace setup a bit; to comment > implicit knowledge which is not immediately clear; to cleanly abstract > platform concerns like optimized class space placement; and to disentangle > cds from metaspace to solve issues which may bite us later with Elastic > Metaspace [4]. > > --- > > The main change is the reworked reservation mechanism. This is based on > Ioi's proposal [5]. > > When reserving class space, three things must happen: > > 1) reservation of the space obviously. If cds is active that space must be > in the vicinity of cds archives to be covered by compressed class pointer > encoding. > 2) setting up the internal Metaspace structures atop of that space > 3) setting up compressed class pointer encoding. > > In its current form, Metaspace may or may not do some or all of that in > one function > (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace > metaspace_rs, char* requested_addr, address cds_base);) - if cds is active, > it will reserve the space for Metaspace and hand it in, otherwise it will > create it itself. > > When discussing this in [2], Ioi proposed to move the reservation of the > class space completely out of Metaspace and make it a responsibility of the > caller always. This would reduce some complexity, and this patch follows > the proposal. > > I removed > Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace > metaspace_rs, char* requested_addr, address cds_base); and all its sub > functions. > > (1) now has to always be done outside - a ReservedSpace for class space > has to be provided by the caller. However, Metaspace now offers a utility > function for reserving space at a "nice" location, and explicitly doing > nothing else: > > ReservedSpace > Metaspace::reserve_address_space_for_compressed_classes(size_t size); > > this function can be redefined on a platform level for platform optimized > reservation, see below for details. > > (2) is taken care of by a new function, > Metaspace::initialize_class_space(ReservedSpace rs) > > (3) is taken care of a new function CompressedKlassPointers::initialize(), > see below for details. > > > So, class space now is set up three explicit steps: > > - First, reserve a suitable space by however means you want. For > convenience you may use > Metaspace::reserve_address_space_for_compressed_classes(), or you may roll > your own reservation. > - Next, tell Metaspace to use that range as backing storage for class > space: Metaspace::initialize_class_space(ReservedSpace rs) > - Finally, set up encoding. Encoding is independent from the concept of a > ReservedSpace, it just gets an address range, see below for details. > > Separating these steps and moving them out of the responsibility of > Metaspace makes this whole thing more flexible; it also removes unnecessary > knowledge (e.g. Metaspace does not need to know anything about either ccp > encoding or cds). > > --- > > How it comes together: > > If CDS is off, we just reserve a space using > Metaspace::reserve_address_space_for_compressed_classes(), initialize it > with Metaspace::initialize_class_space(ReservedSpace rs), then set up > compressed class pointer encoding covering the range of this class space. > > If CDS is on (dump time), we reserve large 4G space, either at > SharedBaseAddress or using > Metaspace::reserve_address_space_for_compressed_classes(); we then split > that into 3G archive space and 1G class space; we set up that space with > Metaspace as class space; then we set up compressed class pointer encoding > covering both archive space and cds. > > If CDS is on (run time), we reserve a large space, split it into archive > space (large enough to hold both archives) and class space, then basically > proceed as above. > > Note that this is almost exactly how things worked before (modulo some > minor fixes, e.g. alignment issues), only the code is reformed and made > more explicit. > > --- > > I moved compressed class pointer setup over to CompressedKlassPointers and > changed the interface: > > -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace > metaspace_rs, address cds_base) > +void CompressedKlassPointers::initialize(address addr, size_t len); > > Instead of feeding it a single ReservedSpace, which is supposed to > represent class space, and an optional alternate base if cds is on, now we > give it just an numeric address range. That range marks the limits to where > Klass structures are to be expected, and is the implicit promise that > outside that range no Klass structures will exist, so encoding has to cover > only this range. > > This range may contain just the class space; or class space+cds; or > whatever allocation scheme we come up with in the future. Encoding does not > really care how the memory is organized as long as the input range covers > all possible Klass locations. That way we remove knowledge about class > space/cds from compressed class pointer encoding. > > Moving it away from metaspace.cpp into the CompressedKlassPointers class > also mirrors CompressedOops::initialize(). > > --- > > I renamed _narrow_klass_range to just _range, because strictly speaking > this is the range un-narrow Klass pointers can have. > > As for the implementation of CompressedKlassPointers::initialize(address > addr, size_t len), I mimicked very closely what happened before, so there > should be almost no differences. Since "almost no differences" sounds scary > :) here are the differences: > > - When CDS is active (dump or run time) we now always, unconditionally, > set the encoding range to 4G. This fixes a theoretical bug discussed on > aarch64-port-dev [1]. > > - When CDS is not active, we set the encoding range to the minimum > required length. Before, it was left at its default value of 4G. > > Both differences only affect aarch64, since they are currently the only > one using the range field in CompressedKlassPointers. > > I wanted to add an assert somewhere to test encoding of the very last > address of the CompressedKlassPointers range, again to prevent errors like > [3]. But I did not come up with a good place for this assert which would > cover also the encoding done by C1/C2. > > For the same reason I thought about introducing a mode where Klass > structures would be allocated in reverse order, starting at the end of the > ccs, but again left it out as too big a change. > > --- > > OS abstraction: platforms may have restrictions of what constitutes a > valid compressed class pointer encoding base. Or if not, they may have at > least preferences. There was logic like this in metaspace.cpp, which I > removed and cleanly factored out into platform dependent files, giving each > platform the option to add special logic. > > These are two new methods: > > - bool CompressedKlassPointers::is_valid_base(address p) > > to let the platform tell you whether it considers p to be a valid encoding > base. The only platform having these restrictions currently is aarch64. > > - ReservedSpace > Metaspace::reserve_address_space_for_compressed_classes(size_t size); > > this hands over the process of allocating a range suitable for compressed > class pointer encoding to the platform. Most platforms will allocate just > anywhere, but some platforms may have a better strategy (e.g. trying low > memory first, trying only correctly aligned addresses and so on). > > Beforehand, this coding existed in a similar form in metaspace.cpp for > aarch64 and AIX. For now, I left the AIX part out - it seems only half > done, and I want to check further if we even need it, if yes why not on > Linux ppc, and C1 does not seem to support anything other than base+offset > with shift either, but I may be mistaken. > > These two methods should give the platform enough control to implement > their own scheme for optimized class space placement without bothering any > shared code about it. > > Note about the form, I introduced two new platform dependent files, > "metaspace_.cpp" and "compressedOops_.cpp". I am not happy about > this but this seems to be what we generally do in hotspot, right? > > --- > > Metaspace reserve alignment vs cds alignment > > CDS was using Metaspace reserve alignment for CDS internal purposes. I > guess this was just a copy paste issue. It never caused problems since > Metaspace reserve alignment == page size, but that is not true anymore in > the upcoming Elastic Metaspace where reserve alignment will be larger. This > causes a number of issues. > > I separated those two cleanly. CDS now uses os::vm_allocation_granularity. > Metaspace::reserve_alignment is only used in those two places where it is > needed, when CDS creates the address space for class space on behalf of the > Metaspace. > > --- > > Windows special handling in CDS > > To simplify coding I removed the windows specific handling which left out > reservation of the archive. This was needed because windows cannot mmap > files into reserved regions. But fallback code exists in filemap.cpp for > this case which just reads in the region instead of mapping it. > > Should that turn out to be a performance problem, I will reinstate the > feature. But a simpler way would be reserve the archive and later just > before mmapping the archive file to release the archive space. That would > not only be simpler but give us the best guarantee that that address space > is actually available. But I'd be happy to leave that part out completely > if we do not see any performance problems on windows x64. > > --- > > NMT cannot deal with spaces which are split. This problem manifests in > that bookkeeping for class space is done under "Shared Classes", not > "Classes" as it should. This problem exists today too at dump time and > randomly at run time. But since I simplified the reservation, this problem > now shows up always, whether or not we map at the SharedBaseAddress. > While I could work around this problem, I'd prefer this problem to be > solved at the core, and NMT to have an option to recognize reservation > splits. So I'd rather not put a workaround for this into the patch but > leave it for fixing as a separate issue. I opened this issue to track it > [6]. > > --- > > Jtreg tests: > > I expanded the CompressedOops/CompressedClassPointers.java. I also > extended them to Windows. The tests now optionally omit strict class space > placement tests, since these tests heavily depend on ASLR and were the > reason they were excluded on Windows. However I think even without checking > for class space placement they make sense, just to see that the VM comes up > and lives with the many different settings we can run in. > > --- > > Tests: > > - I ran the patch through Oracles submit repo > - I ran tests manually for aarch64, zero, linux 32bit and windows x64 > - The whole battery of nightly tests at SAP, including ppc, ppcle and > aarch64, unfortunately excluding windows because of unrelated errors. > Windows x64 tests will be redone tonight. > > > Thank you, > > Thomas > > [1] > https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html > [2] > https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html > [3] https://bugs.openjdk.java.net/browse/JDK-8193266 > [4] https://bugs.openjdk.java.net/browse/JDK-8221173 > [5] > https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html > [6] https://bugs.openjdk.java.net/browse/JDK-8243535 > > From igor.ignatyev at oracle.com Thu May 7 14:39:50 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Thu, 7 May 2020 07:39:50 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: <7F17A7EF-C9B3-40A0-816B-53614A56B7CA@oracle.com> References: <7F17A7EF-C9B3-40A0-816B-53614A56B7CA@oracle.com> Message-ID: <95E6D883-D696-466A-80BD-4ADE420DBA43@oracle.com> Hi Mikael, yes, Vladimir's reply made it clear, let's hope all the needed changes got upstream before the next graal update so it goes smoothly. Cheers, -- Igor > On May 6, 2020, at 10:27 PM, Mikael Vidstedt wrote: > > > Igor, thank you for the review, and again for helping make the test changes in the first place! :) > > I hope Vladimir?s reply clarifies how we?re planning on handling the Graal related changes. > > Cheers, > Mikael > >> On May 4, 2020, at 2:29 PM, Igor Ignatyev wrote: >> >> Hi Mikael, >> >> the changes in /test/ look good to me. >> >> I have a question regarding src/jdk.internal.vm.compiler/*, aren't these files part of graal-compiler and hence will be brought back by the next graal update? >> >> Thanks, >> -- Igor >> >>> On May 3, 2020, at 10:12 PM, Mikael Vidstedt wrote: >>> >>> >>> Please review this change which implements part of JEP 381: >>> >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >>> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ >>> JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 >>> >>> >>> Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! >>> >>> >>> Background: >>> >>> Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. >>> >>> For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. >>> >>> In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. >>> >>> A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! >>> >>> Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. >>> >>> Testing: >>> >>> A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. >>> >>> Cheers, >>> Mikael >>> >>> [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ >>> >> > From sgehwolf at redhat.com Thu May 7 15:09:05 2020 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Thu, 07 May 2020 17:09:05 +0200 Subject: [8u] RFR(S): 8244548: JDK 8u: sun.misc.Version.jdkUpdateVersion() returns wrong result Message-ID: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> Hi, Please review this OpenJDK 8u fix for an issue where the update version as configured via --with-update-version=XXX might overflow in internal JDK structures and thus, will get reported wrong. In particular, only 1 byte is being reserved for the update versions internally. That is, it works fine up to a configured update version of 255 (2^8 - 1). We've passed that in OpenJDK 8u with the 8u262 cycle currently in EA. Hence, we are now seeing this issue. Bug: https://bugs.openjdk.java.net/browse/JDK-8244548 webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8244548/02/ testing: Linux tier 1 The proposed fix is to extend the update_version field in jdk_version_info from 8 to 16 bit and to use 16 bit of the jvm_version integer in jvm_version_info. Thus, the new upper bound for update version number is 2^16-1 => 65535 which should be sufficient. I don't think OpenJDK 8u will live that long ;-) jvm_version_info.jvm_version currently holds this quadruplet: Most significant 8 bits => major version, followed by 8 bits => minor version, followed by 8 bits => micro version, followed by 8 bits => build version. Note that JVM minor version represents the update version as passed in via configure and the micro version is currently not used (always 0). See vm_version.cpp lines 100-102 where only major, minor and build number are ever been set. Knowing this, we can still preserve the same behavior after patch by defining JVM_VERSION_MICRO to 0 for any version. For jdk_version_info the fix is simpler, since the update_version is a separate field for which I've extended it to 16 bit. Andrew Brygin suggested to reduce the reserved1 field from currently 16 bit to 8 bit since we are extending update_version by 8 bits, thus making the whole structure grow. I'm not sure reducing reserved1 to 8 bits so as to not grow the structure would be necessary, but I'd be happy to do so if there is such consensus. Thoughts? Thanks, Severin From magnus.ihse.bursie at oracle.com Thu May 7 15:24:01 2020 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Thu, 7 May 2020 17:24:01 +0200 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: References: Message-ID: On 2020-05-07 07:16, Mikael Vidstedt wrote: > Thank you for the review! Comments inline.. > >> On May 4, 2020, at 1:28 AM, Stefan Karlsson wrote: >> >> Hi Mikael, >> >> On 2020-05-04 07:12, Mikael Vidstedt wrote: >>> Please review this change which implements part of JEP 381: >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >>> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ >> I went over this patch and collected some comments: >> >> src/hotspot/share/adlc/output_c.cpp >> src/hotspot/share/adlc/output_h.cpp >> >> Awkward code layout after change to. > Indeed - fixed! > >> src/hotspot/share/c1/c1_Runtime1.cpp >> src/hotspot/share/classfile/classListParser.cpp >> src/hotspot/share/memory/arena.hpp >> src/hotspot/share/opto/chaitin.cpp >> test/hotspot/jtreg/gc/TestCardTablePageCommits.java >> >> Surrounding comments still refers to Sparc and/or Solaris. >> >> There are even more places if you search in the rest of the HotSpot source. Are we leaving those for a separate cleanup pass? > Correct - I deliberately avoided changing comments that were not immediately ?obvious? how to address and/or that were pre-existing issues, since it?s not necessarily wrong for a comment to refer to Solaris or SPARC even after these changes. I would prefer to do that as follow-ups. Fair? > >> src/hotspot/share/gc/g1/g1HeapRegionAttr.hpp >> >> Remove comment: >> // We use different types to represent the state value depending on platform as >> // some have issues loading parts of words. > Fixed. > >> src/hotspot/share/gc/shared/memset_with_concurrent_readers.hpp >> >> Fuse the declaration and definition, now that we only have one implementation. Maybe even remove function/file at some point. > Fixed (fused). > >> src/hotspot/share/utilities/globalDefinitions.hpp >> >> Now that STACK_BIAS is always 0, should we remove its usages? Follow-up RFE? > Yes, this is one of the things I have on my list to file a follow-up for. > >> src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot/src/org/graalvm/compiler/hotspot/meta/HotSpotGraphBuilderPlugins.java >> >> Maybe remove decryptSuffix? > Fixed. > >> src/utils/hsdis/Makefile >> >> Is this really correct? >> >> Shouldn't: >> ARCH1=$(CPU:x86_64=amd64) >> ARCH2=$(ARCH1:i686=i386) >> ARCH=$(ARCH2:sparc64=sparcv9) >> >> be changed to: >> ARCH1=$(CPU:x86_64=amd64) >> ARCH=$(ARCH1:i686=i386) >> >> so that we have ARCH defined? > Very good catch! This Makefile could use some indentation love or just a plain rewrite.. A rewrite is on my wish list... https://bugs.openjdk.java.net/browse/JDK-8209784 /Magnus > In either case I fixed the ARCH definition and tested it to make sure the end result seemed to do the right thing (and AFAICT it does). > >> Other than that this looks good to me. > Thank you! > > Cheers, > Mikael > >>> JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 >>> Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! >>> Background: >>> Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. >>> For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. >>> In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. >>> A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! >>> Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. >>> Testing: >>> A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. >>> Cheers, >>> Mikael >>> [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ From ioi.lam at oracle.com Fri May 8 05:37:08 2020 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 7 May 2020 22:37:08 -0700 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: Message-ID: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> Hi Thomas, I am running your patch in out CI pipeline now. Some comments: [1] Do we still need to use _class_space_list? It looks like we just have a single range. Is _class_space_list->current_virtual_space() always the same VS? ? if (_class_space_list != NULL) { ??? address base = (address)_class_space_list->current_virtual_space()->bottom(); ??? address top = base + compressed_class_space_size(); ??? st->print("Compressed class space mapped at: " PTR_FORMAT "-" PTR_FORMAT ", size: " SIZE_FORMAT, ?????????????? p2i(base), p2i(top), top - base); [2] Does JDK-8243535 exist with the current jdk/jdk repo? ? // This does currently not work because rs may be the result of a split operation ? // and NMT seems not to be able to handle splits. ? // See JDK-8243535. ? // MemTracker::record_virtual_memory_type((address)rs.base(), mtClass); [3] I think this can be put in header file. bool Metaspace::class_space_is_initialized() { ? return _class_space_list != NULL; } [4] Why does the CCS need to follow UseLargePagesInMetaspace? This is the reason for the ??? gap in the following chart: ??? // We do this by reserving space for the ccs behind the archives. Note however that ??? //? ccs follows a different alignment (Metaspace::reserve_alignment), so there may ??? //? be a gap between ccs and cds. ??? // We use a similar layout at runtime, see reserve_address_space_for_archives(). ??? // ??? //????????????????????????????? +-- SharedBaseAddress (default = 0x800000000) ??? //????????????????????????????? v ??? // +-..---------+---------+ ... +----+----+----+--------+--------------------+ ??? // |??? Heap??? | Archive |???? | MC | RW | RO | [gap]? | class space???? | ??? // +-..---------+---------+ ... +----+----+----+--------+--------------------+ ??? // |<--?? MaxHeapSize? -->|???? |<-- UnscaledClassSpaceMax = 4GB -->| _reserve_alignment is determined here: void Metaspace::ergo_initialize() { ? if (DumpSharedSpaces) { ??? // Using large pages when dumping the shared archive is currently not implemented. ??? FLAG_SET_ERGO(UseLargePagesInMetaspace, false); ? } ? size_t page_size = os::vm_page_size(); ? if (UseLargePages && UseLargePagesInMetaspace) { ??? page_size = os::large_page_size(); ? } ? _commit_alignment? = page_size; ? _reserve_alignment = MAX2(page_size, (size_t)os::vm_allocation_granularity()); But when CDS is enabled, the RS is reserved without large pages, so it should be possible to incrementally commit the CCS using just os::vm_allocation_granularity(). [5] I see the aarch64 code in Metaspace::reserve_address_space_for_compressed_classes has been changed. Is this necessary for this RFE, or is it a separate improvement that can be done in a different RFE? [6] For AARCH64, should we skip the part that reserves it right above the heap? Or does AARCH64 always allocate the heap such that it's a preferable address? ??? // case (b) ??? ReservedSpace rs; ??? // Try right above the Java heap... ??? address base = align_up(CompressedOops::end(), Metaspace::reserve_alignment()); ??? assert(base != NULL, "Sanity"); ??? const size_t size = align_up(CompressedClassSpaceSize, Metaspace::reserve_alignment()); ??? if (CompressedKlassPointers::is_valid_base(base)) { ????? rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* large */, (char*)base); ??? } ??? // ...failing that, reserve anywhere, but let platform do optimized placement: ??? if (!rs.is_reserved()) { ????? rs = Metaspace::reserve_address_space_for_compressed_classes(size); ??? } [7] argument.cpp checks this: ? if (UseSharedSpaces || DumpSharedSpaces) { ??? if (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { ????? log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for this platform, option will be ignored.", ?????????????????????? p2i((address)SharedBaseAddress)); ????? FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); ??? } ? } but initialize_dumptime_shared_and_meta_spaces aligns up the SharedBaseAddress before doing the assert. ? void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { ??? assert(DumpSharedSpaces, "should be called for dump time only"); ??? const size_t reserve_alignment = MetaspaceShared::reserved_space_alignment(); ??? char* shared_base = (char*)align_up((char*)SharedBaseAddress, reserve_alignment); ? #ifdef _LP64 assert(CompressedKlassPointers::is_valid_base((address)shared_base), "Sanity"); So theoretically shared_base may no longer be is_valid_base after the align_up. It probably won't happen, but I think the code will be much clearer tofirst to align_up, then check for is_valid_base and reset to default, and finally assert. [8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the following block can be simplified: ? if (!_shared_rs.is_reserved()) { ??? // Get a reserved space anywhere if attaching at the SharedBaseAddress fails: ??? if (UseCompressedClassPointers) { ????? // If we need a compressed class space too, let the platform handle the reservation. ????? _shared_rs = Metaspace::reserve_address_space_for_compressed_classes(cds_total); ??? } else { ????? // anywhere is fine. ????? _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* large */, (char*)NULL); ??? } ? } ?? ... if you change the declaration to: ? static ReservedSpace reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return ReservedSpace();}); [9] I think the #ifdef _LP64 is not necessary: #ifdef _LP64 ??? // Some sanity checks after reserving address spaces for archives and class space. ??? assert(archive_space_rs.is_reserved(), "Sanity"); ??? if (Metaspace::using_class_space()) { ????? // Class space must closely follow the archive space. Both spaces must be aligned correctly. ????? assert(class_space_rs.is_reserved(), "A class space should have been reserved"); ????? assert(class_space_rs.base() >= archive_space_rs.end(), "class space should follow the cds archive space"); ????? assert(is_aligned(archive_space_rs.base(), MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); ????? assert(is_aligned(class_space_rs.base(), Metaspace::reserve_alignment()), "class space misaligned"); ??? } #endif The rest of the code looks OK to me, but I may take a look at it again after getting more sleep :-) Thanks - Ioi On 5/7/20 7:21 AM, Thomas St?fe wrote: > Hi all, > > please take a look at the third iteration of this change: > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ > > Changes in this version: > > - at the request of Coleen (sorry I did not see your post earlier) I > removed all platform dependent?files and put the platform dependent > coding back to metaspace.cpp and compressedOops.cpp to ease the review > pain. I used plain platform defines though (#ifdef AARCH64) instead of > hiding them behind another macro to make things clearer. Note that I > still intent to put this code away into the platform corners but will > do so in a follow up RFE. > > - I reinstated, in a fashion, the special handling of reservations on > Windows. On all platforms we reserve address space to map the archive > files in with a subsequent mapping operation. However, on Windows, we > cannot use MapViewOfFile() into an existing mapping. So I remove the > mapping again before mapping the archives - see comment in code for > details. > > All CI tests at SAP run through without problems, including on Win64 > and aarch64, but I would be happy if others were to run test too. > > Thank you, Thomas > > On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe > wrote: > > Hi all, > > Could I have reviews for the following proposal of reworking > cds/class space reservation? > > Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 > > Webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ > > (Many thanks to Ioi Lam for so patiently explaining CDS internals > to me, and to Andrew Haley and Nick Gasson for help with aarch64!) > > Reservation of the compressed class space is needlessly > complicated and has some minor issues. It can be simplified and > made clearer. > > The complexity stems from the fact that this area lives at the > intersection of two to three sub systems, depending on how one > counts. Metaspace, CDS, and the platform which may or may not its > own view of how to reserve class space. And this code has been > growing organically over time. > > One small example: > > ReservedSpace Metaspace::reserve_preferred_space(size_t size, > size_t alignment, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?bool large_pages, > char *requested_addr, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?bool > use_requested_addr) > > which I spent hours decoding, resulting in a very confused mail to > hs-runtime and aarch64-port-dev [2]. > > This patch attempts to simplify cds and metaspace setup a bit; to > comment implicit knowledge which is not immediately clear; to > cleanly abstract platform concerns like optimized class space > placement; and to disentangle cds from metaspace to solve issues > which may bite us later with Elastic Metaspace [4]. > > --- > > The main change is the reworked reservation mechanism. This is > based on Ioi's proposal [5]. > > When reserving class space, three things must happen: > > 1) reservation of the space obviously. If cds is active that space > must be in the vicinity of cds archives to be covered by > compressed class pointer encoding. > 2) setting up the internal Metaspace structures atop of that space > 3) setting up compressed class pointer encoding. > > In its current form, Metaspace may or may not do some or all of > that in one function > (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace > metaspace_rs, char* requested_addr, address cds_base);) - if cds > is active, it will reserve the space for Metaspace and hand it in, > otherwise it will create it itself. > > When discussing this in [2], Ioi proposed to move the reservation > of the class space completely out of Metaspace and make it a > responsibility of the caller always. This would reduce some > complexity, and this patch follows the proposal. > > I removed > Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace > metaspace_rs, char* requested_addr, address cds_base); and all its > sub functions. > > (1) now has to always be done outside - a ReservedSpace for class > space has to be provided by the caller. However, Metaspace now > offers a utility function for reserving space at a "nice" > location, and explicitly doing nothing else: > > ReservedSpace > Metaspace::reserve_address_space_for_compressed_classes(size_t size); > > this function can be redefined on a platform level for platform > optimized reservation, see below for details. > > (2) is taken care of by a new function, > Metaspace::initialize_class_space(ReservedSpace rs) > > (3) is taken care of a new function > CompressedKlassPointers::initialize(), see below for details. > > > So, class space now is set up three explicit steps: > > - First, reserve a suitable space by however means you want. For > convenience you may use > Metaspace::reserve_address_space_for_compressed_classes(), or you > may roll your own reservation. > - Next, tell Metaspace to use that range as backing storage for > class space: Metaspace::initialize_class_space(ReservedSpace rs) > - Finally, set up encoding. Encoding is independent from the > concept of a ReservedSpace, it just gets an address range, see > below for details. > > Separating these steps and moving them out of the responsibility > of Metaspace makes this whole thing more flexible; it also removes > unnecessary knowledge (e.g. Metaspace does not need to know > anything about either ccp encoding or cds). > > --- > > How it comes together: > > If CDS is off, we just reserve a space using > Metaspace::reserve_address_space_for_compressed_classes(), > initialize it with Metaspace::initialize_class_space(ReservedSpace > rs), then set up compressed class pointer encoding covering the > range of this class space. > > If CDS is on (dump time), we reserve large 4G space, either at > SharedBaseAddress or using > Metaspace::reserve_address_space_for_compressed_classes(); we then > split that into 3G archive space and 1G class space; we set up > that space with Metaspace as class space; then we set > up?compressed class pointer encoding covering both archive space > and cds. > > If CDS is on (run time), we reserve a large space, split it into > archive space (large enough to hold both archives) and class > space, then basically proceed as above. > > Note that this is almost exactly how things worked before (modulo > some minor fixes, e.g. alignment issues), only the code is > reformed and made more explicit. > > --- > > I moved compressed class pointer setup over to > CompressedKlassPointers and changed the interface: > > -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace > metaspace_rs, address cds_base) > +void CompressedKlassPointers::initialize(address addr, size_t len); > > Instead of feeding it a single ReservedSpace, which is supposed to > represent class space, and an optional alternate base if cds is > on, now we give it just an numeric address range. That range marks > the limits to where Klass structures are to be expected, and is > the implicit promise that outside that range no Klass structures > will exist, so encoding has to cover only this range. > > This range may contain just the class space; or class space+cds; > or whatever allocation scheme we come up with in the future. > Encoding does not really care how the memory is organized as long > as the input range covers all possible Klass locations. That way > we remove knowledge about class space/cds from compressed class > pointer encoding. > > Moving it away from metaspace.cpp into the CompressedKlassPointers > class also mirrors CompressedOops::initialize(). > > --- > > I renamed _narrow_klass_range to just _range, because strictly > speaking this is the range un-narrow Klass pointers can have. > > As for the implementation of > CompressedKlassPointers::initialize(address addr, size_t len), I > mimicked very closely what happened before, so there should be > almost no differences. Since "almost no differences" sounds scary > :) here are the differences: > > - When CDS is active (dump or run time) we now always, > unconditionally, set the encoding range to 4G. This fixes a > theoretical bug discussed on aarch64-port-dev [1]. > > - When CDS is not active, we set the encoding range to the minimum > required length. Before, it was left at its default value of 4G. > > Both differences only affect aarch64, since they are currently the > only one using the range field in CompressedKlassPointers. > > I wanted to add an assert somewhere to test encoding of the very > last address of the CompressedKlassPointers range, again to > prevent errors like [3]. But I did not come up with a good place > for this assert which would cover also the encoding done by C1/C2. > > For the same reason I thought about introducing a mode where Klass > structures would be allocated in reverse order, starting at the > end of the ccs, but again left it out as too big a change. > > --- > > OS abstraction: platforms may have restrictions of what > constitutes a valid compressed class pointer encoding base. Or if > not, they may have at least preferences. There was logic like this > in metaspace.cpp, which I removed and cleanly factored out into > platform dependent files, giving each platform the option to add > special logic. > > These are two new methods: > > - bool CompressedKlassPointers::is_valid_base(address p) > > to let the platform tell you whether it considers p to be a valid > encoding base. The only platform having these restrictions > currently is aarch64. > > - ReservedSpace > Metaspace::reserve_address_space_for_compressed_classes(size_t size); > > this hands over the process of allocating a range suitable for > compressed class pointer encoding to the platform. Most platforms > will allocate just anywhere, but some platforms may have a better > strategy (e.g. trying low memory first, trying only correctly > aligned addresses and so on). > > Beforehand, this coding existed in a similar form in metaspace.cpp > for aarch64 and AIX. For now, I left the AIX part out - it seems > only half done, and I want to check further if we even need it, if > yes why not on Linux ppc, and C1 does not seem to support anything > other than base+offset with shift either, but I may be mistaken. > > These two methods should give the platform enough control to > implement their own scheme for optimized class space placement > without bothering any shared code about it. > > Note about the form, I introduced two new platform dependent > files, "metaspace_.cpp" and "compressedOops_.cpp". I am > not happy about this but this seems to be what we generally do in > hotspot, right? > > --- > > Metaspace reserve alignment vs cds alignment > > CDS was using Metaspace reserve alignment for CDS internal > purposes. I guess this was just a copy paste issue. It never > caused problems since Metaspace reserve alignment == page size, > but that is not true anymore in the upcoming Elastic Metaspace > where reserve alignment will be larger. This causes a number of > issues. > > I separated those two cleanly. CDS now uses > os::vm_allocation_granularity. Metaspace::reserve_alignment is > only used in those two places where it is needed, when CDS creates > the address space for class space on behalf of the Metaspace. > > --- > > Windows special handling in CDS > > To simplify coding I removed the windows specific handling which > left out reservation of the archive. This was needed because > windows cannot mmap files into reserved regions. But fallback code > exists in filemap.cpp for this case which just reads in the region > instead of mapping?it. > > Should that turn out to be a performance problem, I will reinstate > the feature. But a simpler way would be reserve the archive and > later just before mmapping?the archive file to release the archive > space. That would not only be simpler but give us the best > guarantee that that address space is actually available. But I'd > be happy to leave that part out completely if we do not see any > performance problems on windows x64. > > --- > > NMT cannot deal with spaces which are split. This problem > manifests in that bookkeeping for class space is done under > "Shared Classes", not "Classes" as it should. This problem exists > today too at dump?time and randomly at run time. But since I > simplified the reservation, this problem now shows up always, > whether or not we map at the SharedBaseAddress. > While I could work around this problem, I'd prefer this problem to > be solved at the core, and NMT to have an option to recognize > reservation splits. So I'd rather not put a workaround for this > into the patch but leave it for fixing as a separate issue. I > opened this issue to track it [6]. > > --- > > Jtreg tests: > > I expanded the CompressedOops/CompressedClassPointers.java. I also > extended them to Windows. The tests now optionally omit strict > class space placement tests, since these tests heavily depend on > ASLR and were the reason they were excluded on Windows. However I > think even without checking for class space placement they make > sense, just to see that the VM comes up and lives with the many > different settings we can run in. > > --- > > Tests: > > - I ran the patch through Oracles submit repo > - I ran tests manually for aarch64, zero, linux 32bit and windows x64 > - The whole battery of nightly tests at SAP, including ppc, ppcle > and aarch64, unfortunately excluding windows because of unrelated > errors. Windows x64 tests will be redone tonight. > > > Thank you, > > Thomas > > [1] > https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html > [2] > https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html > [3] https://bugs.openjdk.java.net/browse/JDK-8193266 > [4] https://bugs.openjdk.java.net/browse/JDK-8221173 > [5] > https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html > [6] https://bugs.openjdk.java.net/browse/JDK-8243535 > From thomas.stuefe at gmail.com Fri May 8 07:33:35 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 8 May 2020 09:33:35 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> Message-ID: Hi Ioi, thanks for taking the time to look at this! On Fri, May 8, 2020 at 7:37 AM Ioi Lam wrote: > Hi Thomas, > > I am running your patch in out CI pipeline now. Some comments: > > [1] Do we still need to use _class_space_list? It looks like we just have > a single range. Is _class_space_list->current_virtual_space() always the > same VS? > > if (_class_space_list != NULL) { > address base = > (address)_class_space_list->current_virtual_space()->bottom(); > address top = base + compressed_class_space_size(); > st->print("Compressed class space mapped at: " PTR_FORMAT "-" > PTR_FORMAT ", size: " SIZE_FORMAT, > p2i(base), p2i(top), top - base); > > Yes it is. Has been bugging me for years too. With ccs, the VirtualSpaceList is degenerated and only contains one node. It still makes some sense code wise. But maybe VirtualSpace*List* is a misnomer. It also ties in into the question whether Metaspace reservations should be able to grow on demand. Oracle folks I talk to tend to say yes. I personally think the "Metaspace is infinite don't worry" meme was broken since ccs was a thing, since CompressedClassSpaceSize is an absolute limit already. I may tweak and simplify the code a bit with Elastic Metaspace. Maybe, as you say, with ccs we can get rid of VirtualSpaceList and deal with the range directly. Have to check. [2] Does JDK-8243535 exist with the current jdk/jdk repo? > > // This does currently not work because rs may be the result of a split > operation > // and NMT seems not to be able to handle splits. > // See JDK-8243535. > // MemTracker::record_virtual_memory_type((address)rs.base(), mtClass); > > Yes. It shows up more rarely. With this patch, it will show up always. I had a conversation with Zhengyu about this. I think there may be workarounds, like removing the original mapping in NMT and replacing it with the two new ones for archive and ccs. But it seems iffy, I'd rather have this fixed in NMT. I also think it is not such a big deal, but if you think otherwise, I can integrate a workaround into this patch. > > [3] I think this can be put in header file. > > bool Metaspace::class_space_is_initialized() { > return _class_space_list != NULL; > } > > Will do. > [4] Why does the CCS need to follow UseLargePagesInMetaspace? This is the > reason for the > gap in the following chart: > > // We do this by reserving space for the ccs behind the archives. Note > however that > // ccs follows a different alignment (Metaspace::reserve_alignment), > so there may > // be a gap between ccs and cds. > // We use a similar layout at runtime, see > reserve_address_space_for_archives(). > // > // +-- SharedBaseAddress (default = > 0x800000000) > // v > // +-..---------+---------+ ... > +----+----+----+--------+--------------------+ > // | Heap | Archive | | MC | RW | RO | [gap] | class > space | > // +-..---------+---------+ ... > +----+----+----+--------+--------------------+ > // |<-- MaxHeapSize -->| |<-- UnscaledClassSpaceMax = 4GB -->| > > No, nothing to do with large pages. CCS does not have large pages. The gap can result from different alignment rules between metaspace and cds. With Elastic Metaspace, metaspace reservations will be aligned to 4M. So, ccs will have to start at a 4M aligned border. > > _reserve_alignment is determined here: > > void Metaspace::ergo_initialize() { > if (DumpSharedSpaces) { > // Using large pages when dumping the shared archive is currently not > implemented. > FLAG_SET_ERGO(UseLargePagesInMetaspace, false); > } > > size_t page_size = os::vm_page_size(); > if (UseLargePages && UseLargePagesInMetaspace) { > page_size = os::large_page_size(); > } > > _commit_alignment = page_size; > _reserve_alignment = MAX2(page_size, > (size_t)os::vm_allocation_granularity()); > > But when CDS is enabled, the RS is reserved without large pages, so it > should be possible to incrementally commit the CCS using just > os::vm_allocation_granularity(). > > Absolutely. CCS does not use large pages for that reason. As I wrote, this is about the new reservation granularity of 4M. > [5] I see the aarch64 code in > Metaspace::reserve_address_space_for_compressed_classes has been changed. > Is this necessary for this RFE, or is it a separate improvement that can be > done in a different RFE? > Well, it had to be changed somewhat, since the prototype differs. I swapped Metaspace::reserve_preferred_space() for Metaspace::reserve_address_space_for_compressed_classes(), which follows a somewhat different protocol. That aside, the functionality for AARCH64 should be as unchanged as possible. I removed the AIX specific coding, since it was subtly wrong, but that's fine since we are AIX/ppc maintainers. > [6] For AARCH64, should we skip the part that reserves it right above the > heap? Or does AARCH64 always allocate the heap such that it's a preferable > address? > > Yes and yes :) We do skip that for aarch64 (at least if the address is wrong): 1157 // Try right above the Java heap... 1158 address base = align_up(CompressedOops::end(), Metaspace::reserve_alignment()); 1159 assert(base != NULL, "Sanity"); 1160 1161 const size_t size = align_up(CompressedClassSpaceSize, Metaspace::reserve_alignment()); 1162 if (CompressedKlassPointers::is_valid_base(base)) { 1163 rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* large */, (char*)base); 1164 } At 1162, we check if the attach address would be valid for this platform. If aarch64 says no, we won't reserve there but go straight to the "attach wherever you think is good" part. Which is implemented by Metaspace::reserve_address_space_for_compressed_classes(), which on aarch64 does search a suitable address. > // case (b) > ReservedSpace rs; > > // Try right above the Java heap... > address base = align_up(CompressedOops::end(), > Metaspace::reserve_alignment()); > assert(base != NULL, "Sanity"); > > const size_t size = align_up(CompressedClassSpaceSize, > Metaspace::reserve_alignment()); > if (CompressedKlassPointers::is_valid_base(base)) { > rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* > large */, (char*)base); > } > > // ...failing that, reserve anywhere, but let platform do optimized > placement: > if (!rs.is_reserved()) { > rs = Metaspace::reserve_address_space_for_compressed_classes(size); > } > > [7] argument.cpp checks this: > > if (UseSharedSpaces || DumpSharedSpaces) { > if > (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { > log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for > this platform, option will be ignored.", > p2i((address)SharedBaseAddress)); > FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); > } > } > > but initialize_dumptime_shared_and_meta_spaces aligns up the > SharedBaseAddress before doing the assert. > > void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { > assert(DumpSharedSpaces, "should be called for dump time only"); > > const size_t reserve_alignment = > MetaspaceShared::reserved_space_alignment(); > char* shared_base = (char*)align_up((char*)SharedBaseAddress, > reserve_alignment); > > #ifdef _LP64 > assert(CompressedKlassPointers::is_valid_base((address)shared_base), > "Sanity"); > > So theoretically shared_base may no longer be is_valid_base after the > align_up. It probably won't happen, but I think the code will be much > clearer tofirst to align_up, then check for is_valid_base and reset to > default, and finally assert. > > Okay, I'll change that. > [8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the > following block can be simplified: > > if (!_shared_rs.is_reserved()) { > // Get a reserved space anywhere if attaching at the SharedBaseAddress > fails: > if (UseCompressedClassPointers) { > // If we need a compressed class space too, let the platform handle > the reservation. > _shared_rs = > Metaspace::reserve_address_space_for_compressed_classes(cds_total); > } else { > // anywhere is fine. > _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* > large */, (char*)NULL); > } > } > > ... if you change the declaration to: > > static ReservedSpace reserve_address_space_for_compressed_classes(size_t > size) NOT_LP64({ return ReservedSpace();}); > > [9] I think the #ifdef _LP64 is not necessary: > > #ifdef _LP64 > // Some sanity checks after reserving address spaces for archives and > class space. > assert(archive_space_rs.is_reserved(), "Sanity"); > if (Metaspace::using_class_space()) { > // Class space must closely follow the archive space. Both spaces > must be aligned correctly. > assert(class_space_rs.is_reserved(), "A class space should have been > reserved"); > assert(class_space_rs.base() >= archive_space_rs.end(), "class space > should follow the cds archive space"); > assert(is_aligned(archive_space_rs.base(), > MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); > assert(is_aligned(class_space_rs.base(), > Metaspace::reserve_alignment()), "class space misaligned"); > } > #endif > > I'll take a look. > The rest of the code looks OK to me, but I may take a look at it again > after getting more sleep :-) > > Sure. Thanks man! :) ..Thomas Thanks > - Ioi > > > > > On 5/7/20 7:21 AM, Thomas St?fe wrote: > > Hi all, > > please take a look at the third iteration of this change: > > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ > > Changes in this version: > > - at the request of Coleen (sorry I did not see your post earlier) I > removed all platform dependent files and put the platform dependent coding > back to metaspace.cpp and compressedOops.cpp to ease the review pain. I > used plain platform defines though (#ifdef AARCH64) instead of hiding them > behind another macro to make things clearer. Note that I still intent to > put this code away into the platform corners but will do so in a follow up > RFE. > > - I reinstated, in a fashion, the special handling of reservations on > Windows. On all platforms we reserve address space to map the archive files > in with a subsequent mapping operation. However, on Windows, we cannot use > MapViewOfFile() into an existing mapping. So I remove the mapping again > before mapping the archives - see comment in code for details. > > All CI tests at SAP run through without problems, including on Win64 and > aarch64, but I would be happy if others were to run test too. > > Thank you, Thomas > > On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe > wrote: > >> Hi all, >> >> Could I have reviews for the following proposal of reworking cds/class >> space reservation? >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >> >> Webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >> >> (Many thanks to Ioi Lam for so patiently explaining CDS internals to me, >> and to Andrew Haley and Nick Gasson for help with aarch64!) >> >> Reservation of the compressed class space is needlessly complicated and >> has some minor issues. It can be simplified and made clearer. >> >> The complexity stems from the fact that this area lives at the >> intersection of two to three sub systems, depending on how one counts. >> Metaspace, CDS, and the platform which may or may not its own view of how >> to reserve class space. And this code has been growing organically over >> time. >> >> One small example: >> >> ReservedSpace Metaspace::reserve_preferred_space(size_t size, size_t >> alignment, >> bool large_pages, char >> *requested_addr, >> bool use_requested_addr) >> >> which I spent hours decoding, resulting in a very confused mail to >> hs-runtime and aarch64-port-dev [2]. >> >> This patch attempts to simplify cds and metaspace setup a bit; to comment >> implicit knowledge which is not immediately clear; to cleanly abstract >> platform concerns like optimized class space placement; and to disentangle >> cds from metaspace to solve issues which may bite us later with Elastic >> Metaspace [4]. >> >> --- >> >> The main change is the reworked reservation mechanism. This is based on >> Ioi's proposal [5]. >> >> When reserving class space, three things must happen: >> >> 1) reservation of the space obviously. If cds is active that space must >> be in the vicinity of cds archives to be covered by compressed class >> pointer encoding. >> 2) setting up the internal Metaspace structures atop of that space >> 3) setting up compressed class pointer encoding. >> >> In its current form, Metaspace may or may not do some or all of that in >> one function >> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base);) - if cds is active, >> it will reserve the space for Metaspace and hand it in, otherwise it will >> create it itself. >> >> When discussing this in [2], Ioi proposed to move the reservation of the >> class space completely out of Metaspace and make it a responsibility of the >> caller always. This would reduce some complexity, and this patch follows >> the proposal. >> >> I removed >> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base); and all its sub >> functions. >> >> (1) now has to always be done outside - a ReservedSpace for class space >> has to be provided by the caller. However, Metaspace now offers a utility >> function for reserving space at a "nice" location, and explicitly doing >> nothing else: >> >> ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >> >> this function can be redefined on a platform level for platform optimized >> reservation, see below for details. >> >> (2) is taken care of by a new function, >> Metaspace::initialize_class_space(ReservedSpace rs) >> >> (3) is taken care of a new function >> CompressedKlassPointers::initialize(), see below for details. >> >> >> So, class space now is set up three explicit steps: >> >> - First, reserve a suitable space by however means you want. For >> convenience you may use >> Metaspace::reserve_address_space_for_compressed_classes(), or you may roll >> your own reservation. >> - Next, tell Metaspace to use that range as backing storage for class >> space: Metaspace::initialize_class_space(ReservedSpace rs) >> - Finally, set up encoding. Encoding is independent from the concept of a >> ReservedSpace, it just gets an address range, see below for details. >> >> Separating these steps and moving them out of the responsibility of >> Metaspace makes this whole thing more flexible; it also removes unnecessary >> knowledge (e.g. Metaspace does not need to know anything about either ccp >> encoding or cds). >> >> --- >> >> How it comes together: >> >> If CDS is off, we just reserve a space using >> Metaspace::reserve_address_space_for_compressed_classes(), initialize it >> with Metaspace::initialize_class_space(ReservedSpace rs), then set up >> compressed class pointer encoding covering the range of this class space. >> >> If CDS is on (dump time), we reserve large 4G space, either at >> SharedBaseAddress or using >> Metaspace::reserve_address_space_for_compressed_classes(); we then split >> that into 3G archive space and 1G class space; we set up that space with >> Metaspace as class space; then we set up compressed class pointer encoding >> covering both archive space and cds. >> >> If CDS is on (run time), we reserve a large space, split it into archive >> space (large enough to hold both archives) and class space, then basically >> proceed as above. >> >> Note that this is almost exactly how things worked before (modulo some >> minor fixes, e.g. alignment issues), only the code is reformed and made >> more explicit. >> >> --- >> >> I moved compressed class pointer setup over to CompressedKlassPointers >> and changed the interface: >> >> -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >> metaspace_rs, address cds_base) >> +void CompressedKlassPointers::initialize(address addr, size_t len); >> >> Instead of feeding it a single ReservedSpace, which is supposed to >> represent class space, and an optional alternate base if cds is on, now we >> give it just an numeric address range. That range marks the limits to where >> Klass structures are to be expected, and is the implicit promise that >> outside that range no Klass structures will exist, so encoding has to cover >> only this range. >> >> This range may contain just the class space; or class space+cds; or >> whatever allocation scheme we come up with in the future. Encoding does not >> really care how the memory is organized as long as the input range covers >> all possible Klass locations. That way we remove knowledge about class >> space/cds from compressed class pointer encoding. >> >> Moving it away from metaspace.cpp into the CompressedKlassPointers class >> also mirrors CompressedOops::initialize(). >> >> --- >> >> I renamed _narrow_klass_range to just _range, because strictly speaking >> this is the range un-narrow Klass pointers can have. >> >> As for the implementation of CompressedKlassPointers::initialize(address >> addr, size_t len), I mimicked very closely what happened before, so there >> should be almost no differences. Since "almost no differences" sounds scary >> :) here are the differences: >> >> - When CDS is active (dump or run time) we now always, unconditionally, >> set the encoding range to 4G. This fixes a theoretical bug discussed on >> aarch64-port-dev [1]. >> >> - When CDS is not active, we set the encoding range to the minimum >> required length. Before, it was left at its default value of 4G. >> >> Both differences only affect aarch64, since they are currently the only >> one using the range field in CompressedKlassPointers. >> >> I wanted to add an assert somewhere to test encoding of the very last >> address of the CompressedKlassPointers range, again to prevent errors like >> [3]. But I did not come up with a good place for this assert which would >> cover also the encoding done by C1/C2. >> >> For the same reason I thought about introducing a mode where Klass >> structures would be allocated in reverse order, starting at the end of the >> ccs, but again left it out as too big a change. >> >> --- >> >> OS abstraction: platforms may have restrictions of what constitutes a >> valid compressed class pointer encoding base. Or if not, they may have at >> least preferences. There was logic like this in metaspace.cpp, which I >> removed and cleanly factored out into platform dependent files, giving each >> platform the option to add special logic. >> >> These are two new methods: >> >> - bool CompressedKlassPointers::is_valid_base(address p) >> >> to let the platform tell you whether it considers p to be a valid >> encoding base. The only platform having these restrictions currently is >> aarch64. >> >> - ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >> >> this hands over the process of allocating a range suitable for compressed >> class pointer encoding to the platform. Most platforms will allocate just >> anywhere, but some platforms may have a better strategy (e.g. trying low >> memory first, trying only correctly aligned addresses and so on). >> >> Beforehand, this coding existed in a similar form in metaspace.cpp for >> aarch64 and AIX. For now, I left the AIX part out - it seems only half >> done, and I want to check further if we even need it, if yes why not on >> Linux ppc, and C1 does not seem to support anything other than base+offset >> with shift either, but I may be mistaken. >> >> These two methods should give the platform enough control to implement >> their own scheme for optimized class space placement without bothering any >> shared code about it. >> >> Note about the form, I introduced two new platform dependent files, >> "metaspace_.cpp" and "compressedOops_.cpp". I am not happy about >> this but this seems to be what we generally do in hotspot, right? >> >> --- >> >> Metaspace reserve alignment vs cds alignment >> >> CDS was using Metaspace reserve alignment for CDS internal purposes. I >> guess this was just a copy paste issue. It never caused problems since >> Metaspace reserve alignment == page size, but that is not true anymore in >> the upcoming Elastic Metaspace where reserve alignment will be larger. This >> causes a number of issues. >> >> I separated those two cleanly. CDS now uses >> os::vm_allocation_granularity. Metaspace::reserve_alignment is only used in >> those two places where it is needed, when CDS creates the address space for >> class space on behalf of the Metaspace. >> >> --- >> >> Windows special handling in CDS >> >> To simplify coding I removed the windows specific handling which left out >> reservation of the archive. This was needed because windows cannot mmap >> files into reserved regions. But fallback code exists in filemap.cpp for >> this case which just reads in the region instead of mapping it. >> >> Should that turn out to be a performance problem, I will reinstate the >> feature. But a simpler way would be reserve the archive and later just >> before mmapping the archive file to release the archive space. That would >> not only be simpler but give us the best guarantee that that address space >> is actually available. But I'd be happy to leave that part out completely >> if we do not see any performance problems on windows x64. >> >> --- >> >> NMT cannot deal with spaces which are split. This problem manifests in >> that bookkeeping for class space is done under "Shared Classes", not >> "Classes" as it should. This problem exists today too at dump time and >> randomly at run time. But since I simplified the reservation, this problem >> now shows up always, whether or not we map at the SharedBaseAddress. >> While I could work around this problem, I'd prefer this problem to be >> solved at the core, and NMT to have an option to recognize reservation >> splits. So I'd rather not put a workaround for this into the patch but >> leave it for fixing as a separate issue. I opened this issue to track it >> [6]. >> >> --- >> >> Jtreg tests: >> >> I expanded the CompressedOops/CompressedClassPointers.java. I also >> extended them to Windows. The tests now optionally omit strict class space >> placement tests, since these tests heavily depend on ASLR and were the >> reason they were excluded on Windows. However I think even without checking >> for class space placement they make sense, just to see that the VM comes up >> and lives with the many different settings we can run in. >> >> --- >> >> Tests: >> >> - I ran the patch through Oracles submit repo >> - I ran tests manually for aarch64, zero, linux 32bit and windows x64 >> - The whole battery of nightly tests at SAP, including ppc, ppcle and >> aarch64, unfortunately excluding windows because of unrelated errors. >> Windows x64 tests will be redone tonight. >> >> >> Thank you, >> >> Thomas >> >> [1] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >> [2] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >> [5] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >> >> > From goetz.lindenmaier at sap.com Fri May 8 07:37:22 2020 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Fri, 8 May 2020 07:37:22 +0000 Subject: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages per default. In-Reply-To: <47f7b53e-1306-96a2-89d7-16d7dec2ee9f@oracle.com> References: <47f7b53e-1306-96a2-89d7-16d7dec2ee9f@oracle.com> Message-ID: Hi David, Ok, I will file a CSR. Best regards, Goetz. > -----Original Message----- > From: David Holmes > Sent: Monday, April 27, 2020 12:03 AM > To: Lindenmaier, Goetz ; 'hotspot- > dev at openjdk.java.net' > Subject: Re: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages > per default. > > Hi Goetz, > > This will need a CSR request to change the default flag value. > > Otherwise change looks okay to me. > > Thanks, > David > > On 25/04/2020 12:36 am, Lindenmaier, Goetz wrote: > > Hi, > > > > I would like to enable JEP 358 Helpful NullPointerExceptions by default: > > > > http://cr.openjdk.java.net/~goetz/wr19/8233014- > enable_NPE_per_default/02/ > > > > Please review. > > > > The change passed our testing and jdk-submit. > > > > Best regards, > > Goetz. > > From goetz.lindenmaier at sap.com Fri May 8 07:48:47 2020 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Fri, 8 May 2020 07:48:47 +0000 Subject: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages per default. In-Reply-To: References: <47f7b53e-1306-96a2-89d7-16d7dec2ee9f@oracle.com> Message-ID: Hi David, > are we ready to flip the flag in 15? We have no feedback on the feature > and would not likely get any for some time. That is basically the point. We have no negative feedback on the issue. As you can see in the bug, the feature is live in SapMachine Since 13.0.2 and 11.0.6. That is quite a time to collect feedback. Also, if we enable it now, - we have the 15 development frame to get immediate feedback I don't want to enable it at the end of the development time (which would give more time to collect feedback from 14). - there is time to incorporate feedback before LTE 17. The time frame people are using 15 is long enough before release of LTE 17 so that I can react on feedback and improve the implementation. If I enable it in 16, which will be in use while 17 is developed and stabilized, this might not be possible. So I think this is the right point of time to enable it. For testing, as you know we have quite a test infrastructure. I ran it through all these tests, which incorporate most jck tests and all headless jtreg tests. Also, if I remember correctly, the change has been run through your testing landscape when we first pushed the feature to have it tested in enabled mode. But I have so many mails on this that I could not find prove of this. So I'd assume there are no correctness problems. There might be a new test somewhere that checks the content of the message (before: null) and which needs to be adapted. Best regards, Goetz. > -----Original Message----- > From: David Holmes > Sent: Monday, April 27, 2020 1:08 AM > To: Lindenmaier, Goetz ; 'hotspot- > dev at openjdk.java.net' > Subject: Re: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages > per default. > > Hi Goetz, > > On 27/04/2020 8:02 am, David Holmes wrote: > > Hi Goetz, > > > > This will need a CSR request to change the default flag value. > > On that front, given this feature only just shipped in 14, on what basis > are we ready to flip the flag in 15? We have no feedback on the feature > and would not likely get any for some time. > > > Otherwise change looks okay to me. > > With regards to testing however, please ensure that turning this on does > not inadvertently disrupt any tests. Passing jdk-submit is a minimal > requirement here and more extensive testing is desirable. > > Thanks, > David > > > Thanks, > > David > > > > On 25/04/2020 12:36 am, Lindenmaier, Goetz wrote: > >> Hi, > >> > >> I would like to enable JEP 358 Helpful NullPointerExceptions by default: > >> > >> http://cr.openjdk.java.net/~goetz/wr19/8233014- > enable_NPE_per_default/02/ > >> > >> Please review. > >> > >> The change passed our testing and jdk-submit. > >> > >> Best regards, > >> ? Goetz. > >> From goetz.lindenmaier at sap.com Fri May 8 09:07:44 2020 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Fri, 8 May 2020 09:07:44 +0000 Subject: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages per default. In-Reply-To: <0fb381bf-0e86-3f9c-d8b3-d9fc3f0a46b2@oracle.com> References: <0fb381bf-0e86-3f9c-d8b3-d9fc3f0a46b2@oracle.com> Message-ID: Hi Yes I think my webrev is ok. In the original file Oracle comes first, and my patch does not reverse that. The SAP copyright never has a comma after the last year. We decided to do it that way long ago when we edited all the files we contributed for the ppc port. But there are no strict rules on that by SAP. It's just nice if it is consistent, but I think there are both variants by now. Best regards, Goetz. > -----Original Message----- > From: David Holmes > Sent: Monday, April 27, 2020 11:58 AM > To: Reingruber, Richard ; Lindenmaier, Goetz > ; 'hotspot-dev at openjdk.java.net' dev at openjdk.java.net> > Subject: Re: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages > per default. > > On 27/04/2020 7:49 pm, Reingruber, Richard wrote: > > Hi David, Goetz, > > > >>> Just a minor detail: the license check fails for the file > SuppressMessagesTest.java > >>> > >>> Try > >>> > >>> sh make/scripts/lic_check.sh -gpl $(hg log -v --rev tip |sed -n > '/files:[[:space:]]*/s///p') > > > >> Why did it fail? > > > > Because there is no match for make/templates/gpl-header I reckon. > > > >>> With > >>> > >>> --- > a/test/hotspot/jtreg/runtime/exceptionMsgs/NullPointerException/Suppress > MessagesTest.java > >>> +++ > b/test/hotspot/jtreg/runtime/exceptionMsgs/NullPointerException/Suppress > MessagesTest.java > >>> @@ -1,6 +1,6 @@ > >>> /* > >>> + * Copyright (c) 2019, 2020, SAP SE. All rights reserved. > >>> * Copyright (c) 2019, 2020, Oracle and/or its affiliates. All rights > reserved. > >>> - * Copyright (c) 2019, 2020 SAP SE. All rights reserved. > >>> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE > HEADER. > >>> * > >>> * This code is free software; you can redistribute it and/or modify it > >>> > >>> it succeeds. Note that I added a comma after 2020 also. > > > >> That doesn't seem to be the SAP format. (Though it is not 100% > consistent.) > > > > Right. In all files I checked the Oracle copyright line came before other > copyright lines (SAP, Red > > Hat). > > I was referring specifically to the comma after the second copyright > year. It's an Oracle requirement, but not apparently a SAP one. > > Most files with dual copyrights do have Oracle first, but that should > not be a requirement for the header. There are many files without an > Oracle copyright and if an Oracle engineer were to make significant > changes and thus add the Oracle copyright, there's no rule I know that > says the Oracle copyright should then come first. Sounds like a bug in > the script to me. > > Cheers, > David > > > Maybe it is not applicable here? > > > > Cheers, Richard. > > > > -----Original Message----- > > From: David Holmes > > Sent: Montag, 27. April 2020 11:02 > > To: Reingruber, Richard ; Lindenmaier, > Goetz ; 'hotspot-dev at openjdk.java.net' > > > Subject: Re: RFR(S): 8233014: Enable > ShowCodeDetailsInExceptionMessages per default. > > > > On 27/04/2020 6:20 pm, Reingruber, Richard wrote: > >> Hi Goetz, > >> > >> Just a minor detail: the license check fails for the file > SuppressMessagesTest.java > >> > >> Try > >> > >> sh make/scripts/lic_check.sh -gpl $(hg log -v --rev tip |sed -n > '/files:[[:space:]]*/s///p') > > > > Why did it fail? > > > >> With > >> > >> --- > a/test/hotspot/jtreg/runtime/exceptionMsgs/NullPointerException/Suppress > MessagesTest.java > >> +++ > b/test/hotspot/jtreg/runtime/exceptionMsgs/NullPointerException/Suppress > MessagesTest.java > >> @@ -1,6 +1,6 @@ > >> /* > >> + * Copyright (c) 2019, 2020, SAP SE. All rights reserved. > >> * Copyright (c) 2019, 2020, Oracle and/or its affiliates. All rights > reserved. > >> - * Copyright (c) 2019, 2020 SAP SE. All rights reserved. > >> * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE > HEADER. > >> * > >> * This code is free software; you can redistribute it and/or modify it > >> > >> it succeeds. Note that I added a comma after 2020 also. > > > > That doesn't seem to be the SAP format. (Though it is not 100% consistent.) > > > > Cheers, > > David > > > >> Cheers, Richard. > >> > >> -----Original Message----- > >> From: hotspot-dev On Behalf > Of Lindenmaier, Goetz > >> Sent: Freitag, 24. April 2020 16:36 > >> To: 'hotspot-dev at openjdk.java.net' > >> Subject: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages > per default. > >> > >> Hi, > >> > >> I would like to enable JEP 358 Helpful NullPointerExceptions by default: > >> > >> http://cr.openjdk.java.net/~goetz/wr19/8233014- > enable_NPE_per_default/02/ > >> > >> Please review. > >> > >> The change passed our testing and jdk-submit. > >> > >> Best regards, > >> Goetz. > >> From goetz.lindenmaier at sap.com Fri May 8 09:11:43 2020 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Fri, 8 May 2020 09:11:43 +0000 Subject: RFR: CSR 8244639 Enable ShowCodeDetailsInExceptionMessages per default. Message-ID: Hi, I filed this CSR as requested by David and Alan. Please review. https://bugs.openjdk.java.net/browse/JDK-8244639 Best regards, Goetz. From thomas.schatzl at oracle.com Fri May 8 09:34:15 2020 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 8 May 2020 11:34:15 +0200 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: <394BD86E-EC91-440E-9936-696B2B453093@oracle.com> References: <394BD86E-EC91-440E-9936-696B2B453093@oracle.com> Message-ID: <28de2cd7-6209-d780-0975-ab701e6f24f9@oracle.com> Hi, On 07.05.20 07:35, Mikael Vidstedt wrote: > > New webrev here: > > webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.01/hotspot/open/webrev/ > incremental: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.01/hotspot.incr/open/webrev/ > > Remaining items: > > * File follow-up to remove STACK_BIAS > > * File follow-ups to change/update/remove flags and/or flag documentation: UseLWPSynchronization, BranchOnRegister, LIRFillDelaySlots, ArrayAllocatorMallocLimit, ThreadPriorityPolicy > > * File follow-up(s) to update comments ("solaris", ?sparc?, ?solstudio?, ?sunos?, ?sun studio?, ?s compiler bug?, ?niagara?, ?) > > > Please let me know if there?s something I have missed! Looks good. Thomas From david.holmes at oracle.com Fri May 8 09:58:01 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 8 May 2020 19:58:01 +1000 Subject: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages per default. In-Reply-To: References: <47f7b53e-1306-96a2-89d7-16d7dec2ee9f@oracle.com> Message-ID: <03601511-fa52-38c8-1c4c-696ad3fdc4ab@oracle.com> Hi Goetz, I have reviewed the CSR. On 8/05/2020 5:48 pm, Lindenmaier, Goetz wrote: > Hi David, > >> are we ready to flip the flag in 15? We have no feedback on the feature >> and would not likely get any for some time. > That is basically the point. > We have no negative feedback on the issue. > > As you can see in the bug, the feature is live in SapMachine > Since 13.0.2 and 11.0.6. That is quite a time to collect feedback. > > Also, if we enable it now, > - we have the 15 development frame to get immediate feedback > I don't want to enable it at the end of the development time > (which would give more time to collect feedback from 14). > - there is time to incorporate feedback before LTE 17. > The time frame people are using 15 is long enough before > release of LTE 17 so that I can react on feedback and improve > the implementation. If I enable it in 16, which will be in use > while 17 is developed and stabilized, this might not be possible. > > So I think this is the right point of time to enable it. Okay that sounds quite reasonable. > For testing, as you know we have quite a test infrastructure. > I ran it through all these tests, which incorporate most jck tests > and all headless jtreg tests. > Also, if I remember correctly, the change has been run through your > testing landscape when we first pushed the feature to have it tested > in enabled mode. But I have so many mails on this that I could not > find prove of this. > > So I'd assume there are no correctness problems. There might be a > new test somewhere that checks the content of the message > (before: null) and which needs to be adapted. Yeah I don't know what was done before. I'll run it through our testing again while you wait for the CSR approval. Thanks, David > Best regards, > Goetz. > > > > > > >> -----Original Message----- >> From: David Holmes >> Sent: Monday, April 27, 2020 1:08 AM >> To: Lindenmaier, Goetz ; 'hotspot- >> dev at openjdk.java.net' >> Subject: Re: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages >> per default. >> >> Hi Goetz, >> >> On 27/04/2020 8:02 am, David Holmes wrote: >>> Hi Goetz, >>> >>> This will need a CSR request to change the default flag value. >> >> On that front, given this feature only just shipped in 14, on what basis >> are we ready to flip the flag in 15? We have no feedback on the feature >> and would not likely get any for some time. >> >>> Otherwise change looks okay to me. >> >> With regards to testing however, please ensure that turning this on does >> not inadvertently disrupt any tests. Passing jdk-submit is a minimal >> requirement here and more extensive testing is desirable. >> >> Thanks, >> David >> >>> Thanks, >>> David >>> >>> On 25/04/2020 12:36 am, Lindenmaier, Goetz wrote: >>>> Hi, >>>> >>>> I would like to enable JEP 358 Helpful NullPointerExceptions by default: >>>> >>>> http://cr.openjdk.java.net/~goetz/wr19/8233014- >> enable_NPE_per_default/02/ >>>> >>>> Please review. >>>> >>>> The change passed our testing and jdk-submit. >>>> >>>> Best regards, >>>> ? Goetz. >>>> From goetz.lindenmaier at sap.com Fri May 8 10:10:28 2020 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Fri, 8 May 2020 10:10:28 +0000 Subject: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages per default. In-Reply-To: <03601511-fa52-38c8-1c4c-696ad3fdc4ab@oracle.com> References: <47f7b53e-1306-96a2-89d7-16d7dec2ee9f@oracle.com> <03601511-fa52-38c8-1c4c-696ad3fdc4ab@oracle.com> Message-ID: Hi David, > I have reviewed the CSR. Thanks a lot! Should I move it back to draft? > > ... > > So I think this is the right point of time to enable it. > Okay that sounds quite reasonable. Thanks! > Yeah I don't know what was done before. I'll run it through our testing > again while you wait for the CSR approval. That's great, thanks a lot. Best regards, Goetz. > > Thanks, > David > > > Best regards, > > Goetz. > > > > > > > > > > > > > >> -----Original Message----- > >> From: David Holmes > >> Sent: Monday, April 27, 2020 1:08 AM > >> To: Lindenmaier, Goetz ; 'hotspot- > >> dev at openjdk.java.net' > >> Subject: Re: RFR(S): 8233014: Enable > ShowCodeDetailsInExceptionMessages > >> per default. > >> > >> Hi Goetz, > >> > >> On 27/04/2020 8:02 am, David Holmes wrote: > >>> Hi Goetz, > >>> > >>> This will need a CSR request to change the default flag value. > >> > >> On that front, given this feature only just shipped in 14, on what basis > >> are we ready to flip the flag in 15? We have no feedback on the feature > >> and would not likely get any for some time. > >> > >>> Otherwise change looks okay to me. > >> > >> With regards to testing however, please ensure that turning this on does > >> not inadvertently disrupt any tests. Passing jdk-submit is a minimal > >> requirement here and more extensive testing is desirable. > >> > >> Thanks, > >> David > >> > >>> Thanks, > >>> David > >>> > >>> On 25/04/2020 12:36 am, Lindenmaier, Goetz wrote: > >>>> Hi, > >>>> > >>>> I would like to enable JEP 358 Helpful NullPointerExceptions by default: > >>>> > >>>> http://cr.openjdk.java.net/~goetz/wr19/8233014- > >> enable_NPE_per_default/02/ > >>>> > >>>> Please review. > >>>> > >>>> The change passed our testing and jdk-submit. > >>>> > >>>> Best regards, > >>>> ? Goetz. > >>>> From david.holmes at oracle.com Fri May 8 10:22:55 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 8 May 2020 20:22:55 +1000 Subject: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages per default. In-Reply-To: References: <47f7b53e-1306-96a2-89d7-16d7dec2ee9f@oracle.com> <03601511-fa52-38c8-1c4c-696ad3fdc4ab@oracle.com> Message-ID: On 8/05/2020 8:10 pm, Lindenmaier, Goetz wrote: > Hi David, > >> I have reviewed the CSR. > Thanks a lot! > Should I move it back to draft? Back to Draft then to Finalized. Cheers, David >>> ... >>> So I think this is the right point of time to enable it. >> Okay that sounds quite reasonable. > Thanks! > >> Yeah I don't know what was done before. I'll run it through our testing >> again while you wait for the CSR approval. > That's great, thanks a lot. > > Best regards, > Goetz. > > >> >> Thanks, >> David >> >>> Best regards, >>> Goetz. >>> >>> >>> >>> >>> >>> >>>> -----Original Message----- >>>> From: David Holmes >>>> Sent: Monday, April 27, 2020 1:08 AM >>>> To: Lindenmaier, Goetz ; 'hotspot- >>>> dev at openjdk.java.net' >>>> Subject: Re: RFR(S): 8233014: Enable >> ShowCodeDetailsInExceptionMessages >>>> per default. >>>> >>>> Hi Goetz, >>>> >>>> On 27/04/2020 8:02 am, David Holmes wrote: >>>>> Hi Goetz, >>>>> >>>>> This will need a CSR request to change the default flag value. >>>> >>>> On that front, given this feature only just shipped in 14, on what basis >>>> are we ready to flip the flag in 15? We have no feedback on the feature >>>> and would not likely get any for some time. >>>> >>>>> Otherwise change looks okay to me. >>>> >>>> With regards to testing however, please ensure that turning this on does >>>> not inadvertently disrupt any tests. Passing jdk-submit is a minimal >>>> requirement here and more extensive testing is desirable. >>>> >>>> Thanks, >>>> David >>>> >>>>> Thanks, >>>>> David >>>>> >>>>> On 25/04/2020 12:36 am, Lindenmaier, Goetz wrote: >>>>>> Hi, >>>>>> >>>>>> I would like to enable JEP 358 Helpful NullPointerExceptions by default: >>>>>> >>>>>> http://cr.openjdk.java.net/~goetz/wr19/8233014- >>>> enable_NPE_per_default/02/ >>>>>> >>>>>> Please review. >>>>>> >>>>>> The change passed our testing and jdk-submit. >>>>>> >>>>>> Best regards, >>>>>> ? Goetz. >>>>>> From goetz.lindenmaier at sap.com Fri May 8 10:51:05 2020 From: goetz.lindenmaier at sap.com (Lindenmaier, Goetz) Date: Fri, 8 May 2020 10:51:05 +0000 Subject: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages per default. In-Reply-To: References: <47f7b53e-1306-96a2-89d7-16d7dec2ee9f@oracle.com> <03601511-fa52-38c8-1c4c-696ad3fdc4ab@oracle.com> Message-ID: Ok, thanks ?? Done. Best regards, Goetz. > -----Original Message----- > From: David Holmes > Sent: Friday, May 8, 2020 12:23 PM > To: Lindenmaier, Goetz ; 'hotspot- > dev at openjdk.java.net' > Subject: Re: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages > per default. > > On 8/05/2020 8:10 pm, Lindenmaier, Goetz wrote: > > Hi David, > > > >> I have reviewed the CSR. > > Thanks a lot! > > Should I move it back to draft? > > Back to Draft then to Finalized. > > Cheers, > David > > >>> ... > >>> So I think this is the right point of time to enable it. > >> Okay that sounds quite reasonable. > > Thanks! > > > >> Yeah I don't know what was done before. I'll run it through our testing > >> again while you wait for the CSR approval. > > That's great, thanks a lot. > > > > Best regards, > > Goetz. > > > > > >> > >> Thanks, > >> David > >> > >>> Best regards, > >>> Goetz. > >>> > >>> > >>> > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: David Holmes > >>>> Sent: Monday, April 27, 2020 1:08 AM > >>>> To: Lindenmaier, Goetz ; 'hotspot- > >>>> dev at openjdk.java.net' > >>>> Subject: Re: RFR(S): 8233014: Enable > >> ShowCodeDetailsInExceptionMessages > >>>> per default. > >>>> > >>>> Hi Goetz, > >>>> > >>>> On 27/04/2020 8:02 am, David Holmes wrote: > >>>>> Hi Goetz, > >>>>> > >>>>> This will need a CSR request to change the default flag value. > >>>> > >>>> On that front, given this feature only just shipped in 14, on what basis > >>>> are we ready to flip the flag in 15? We have no feedback on the feature > >>>> and would not likely get any for some time. > >>>> > >>>>> Otherwise change looks okay to me. > >>>> > >>>> With regards to testing however, please ensure that turning this on > does > >>>> not inadvertently disrupt any tests. Passing jdk-submit is a minimal > >>>> requirement here and more extensive testing is desirable. > >>>> > >>>> Thanks, > >>>> David > >>>> > >>>>> Thanks, > >>>>> David > >>>>> > >>>>> On 25/04/2020 12:36 am, Lindenmaier, Goetz wrote: > >>>>>> Hi, > >>>>>> > >>>>>> I would like to enable JEP 358 Helpful NullPointerExceptions by > default: > >>>>>> > >>>>>> http://cr.openjdk.java.net/~goetz/wr19/8233014- > >>>> enable_NPE_per_default/02/ > >>>>>> > >>>>>> Please review. > >>>>>> > >>>>>> The change passed our testing and jdk-submit. > >>>>>> > >>>>>> Best regards, > >>>>>> ? Goetz. > >>>>>> From daniel.daugherty at oracle.com Fri May 8 19:48:02 2020 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Fri, 8 May 2020 15:48:02 -0400 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: <394BD86E-EC91-440E-9936-696B2B453093@oracle.com> References: <394BD86E-EC91-440E-9936-696B2B453093@oracle.com> Message-ID: <048a8e25-fec4-5304-8763-6a4472bfbf54@oracle.com> On 5/7/20 1:35 AM, Mikael Vidstedt wrote: > New webrev here: > > webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.01/hotspot/open/webrev/ This pretty much says it all: > Summary of changes:??? 90904 lines changed: 8 ins; 90725 del; 171 mod; 103780 unchg My review is focused on looking at the changes and not looking for missed changes. I figure there's enough work here just looking at the changes to keep me occupied for a while and enough people have posted comments about finding other things to be examined, etc... Unlike my normal reviews, I won't be listing all the touched files; (there's _only_ 427 of them...) Don't forget to make a copyright year update pass before you push. src/hotspot/os/posix/os_posix.hpp ??????? L174 ??? old L175 #ifndef SOLARIS ??????? L176 ?????? nit - on most of this style of deletion you also got rid of ?????? one of the blank lines, but not here. src/hotspot/share/utilities/dtrace.hpp ??? old L42: #elif defined(__APPLE__) ??? old L44: #include ??? old L45: #else ??? new L32: #include ??????? was previous included only for __APPLE__ and it ??????? is now there for every platform. Any particular reason? Thumbs up! Dan > incremental: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.01/hotspot.incr/open/webrev/ > > Remaining items: > > * File follow-up to remove STACK_BIAS > > * File follow-ups to change/update/remove flags and/or flag documentation: UseLWPSynchronization, BranchOnRegister, LIRFillDelaySlots, ArrayAllocatorMallocLimit, ThreadPriorityPolicy > > * File follow-up(s) to update comments ("solaris", ?sparc?, ?solstudio?, ?sunos?, ?sun studio?, ?s compiler bug?, ?niagara?, ?) > > > Please let me know if there?s something I have missed! > > Cheers, > Mikael > >> On May 3, 2020, at 10:12 PM, Mikael Vidstedt wrote: >> >> >> Please review this change which implements part of JEP 381: >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ >> JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 >> >> >> Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! >> >> >> Background: >> >> Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. >> >> For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. >> >> In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. >> >> A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! >> >> Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. >> >> Testing: >> >> A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. >> >> Cheers, >> Mikael >> >> [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ >> From coleen.phillimore at oracle.com Fri May 8 22:59:19 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 8 May 2020 18:59:19 -0400 Subject: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages per default. In-Reply-To: <03601511-fa52-38c8-1c4c-696ad3fdc4ab@oracle.com> References: <47f7b53e-1306-96a2-89d7-16d7dec2ee9f@oracle.com> <03601511-fa52-38c8-1c4c-696ad3fdc4ab@oracle.com> Message-ID: <7c2f51fe-80ca-16be-a700-2833bc63ea9c@oracle.com> On 5/8/20 5:58 AM, David Holmes wrote: > Hi Goetz, > > I have reviewed the CSR. > > On 8/05/2020 5:48 pm, Lindenmaier, Goetz wrote: >> Hi David, >> >>> are we ready to flip the flag in 15? We have no feedback on the feature >>> and would not likely get any for some time. >> That is basically the point. >> We have no negative feedback on the issue. >> >> As you can see in the bug, the feature is live in SapMachine >> Since 13.0.2 and 11.0.6. That is quite a time to collect feedback. >> >> Also, if we enable it now, >> ?? - we have the 15 development frame to get immediate feedback >> ???? I don't want to enable it at the end of the development time >> ??? (which would give more time to collect feedback from 14). >> ?? - there is time to incorporate feedback before LTE 17. >> ???? The time frame people are using 15 is long enough before >> ???? release of LTE 17 so that I can react on feedback and improve >> ???? the implementation.? If I enable it in 16, which will be in use >> ???? while 17 is developed and stabilized, this might not be possible. >> >> So I think this is the right point of time to enable it. > > Okay that sounds quite reasonable. Thanks Goetz for the justification.? I agree with this.? With the short release cycle, we're not likely to get feedback unless a customer is staying current on JDK versions and knows about the option. There's also probably a lag until people transition from JDK 14 to 15, so that gives less time to have users experience the option.? And so far, as you report the SAP users have had a positive experiences, and it's something we want for the platform. > >> For testing, as you know we have quite a test infrastructure. >> I ran it through all these tests, which incorporate most jck tests >> and all headless jtreg tests. >> Also, if I remember correctly, the change has been run through your >> testing landscape when we first pushed the feature to have it tested >> in enabled mode.? But I have so many mails on this that I could not >> find prove of this. >> >> So I'd assume there are no correctness problems. There might be a >> new test somewhere that checks the content of the message >> (before: null) and which needs to be adapted. > > Yeah I don't know what was done before. I'll run it through our > testing again while you wait for the CSR approval. > I ran our tier1 testing on this patch, but David, please see the JEP, I posted internal links to test results for the option on and off.? I also completed the security tasks for the JEP which has internal links as well. https://bugs.openjdk.java.net/browse/JDK-8220715 Thanks, Coleen > Thanks, > David > >> Best regards, >> ?? Goetz. >> >> >> >> >> >> >>> -----Original Message----- >>> From: David Holmes >>> Sent: Monday, April 27, 2020 1:08 AM >>> To: Lindenmaier, Goetz ; 'hotspot- >>> dev at openjdk.java.net' >>> Subject: Re: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages >>> per default. >>> >>> Hi Goetz, >>> >>> On 27/04/2020 8:02 am, David Holmes wrote: >>>> Hi Goetz, >>>> >>>> This will need a CSR request to change the default flag value. >>> >>> On that front, given this feature only just shipped in 14, on what >>> basis >>> are we ready to flip the flag in 15? We have no feedback on the feature >>> and would not likely get any for some time. >>> >>>> Otherwise change looks okay to me. >>> >>> With regards to testing however, please ensure that turning this on >>> does >>> not inadvertently disrupt any tests. Passing jdk-submit is a minimal >>> requirement here and more extensive testing is desirable. >>> >>> Thanks, >>> David >>> >>>> Thanks, >>>> David >>>> >>>> On 25/04/2020 12:36 am, Lindenmaier, Goetz wrote: >>>>> Hi, >>>>> >>>>> I would like to enable JEP 358 Helpful NullPointerExceptions by >>>>> default: >>>>> >>>>> http://cr.openjdk.java.net/~goetz/wr19/8233014- >>> enable_NPE_per_default/02/ >>>>> >>>>> Please review. >>>>> >>>>> The change passed our testing and jdk-submit. >>>>> >>>>> Best regards, >>>>> ?? Goetz. >>>>> From david.holmes at oracle.com Sat May 9 05:29:37 2020 From: david.holmes at oracle.com (David Holmes) Date: Sat, 9 May 2020 15:29:37 +1000 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: References: Message-ID: <81b1d335-d898-5bc9-8aed-b105f8d73a2e@oracle.com> Hi Matthias, On 6/05/2020 9:18 pm, Baesken, Matthias wrote: > Hello, > > I noticed issues on one of our Linux test machines with jtreg test test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > After 8231111: Cgroups v2: Rework Metrics in java.base so as to recognize unified hierarchy . > we see errors on one of our Linux test machines in jtreg test test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java . > >> Error: >> test result: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: 'OperatingSystemMXBean.getTotalSwapSpaceSize: 0' missing from stdout/stderr >> Test output reports: >> OperatingSystemMXBean.getTotalSwapSpaceSize: 16106061824 >> OperatingSystemMXBean.getFreeSwapSpaceSize: 15958736896 >> We also see this output: >> WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. >> > > A check for the warning message "Your kernel does not support swap limit capabilities" could be added to the test. But according to the comment immediately before your changes: // in case of warnings like : "Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap." // the getTotalSwapSpaceSize does not return the expected result, but 0 we should be returning zero. So this seems to be a bug in the implementation not in the test. David ----- > Bug/webrev : > https://bugs.openjdk.java.net/browse/JDK-8244500 > > http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.0/ > > Thanks, Matthias > > From david.holmes at oracle.com Mon May 11 02:41:11 2020 From: david.holmes at oracle.com (David Holmes) Date: Mon, 11 May 2020 12:41:11 +1000 Subject: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages per default. In-Reply-To: <7c2f51fe-80ca-16be-a700-2833bc63ea9c@oracle.com> References: <47f7b53e-1306-96a2-89d7-16d7dec2ee9f@oracle.com> <03601511-fa52-38c8-1c4c-696ad3fdc4ab@oracle.com> <7c2f51fe-80ca-16be-a700-2833bc63ea9c@oracle.com> Message-ID: <246cae48-f195-4d75-83e1-0522c483d8dc@oracle.com> On 9/05/2020 8:59 am, coleen.phillimore at oracle.com wrote: > On 5/8/20 5:58 AM, David Holmes wrote: >> Hi Goetz, >> >> I have reviewed the CSR. >> >> On 8/05/2020 5:48 pm, Lindenmaier, Goetz wrote: >>> Hi David, >>> >>>> are we ready to flip the flag in 15? We have no feedback on the feature >>>> and would not likely get any for some time. >>> That is basically the point. >>> We have no negative feedback on the issue. >>> >>> As you can see in the bug, the feature is live in SapMachine >>> Since 13.0.2 and 11.0.6. That is quite a time to collect feedback. >>> >>> Also, if we enable it now, >>> ?? - we have the 15 development frame to get immediate feedback >>> ???? I don't want to enable it at the end of the development time >>> ??? (which would give more time to collect feedback from 14). >>> ?? - there is time to incorporate feedback before LTE 17. >>> ???? The time frame people are using 15 is long enough before >>> ???? release of LTE 17 so that I can react on feedback and improve >>> ???? the implementation.? If I enable it in 16, which will be in use >>> ???? while 17 is developed and stabilized, this might not be possible. >>> >>> So I think this is the right point of time to enable it. >> >> Okay that sounds quite reasonable. > > Thanks Goetz for the justification.? I agree with this.? With the short > release cycle, we're not likely to get feedback unless a customer is > staying current on JDK versions and knows about the option. There's also > probably a lag until people transition from JDK 14 to 15, so that gives > less time to have users experience the option.? And so far, as you > report the SAP users have had a positive experiences, and it's something > we want for the platform. >> >>> For testing, as you know we have quite a test infrastructure. >>> I ran it through all these tests, which incorporate most jck tests >>> and all headless jtreg tests. >>> Also, if I remember correctly, the change has been run through your >>> testing landscape when we first pushed the feature to have it tested >>> in enabled mode.? But I have so many mails on this that I could not >>> find prove of this. >>> >>> So I'd assume there are no correctness problems. There might be a >>> new test somewhere that checks the content of the message >>> (before: null) and which needs to be adapted. >> >> Yeah I don't know what was done before. I'll run it through our >> testing again while you wait for the CSR approval. >> > > I ran our tier1 testing on this patch, but David, please see the JEP, I > posted internal links to test results for the option on and off.? I also > completed the security tasks for the JEP which has internal links as well. > > https://bugs.openjdk.java.net/browse/JDK-8220715 I was just checking nothing had slipped in since then that might be affected. tiers 1-3 are clear. Cheers, David > Thanks, > Coleen > > >> Thanks, >> David >> >>> Best regards, >>> ?? Goetz. >>> >>> >>> >>> >>> >>> >>>> -----Original Message----- >>>> From: David Holmes >>>> Sent: Monday, April 27, 2020 1:08 AM >>>> To: Lindenmaier, Goetz ; 'hotspot- >>>> dev at openjdk.java.net' >>>> Subject: Re: RFR(S): 8233014: Enable ShowCodeDetailsInExceptionMessages >>>> per default. >>>> >>>> Hi Goetz, >>>> >>>> On 27/04/2020 8:02 am, David Holmes wrote: >>>>> Hi Goetz, >>>>> >>>>> This will need a CSR request to change the default flag value. >>>> >>>> On that front, given this feature only just shipped in 14, on what >>>> basis >>>> are we ready to flip the flag in 15? We have no feedback on the feature >>>> and would not likely get any for some time. >>>> >>>>> Otherwise change looks okay to me. >>>> >>>> With regards to testing however, please ensure that turning this on >>>> does >>>> not inadvertently disrupt any tests. Passing jdk-submit is a minimal >>>> requirement here and more extensive testing is desirable. >>>> >>>> Thanks, >>>> David >>>> >>>>> Thanks, >>>>> David >>>>> >>>>> On 25/04/2020 12:36 am, Lindenmaier, Goetz wrote: >>>>>> Hi, >>>>>> >>>>>> I would like to enable JEP 358 Helpful NullPointerExceptions by >>>>>> default: >>>>>> >>>>>> http://cr.openjdk.java.net/~goetz/wr19/8233014- >>>> enable_NPE_per_default/02/ >>>>>> >>>>>> Please review. >>>>>> >>>>>> The change passed our testing and jdk-submit. >>>>>> >>>>>> Best regards, >>>>>> ?? Goetz. >>>>>> > From martinrb at google.com Mon May 11 17:54:53 2020 From: martinrb at google.com (Martin Buchholz) Date: Mon, 11 May 2020 10:54:53 -0700 Subject: [8u] RFR(S): 8244548: JDK 8u: sun.misc.Version.jdkUpdateVersion() returns wrong result In-Reply-To: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> References: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> Message-ID: On Thu, May 7, 2020 at 8:18 AM Severin Gehwolf wrote: > Thus, the new upper bound for update > version number is 2^16-1 => 65535 which should be sufficient. I don't > think OpenJDK 8u will live that long ;-) Everyone expected 256 to be sufficient for update release numbering, before the gaps in the numbering scheme were introduced! I don't know how we ended up with 2 mod 10. From mikael.vidstedt at oracle.com Mon May 11 18:18:30 2020 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Mon, 11 May 2020 11:18:30 -0700 Subject: RFR: 8244224: Implementation of JEP 381: Remove the Solaris and SPARC Ports (hotspot) In-Reply-To: <048a8e25-fec4-5304-8763-6a4472bfbf54@oracle.com> References: <394BD86E-EC91-440E-9936-696B2B453093@oracle.com> <048a8e25-fec4-5304-8763-6a4472bfbf54@oracle.com> Message-ID: <3810FA34-8AC5-45F0-B0DB-57C28324FECB@oracle.com> > On May 8, 2020, at 12:48 PM, Daniel D. Daugherty wrote: > > On 5/7/20 1:35 AM, Mikael Vidstedt wrote: >> New webrev here: >> >> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.01/hotspot/open/webrev/ > > This pretty much says it all: > > > Summary of changes: 90904 lines changed: 8 ins; 90725 del; 171 mod; 103780 unchg > > > My review is focused on looking at the changes and not looking for missed > changes. I figure there's enough work here just looking at the changes to > keep me occupied for a while and enough people have posted comments about > finding other things to be examined, etc... > > Unlike my normal reviews, I won't be listing all the touched files; > (there's _only_ 427 of them...) > > Don't forget to make a copyright year update pass before you push. Yup - I have added it in 10 different places on my TODO list to maximize the likelihood of me remembering it :) > > src/hotspot/os/posix/os_posix.hpp > L174 > old L175 #ifndef SOLARIS > L176 > nit - on most of this style of deletion you also got rid of > one of the blank lines, but not here. Oops, will fix. > src/hotspot/share/utilities/dtrace.hpp > old L42: #elif defined(__APPLE__) > old L44: #include > old L45: #else > new L32: #include > was previous included only for __APPLE__ and it > is now there for every platform. Any particular reason? No particular reason other than "it looks cleaner". I guess we could see if the include can be removed altogether. > Thumbs up! Thanks for the review!! Cheers, Mikael > >> incremental: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.01/hotspot.incr/open/webrev/ >> >> Remaining items: >> >> * File follow-up to remove STACK_BIAS >> >> * File follow-ups to change/update/remove flags and/or flag documentation: UseLWPSynchronization, BranchOnRegister, LIRFillDelaySlots, ArrayAllocatorMallocLimit, ThreadPriorityPolicy >> >> * File follow-up(s) to update comments ("solaris", ?sparc?, ?solstudio?, ?sunos?, ?sun studio?, ?s compiler bug?, ?niagara?, ?) >> >> >> Please let me know if there?s something I have missed! >> >> Cheers, >> Mikael >> >>> On May 3, 2020, at 10:12 PM, Mikael Vidstedt wrote: >>> >>> >>> Please review this change which implements part of JEP 381: >>> >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8244224 >>> webrev: http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/hotspot/open/webrev/ >>> JEP: https://bugs.openjdk.java.net/browse/JDK-8241787 >>> >>> >>> Note: When reviewing this, please be aware that this exercise was *extremely* mind-numbing, so I appreciate your help reviewing all the individual changes carefully. You may want to get that coffee cup filled up (or whatever keeps you awake)! >>> >>> >>> Background: >>> >>> Because of the size of the total patch and wide range of areas touched, this patch is one out of in total six partial patches which together make up the necessary changes to remove the Solaris and SPARC ports. The other patches are being sent out for review to mailing lists appropriate for the respective areas the touch. An email will be sent to jdk-dev summarizing all the patches/reviews. To be clear: this patch is *not* in itself complete and stand-alone - all of the (six) patches are needed to form a complete patch. Some changes in this patch may look wrong or incomplete unless also looking at the corresponding changes in other areas. >>> >>> For convenience, I?m including a link below[1] to the full webrev, but in case you have comments on changes in other areas, outside of the files included in this thread, please provide those comments directly in the thread on the appropriate mailing list for that area if possible. >>> >>> In case it helps, the changes were effectively produced by searching for and updating any code mentioning ?solaris", ?sparc?, ?solstudio?, ?sunos?, etc. More information about the areas impacted can be found in the JEP itself. >>> >>> A big thank you to Igor Ignatyev for helping make the changes to the hotspot tests! >>> >>> Also, I have a short list of follow-ups which I?m going to look at separately from this JEP/patch, mainly related to command line options/flags which are no longer relevant and should be deprecated/obsoleted/removed. >>> >>> Testing: >>> >>> A slightly earlier version of this change successfully passed tier1-8, as well as client tier1-2. Additional testing will be done after the first round of reviews has been completed. >>> >>> Cheers, >>> Mikael >>> >>> [1] http://cr.openjdk.java.net/~mikael/webrevs/8244224/webrev.00/all/open/webrev/ >>> > From ioi.lam at oracle.com Mon May 11 22:01:39 2020 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 11 May 2020 15:01:39 -0700 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> Message-ID: On 5/8/20 12:33 AM, Thomas St?fe wrote: > Hi Ioi, > > thanks for taking the time to look at this! > > On Fri, May 8, 2020 at 7:37 AM Ioi Lam > wrote: > > Hi Thomas, > > I am running your patch in out CI pipeline now. Some comments: > > [1] Do we still need to use _class_space_list? It looks like we > just have a single range. Is > _class_space_list->current_virtual_space() always the same VS? > > ? if (_class_space_list != NULL) { > ??? address base = > (address)_class_space_list->current_virtual_space()->bottom(); > ??? address top = base + compressed_class_space_size(); > ??? st->print("Compressed class space mapped at: " PTR_FORMAT "-" > PTR_FORMAT ", size: " SIZE_FORMAT, > ?????????????? p2i(base), p2i(top), top - base); > > > Yes it is. Has been bugging me for years too. With ccs, the > VirtualSpaceList is degenerated and only contains one node. It still > makes some sense code wise. But maybe VirtualSpace*List* is a misnomer. > > It also ties in into the question whether Metaspace reservations > should be able to grow on demand. Oracle folks I talk to tend to say > yes. I personally think the "Metaspace is infinite don't worry" meme > was broken since ccs was a thing, since CompressedClassSpaceSize is an > absolute limit already. > > I may tweak and simplify the code a bit with Elastic Metaspace. Maybe, > as you say, with ccs we can get rid of VirtualSpaceList and deal with > the range directly. Have to check. > > [2] Does JDK-8243535 exist with the current jdk/jdk repo? > > ? // This does currently not work because rs may be the result of > a split operation > ? // and NMT seems not to be able to handle splits. > ? // See JDK-8243535. > ? // MemTracker::record_virtual_memory_type((address)rs.base(), > mtClass); > > > Yes. It shows up more rarely. With this patch, it will show up always. > I had a conversation with Zhengyu about this. I think there may be > workarounds, like removing the original mapping in NMT and replacing > it with the two new ones for archive and ccs. But it seems iffy, I'd > rather have this fixed in NMT. > > I also think it is not such a big deal, but if you think otherwise, I > can integrate a workaround into this patch. My question is -- will NMT (or control-backspace) now always report incorrect memory size for the metaspace when CDS is enabled? If so, I think a work-around is necessary. > > [3] I think this can be put in header file. > > bool Metaspace::class_space_is_initialized() { > ? return _class_space_list != NULL; > } > > > Will do. > > [4] Why does the CCS need to follow UseLargePagesInMetaspace? This > is the reason for the > ??? gap in the following chart: > > ??? // We do this by reserving space for the ccs behind the > archives. Note however that > ??? //? ccs follows a different alignment > (Metaspace::reserve_alignment), so there may > ??? //? be a gap between ccs and cds. > ??? // We use a similar layout at runtime, see > reserve_address_space_for_archives(). > ??? // > ??? //????????????????????????????? +-- SharedBaseAddress (default > = 0x800000000) > ??? //????????????????????????????? v > ??? // +-..---------+---------+ ... > +----+----+----+--------+--------------------+ > ??? // |??? Heap??? | Archive |???? | MC | RW | RO | [gap]? |??? > class space???? | > ??? // +-..---------+---------+ ... > +----+----+----+--------+--------------------+ > ??? // |<--?? MaxHeapSize? -->|???? |<-- UnscaledClassSpaceMax = > 4GB -->| > > > No, nothing to do with large pages. CCS does not have large pages. > > The gap can result from different alignment rules between metaspace > and cds. With Elastic Metaspace, metaspace reservations will be > aligned to 4M. So, ccs will have to start at a 4M aligned border. > > > _reserve_alignment is determined here: > > void Metaspace::ergo_initialize() { > ? if (DumpSharedSpaces) { > ??? // Using large pages when dumping the shared archive is > currently not implemented. > ??? FLAG_SET_ERGO(UseLargePagesInMetaspace, false); > ? } > > ? size_t page_size = os::vm_page_size(); > ? if (UseLargePages && UseLargePagesInMetaspace) { > ??? page_size = os::large_page_size(); > ? } > > ? _commit_alignment? = page_size; > ? _reserve_alignment = MAX2(page_size, > (size_t)os::vm_allocation_granularity()); > > But when CDS is enabled, the RS is reserved without large pages, > so it should be possible to incrementally commit the CCS using > just os::vm_allocation_granularity(). > > > Absolutely. CCS does not use large pages for that reason. As I wrote, > this is about the new reservation granularity of 4M. OK, if I understand correctly: + With the current patch, the gap exists only when UseLargePagesInMetaspace is used at run time. The gap actually is unnecessary, since ccs doesn't use large pages. + With the upcoming Elastic Metaspace, _reserve_alignment will be 4MB, so we will almost always have a gap. > [5] I see the aarch64 code in > Metaspace::reserve_address_space_for_compressed_classes has been > changed. Is this necessary for this RFE, or is it a separate > improvement that can be done in a different RFE? > > > Well, it had to be changed somewhat, since the prototype differs. I > swapped Metaspace::reserve_preferred_space() for > Metaspace::reserve_address_space_for_compressed_classes(), which > follows a somewhat different protocol. > > That aside, the functionality for AARCH64 should be as unchanged?as > possible. I removed the AIX specific coding, since it was > subtly?wrong, but that's fine since we are AIX/ppc maintainers. > > > [6] For AARCH64, should we skip the part that reserves it right > above the heap? Or does AARCH64 always allocate the heap such that > it's a preferable address? > > > Yes and yes :) > > We do skip that for aarch64 (at least if the address is wrong): > > 1157 // Try right above the Java heap... 1158 address base = > align_up(CompressedOops::end(), Metaspace::reserve_alignment()); 1159 > assert(base != NULL, "Sanity"); 1160 1161 const size_t size = > align_up(CompressedClassSpaceSize, Metaspace::reserve_alignment()); > 1162 if (CompressedKlassPointers::is_valid_base(base)) { 1163 rs = > ReservedSpace(size, Metaspace::reserve_alignment(), false /* large */, > (char*)base); 1164 } > At 1162, we check if the attach address would be valid for this > platform. If aarch64 says no, we won't reserve there but go straight > to the "attach wherever you think is good" part. Which is implemented > by?Metaspace::reserve_address_space_for_compressed_classes(), which on > aarch64 does search a suitable address. > > ??? // case (b) > ??? ReservedSpace rs; > > ??? // Try right above the Java heap... > ??? address base = align_up(CompressedOops::end(), > Metaspace::reserve_alignment()); > ??? assert(base != NULL, "Sanity"); > > ??? const size_t size = align_up(CompressedClassSpaceSize, > Metaspace::reserve_alignment()); > ??? if (CompressedKlassPointers::is_valid_base(base)) { > ????? rs = ReservedSpace(size, Metaspace::reserve_alignment(), > false /* large */, (char*)base); > ??? } > > ??? // ...failing that, reserve anywhere, but let platform do > optimized placement: > ??? if (!rs.is_reserved()) { > ????? rs = > Metaspace::reserve_address_space_for_compressed_classes(size); > ??? } > > [7] argument.cpp checks this: > > ? if (UseSharedSpaces || DumpSharedSpaces) { > ??? if > (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) > { > ????? log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is > invalid for this platform, option will be ignored.", > ?????????????????????? p2i((address)SharedBaseAddress)); > ????? FLAG_SET_DEFAULT(SharedBaseAddress, > default_SharedBaseAddress()); > ??? } > ? } > > but initialize_dumptime_shared_and_meta_spaces aligns up the > SharedBaseAddress before doing the assert. > > ? void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { > ??? assert(DumpSharedSpaces, "should be called for dump time only"); > > ??? const size_t reserve_alignment = > MetaspaceShared::reserved_space_alignment(); > ??? char* shared_base = (char*)align_up((char*)SharedBaseAddress, > reserve_alignment); > > ? #ifdef _LP64 > assert(CompressedKlassPointers::is_valid_base((address)shared_base), > "Sanity"); > > So theoretically shared_base may no longer be is_valid_base after > the align_up. It probably won't happen, but I think the code will > be much clearer tofirst to align_up, then check for is_valid_base > and reset to default, and finally assert. > > > Okay, I'll change that. > > [8] In > MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the > following block can be simplified: > > ? if (!_shared_rs.is_reserved()) { > ??? // Get a reserved space anywhere if attaching at the > SharedBaseAddress fails: > ??? if (UseCompressedClassPointers) { > ????? // If we need a compressed class space too, let the platform > handle the reservation. > ????? _shared_rs = > Metaspace::reserve_address_space_for_compressed_classes(cds_total); > ??? } else { > ????? // anywhere is fine. > ????? _shared_rs = ReservedSpace(cds_total, reserve_alignment, > false /* large */, (char*)NULL); > ??? } > ? } > > ?? ... if you change the declaration to: > > ? static ReservedSpace > reserve_address_space_for_compressed_classes(size_t size) > NOT_LP64({ return ReservedSpace();}); > > [9] I think the #ifdef _LP64 is not necessary: > > #ifdef _LP64 > ??? // Some sanity checks after reserving address spaces for > archives and class space. > ??? assert(archive_space_rs.is_reserved(), "Sanity"); > ??? if (Metaspace::using_class_space()) { > ????? // Class space must closely follow the archive space. Both > spaces must be aligned correctly. > ????? assert(class_space_rs.is_reserved(), "A class space should > have been reserved"); > ????? assert(class_space_rs.base() >= archive_space_rs.end(), > "class space should follow the cds archive space"); > ????? assert(is_aligned(archive_space_rs.base(), > MetaspaceShared::reserved_space_alignment()), "Archive space > misaligned"); > ????? assert(is_aligned(class_space_rs.base(), > Metaspace::reserve_alignment()), "class space misaligned"); > ??? } > #endif > > > I'll take a look. > > The rest of the code looks OK to me, but I may take a look at it > again after getting more sleep :-) > > > Sure. Thanks man! :) Good news. I ran our tiers 1-4 and there were no failures related to your changes! Thanks - Ioi > > ..Thomas > > Thanks > - Ioi > > > > > On 5/7/20 7:21 AM, Thomas St?fe wrote: >> Hi all, >> >> please take a look at the third iteration of this change: >> >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >> >> Changes in this version: >> >> - at the request of Coleen (sorry I did not see your post >> earlier) I removed all platform dependent?files and put the >> platform dependent coding back to metaspace.cpp and >> compressedOops.cpp to ease the review pain. I used plain platform >> defines though (#ifdef AARCH64) instead of hiding them behind >> another macro to make things clearer. Note that I still intent to >> put this code away into the platform corners but will do so in a >> follow up RFE. >> >> - I reinstated, in a fashion, the special handling of >> reservations on Windows. On all platforms we reserve address >> space to map the archive files in with a subsequent mapping >> operation. However, on Windows, we cannot use MapViewOfFile() >> into an existing mapping. So I remove the mapping again before >> mapping the archives - see comment in code for details. >> >> All CI tests at SAP run through without problems, including on >> Win64 and aarch64, but I would be happy if others were to run >> test too. >> >> Thank you, Thomas >> >> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >> > wrote: >> >> Hi all, >> >> Could I have reviews for the following proposal of reworking >> cds/class space reservation? >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >> >> Webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >> >> (Many thanks to Ioi Lam for so patiently explaining CDS >> internals to me, and to Andrew Haley and Nick Gasson for help >> with aarch64!) >> >> Reservation of the compressed class space is needlessly >> complicated and has some minor issues. It can be simplified >> and made clearer. >> >> The complexity stems from the fact that this area lives at >> the intersection of two to three sub systems, depending on >> how one counts. Metaspace, CDS, and the platform which may or >> may not its own view of how to reserve class space. And this >> code has been growing organically over time. >> >> One small example: >> >> ReservedSpace Metaspace::reserve_preferred_space(size_t size, >> size_t alignment, >> ?bool large_pages, char *requested_addr, >> ?bool use_requested_addr) >> >> which I spent hours decoding, resulting in a very confused >> mail to hs-runtime and aarch64-port-dev [2]. >> >> This patch attempts to simplify cds and metaspace setup a >> bit; to comment implicit knowledge which is not immediately >> clear; to cleanly abstract platform concerns like optimized >> class space placement; and to disentangle cds from metaspace >> to solve issues which may bite us later with Elastic >> Metaspace [4]. >> >> --- >> >> The main change is the reworked reservation mechanism. This >> is based on Ioi's proposal [5]. >> >> When reserving class space, three things must happen: >> >> 1) reservation of the space obviously. If cds is active that >> space must be in the vicinity of cds archives to be covered >> by compressed class pointer encoding. >> 2) setting up the internal Metaspace structures atop of that >> space >> 3) setting up compressed class pointer encoding. >> >> In its current form, Metaspace may or may not do some or all >> of that in one function >> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base);) - if >> cds is active, it will reserve the space for Metaspace and >> hand it in, otherwise it will create it itself. >> >> When discussing this in [2], Ioi proposed to move the >> reservation of the class space completely out of Metaspace >> and make it a responsibility of the caller always. This would >> reduce some complexity, and this patch follows the proposal. >> >> I removed >> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base); and >> all its sub functions. >> >> (1) now has to always be done outside - a ReservedSpace for >> class space has to be provided by the caller. However, >> Metaspace now offers a utility function for reserving space >> at a "nice" location, and explicitly doing nothing else: >> >> ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t >> size); >> >> this function can be redefined on a platform level for >> platform optimized reservation, see below for details. >> >> (2) is taken care of by a new function, >> Metaspace::initialize_class_space(ReservedSpace rs) >> >> (3) is taken care of a new function >> CompressedKlassPointers::initialize(), see below for details. >> >> >> So, class space now is set up three explicit steps: >> >> - First, reserve a suitable space by however means you want. >> For convenience you may use >> Metaspace::reserve_address_space_for_compressed_classes(), or >> you may roll your own reservation. >> - Next, tell Metaspace to use that range as backing storage >> for class space: >> Metaspace::initialize_class_space(ReservedSpace rs) >> - Finally, set up encoding. Encoding is independent from the >> concept of a ReservedSpace, it just gets an address range, >> see below for details. >> >> Separating these steps and moving them out of the >> responsibility of Metaspace makes this whole thing more >> flexible; it also removes unnecessary knowledge (e.g. >> Metaspace does not need to know anything about either ccp >> encoding or cds). >> >> --- >> >> How it comes together: >> >> If CDS is off, we just reserve a space using >> Metaspace::reserve_address_space_for_compressed_classes(), >> initialize it with >> Metaspace::initialize_class_space(ReservedSpace rs), then set >> up compressed class pointer encoding covering the range of >> this class space. >> >> If CDS is on (dump time), we reserve large 4G space, either >> at SharedBaseAddress or using >> Metaspace::reserve_address_space_for_compressed_classes(); we >> then split that into 3G archive space and 1G class space; we >> set up that space with Metaspace as class space; then we set >> up?compressed class pointer encoding covering both archive >> space and cds. >> >> If CDS is on (run time), we reserve a large space, split it >> into archive space (large enough to hold both archives) and >> class space, then basically proceed as above. >> >> Note that this is almost exactly how things worked before >> (modulo some minor fixes, e.g. alignment issues), only the >> code is reformed and made more explicit. >> >> --- >> >> I moved compressed class pointer setup over to >> CompressedKlassPointers and changed the interface: >> >> -void >> Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >> metaspace_rs, address cds_base) >> +void CompressedKlassPointers::initialize(address addr, >> size_t len); >> >> Instead of feeding it a single ReservedSpace, which is >> supposed to represent class space, and an optional alternate >> base if cds is on, now we give it just an numeric address >> range. That range marks the limits to where Klass structures >> are to be expected, and is the implicit promise that outside >> that range no Klass structures will exist, so encoding has to >> cover only this range. >> >> This range may contain just the class space; or class >> space+cds; or whatever allocation scheme we come up with in >> the future. Encoding does not really care how the memory is >> organized as long as the input range covers all possible >> Klass locations. That way we remove knowledge about class >> space/cds from compressed class pointer encoding. >> >> Moving it away from metaspace.cpp into the >> CompressedKlassPointers class also mirrors >> CompressedOops::initialize(). >> >> --- >> >> I renamed _narrow_klass_range to just _range, because >> strictly speaking this is the range un-narrow Klass pointers >> can have. >> >> As for the implementation of >> CompressedKlassPointers::initialize(address addr, size_t >> len), I mimicked very closely what happened before, so there >> should be almost no differences. Since "almost no >> differences" sounds scary :) here are the differences: >> >> - When CDS is active (dump or run time) we now always, >> unconditionally, set the encoding range to 4G. This fixes a >> theoretical bug discussed on aarch64-port-dev [1]. >> >> - When CDS is not active, we set the encoding range to the >> minimum required length. Before, it was left at its default >> value of 4G. >> >> Both differences only affect aarch64, since they are >> currently the only one using the range field in >> CompressedKlassPointers. >> >> I wanted to add an assert somewhere to test encoding of the >> very last address of the CompressedKlassPointers range, again >> to prevent errors like [3]. But I did not come up with a good >> place for this assert which would cover also the encoding >> done by C1/C2. >> >> For the same reason I thought about introducing a mode where >> Klass structures would be allocated in reverse order, >> starting at the end of the ccs, but again left it out as too >> big a change. >> >> --- >> >> OS abstraction: platforms may have restrictions of what >> constitutes a valid compressed class pointer encoding base. >> Or if not, they may have at least preferences. There was >> logic like this in metaspace.cpp, which I removed and cleanly >> factored out into platform dependent files, giving each >> platform the option to add special logic. >> >> These are two new methods: >> >> - bool CompressedKlassPointers::is_valid_base(address p) >> >> to let the platform tell you whether it considers p to be a >> valid encoding base. The only platform having these >> restrictions currently is aarch64. >> >> - ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t >> size); >> >> this hands over the process of allocating a range suitable >> for compressed class pointer encoding to the platform. Most >> platforms will allocate just anywhere, but some platforms may >> have a better strategy (e.g. trying low memory first, trying >> only correctly aligned addresses and so on). >> >> Beforehand, this coding existed in a similar form in >> metaspace.cpp for aarch64 and AIX. For now, I left the AIX >> part out - it seems only half done, and I want to check >> further if we even need it, if yes why not on Linux ppc, and >> C1 does not seem to support anything other than base+offset >> with shift either, but I may be mistaken. >> >> These two methods should give the platform enough control to >> implement their own scheme for optimized class space >> placement without bothering any shared code about it. >> >> Note about the form, I introduced two new platform dependent >> files, "metaspace_.cpp" and "compressedOops_.cpp". >> I am not happy about this but this seems to be what we >> generally do in hotspot, right? >> >> --- >> >> Metaspace reserve alignment vs cds alignment >> >> CDS was using Metaspace reserve alignment for CDS internal >> purposes. I guess this was just a copy paste issue. It never >> caused problems since Metaspace reserve alignment == page >> size, but that is not true anymore in the upcoming Elastic >> Metaspace where reserve alignment will be larger. This causes >> a number of issues. >> >> I separated those two cleanly. CDS now uses >> os::vm_allocation_granularity. Metaspace::reserve_alignment >> is only used in those two places where it is needed, when CDS >> creates the address space for class space on behalf of the >> Metaspace. >> >> --- >> >> Windows special handling in CDS >> >> To simplify coding I removed the windows specific handling >> which left out reservation of the archive. This was needed >> because windows cannot mmap files into reserved regions. But >> fallback code exists in filemap.cpp for this case which just >> reads in the region instead of mapping?it. >> >> Should that turn out to be a performance problem, I will >> reinstate the feature. But a simpler way would be reserve the >> archive and later just before mmapping?the archive file to >> release the archive space. That would not only be simpler but >> give us the best guarantee that that address space is >> actually available. But I'd be happy to leave that part out >> completely if we do not see any performance problems on >> windows x64. >> >> --- >> >> NMT cannot deal with spaces which are split. This problem >> manifests in that bookkeeping for class space is done under >> "Shared Classes", not "Classes" as it should. This problem >> exists today too at dump?time and randomly at run time. But >> since I simplified the reservation, this problem now shows up >> always, whether or not we map at the SharedBaseAddress. >> While I could work around this problem, I'd prefer this >> problem to be solved at the core, and NMT to have an option >> to recognize reservation splits. So I'd rather not put a >> workaround for this into the patch but leave it for fixing as >> a separate issue. I opened this issue to track it [6]. >> >> --- >> >> Jtreg tests: >> >> I expanded the CompressedOops/CompressedClassPointers.java. I >> also extended them to Windows. The tests now optionally omit >> strict class space placement tests, since these tests heavily >> depend on ASLR and were the reason they were excluded on >> Windows. However I think even without checking for class >> space placement they make sense, just to see that the VM >> comes up and lives with the many different settings we can >> run in. >> >> --- >> >> Tests: >> >> - I ran the patch through Oracles submit repo >> - I ran tests manually for aarch64, zero, linux 32bit and >> windows x64 >> - The whole battery of nightly tests at SAP, including ppc, >> ppcle and aarch64, unfortunately excluding windows because of >> unrelated errors. Windows x64 tests will be redone tonight. >> >> >> Thank you, >> >> Thomas >> >> [1] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >> [2] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >> [5] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >> > From martin.doerr at sap.com Tue May 12 08:42:31 2020 From: martin.doerr at sap.com (Doerr, Martin) Date: Tue, 12 May 2020 08:42:31 +0000 Subject: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant In-Reply-To: References: <3c59b9f9-ec38-18c9-8f24-e1186a08a04a@oracle.com> <410eed04-e2ef-0f4f-1c56-19e6734a10f6@oracle.com> <81d7caa8-4244-85f3-4d4e-78117fe5e25b@oss.nttdata.com> <550b95ac-8b29-1eb8-a507-533e81d02322@oracle.com> <9c49ea2d-e3b8-b576-1d17-d18ad87cd6ed@oracle.com> Message-ID: Hi Richard, I had already reviewed webrev.1. Looks good to me. Thanks for contributing it. Best regards, Martin > -----Original Message----- > From: hotspot-runtime-dev bounces at openjdk.java.net> On Behalf Of Reingruber, Richard > Sent: Montag, 4. Mai 2020 12:33 > To: David Holmes ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > Subject: RE: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > > // Trimmed the list of recipients. If the list gets too long then the message > needs to be approved > // by a moderator. > > Hi David, > > > On 28/04/2020 12:09 am, Reingruber, Richard wrote: > > > Hi David, > > > > > >> Not a review but some general commentary ... > > > > > > That's welcome. > > > Having had to take an even closer look now I have a review comment too :) > > > src/hotspot/share/prims/jvmtiThreadState.cpp > > > void JvmtiThreadState::invalidate_cur_stack_depth() { > > ! assert(SafepointSynchronize::is_at_safepoint() || > > ! (Thread::current()->is_VM_thread() && > > get_thread()->is_vmthread_processing_handshake()) || > > (JavaThread *)Thread::current() == get_thread(), > > "must be current thread or at safepoint"); > > You're looking at an outdated webrev, I'm afraid. > > This would be the post with the current webrev.1 > > http://mail.openjdk.java.net/pipermail/serviceability-dev/2020- > April/031245.html > > Thanks, Richard. > > -----Original Message----- > From: David Holmes > Sent: Montag, 4. Mai 2020 08:51 > To: Reingruber, Richard ; Yasumasa Suenaga > ; Patricio Chilano > ; serguei.spitsyn at oracle.com; Vladimir > Ivanov ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > > Hi Richard, > > On 28/04/2020 12:09 am, Reingruber, Richard wrote: > > Hi David, > > > >> Not a review but some general commentary ... > > > > That's welcome. > > Having had to take an even closer look now I have a review comment too :) > > src/hotspot/share/prims/jvmtiThreadState.cpp > > void JvmtiThreadState::invalidate_cur_stack_depth() { > ! assert(SafepointSynchronize::is_at_safepoint() || > ! (Thread::current()->is_VM_thread() && > get_thread()->is_vmthread_processing_handshake()) || > (JavaThread *)Thread::current() == get_thread(), > "must be current thread or at safepoint"); > > The message needs updating to include handshakes. > > More below ... > > >> On 25/04/2020 2:08 am, Reingruber, Richard wrote: > >>> Hi Yasumasa, Patricio, > >>> > >>>>>> I will send review request to replace VM_SetFramePop to > handshake in early next week in JDK-8242427. > >>>>>> Does it help you? I think it gives you to remove workaround. > >>>>> > >>>>> I think it would not help that much. Note that when replacing > VM_SetFramePop with a direct handshake > >>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm > operation [1]. So you would have to > >>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt > to these changes. > >>> > >>>> Thanks for your information. > >>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and > vmTestbase/nsk/jvmti/NotifyFramePop. > >>>> I will modify and will test it after yours. > >>> > >>> Thanks :) > >>> > >>>>> Also my first impression was that it won't be that easy from a > synchronization point of view to > >>>>> replace VM_SetFramePop with a direct handshake. E.g. > VM_SetFramePop::doit() indirectly calls > >>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, > JvmtiFramePop fpop) where > >>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at > safepoint. It's not directly clear > >>>>> to me, how this has to be handled. > >>> > >>>> I think JvmtiEventController::set_frame_pop() should hold > JvmtiThreadState_lock because it affects other JVMTI operation especially > FramePop event. > >>> > >>> Yes. To me it is unclear what synchronization is necessary, if it is called > during a handshake. And > >>> also I'm unsure if a thread should do safepoint checks while executing a > handshake. > > > >> I'm growing increasingly concerned that use of direct handshakes to > >> replace VM operations needs a much greater examination for correctness > >> than might initially be thought. I see a number of issues: > > > > I agree. I'll address your concerns in the context of this review thread for > JDK-8238585 below. > > > > In addition I would suggest to take the general part of the discussion to a > dedicated thread or to > > the review thread for JDK-8242427. I would like to keep this thread closer > to its subject. > > I will focus on the issues in the context of this particular change > then, though the issues themselves are applicable to all handshake > situations (and more so with direct handshakes). This is mostly just > discussion. > > >> First, the VMThread executes (most) VM operations with a clean stack in > >> a clean state, so it has lots of room to work. If we now execute the > >> same logic in a JavaThread then we risk hitting stackoverflows if > >> nothing else. But we are also now executing code in a JavaThread and so > >> we have to be sure that code is not going to act differently (in a bad > >> way) if executed by a JavaThread rather than the VMThread. For > example, > >> may it be possible that if executing in the VMThread we defer some > >> activity that might require execution of Java code, or else hand it off > >> to one of the service threads? If we execute that code directly in the > >> current JavaThread instead we may not be in a valid state (e.g. consider > >> re-entrancy to various subsystems that is not allowed). > > > > It is not too complex, what EnterInterpOnlyModeClosure::do_thread() is > doing. I already added a > > paragraph to the JBS-Item [1] explaining why the direct handshake is > sufficient from a > > synchronization point of view. > > Just to be clear, your proposed change is not using a direct handshake. > > > Furthermore the stack is walked and the return pc of compiled frames is > replaced with the address of > > the deopt handler. > > > > I can't see why this cannot be done with a direct handshake. Something > very similar is already done > > in JavaThread::deoptimize_marked_methods() which is executed as part > of an ordinary handshake. > > Note that existing non-direct handshakes may also have issues that not > have been fully investigated. > > > The demand on stack-space should be very modest. I would not expect a > higher risk for stackoverflow. > > For the target thread if you use more stack than would be used stopping > at a safepoint then you are at risk. For the thread initiating the > direct handshake if you use more stack than would be used enqueuing a VM > operation, then you are at risk. As we have not quantified these > numbers, nor have any easy way to establish the stack use of the actual > code to be executed, we're really just hoping for the best. This is a > general problem with handshakes that needs to be investigated more > deeply. As a simple, general, example just imagine if the code involves > logging that might utilise an on-stack buffer. > > >> Second, we have this question mark over what happens if the operation > >> hits further safepoint or handshake polls/checks? Are there constraints > >> on what is allowed here? How can we recognise this problem may exist > and > >> so deal with it? > > > > The thread in EnterInterpOnlyModeClosure::do_thread() can't become > safepoint/handshake safe. I > > tested locally test/hotspot/jtreg:vmTestbase_nsk_jvmti with a > NoSafepointVerifier. > > That's good to hear but such tests are not exhaustive, they will detect > if you do reach a safepoint/handshake but they can't prove that you > cannot reach one. What you have done is necessary but may not be > sufficient. Plus you didn't actually add the NSV to the code - is there > a reason we can't actually keep it in do_thread? (I'm not sure if the > NSV also acts as a NoHandshakeVerifier?) > > >> Third, while we are generally considering what appear to be > >> single-thread operations, which should be amenable to a direct > >> handshake, we also have to be careful that some of the code involved > >> doesn't already expect/assume we are at a safepoint - e.g. a VM op may > >> not need to take a lock where a direct handshake might! > > > > See again my arguments in the JBS item [1]. > > Yes I see the reasoning and that is good. My point is a general one as > it may not be obvious when such assumptions exist in the current code. > > Thanks, > David > > > Thanks, > > Richard. > > > > [1] https://bugs.openjdk.java.net/browse/JDK-8238585 > > > > -----Original Message----- > > From: David Holmes > > Sent: Montag, 27. April 2020 07:16 > > To: Reingruber, Richard ; Yasumasa Suenaga > ; Patricio Chilano > ; serguei.spitsyn at oracle.com; Vladimir > Ivanov ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > > Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > > > > Hi all, > > > > Not a review but some general commentary ... > > > > On 25/04/2020 2:08 am, Reingruber, Richard wrote: > >> Hi Yasumasa, Patricio, > >> > >>>>> I will send review request to replace VM_SetFramePop to handshake > in early next week in JDK-8242427. > >>>>> Does it help you? I think it gives you to remove workaround. > >>>> > >>>> I think it would not help that much. Note that when replacing > VM_SetFramePop with a direct handshake > >>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm > operation [1]. So you would have to > >>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt > to these changes. > >> > >>> Thanks for your information. > >>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and > vmTestbase/nsk/jvmti/NotifyFramePop. > >>> I will modify and will test it after yours. > >> > >> Thanks :) > >> > >>>> Also my first impression was that it won't be that easy from a > synchronization point of view to > >>>> replace VM_SetFramePop with a direct handshake. E.g. > VM_SetFramePop::doit() indirectly calls > >>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, > JvmtiFramePop fpop) where > >>>> JvmtiThreadState_lock is acquired with safepoint check, if not at > safepoint. It's not directly clear > >>>> to me, how this has to be handled. > >> > >>> I think JvmtiEventController::set_frame_pop() should hold > JvmtiThreadState_lock because it affects other JVMTI operation especially > FramePop event. > >> > >> Yes. To me it is unclear what synchronization is necessary, if it is called > during a handshake. And > >> also I'm unsure if a thread should do safepoint checks while executing a > handshake. > > > > I'm growing increasingly concerned that use of direct handshakes to > > replace VM operations needs a much greater examination for correctness > > than might initially be thought. I see a number of issues: > > > > First, the VMThread executes (most) VM operations with a clean stack in > > a clean state, so it has lots of room to work. If we now execute the > > same logic in a JavaThread then we risk hitting stackoverflows if > > nothing else. But we are also now executing code in a JavaThread and so > > we have to be sure that code is not going to act differently (in a bad > > way) if executed by a JavaThread rather than the VMThread. For example, > > may it be possible that if executing in the VMThread we defer some > > activity that might require execution of Java code, or else hand it off > > to one of the service threads? If we execute that code directly in the > > current JavaThread instead we may not be in a valid state (e.g. consider > > re-entrancy to various subsystems that is not allowed). > > > > Second, we have this question mark over what happens if the operation > > hits further safepoint or handshake polls/checks? Are there constraints > > on what is allowed here? How can we recognise this problem may exist and > > so deal with it? > > > > Third, while we are generally considering what appear to be > > single-thread operations, which should be amenable to a direct > > handshake, we also have to be careful that some of the code involved > > doesn't already expect/assume we are at a safepoint - e.g. a VM op may > > not need to take a lock where a direct handshake might! > > > > Cheers, > > David > > ----- > > > >> @Patricio, coming back to my question [1]: > >> > >> In the example you gave in your answer [2]: the java thread would > execute a vm operation during a > >> direct handshake operation, while the VMThread is actually in the middle > of a VM_HandshakeAllThreads > >> operation, waiting to handshake the same handshakee: why can't the > VMThread just proceed? The > >> handshakee would be safepoint safe, wouldn't it? > >> > >> Thanks, Richard. > >> > >> [1] https://bugs.openjdk.java.net/browse/JDK- > 8230594?focusedCommentId=14301677&page=com.atlassian.jira.plugin.syst > em.issuetabpanels:comment-tabpanel#comment-14301677 > >> > >> [2] https://bugs.openjdk.java.net/browse/JDK- > 8230594?focusedCommentId=14301763&page=com.atlassian.jira.plugin.syst > em.issuetabpanels:comment-tabpanel#comment-14301763 > >> > >> -----Original Message----- > >> From: Yasumasa Suenaga > >> Sent: Freitag, 24. April 2020 17:23 > >> To: Reingruber, Richard ; Patricio Chilano > ; serguei.spitsyn at oracle.com; Vladimir > Ivanov ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > >> Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > >> > >> Hi Richard, > >> > >> On 2020/04/24 23:44, Reingruber, Richard wrote: > >>> Hi Yasumasa, > >>> > >>>> I will send review request to replace VM_SetFramePop to handshake > in early next week in JDK-8242427. > >>>> Does it help you? I think it gives you to remove workaround. > >>> > >>> I think it would not help that much. Note that when replacing > VM_SetFramePop with a direct handshake > >>> you could not just execute VM_EnterInterpOnlyMode as a nested vm > operation [1]. So you would have to > >>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to > these changes. > >> > >> Thanks for your information. > >> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and > vmTestbase/nsk/jvmti/NotifyFramePop. > >> I will modify and will test it after yours. > >> > >> > >>> Also my first impression was that it won't be that easy from a > synchronization point of view to > >>> replace VM_SetFramePop with a direct handshake. E.g. > VM_SetFramePop::doit() indirectly calls > >>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, > JvmtiFramePop fpop) where > >>> JvmtiThreadState_lock is acquired with safepoint check, if not at > safepoint. It's not directly clear > >>> to me, how this has to be handled. > >> > >> I think JvmtiEventController::set_frame_pop() should hold > JvmtiThreadState_lock because it affects other JVMTI operation especially > FramePop event. > >> > >> > >> Thanks, > >> > >> Yasumasa > >> > >> > >>> So it appears to me that it would be easier to push JDK-8242427 after this > (JDK-8238585). > >>> > >>>> (The patch is available, but I want to see the result of PIT in this > weekend whether JDK-8242425 works fine.) > >>> > >>> Would be interesting to see how you handled the issues above :) > >>> > >>> Thanks, Richard. > >>> > >>> [1] See question in comment > https://bugs.openjdk.java.net/browse/JDK- > 8230594?focusedCommentId=14302030&page=com.atlassian.jira.plugin.syst > em.issuetabpanels:comment-tabpanel#comment-14302030 > >>> > >>> -----Original Message----- > >>> From: Yasumasa Suenaga > >>> Sent: Freitag, 24. April 2020 13:34 > >>> To: Reingruber, Richard ; Patricio Chilano > ; serguei.spitsyn at oracle.com; Vladimir > Ivanov ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > >>> Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > >>> > >>> Hi Richard, > >>> > >>> I will send review request to replace VM_SetFramePop to handshake in > early next week in JDK-8242427. > >>> Does it help you? I think it gives you to remove workaround. > >>> > >>> (The patch is available, but I want to see the result of PIT in this weekend > whether JDK-8242425 works fine.) > >>> > >>> > >>> Thanks, > >>> > >>> Yasumasa > >>> > >>> > >>> On 2020/04/24 17:18, Reingruber, Richard wrote: > >>>> Hi Patricio, Vladimir, and Serguei, > >>>> > >>>> now that direct handshakes are available, I've updated the patch to > make use of them. > >>>> > >>>> In addition I have done some clean-up changes I missed in the first > webrev. > >>>> > >>>> Finally I have implemented the workaround suggested by Patricio to > avoid nesting the handshake > >>>> into the vm operation VM_SetFramePop [1] > >>>> > >>>> Kindly review again: > >>>> > >>>> Webrev: > http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1/ > >>>> Webrev(delta): > http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1.inc/ > >>>> > >>>> I updated the JBS item explaining why the vm operation > VM_EnterInterpOnlyMode can be replaced with a > >>>> direct handshake: > >>>> > >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8238585 > >>>> > >>>> Testing: > >>>> > >>>> * JCK and JTREG tests, also in Xcomp mode with fastdebug and release > builds on all platforms. > >>>> > >>>> * Submit-repo: mach5-one-rrich-JDK-8238585-20200423-1436-10441737 > >>>> > >>>> Thanks, > >>>> Richard. > >>>> > >>>> [1] An assertion in Handshake::execute_direct() fails, if called be > VMThread, because it is no JavaThread. > >>>> > >>>> -----Original Message----- > >>>> From: hotspot-dev On > Behalf Of Reingruber, Richard > >>>> Sent: Freitag, 14. Februar 2020 19:47 > >>>> To: Patricio Chilano ; > serviceability-dev at openjdk.java.net; hotspot-compiler- > dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime- > dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net > >>>> Subject: RE: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > >>>> > >>>> Hi Patricio, > >>>> > >>>> > > I'm really glad you noticed the problematic nesting. This seems > to be a general issue: currently a > >>>> > > handshake cannot be nested in a vm operation. Maybe it should > be asserted in the > >>>> > > Handshake::execute() methods that they are not called by the > vm thread evaluating a vm operation? > >>>> > > > >>>> > > > Alternatively I think you could do something similar to what > we do in > >>>> > > > Deoptimization::deoptimize_all_marked(): > >>>> > > > > >>>> > > > EnterInterpOnlyModeClosure hs; > >>>> > > > if (SafepointSynchronize::is_at_safepoint()) { > >>>> > > > hs.do_thread(state->get_thread()); > >>>> > > > } else { > >>>> > > > Handshake::execute(&hs, state->get_thread()); > >>>> > > > } > >>>> > > > (you could pass ?EnterInterpOnlyModeClosure? directly to > the > >>>> > > > HandshakeClosure() constructor) > >>>> > > > >>>> > > Maybe this could be used also in the Handshake::execute() > methods as general solution? > >>>> > Right, we could also do that. Avoiding to clear the polling page in > >>>> > HandshakeState::clear_handshake() should be enough to fix this > issue and > >>>> > execute a handshake inside a safepoint, but adding that "if" > statement > >>>> > in Hanshake::execute() sounds good to avoid all the extra code > that we > >>>> > go through when executing a handshake. I filed 8239084 to make > that change. > >>>> > >>>> Thanks for taking care of this and creating the RFE. > >>>> > >>>> > > >>>> > > > I don?t know JVMTI code so I?m not sure if > VM_EnterInterpOnlyMode is > >>>> > > > always called in a nested operation or just sometimes. > >>>> > > > >>>> > > At least one execution path without vm operation exists: > >>>> > > > >>>> > > > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) > : void > >>>> > > > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState > *) : jlong > >>>> > > JvmtiEventControllerPrivate::recompute_enabled() : void > >>>> > > > JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 > matches) > >>>> > > JvmtiEventController::change_field_watch(jvmtiEvent, > bool) : void > >>>> > > JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : > jvmtiError > >>>> > > jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : > jvmtiError > >>>> > > > >>>> > > I tend to revert back to VM_EnterInterpOnlyMode as it wasn't > my main intent to replace it with a > >>>> > > handshake, but to avoid making the compiled methods on stack > not_entrant.... unless I'm further > >>>> > > encouraged to do it with a handshake :) > >>>> > Ah! I think you can still do it with a handshake with the > >>>> > Deoptimization::deoptimize_all_marked() like solution. I can > change the > >>>> > if-else statement with just the Handshake::execute() call in > 8239084. > >>>> > But up to you. : ) > >>>> > >>>> Well, I think that's enough encouragement :) > >>>> I'll wait for 8239084 and try then again. > >>>> (no urgency and all) > >>>> > >>>> Thanks, > >>>> Richard. > >>>> > >>>> -----Original Message----- > >>>> From: Patricio Chilano > >>>> Sent: Freitag, 14. Februar 2020 15:54 > >>>> To: Reingruber, Richard ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > >>>> Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > >>>> > >>>> Hi Richard, > >>>> > >>>> On 2/14/20 9:58 AM, Reingruber, Richard wrote: > >>>>> Hi Patricio, > >>>>> > >>>>> thanks for having a look. > >>>>> > >>>>> > I?m only commenting on the handshake changes. > >>>>> > I see that operation VM_EnterInterpOnlyMode can be called > inside > >>>>> > operation VM_SetFramePop which also allows nested > operations. Here is a > >>>>> > comment in VM_SetFramePop definition: > >>>>> > > >>>>> > // Nested operation must be allowed for the > VM_EnterInterpOnlyMode that is > >>>>> > // called from the > JvmtiEventControllerPrivate::recompute_thread_enabled. > >>>>> > > >>>>> > So if we change VM_EnterInterpOnlyMode to be a handshake, > then now we > >>>>> > could have a handshake inside a safepoint operation. The issue I > see > >>>>> > there is that at the end of the handshake the polling page of the > target > >>>>> > thread could be disarmed. So if the target thread happens to be > in a > >>>>> > blocked state just transiently and wakes up then it will not stop > for > >>>>> > the ongoing safepoint. Maybe I can file an RFE to assert that the > >>>>> > polling page is armed at the beginning of disarm_safepoint(). > >>>>> > >>>>> I'm really glad you noticed the problematic nesting. This seems to be a > general issue: currently a > >>>>> handshake cannot be nested in a vm operation. Maybe it should be > asserted in the > >>>>> Handshake::execute() methods that they are not called by the vm > thread evaluating a vm operation? > >>>>> > >>>>> > Alternatively I think you could do something similar to what we > do in > >>>>> > Deoptimization::deoptimize_all_marked(): > >>>>> > > >>>>> > EnterInterpOnlyModeClosure hs; > >>>>> > if (SafepointSynchronize::is_at_safepoint()) { > >>>>> > hs.do_thread(state->get_thread()); > >>>>> > } else { > >>>>> > Handshake::execute(&hs, state->get_thread()); > >>>>> > } > >>>>> > (you could pass ?EnterInterpOnlyModeClosure? directly to the > >>>>> > HandshakeClosure() constructor) > >>>>> > >>>>> Maybe this could be used also in the Handshake::execute() methods > as general solution? > >>>> Right, we could also do that. Avoiding to clear the polling page in > >>>> HandshakeState::clear_handshake() should be enough to fix this issue > and > >>>> execute a handshake inside a safepoint, but adding that "if" statement > >>>> in Hanshake::execute() sounds good to avoid all the extra code that we > >>>> go through when executing a handshake. I filed 8239084 to make that > change. > >>>> > >>>>> > I don?t know JVMTI code so I?m not sure if > VM_EnterInterpOnlyMode is > >>>>> > always called in a nested operation or just sometimes. > >>>>> > >>>>> At least one execution path without vm operation exists: > >>>>> > >>>>> > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) > : void > >>>>> > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState > *) : jlong > >>>>> JvmtiEventControllerPrivate::recompute_enabled() : void > >>>>> JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, > bool) : void (2 matches) > >>>>> JvmtiEventController::change_field_watch(jvmtiEvent, bool) : > void > >>>>> JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError > >>>>> jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : > jvmtiError > >>>>> > >>>>> I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my > main intent to replace it with a > >>>>> handshake, but to avoid making the compiled methods on stack > not_entrant.... unless I'm further > >>>>> encouraged to do it with a handshake :) > >>>> Ah! I think you can still do it with a handshake with the > >>>> Deoptimization::deoptimize_all_marked() like solution. I can change > the > >>>> if-else statement with just the Handshake::execute() call in 8239084. > >>>> But up to you.? : ) > >>>> > >>>> Thanks, > >>>> Patricio > >>>>> Thanks again, > >>>>> Richard. > >>>>> > >>>>> -----Original Message----- > >>>>> From: Patricio Chilano > >>>>> Sent: Donnerstag, 13. Februar 2020 18:47 > >>>>> To: Reingruber, Richard ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > >>>>> Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > >>>>> > >>>>> Hi Richard, > >>>>> > >>>>> I?m only commenting on the handshake changes. > >>>>> I see that operation VM_EnterInterpOnlyMode can be called inside > >>>>> operation VM_SetFramePop which also allows nested operations. > Here is a > >>>>> comment in VM_SetFramePop definition: > >>>>> > >>>>> // Nested operation must be allowed for the > VM_EnterInterpOnlyMode that is > >>>>> // called from the > JvmtiEventControllerPrivate::recompute_thread_enabled. > >>>>> > >>>>> So if we change VM_EnterInterpOnlyMode to be a handshake, then > now we > >>>>> could have a handshake inside a safepoint operation. The issue I see > >>>>> there is that at the end of the handshake the polling page of the > target > >>>>> thread could be disarmed. So if the target thread happens to be in a > >>>>> blocked state just transiently and wakes up then it will not stop for > >>>>> the ongoing safepoint. Maybe I can file an RFE to assert that the > >>>>> polling page is armed at the beginning of disarm_safepoint(). > >>>>> > >>>>> I think one option could be to remove > >>>>> SafepointMechanism::disarm_if_needed() in > >>>>> HandshakeState::clear_handshake() and let each JavaThread disarm > itself > >>>>> for the handshake case. > >>>>> > >>>>> Alternatively I think you could do something similar to what we do in > >>>>> Deoptimization::deoptimize_all_marked(): > >>>>> > >>>>> ? EnterInterpOnlyModeClosure hs; > >>>>> ? if (SafepointSynchronize::is_at_safepoint()) { > >>>>> ??? hs.do_thread(state->get_thread()); > >>>>> ? } else { > >>>>> ??? Handshake::execute(&hs, state->get_thread()); > >>>>> ? } > >>>>> (you could pass ?EnterInterpOnlyModeClosure? directly to the > >>>>> HandshakeClosure() constructor) > >>>>> > >>>>> I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode > is > >>>>> always called in a nested operation or just sometimes. > >>>>> > >>>>> Thanks, > >>>>> Patricio > >>>>> > >>>>> On 2/12/20 7:23 AM, Reingruber, Richard wrote: > >>>>>> // Repost including hotspot runtime and gc lists. > >>>>>> // Dean Long suggested to do so, because the enhancement > replaces a vm operation > >>>>>> // with a handshake. > >>>>>> // Original thread: > http://mail.openjdk.java.net/pipermail/serviceability-dev/2020- > February/030359.html > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> could I please get reviews for this small enhancement in hotspot's > jvmti implementation: > >>>>>> > >>>>>> Webrev: > http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.0/ > >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8238585 > >>>>>> > >>>>>> The change avoids making all compiled methods on stack > not_entrant when switching a java thread to > >>>>>> interpreter only execution for jvmti purposes. It is sufficient to > deoptimize the compiled frames on stack. > >>>>>> > >>>>>> Additionally a handshake is used instead of a vm operation to walk > the stack and do the deoptimizations. > >>>>>> > >>>>>> Testing: JCK and JTREG tests, also in Xcomp mode with fastdebug > and release builds on all platforms. > >>>>>> > >>>>>> Thanks, Richard. > >>>>>> > >>>>>> See also my question if anyone knows a reason for making the > compiled methods not_entrant: > >>>>>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020- > January/030339.html > >>>> From richard.reingruber at sap.com Tue May 12 09:42:39 2020 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Tue, 12 May 2020 09:42:39 +0000 Subject: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant In-Reply-To: References: <3c59b9f9-ec38-18c9-8f24-e1186a08a04a@oracle.com> <410eed04-e2ef-0f4f-1c56-19e6734a10f6@oracle.com> <81d7caa8-4244-85f3-4d4e-78117fe5e25b@oss.nttdata.com> <550b95ac-8b29-1eb8-a507-533e81d02322@oracle.com> <9c49ea2d-e3b8-b576-1d17-d18ad87cd6ed@oracle.com> Message-ID: Thanks Martin. Cheers, Richard. -----Original Message----- From: Doerr, Martin Sent: Dienstag, 12. Mai 2020 10:43 To: Reingruber, Richard ; David Holmes ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net Subject: RE: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant Hi Richard, I had already reviewed webrev.1. Looks good to me. Thanks for contributing it. Best regards, Martin > -----Original Message----- > From: hotspot-runtime-dev bounces at openjdk.java.net> On Behalf Of Reingruber, Richard > Sent: Montag, 4. Mai 2020 12:33 > To: David Holmes ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > Subject: RE: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > > // Trimmed the list of recipients. If the list gets too long then the message > needs to be approved > // by a moderator. > > Hi David, > > > On 28/04/2020 12:09 am, Reingruber, Richard wrote: > > > Hi David, > > > > > >> Not a review but some general commentary ... > > > > > > That's welcome. > > > Having had to take an even closer look now I have a review comment too :) > > > src/hotspot/share/prims/jvmtiThreadState.cpp > > > void JvmtiThreadState::invalidate_cur_stack_depth() { > > ! assert(SafepointSynchronize::is_at_safepoint() || > > ! (Thread::current()->is_VM_thread() && > > get_thread()->is_vmthread_processing_handshake()) || > > (JavaThread *)Thread::current() == get_thread(), > > "must be current thread or at safepoint"); > > You're looking at an outdated webrev, I'm afraid. > > This would be the post with the current webrev.1 > > http://mail.openjdk.java.net/pipermail/serviceability-dev/2020- > April/031245.html > > Thanks, Richard. > > -----Original Message----- > From: David Holmes > Sent: Montag, 4. Mai 2020 08:51 > To: Reingruber, Richard ; Yasumasa Suenaga > ; Patricio Chilano > ; serguei.spitsyn at oracle.com; Vladimir > Ivanov ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > > Hi Richard, > > On 28/04/2020 12:09 am, Reingruber, Richard wrote: > > Hi David, > > > >> Not a review but some general commentary ... > > > > That's welcome. > > Having had to take an even closer look now I have a review comment too :) > > src/hotspot/share/prims/jvmtiThreadState.cpp > > void JvmtiThreadState::invalidate_cur_stack_depth() { > ! assert(SafepointSynchronize::is_at_safepoint() || > ! (Thread::current()->is_VM_thread() && > get_thread()->is_vmthread_processing_handshake()) || > (JavaThread *)Thread::current() == get_thread(), > "must be current thread or at safepoint"); > > The message needs updating to include handshakes. > > More below ... > > >> On 25/04/2020 2:08 am, Reingruber, Richard wrote: > >>> Hi Yasumasa, Patricio, > >>> > >>>>>> I will send review request to replace VM_SetFramePop to > handshake in early next week in JDK-8242427. > >>>>>> Does it help you? I think it gives you to remove workaround. > >>>>> > >>>>> I think it would not help that much. Note that when replacing > VM_SetFramePop with a direct handshake > >>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm > operation [1]. So you would have to > >>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt > to these changes. > >>> > >>>> Thanks for your information. > >>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and > vmTestbase/nsk/jvmti/NotifyFramePop. > >>>> I will modify and will test it after yours. > >>> > >>> Thanks :) > >>> > >>>>> Also my first impression was that it won't be that easy from a > synchronization point of view to > >>>>> replace VM_SetFramePop with a direct handshake. E.g. > VM_SetFramePop::doit() indirectly calls > >>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, > JvmtiFramePop fpop) where > >>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at > safepoint. It's not directly clear > >>>>> to me, how this has to be handled. > >>> > >>>> I think JvmtiEventController::set_frame_pop() should hold > JvmtiThreadState_lock because it affects other JVMTI operation especially > FramePop event. > >>> > >>> Yes. To me it is unclear what synchronization is necessary, if it is called > during a handshake. And > >>> also I'm unsure if a thread should do safepoint checks while executing a > handshake. > > > >> I'm growing increasingly concerned that use of direct handshakes to > >> replace VM operations needs a much greater examination for correctness > >> than might initially be thought. I see a number of issues: > > > > I agree. I'll address your concerns in the context of this review thread for > JDK-8238585 below. > > > > In addition I would suggest to take the general part of the discussion to a > dedicated thread or to > > the review thread for JDK-8242427. I would like to keep this thread closer > to its subject. > > I will focus on the issues in the context of this particular change > then, though the issues themselves are applicable to all handshake > situations (and more so with direct handshakes). This is mostly just > discussion. > > >> First, the VMThread executes (most) VM operations with a clean stack in > >> a clean state, so it has lots of room to work. If we now execute the > >> same logic in a JavaThread then we risk hitting stackoverflows if > >> nothing else. But we are also now executing code in a JavaThread and so > >> we have to be sure that code is not going to act differently (in a bad > >> way) if executed by a JavaThread rather than the VMThread. For > example, > >> may it be possible that if executing in the VMThread we defer some > >> activity that might require execution of Java code, or else hand it off > >> to one of the service threads? If we execute that code directly in the > >> current JavaThread instead we may not be in a valid state (e.g. consider > >> re-entrancy to various subsystems that is not allowed). > > > > It is not too complex, what EnterInterpOnlyModeClosure::do_thread() is > doing. I already added a > > paragraph to the JBS-Item [1] explaining why the direct handshake is > sufficient from a > > synchronization point of view. > > Just to be clear, your proposed change is not using a direct handshake. > > > Furthermore the stack is walked and the return pc of compiled frames is > replaced with the address of > > the deopt handler. > > > > I can't see why this cannot be done with a direct handshake. Something > very similar is already done > > in JavaThread::deoptimize_marked_methods() which is executed as part > of an ordinary handshake. > > Note that existing non-direct handshakes may also have issues that not > have been fully investigated. > > > The demand on stack-space should be very modest. I would not expect a > higher risk for stackoverflow. > > For the target thread if you use more stack than would be used stopping > at a safepoint then you are at risk. For the thread initiating the > direct handshake if you use more stack than would be used enqueuing a VM > operation, then you are at risk. As we have not quantified these > numbers, nor have any easy way to establish the stack use of the actual > code to be executed, we're really just hoping for the best. This is a > general problem with handshakes that needs to be investigated more > deeply. As a simple, general, example just imagine if the code involves > logging that might utilise an on-stack buffer. > > >> Second, we have this question mark over what happens if the operation > >> hits further safepoint or handshake polls/checks? Are there constraints > >> on what is allowed here? How can we recognise this problem may exist > and > >> so deal with it? > > > > The thread in EnterInterpOnlyModeClosure::do_thread() can't become > safepoint/handshake safe. I > > tested locally test/hotspot/jtreg:vmTestbase_nsk_jvmti with a > NoSafepointVerifier. > > That's good to hear but such tests are not exhaustive, they will detect > if you do reach a safepoint/handshake but they can't prove that you > cannot reach one. What you have done is necessary but may not be > sufficient. Plus you didn't actually add the NSV to the code - is there > a reason we can't actually keep it in do_thread? (I'm not sure if the > NSV also acts as a NoHandshakeVerifier?) > > >> Third, while we are generally considering what appear to be > >> single-thread operations, which should be amenable to a direct > >> handshake, we also have to be careful that some of the code involved > >> doesn't already expect/assume we are at a safepoint - e.g. a VM op may > >> not need to take a lock where a direct handshake might! > > > > See again my arguments in the JBS item [1]. > > Yes I see the reasoning and that is good. My point is a general one as > it may not be obvious when such assumptions exist in the current code. > > Thanks, > David > > > Thanks, > > Richard. > > > > [1] https://bugs.openjdk.java.net/browse/JDK-8238585 > > > > -----Original Message----- > > From: David Holmes > > Sent: Montag, 27. April 2020 07:16 > > To: Reingruber, Richard ; Yasumasa Suenaga > ; Patricio Chilano > ; serguei.spitsyn at oracle.com; Vladimir > Ivanov ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > > Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > > > > Hi all, > > > > Not a review but some general commentary ... > > > > On 25/04/2020 2:08 am, Reingruber, Richard wrote: > >> Hi Yasumasa, Patricio, > >> > >>>>> I will send review request to replace VM_SetFramePop to handshake > in early next week in JDK-8242427. > >>>>> Does it help you? I think it gives you to remove workaround. > >>>> > >>>> I think it would not help that much. Note that when replacing > VM_SetFramePop with a direct handshake > >>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm > operation [1]. So you would have to > >>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt > to these changes. > >> > >>> Thanks for your information. > >>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and > vmTestbase/nsk/jvmti/NotifyFramePop. > >>> I will modify and will test it after yours. > >> > >> Thanks :) > >> > >>>> Also my first impression was that it won't be that easy from a > synchronization point of view to > >>>> replace VM_SetFramePop with a direct handshake. E.g. > VM_SetFramePop::doit() indirectly calls > >>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, > JvmtiFramePop fpop) where > >>>> JvmtiThreadState_lock is acquired with safepoint check, if not at > safepoint. It's not directly clear > >>>> to me, how this has to be handled. > >> > >>> I think JvmtiEventController::set_frame_pop() should hold > JvmtiThreadState_lock because it affects other JVMTI operation especially > FramePop event. > >> > >> Yes. To me it is unclear what synchronization is necessary, if it is called > during a handshake. And > >> also I'm unsure if a thread should do safepoint checks while executing a > handshake. > > > > I'm growing increasingly concerned that use of direct handshakes to > > replace VM operations needs a much greater examination for correctness > > than might initially be thought. I see a number of issues: > > > > First, the VMThread executes (most) VM operations with a clean stack in > > a clean state, so it has lots of room to work. If we now execute the > > same logic in a JavaThread then we risk hitting stackoverflows if > > nothing else. But we are also now executing code in a JavaThread and so > > we have to be sure that code is not going to act differently (in a bad > > way) if executed by a JavaThread rather than the VMThread. For example, > > may it be possible that if executing in the VMThread we defer some > > activity that might require execution of Java code, or else hand it off > > to one of the service threads? If we execute that code directly in the > > current JavaThread instead we may not be in a valid state (e.g. consider > > re-entrancy to various subsystems that is not allowed). > > > > Second, we have this question mark over what happens if the operation > > hits further safepoint or handshake polls/checks? Are there constraints > > on what is allowed here? How can we recognise this problem may exist and > > so deal with it? > > > > Third, while we are generally considering what appear to be > > single-thread operations, which should be amenable to a direct > > handshake, we also have to be careful that some of the code involved > > doesn't already expect/assume we are at a safepoint - e.g. a VM op may > > not need to take a lock where a direct handshake might! > > > > Cheers, > > David > > ----- > > > >> @Patricio, coming back to my question [1]: > >> > >> In the example you gave in your answer [2]: the java thread would > execute a vm operation during a > >> direct handshake operation, while the VMThread is actually in the middle > of a VM_HandshakeAllThreads > >> operation, waiting to handshake the same handshakee: why can't the > VMThread just proceed? The > >> handshakee would be safepoint safe, wouldn't it? > >> > >> Thanks, Richard. > >> > >> [1] https://bugs.openjdk.java.net/browse/JDK- > 8230594?focusedCommentId=14301677&page=com.atlassian.jira.plugin.syst > em.issuetabpanels:comment-tabpanel#comment-14301677 > >> > >> [2] https://bugs.openjdk.java.net/browse/JDK- > 8230594?focusedCommentId=14301763&page=com.atlassian.jira.plugin.syst > em.issuetabpanels:comment-tabpanel#comment-14301763 > >> > >> -----Original Message----- > >> From: Yasumasa Suenaga > >> Sent: Freitag, 24. April 2020 17:23 > >> To: Reingruber, Richard ; Patricio Chilano > ; serguei.spitsyn at oracle.com; Vladimir > Ivanov ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > >> Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > >> > >> Hi Richard, > >> > >> On 2020/04/24 23:44, Reingruber, Richard wrote: > >>> Hi Yasumasa, > >>> > >>>> I will send review request to replace VM_SetFramePop to handshake > in early next week in JDK-8242427. > >>>> Does it help you? I think it gives you to remove workaround. > >>> > >>> I think it would not help that much. Note that when replacing > VM_SetFramePop with a direct handshake > >>> you could not just execute VM_EnterInterpOnlyMode as a nested vm > operation [1]. So you would have to > >>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to > these changes. > >> > >> Thanks for your information. > >> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and > vmTestbase/nsk/jvmti/NotifyFramePop. > >> I will modify and will test it after yours. > >> > >> > >>> Also my first impression was that it won't be that easy from a > synchronization point of view to > >>> replace VM_SetFramePop with a direct handshake. E.g. > VM_SetFramePop::doit() indirectly calls > >>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, > JvmtiFramePop fpop) where > >>> JvmtiThreadState_lock is acquired with safepoint check, if not at > safepoint. It's not directly clear > >>> to me, how this has to be handled. > >> > >> I think JvmtiEventController::set_frame_pop() should hold > JvmtiThreadState_lock because it affects other JVMTI operation especially > FramePop event. > >> > >> > >> Thanks, > >> > >> Yasumasa > >> > >> > >>> So it appears to me that it would be easier to push JDK-8242427 after this > (JDK-8238585). > >>> > >>>> (The patch is available, but I want to see the result of PIT in this > weekend whether JDK-8242425 works fine.) > >>> > >>> Would be interesting to see how you handled the issues above :) > >>> > >>> Thanks, Richard. > >>> > >>> [1] See question in comment > https://bugs.openjdk.java.net/browse/JDK- > 8230594?focusedCommentId=14302030&page=com.atlassian.jira.plugin.syst > em.issuetabpanels:comment-tabpanel#comment-14302030 > >>> > >>> -----Original Message----- > >>> From: Yasumasa Suenaga > >>> Sent: Freitag, 24. April 2020 13:34 > >>> To: Reingruber, Richard ; Patricio Chilano > ; serguei.spitsyn at oracle.com; Vladimir > Ivanov ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > >>> Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > >>> > >>> Hi Richard, > >>> > >>> I will send review request to replace VM_SetFramePop to handshake in > early next week in JDK-8242427. > >>> Does it help you? I think it gives you to remove workaround. > >>> > >>> (The patch is available, but I want to see the result of PIT in this weekend > whether JDK-8242425 works fine.) > >>> > >>> > >>> Thanks, > >>> > >>> Yasumasa > >>> > >>> > >>> On 2020/04/24 17:18, Reingruber, Richard wrote: > >>>> Hi Patricio, Vladimir, and Serguei, > >>>> > >>>> now that direct handshakes are available, I've updated the patch to > make use of them. > >>>> > >>>> In addition I have done some clean-up changes I missed in the first > webrev. > >>>> > >>>> Finally I have implemented the workaround suggested by Patricio to > avoid nesting the handshake > >>>> into the vm operation VM_SetFramePop [1] > >>>> > >>>> Kindly review again: > >>>> > >>>> Webrev: > http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1/ > >>>> Webrev(delta): > http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1.inc/ > >>>> > >>>> I updated the JBS item explaining why the vm operation > VM_EnterInterpOnlyMode can be replaced with a > >>>> direct handshake: > >>>> > >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8238585 > >>>> > >>>> Testing: > >>>> > >>>> * JCK and JTREG tests, also in Xcomp mode with fastdebug and release > builds on all platforms. > >>>> > >>>> * Submit-repo: mach5-one-rrich-JDK-8238585-20200423-1436-10441737 > >>>> > >>>> Thanks, > >>>> Richard. > >>>> > >>>> [1] An assertion in Handshake::execute_direct() fails, if called be > VMThread, because it is no JavaThread. > >>>> > >>>> -----Original Message----- > >>>> From: hotspot-dev On > Behalf Of Reingruber, Richard > >>>> Sent: Freitag, 14. Februar 2020 19:47 > >>>> To: Patricio Chilano ; > serviceability-dev at openjdk.java.net; hotspot-compiler- > dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime- > dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net > >>>> Subject: RE: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > >>>> > >>>> Hi Patricio, > >>>> > >>>> > > I'm really glad you noticed the problematic nesting. This seems > to be a general issue: currently a > >>>> > > handshake cannot be nested in a vm operation. Maybe it should > be asserted in the > >>>> > > Handshake::execute() methods that they are not called by the > vm thread evaluating a vm operation? > >>>> > > > >>>> > > > Alternatively I think you could do something similar to what > we do in > >>>> > > > Deoptimization::deoptimize_all_marked(): > >>>> > > > > >>>> > > > EnterInterpOnlyModeClosure hs; > >>>> > > > if (SafepointSynchronize::is_at_safepoint()) { > >>>> > > > hs.do_thread(state->get_thread()); > >>>> > > > } else { > >>>> > > > Handshake::execute(&hs, state->get_thread()); > >>>> > > > } > >>>> > > > (you could pass ?EnterInterpOnlyModeClosure? directly to > the > >>>> > > > HandshakeClosure() constructor) > >>>> > > > >>>> > > Maybe this could be used also in the Handshake::execute() > methods as general solution? > >>>> > Right, we could also do that. Avoiding to clear the polling page in > >>>> > HandshakeState::clear_handshake() should be enough to fix this > issue and > >>>> > execute a handshake inside a safepoint, but adding that "if" > statement > >>>> > in Hanshake::execute() sounds good to avoid all the extra code > that we > >>>> > go through when executing a handshake. I filed 8239084 to make > that change. > >>>> > >>>> Thanks for taking care of this and creating the RFE. > >>>> > >>>> > > >>>> > > > I don?t know JVMTI code so I?m not sure if > VM_EnterInterpOnlyMode is > >>>> > > > always called in a nested operation or just sometimes. > >>>> > > > >>>> > > At least one execution path without vm operation exists: > >>>> > > > >>>> > > > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) > : void > >>>> > > > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState > *) : jlong > >>>> > > JvmtiEventControllerPrivate::recompute_enabled() : void > >>>> > > > JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 > matches) > >>>> > > JvmtiEventController::change_field_watch(jvmtiEvent, > bool) : void > >>>> > > JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : > jvmtiError > >>>> > > jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : > jvmtiError > >>>> > > > >>>> > > I tend to revert back to VM_EnterInterpOnlyMode as it wasn't > my main intent to replace it with a > >>>> > > handshake, but to avoid making the compiled methods on stack > not_entrant.... unless I'm further > >>>> > > encouraged to do it with a handshake :) > >>>> > Ah! I think you can still do it with a handshake with the > >>>> > Deoptimization::deoptimize_all_marked() like solution. I can > change the > >>>> > if-else statement with just the Handshake::execute() call in > 8239084. > >>>> > But up to you. : ) > >>>> > >>>> Well, I think that's enough encouragement :) > >>>> I'll wait for 8239084 and try then again. > >>>> (no urgency and all) > >>>> > >>>> Thanks, > >>>> Richard. > >>>> > >>>> -----Original Message----- > >>>> From: Patricio Chilano > >>>> Sent: Freitag, 14. Februar 2020 15:54 > >>>> To: Reingruber, Richard ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > >>>> Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > >>>> > >>>> Hi Richard, > >>>> > >>>> On 2/14/20 9:58 AM, Reingruber, Richard wrote: > >>>>> Hi Patricio, > >>>>> > >>>>> thanks for having a look. > >>>>> > >>>>> > I?m only commenting on the handshake changes. > >>>>> > I see that operation VM_EnterInterpOnlyMode can be called > inside > >>>>> > operation VM_SetFramePop which also allows nested > operations. Here is a > >>>>> > comment in VM_SetFramePop definition: > >>>>> > > >>>>> > // Nested operation must be allowed for the > VM_EnterInterpOnlyMode that is > >>>>> > // called from the > JvmtiEventControllerPrivate::recompute_thread_enabled. > >>>>> > > >>>>> > So if we change VM_EnterInterpOnlyMode to be a handshake, > then now we > >>>>> > could have a handshake inside a safepoint operation. The issue I > see > >>>>> > there is that at the end of the handshake the polling page of the > target > >>>>> > thread could be disarmed. So if the target thread happens to be > in a > >>>>> > blocked state just transiently and wakes up then it will not stop > for > >>>>> > the ongoing safepoint. Maybe I can file an RFE to assert that the > >>>>> > polling page is armed at the beginning of disarm_safepoint(). > >>>>> > >>>>> I'm really glad you noticed the problematic nesting. This seems to be a > general issue: currently a > >>>>> handshake cannot be nested in a vm operation. Maybe it should be > asserted in the > >>>>> Handshake::execute() methods that they are not called by the vm > thread evaluating a vm operation? > >>>>> > >>>>> > Alternatively I think you could do something similar to what we > do in > >>>>> > Deoptimization::deoptimize_all_marked(): > >>>>> > > >>>>> > EnterInterpOnlyModeClosure hs; > >>>>> > if (SafepointSynchronize::is_at_safepoint()) { > >>>>> > hs.do_thread(state->get_thread()); > >>>>> > } else { > >>>>> > Handshake::execute(&hs, state->get_thread()); > >>>>> > } > >>>>> > (you could pass ?EnterInterpOnlyModeClosure? directly to the > >>>>> > HandshakeClosure() constructor) > >>>>> > >>>>> Maybe this could be used also in the Handshake::execute() methods > as general solution? > >>>> Right, we could also do that. Avoiding to clear the polling page in > >>>> HandshakeState::clear_handshake() should be enough to fix this issue > and > >>>> execute a handshake inside a safepoint, but adding that "if" statement > >>>> in Hanshake::execute() sounds good to avoid all the extra code that we > >>>> go through when executing a handshake. I filed 8239084 to make that > change. > >>>> > >>>>> > I don?t know JVMTI code so I?m not sure if > VM_EnterInterpOnlyMode is > >>>>> > always called in a nested operation or just sometimes. > >>>>> > >>>>> At least one execution path without vm operation exists: > >>>>> > >>>>> > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) > : void > >>>>> > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState > *) : jlong > >>>>> JvmtiEventControllerPrivate::recompute_enabled() : void > >>>>> JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, > bool) : void (2 matches) > >>>>> JvmtiEventController::change_field_watch(jvmtiEvent, bool) : > void > >>>>> JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError > >>>>> jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : > jvmtiError > >>>>> > >>>>> I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my > main intent to replace it with a > >>>>> handshake, but to avoid making the compiled methods on stack > not_entrant.... unless I'm further > >>>>> encouraged to do it with a handshake :) > >>>> Ah! I think you can still do it with a handshake with the > >>>> Deoptimization::deoptimize_all_marked() like solution. I can change > the > >>>> if-else statement with just the Handshake::execute() call in 8239084. > >>>> But up to you.? : ) > >>>> > >>>> Thanks, > >>>> Patricio > >>>>> Thanks again, > >>>>> Richard. > >>>>> > >>>>> -----Original Message----- > >>>>> From: Patricio Chilano > >>>>> Sent: Donnerstag, 13. Februar 2020 18:47 > >>>>> To: Reingruber, Richard ; serviceability- > dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot- > dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot- > gc-dev at openjdk.java.net > >>>>> Subject: Re: RFR(S) 8238585: Use handshake for > JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make > compiled methods on stack not_entrant > >>>>> > >>>>> Hi Richard, > >>>>> > >>>>> I?m only commenting on the handshake changes. > >>>>> I see that operation VM_EnterInterpOnlyMode can be called inside > >>>>> operation VM_SetFramePop which also allows nested operations. > Here is a > >>>>> comment in VM_SetFramePop definition: > >>>>> > >>>>> // Nested operation must be allowed for the > VM_EnterInterpOnlyMode that is > >>>>> // called from the > JvmtiEventControllerPrivate::recompute_thread_enabled. > >>>>> > >>>>> So if we change VM_EnterInterpOnlyMode to be a handshake, then > now we > >>>>> could have a handshake inside a safepoint operation. The issue I see > >>>>> there is that at the end of the handshake the polling page of the > target > >>>>> thread could be disarmed. So if the target thread happens to be in a > >>>>> blocked state just transiently and wakes up then it will not stop for > >>>>> the ongoing safepoint. Maybe I can file an RFE to assert that the > >>>>> polling page is armed at the beginning of disarm_safepoint(). > >>>>> > >>>>> I think one option could be to remove > >>>>> SafepointMechanism::disarm_if_needed() in > >>>>> HandshakeState::clear_handshake() and let each JavaThread disarm > itself > >>>>> for the handshake case. > >>>>> > >>>>> Alternatively I think you could do something similar to what we do in > >>>>> Deoptimization::deoptimize_all_marked(): > >>>>> > >>>>> ? EnterInterpOnlyModeClosure hs; > >>>>> ? if (SafepointSynchronize::is_at_safepoint()) { > >>>>> ??? hs.do_thread(state->get_thread()); > >>>>> ? } else { > >>>>> ??? Handshake::execute(&hs, state->get_thread()); > >>>>> ? } > >>>>> (you could pass ?EnterInterpOnlyModeClosure? directly to the > >>>>> HandshakeClosure() constructor) > >>>>> > >>>>> I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode > is > >>>>> always called in a nested operation or just sometimes. > >>>>> > >>>>> Thanks, > >>>>> Patricio > >>>>> > >>>>> On 2/12/20 7:23 AM, Reingruber, Richard wrote: > >>>>>> // Repost including hotspot runtime and gc lists. > >>>>>> // Dean Long suggested to do so, because the enhancement > replaces a vm operation > >>>>>> // with a handshake. > >>>>>> // Original thread: > http://mail.openjdk.java.net/pipermail/serviceability-dev/2020- > February/030359.html > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> could I please get reviews for this small enhancement in hotspot's > jvmti implementation: > >>>>>> > >>>>>> Webrev: > http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.0/ > >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8238585 > >>>>>> > >>>>>> The change avoids making all compiled methods on stack > not_entrant when switching a java thread to > >>>>>> interpreter only execution for jvmti purposes. It is sufficient to > deoptimize the compiled frames on stack. > >>>>>> > >>>>>> Additionally a handshake is used instead of a vm operation to walk > the stack and do the deoptimizations. > >>>>>> > >>>>>> Testing: JCK and JTREG tests, also in Xcomp mode with fastdebug > and release builds on all platforms. > >>>>>> > >>>>>> Thanks, Richard. > >>>>>> > >>>>>> See also my question if anyone knows a reason for making the > compiled methods not_entrant: > >>>>>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020- > January/030339.html > >>>> From thomas.stuefe at gmail.com Tue May 12 12:41:48 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 12 May 2020 14:41:48 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> Message-ID: On Tue, May 12, 2020 at 12:01 AM Ioi Lam wrote: > > > On 5/8/20 12:33 AM, Thomas St?fe wrote: > > Hi Ioi, > > thanks for taking the time to look at this! > > On Fri, May 8, 2020 at 7:37 AM Ioi Lam wrote: > >> Hi Thomas, >> >> I am running your patch in out CI pipeline now. Some comments: >> >> [1] Do we still need to use _class_space_list? It looks like we just have >> a single range. Is _class_space_list->current_virtual_space() always the >> same VS? >> >> if (_class_space_list != NULL) { >> address base = >> (address)_class_space_list->current_virtual_space()->bottom(); >> address top = base + compressed_class_space_size(); >> st->print("Compressed class space mapped at: " PTR_FORMAT "-" >> PTR_FORMAT ", size: " SIZE_FORMAT, >> p2i(base), p2i(top), top - base); >> >> > Yes it is. Has been bugging me for years too. With ccs, the > VirtualSpaceList is degenerated and only contains one node. It still makes > some sense code wise. But maybe VirtualSpace*List* is a misnomer. > > It also ties in into the question whether Metaspace reservations should be > able to grow on demand. Oracle folks I talk to tend to say yes. I > personally think the "Metaspace is infinite don't worry" meme was broken > since ccs was a thing, since CompressedClassSpaceSize is an absolute limit > already. > > I may tweak and simplify the code a bit with Elastic Metaspace. Maybe, as > you say, with ccs we can get rid of VirtualSpaceList and deal with the > range directly. Have to check. > > [2] Does JDK-8243535 exist with the current jdk/jdk repo? >> >> // This does currently not work because rs may be the result of a split >> operation >> // and NMT seems not to be able to handle splits. >> // See JDK-8243535. >> // MemTracker::record_virtual_memory_type((address)rs.base(), mtClass); >> >> > Yes. It shows up more rarely. With this patch, it will show up always. I > had a conversation with Zhengyu about this. I think there may be > workarounds, like removing the original mapping in NMT and replacing it > with the two new ones for archive and ccs. But it seems iffy, I'd rather > have this fixed in NMT. > > I also think it is not such a big deal, but if you think otherwise, I can > integrate a workaround into this patch. > > > > My question is -- will NMT (or control-backspace) now always report > incorrect memory size for the metaspace when CDS is enabled? If so, I think > a work-around is necessary. > > What is control backspace? I'm staring at '^H' output in my terminal and feel like I miss something :) Today NMT reports wrong memory size at dumptime (I think no-one cares probably) and at runtime if the archive got relocated. With the patch NMT will also report wrong numbers at runtime when mapping to default address. But okay, I will add a workaround. > >> [3] I think this can be put in header file. >> >> bool Metaspace::class_space_is_initialized() { >> return _class_space_list != NULL; >> } >> >> > Will do. > > >> [4] Why does the CCS need to follow UseLargePagesInMetaspace? This is the >> reason for the >> gap in the following chart: >> >> // We do this by reserving space for the ccs behind the archives. >> Note however that >> // ccs follows a different alignment (Metaspace::reserve_alignment), >> so there may >> // be a gap between ccs and cds. >> // We use a similar layout at runtime, see >> reserve_address_space_for_archives(). >> // >> // +-- SharedBaseAddress (default = >> 0x800000000) >> // v >> // +-..---------+---------+ ... >> +----+----+----+--------+--------------------+ >> // | Heap | Archive | | MC | RW | RO | [gap] | class >> space | >> // +-..---------+---------+ ... >> +----+----+----+--------+--------------------+ >> // |<-- MaxHeapSize -->| |<-- UnscaledClassSpaceMax = 4GB -->| >> >> > No, nothing to do with large pages. CCS does not have large pages. > > The gap can result from different alignment rules between metaspace and > cds. With Elastic Metaspace, metaspace reservations will be aligned to 4M. > So, ccs will have to start at a 4M aligned border. > > >> >> _reserve_alignment is determined here: >> >> void Metaspace::ergo_initialize() { >> if (DumpSharedSpaces) { >> // Using large pages when dumping the shared archive is currently not >> implemented. >> FLAG_SET_ERGO(UseLargePagesInMetaspace, false); >> } >> >> size_t page_size = os::vm_page_size(); >> if (UseLargePages && UseLargePagesInMetaspace) { >> page_size = os::large_page_size(); >> } >> >> _commit_alignment = page_size; >> _reserve_alignment = MAX2(page_size, >> (size_t)os::vm_allocation_granularity()); >> >> But when CDS is enabled, the RS is reserved without large pages, so it >> should be possible to incrementally commit the CCS using just >> os::vm_allocation_granularity(). >> >> > Absolutely. CCS does not use large pages for that reason. As I wrote, this > is about the new reservation granularity of 4M. > > > OK, if I understand correctly: > > + With the current patch, the gap exists only when > UseLargePagesInMetaspace is used at run time. The gap actually is > unnecessary, since ccs doesn't use large pages. > > + With the upcoming Elastic Metaspace, _reserve_alignment will be 4MB, so > we will almost always have a gap. > Yes. Note that we decided to remove the large page feature from Metaspace with Elastic Metaspace for now, though we may reintroduce it later in a different form. See CSR: https://bugs.openjdk.java.net/browse/JDK-8243161 . > > > >> [5] I see the aarch64 code in >> Metaspace::reserve_address_space_for_compressed_classes has been changed. >> Is this necessary for this RFE, or is it a separate improvement that can be >> done in a different RFE? >> > > Well, it had to be changed somewhat, since the prototype differs. I > swapped Metaspace::reserve_preferred_space() for > Metaspace::reserve_address_space_for_compressed_classes(), which follows a > somewhat different protocol. > > That aside, the functionality for AARCH64 should be as unchanged as > possible. I removed the AIX specific coding, since it was subtly wrong, but > that's fine since we are AIX/ppc maintainers. > > >> [6] For AARCH64, should we skip the part that reserves it right above the >> heap? Or does AARCH64 always allocate the heap such that it's a preferable >> address? >> >> > Yes and yes :) > > We do skip that for aarch64 (at least if the address is wrong): > > 1157 // Try right above the Java heap... > 1158 address base = align_up(CompressedOops::end(), Metaspace::reserve_alignment()); > 1159 assert(base != NULL, "Sanity"); > 1160 > 1161 const size_t size = align_up(CompressedClassSpaceSize, Metaspace::reserve_alignment()); > 1162 if (CompressedKlassPointers::is_valid_base(base)) { > 1163 rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* large */, (char*)base); > 1164 } > > At 1162, we check if the attach address would be valid for this platform. > If aarch64 says no, we won't reserve there but go straight to the "attach > wherever you think is good" part. Which is implemented > by Metaspace::reserve_address_space_for_compressed_classes(), which on > aarch64 does search a suitable address. > > > >> // case (b) >> ReservedSpace rs; >> >> // Try right above the Java heap... >> address base = align_up(CompressedOops::end(), >> Metaspace::reserve_alignment()); >> assert(base != NULL, "Sanity"); >> >> const size_t size = align_up(CompressedClassSpaceSize, >> Metaspace::reserve_alignment()); >> if (CompressedKlassPointers::is_valid_base(base)) { >> rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* >> large */, (char*)base); >> } >> >> // ...failing that, reserve anywhere, but let platform do optimized >> placement: >> if (!rs.is_reserved()) { >> rs = Metaspace::reserve_address_space_for_compressed_classes(size); >> } >> >> [7] argument.cpp checks this: >> >> if (UseSharedSpaces || DumpSharedSpaces) { >> if >> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >> log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for >> this platform, option will be ignored.", >> p2i((address)SharedBaseAddress)); >> FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >> } >> } >> >> but initialize_dumptime_shared_and_meta_spaces aligns up the >> SharedBaseAddress before doing the assert. >> >> void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >> assert(DumpSharedSpaces, "should be called for dump time only"); >> >> const size_t reserve_alignment = >> MetaspaceShared::reserved_space_alignment(); >> char* shared_base = (char*)align_up((char*)SharedBaseAddress, >> reserve_alignment); >> >> #ifdef _LP64 >> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >> "Sanity"); >> >> So theoretically shared_base may no longer be is_valid_base after the >> align_up. It probably won't happen, but I think the code will be much >> clearer tofirst to align_up, then check for is_valid_base and reset to >> default, and finally assert. >> >> > Okay, I'll change that. > > >> [8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >> following block can be simplified: >> >> if (!_shared_rs.is_reserved()) { >> // Get a reserved space anywhere if attaching at the >> SharedBaseAddress fails: >> if (UseCompressedClassPointers) { >> // If we need a compressed class space too, let the platform handle >> the reservation. >> _shared_rs = >> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >> } else { >> // anywhere is fine. >> _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >> large */, (char*)NULL); >> } >> } >> >> ... if you change the declaration to: >> >> static ReservedSpace >> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >> ReservedSpace();}); >> >> [9] I think the #ifdef _LP64 is not necessary: >> >> #ifdef _LP64 >> // Some sanity checks after reserving address spaces for archives and >> class space. >> assert(archive_space_rs.is_reserved(), "Sanity"); >> if (Metaspace::using_class_space()) { >> // Class space must closely follow the archive space. Both spaces >> must be aligned correctly. >> assert(class_space_rs.is_reserved(), "A class space should have >> been reserved"); >> assert(class_space_rs.base() >= archive_space_rs.end(), "class >> space should follow the cds archive space"); >> assert(is_aligned(archive_space_rs.base(), >> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >> assert(is_aligned(class_space_rs.base(), >> Metaspace::reserve_alignment()), "class space misaligned"); >> } >> #endif >> >> > I'll take a look. > > >> The rest of the code looks OK to me, but I may take a look at it again >> after getting more sleep :-) >> >> > Sure. Thanks man! :) > > > Good news. I ran our tiers 1-4 and there were no failures related to your > changes! > > Neat. This mirrors our results. I'll prepare a new webrev. ..Thomas From aph at redhat.com Tue May 12 13:07:17 2020 From: aph at redhat.com (Andrew Haley) Date: Tue, 12 May 2020 14:07:17 +0100 Subject: [8u] RFR(S): 8244548: JDK 8u: sun.misc.Version.jdkUpdateVersion() returns wrong result In-Reply-To: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> References: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> Message-ID: <808419e3-316b-381d-f636-606aadff39b6@redhat.com> On 5/7/20 4:09 PM, Severin Gehwolf wrote: > jvm_version_info.jvm_version currently holds this quadruplet: > > Most significant 8 bits => major version, followed by 8 bits => minor > version, followed by 8 bits => micro version, followed by 8 bits => > build version. Note that JVM minor version represents the update > version as passed in via configure and the micro version is currently > not used (always 0). See vm_version.cpp lines 100-102 where only major, > minor and build number are ever been set. Knowing this, we can still > preserve the same behavior after patch by defining JVM_VERSION_MICRO to > 0 for any version. This is tricky. JVM_GetVersionInfo is a function exported by libjvm.so, and the version is simply encoded as unsigned int Abstract_VM_Version::jvm_version() { return ((Abstract_VM_Version::vm_major_version() & 0xFF) << 24) | ((Abstract_VM_Version::vm_minor_version() & 0xFF) << 16) | (Abstract_VM_Version::vm_build_number() & 0xFF); } I guess we could argue that this is for JVM-JDK internal use only, and no-one else cares. Or we could encode it in a different way such that at least we have a jvm_version that is monotonically increasing, perhaps by (ab)using the lower 8 bits of the version, the vm_build_number. But I guess I'm being paranoid, and no tools are going to care about minor versions anyway,even if they do call JVM_GetVersionInfo. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From erik.osterlund at oracle.com Tue May 12 14:09:47 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 12 May 2020 16:09:47 +0200 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: References: <56a0cf85-9698-64b2-95b4-e5b123c3e9e2@oracle.com> <9a314ebe-5499-ecb6-3c5b-0c8ce7f9c8d2@oracle.com> <44febab6-5703-1c8b-d4ad-0f4c12517afa@oracle.com> Message-ID: Hi, I ran into some merge conflicts in the tests and the new tests behaved a bit differently on different platforms. The new tests naturally do not behave the same way on the platforms that do not yet support compressed class pointers without compressed oops, so I just filtered away those platforms from running the new tests. Will update the filter when they start supporting it. Another interesting thing is that on macOS, sometimes we did not get the expected compressed class pointer encoding without compressed oops. The reason is that without compressed oops, we allocate the heap at some random address where the OS thinks is suitable. Sometimes that is where we would ideally have put the compressed class space. This forces metaspace to be mapped in somewhere else with different characteristics. Ideally, the bootstrapping logic should be disentangled so that when you run without compressed oops, metaspace gets reserved before the heap. But I think that warrants a follow-up RFE, as I do not want to complicate this patch any further. Disentangling that seems like it could become quite tricky on its own. So here is another webrev with the test issues resolved: http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ Incremental: http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ Thanks, /Erik On 2020-04-29 09:39, Erik ?sterlund wrote: > Hi Thomas, > > On 2020-04-29 08:27, Thomas St?fe wrote: >> Hi Erik, >> >> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund >> > wrote: >> >> ??? Hi Thomas, >> >> ??? Thanks for reviewing this. >> >> ??? On 2020-04-28 13:14, Thomas St?fe wrote: >> >>> ??? With compressed oops we continue to squeeze ccs in after the heap >>> ??? in the hope that it all still fits into the lower 4G or 32G. (Do >>> ??? we ever bother to check if heap is actually reserved in low area? >>> ??? It may have given up and reserved upstate instead, leaving free >>> ??? room below...). >> >> ??? Right. And yes I'm pretty sure we do check. The heap reserves >> ??? memory, and sets the CompressedOops::end() based on that, which is >> ??? fed as input to the metaspace/CDS placement. >> >> >> No, what I meant was: since we reserve at CompressedOops::end(), if >> the heap was too large to place it in the lower address regions, the >> class space will not live there either even if it could, since it is >> smaller and might fit. >> >> For example: >> >> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ >> ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace -Xshare:off >> -Xmx31g >> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: >> reserved=1048576.00 KB, committed=0.00 KB ( ?0%), used=0.00 KB ( ?0%) >> [0.003s][trace][gc,metaspace] ?[0x00000017c0400000, >> 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) >> [0.003s][trace][gc,metaspace] Narrow klass base: 0x00000017c0400000, >> Narrow klass shift: 0 >> [0.003s][trace][gc,metaspace] Compressed class space size: 1073741824 >> Address: 0x00000017c0400000 Req Addr: 0x00000017c0400000 >> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: >> reserved=8192.00 KB, committed=0.00 KB ( ?0%), used=0.00 KB ( ?0%) >> [0.003s][trace][gc,metaspace] ?[0x00007fc9cb800000, >> 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) >> >> (large heap and no CDS, since it prevents zero based mode). >> >> In that case narrow klass base is not zero based even it if probably >> could be, if we were to allocate class space in the lower regions, >> which are now unoccupied since the large heap does not fit there. A >> missed opportunity to go zero based. Not a big deal, since we mostly >> run with CDS now anyway. > > I see. Yeah, there is room for improvement there. Although, as you > say, people mostly run with CDS on, so it probably won't matter even > if that is improved. > >>> ??? Now with your patch without compressed oops we attempt to map at >>> ??? HeapBaseMinAddress since this is the lowest address we are >>> ??? allowed to map at and the heap won't obviously need it,?right? >> >> ??? Exactly. >> >>> ??? My only misgiving would be that?HeapBaseMinAddress is even more >>> ??? of a misnomer (its already now a?bit, witness my year long >>> ??? confusion :-), since it now controls not only heap placement but >>> ??? ccs too. Actually if it is that important to avoid mapping to >>> ??? addresses below?HeapBaseMinAddress we should probably prevent it >>> ??? somewhere below in the os layer but I really do not want to open >>> ??? that can of worms. >> >> ??? Yes agreed. Ideally it would be called something like >> ??? MappingMinAddress instead, because it really isn't tied to the >> ??? heap really, it just happened to be the primary user of mapping >> ??? things at low addresses. But I also think changing that, and >> ??? enforcing it more widely in the os layer sounds like a different >> ??? can of worms. >> >>> ??? -- >>> >>> ??? Some more remarks about arguments.cpp: >>> >>> ??? - In Arguments::set_use_compressed_klass_ptrs(), if now >>> ??? compressed class pointers are independent from compressed oops, >>> ??? checking CompressedClassSpaceSize for validity should now always >>> ??? happen for UseCompressedClassPointers=1. >> >> ??? I think it looks like I do that. >> >>> ??? - If HeapBaseMinAddress is now always used, we should check it >>> ??? always too (see Arguments::set_heap_size) >> >> ??? Good idea. Fixed. >> >>> >>>> ??????? Also, could we have some?tests which exercise cds/metaspace >>>> ??????? initialization without coops and with ccps? Easiest way >>>> ??????? would be to extend >>>> ??????? "CompressedOops/CompressedClassPointers.java" by some cases. >>> >>> >>> ??? Thank you. I think this makes sense especially since compressed >>> ??? class pointer encoding is restricted to one mode if CDS is on >>> ??? (base = shared address and shift=3), and since CDS=on is the >>> ??? default zerobased encoding may not happen that often. >> >> ??? Yup. I added some explicit test cases for -XX:-UseCompressedOops >> ??? -XX:+UseCompressedClassPointers as suggested. This will also be >> ??? the default configuration for ZGC, so that should give that >> ??? configuration more test coverage in the various tests we have. >> >> ??? Here is a new webrev: >> ??? http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ >> >> ??? Incremental: >> ??? http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ >> >> ??? I hope I fixed everything you wanted in this patch. I also fixed >> ??? what Frederic and Coleen wanted. >> >> >> arguments.cpp, the metaspace stuff and the test look good to me now. >> Thank you for your work! > > Thanks for the review Thomas! > > /Erik From sgehwolf at redhat.com Tue May 12 14:14:45 2020 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Tue, 12 May 2020 16:14:45 +0200 Subject: [8u] RFR(S): 8244548: JDK 8u: sun.misc.Version.jdkUpdateVersion() returns wrong result In-Reply-To: <808419e3-316b-381d-f636-606aadff39b6@redhat.com> References: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> <808419e3-316b-381d-f636-606aadff39b6@redhat.com> Message-ID: <15a2fa41f80935d8c7d3893aad7d080c1f67aa2a.camel@redhat.com> Hi Andrew, Thanks for looking at this. On Tue, 2020-05-12 at 14:07 +0100, Andrew Haley wrote: > On 5/7/20 4:09 PM, Severin Gehwolf wrote: > > jvm_version_info.jvm_version currently holds this quadruplet: > > > > Most significant 8 bits => major version, followed by 8 bits => minor > > version, followed by 8 bits => micro version, followed by 8 bits => > > build version. Note that JVM minor version represents the update > > version as passed in via configure and the micro version is currently > > not used (always 0). See vm_version.cpp lines 100-102 where only major, > > minor and build number are ever been set. Knowing this, we can still > > preserve the same behavior after patch by defining JVM_VERSION_MICRO to > > 0 for any version. > > This is tricky. JVM_GetVersionInfo is a function exported by > libjvm.so, and the version is simply encoded as > > unsigned int Abstract_VM_Version::jvm_version() { > return ((Abstract_VM_Version::vm_major_version() & 0xFF) << 24) | > ((Abstract_VM_Version::vm_minor_version() & 0xFF) << 16) | > (Abstract_VM_Version::vm_build_number() & 0xFF); > } > > I guess we could argue that this is for JVM-JDK internal use only, and > no-one else cares. > > Or we could encode it in a different way such that at least we have a > jvm_version that is monotonically increasing, perhaps by (ab)using the > lower 8 bits of the version, the vm_build_number. But I guess I'm > being paranoid, and no tools are going to care about minor versions > anyway,even if they do call JVM_GetVersionInfo. Yes, this is indeed tricky. The trouble is that Abstract_VM_Version::vm_build_number() actually holds the configured JDK build number, we can't really use it. It's already being used. (gdb) 99 100 _vm_major_version = atoi(vm_major_ver); 101 _vm_minor_version = atoi(vm_minor_ver); 102 _vm_build_number = atoi(vm_build_num); 103 104 os::free(vm_version); 105 _initialized = true; 106 } 107 108 #if defined(_LP64) (gdb) p _vm_build_number $1 = 2 The only bits which seem unused are bits 8 through 16 of this unsigned int. So even if JVM_GetVersionInfo *is* being used somewhere in the wild, it *should* continue to work. Hence, the proposal to now also use those unused bits for the minor version. Thoughts? Thanks, Severin From ioi.lam at oracle.com Tue May 12 15:42:33 2020 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 12 May 2020 08:42:33 -0700 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> Message-ID: On 5/12/20 5:41 AM, Thomas St?fe wrote: > > > On Tue, May 12, 2020 at 12:01 AM Ioi Lam > wrote: > > > > On 5/8/20 12:33 AM, Thomas St?fe wrote: >> Hi Ioi, >> >> thanks for taking the time to look at this! >> >> On Fri, May 8, 2020 at 7:37 AM Ioi Lam > > wrote: >> >> Hi Thomas, >> >> I am running your patch in out CI pipeline now. Some comments: >> >> [1] Do we still need to use _class_space_list? It looks like >> we just have a single range. Is >> _class_space_list->current_virtual_space() always the same VS? >> >> ? if (_class_space_list != NULL) { >> ??? address base = >> (address)_class_space_list->current_virtual_space()->bottom(); >> ??? address top = base + compressed_class_space_size(); >> ??? st->print("Compressed class space mapped at: " PTR_FORMAT >> "-" PTR_FORMAT ", size: " SIZE_FORMAT, >> ?????????????? p2i(base), p2i(top), top - base); >> >> >> Yes it is. Has been bugging me for years too. With ccs, the >> VirtualSpaceList is degenerated and only contains one node. It >> still makes some sense code wise. But maybe VirtualSpace*List* is >> a misnomer. >> >> It also ties in into the question whether Metaspace reservations >> should be able to grow on demand. Oracle folks I talk to tend to >> say yes. I personally think the "Metaspace is infinite don't >> worry" meme was broken since ccs was a thing, since >> CompressedClassSpaceSize is an absolute limit already. >> >> I may tweak and simplify the code a bit with Elastic Metaspace. >> Maybe, as you say, with ccs we can get rid of VirtualSpaceList >> and deal with the range directly. Have to check. >> >> [2] Does JDK-8243535 exist with the current jdk/jdk repo? >> >> ? // This does currently not work because rs may be the >> result of a split operation >> ? // and NMT seems not to be able to handle splits. >> ? // See JDK-8243535. >> ? // >> MemTracker::record_virtual_memory_type((address)rs.base(), >> mtClass); >> >> >> Yes. It shows up more rarely. With this patch, it will show up >> always. I had a conversation with Zhengyu about this. I think >> there may be workarounds, like removing the original mapping in >> NMT and replacing it with the two new ones for archive and ccs. >> But it seems iffy, I'd rather have this fixed in NMT. >> >> I also think it is not such a big deal, but if you think >> otherwise, I can integrate a workaround into this patch. > > My question is -- will NMT (or control-backspace) now always > report incorrect memory size for the metaspace when CDS is > enabled? If so, I think a work-around is necessary. > > > What is control backspace? I'm staring at '^H' output in my terminal > and feel like I miss something :) If you have a running foreground JVM in your Linux terminal, you can type "Ctrl-\" and see this. E.g., $ java -cp . SleepForever <...> Heap ?garbage-first heap?? total 1048576K, used 824K [0x0000000412000000, 0x0000000800000000) ? region size 8192K, 1 young (8192K), 0 survivors (0K) ?Metaspace?????? used 143K, capacity 4486K, committed 4864K, reserved 1056768K ? class space??? used 6K, capacity 386K, committed 512K, reserved 1048576K Actually I see that the current output is not correct -- class space doesn't include the CDS sizes :-( I am not sure how many people use this anyway, so it may not be important. Thanks - Ioi > > Today NMT reports wrong memory size at dumptime (I think no-one cares > probably) and at runtime if the archive got relocated. With the patch > NMT will also report wrong numbers at runtime when mapping to default > address. > > But okay, I will add a workaround. > >> >> [3] I think this can be put in header file. >> >> bool Metaspace::class_space_is_initialized() { >> ? return _class_space_list != NULL; >> } >> >> >> Will do. >> >> [4] Why does the CCS need to follow UseLargePagesInMetaspace? >> This is the reason for the >> ??? gap in the following chart: >> >> ??? // We do this by reserving space for the ccs behind the >> archives. Note however that >> ??? //? ccs follows a different alignment >> (Metaspace::reserve_alignment), so there may >> ??? //? be a gap between ccs and cds. >> ??? // We use a similar layout at runtime, see >> reserve_address_space_for_archives(). >> ??? // >> ??? //????????????????????????????? +-- SharedBaseAddress >> (default = 0x800000000) >> ??? //????????????????????????????? v >> ??? // +-..---------+---------+ ... >> +----+----+----+--------+--------------------+ >> ??? // |??? Heap??? | Archive |???? | MC | RW | RO | [gap]? >> |??? class space???? | >> ??? // +-..---------+---------+ ... >> +----+----+----+--------+--------------------+ >> ??? // |<--?? MaxHeapSize? -->| |<-- UnscaledClassSpaceMax = >> 4GB -->| >> >> >> No, nothing to do with large pages. CCS does not have large pages. >> >> The gap can result from different alignment rules between >> metaspace and cds. With Elastic Metaspace, metaspace reservations >> will be aligned to 4M. So, ccs will have to start at a 4M aligned >> border. >> >> >> _reserve_alignment is determined here: >> >> void Metaspace::ergo_initialize() { >> ? if (DumpSharedSpaces) { >> ??? // Using large pages when dumping the shared archive is >> currently not implemented. >> ??? FLAG_SET_ERGO(UseLargePagesInMetaspace, false); >> ? } >> >> ? size_t page_size = os::vm_page_size(); >> ? if (UseLargePages && UseLargePagesInMetaspace) { >> ??? page_size = os::large_page_size(); >> ? } >> >> ? _commit_alignment? = page_size; >> ? _reserve_alignment = MAX2(page_size, >> (size_t)os::vm_allocation_granularity()); >> >> But when CDS is enabled, the RS is reserved without large >> pages, so it should be possible to incrementally commit the >> CCS using just os::vm_allocation_granularity(). >> >> >> Absolutely. CCS does not use large pages for that reason. As I >> wrote, this is about the new reservation granularity of 4M. > > OK, if I understand correctly: > > + With the current patch, the gap exists only when > UseLargePagesInMetaspace is used at run time. The gap actually is > unnecessary, since ccs doesn't use large pages. > > + With the upcoming Elastic Metaspace, _reserve_alignment will be > 4MB, so we will almost always have a gap. > > > Yes. > > Note that we decided to remove the large page feature from Metaspace > with Elastic Metaspace for now, though we may reintroduce it later in > a?different form.?See CSR: > https://bugs.openjdk.java.net/browse/JDK-8243161?. > > >> [5] I see the aarch64 code in >> Metaspace::reserve_address_space_for_compressed_classes has >> been changed. Is this necessary for this RFE, or is it a >> separate improvement that can be done in a different RFE? >> >> >> Well, it had to be changed somewhat, since the prototype differs. >> I swapped Metaspace::reserve_preferred_space() for >> Metaspace::reserve_address_space_for_compressed_classes(), which >> follows a somewhat different protocol. >> >> That aside, the functionality for AARCH64 should be as >> unchanged?as possible. I removed the AIX specific coding, since >> it was subtly?wrong, but that's fine since we are AIX/ppc >> maintainers. >> >> >> [6] For AARCH64, should we skip the part that reserves it >> right above the heap? Or does AARCH64 always allocate the >> heap such that it's a preferable address? >> >> >> Yes and yes :) >> >> We do skip that for aarch64 (at least if the address is wrong): >> >> 1157 // Try right above the Java heap... 1158 address base = >> align_up(CompressedOops::end(), Metaspace::reserve_alignment()); >> 1159 assert(base != NULL, "Sanity"); 1160 1161 const size_t size >> = align_up(CompressedClassSpaceSize, >> Metaspace::reserve_alignment()); 1162 if >> (CompressedKlassPointers::is_valid_base(base)) { 1163 rs = >> ReservedSpace(size, Metaspace::reserve_alignment(), false /* >> large */, (char*)base); 1164 } >> At 1162, we check if the attach address would be valid for this >> platform. If aarch64 says no, we won't reserve there but go >> straight to the "attach wherever you think is good" part. Which >> is implemented >> by?Metaspace::reserve_address_space_for_compressed_classes(), >> which on aarch64 does search a suitable address. >> >> ??? // case (b) >> ??? ReservedSpace rs; >> >> ??? // Try right above the Java heap... >> ??? address base = align_up(CompressedOops::end(), >> Metaspace::reserve_alignment()); >> ??? assert(base != NULL, "Sanity"); >> >> ??? const size_t size = align_up(CompressedClassSpaceSize, >> Metaspace::reserve_alignment()); >> ??? if (CompressedKlassPointers::is_valid_base(base)) { >> ????? rs = ReservedSpace(size, >> Metaspace::reserve_alignment(), false /* large */, (char*)base); >> ??? } >> >> ??? // ...failing that, reserve anywhere, but let platform do >> optimized placement: >> ??? if (!rs.is_reserved()) { >> ????? rs = >> Metaspace::reserve_address_space_for_compressed_classes(size); >> ??? } >> >> [7] argument.cpp checks this: >> >> ? if (UseSharedSpaces || DumpSharedSpaces) { >> ??? if >> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) >> { >> ????? log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is >> invalid for this platform, option will be ignored.", >> p2i((address)SharedBaseAddress)); >> ????? FLAG_SET_DEFAULT(SharedBaseAddress, >> default_SharedBaseAddress()); >> ??? } >> ? } >> >> but initialize_dumptime_shared_and_meta_spaces aligns up the >> SharedBaseAddress before doing the assert. >> >> ? void >> MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >> ??? assert(DumpSharedSpaces, "should be called for dump time >> only"); >> >> ??? const size_t reserve_alignment = >> MetaspaceShared::reserved_space_alignment(); >> ??? char* shared_base = >> (char*)align_up((char*)SharedBaseAddress, reserve_alignment); >> >> ? #ifdef _LP64 >> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >> "Sanity"); >> >> So theoretically shared_base may no longer be is_valid_base >> after the align_up. It probably won't happen, but I think the >> code will be much clearer tofirst to align_up, then check for >> is_valid_base and reset to default, and finally assert. >> >> >> Okay, I'll change that. >> >> [8] In >> MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, >> the following block can be simplified: >> >> ? if (!_shared_rs.is_reserved()) { >> ??? // Get a reserved space anywhere if attaching at the >> SharedBaseAddress fails: >> ??? if (UseCompressedClassPointers) { >> ????? // If we need a compressed class space too, let the >> platform handle the reservation. >> ????? _shared_rs = >> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >> ??? } else { >> ????? // anywhere is fine. >> ????? _shared_rs = ReservedSpace(cds_total, >> reserve_alignment, false /* large */, (char*)NULL); >> ??? } >> ? } >> >> ?? ... if you change the declaration to: >> >> ? static ReservedSpace >> reserve_address_space_for_compressed_classes(size_t size) >> NOT_LP64({ return ReservedSpace();}); >> >> [9] I think the #ifdef _LP64 is not necessary: >> >> #ifdef _LP64 >> ??? // Some sanity checks after reserving address spaces for >> archives and class space. >> ??? assert(archive_space_rs.is_reserved(), "Sanity"); >> ??? if (Metaspace::using_class_space()) { >> ????? // Class space must closely follow the archive space. >> Both spaces must be aligned correctly. >> ????? assert(class_space_rs.is_reserved(), "A class space >> should have been reserved"); >> ????? assert(class_space_rs.base() >= archive_space_rs.end(), >> "class space should follow the cds archive space"); >> assert(is_aligned(archive_space_rs.base(), >> MetaspaceShared::reserved_space_alignment()), "Archive space >> misaligned"); >> ????? assert(is_aligned(class_space_rs.base(), >> Metaspace::reserve_alignment()), "class space misaligned"); >> ??? } >> #endif >> >> >> I'll take a look. >> >> The rest of the code looks OK to me, but I may take a look at >> it again after getting more sleep :-) >> >> >> Sure. Thanks man! :) > > Good news. I ran our tiers 1-4 and there were no failures related > to your changes! > > > Neat. This mirrors our results. > > I'll prepare a new webrev. > > ..Thomas From bob.vandette at oracle.com Tue May 12 18:37:33 2020 From: bob.vandette at oracle.com (Bob Vandette) Date: Tue, 12 May 2020 14:37:33 -0400 Subject: RFR: 8244852 GraalVM native-image fails after JDK-8238048 change Message-ID: Please review this fix for a regression in JDK 15 causing native-image to fail. https://bugs.openjdk.java.net/browse/JDK-8244852 diff --git a/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java b/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java --- a/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java +++ b/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java @@ -818,7 +818,7 @@ @Override public String getSourceFileName() { if (isArray()) { - throw new JVMCIError("Cannot call getSourceFileName() on an array klass type: %s", this); + return (String)null; } return getConstantPool().getSourceFileName(); } Bob. From david.holmes at oracle.com Wed May 13 04:37:34 2020 From: david.holmes at oracle.com (David Holmes) Date: Wed, 13 May 2020 14:37:34 +1000 Subject: RFR: 8244852 GraalVM native-image fails after JDK-8238048 change In-Reply-To: References: Message-ID: Hi Bob, On 13/05/2020 4:37 am, Bob Vandette wrote: > Please review this fix for a regression in JDK 15 causing native-image to fail. > > https://bugs.openjdk.java.net/browse/JDK-8244852 > > > diff --git a/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java b/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java > --- a/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java > +++ b/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java > @@ -818,7 +818,7 @@ > @Override > public String getSourceFileName() { > if (isArray()) { > - throw new JVMCIError("Cannot call getSourceFileName() on an array klass type: %s", this); > + return (String)null; Fix seems reasonable. While JavaType.getSourceFileName() seems a little underspecified, JavaType includes array types so the call should be valid. Nit: No cast needed on the null. Thanks, David > } > return getConstantPool().getSourceFileName(); > } > > > Bob. > From thomas.stuefe at gmail.com Wed May 13 05:40:07 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 13 May 2020 07:40:07 +0200 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: References: <56a0cf85-9698-64b2-95b4-e5b123c3e9e2@oracle.com> <9a314ebe-5499-ecb6-3c5b-0c8ce7f9c8d2@oracle.com> <44febab6-5703-1c8b-d4ad-0f4c12517afa@oracle.com> Message-ID: Hi Erik, note that I am changing the CompressClassPointers too as part of "8243392: Remodel CDS/Metaspace storage reservation" which is out for review. That is not a big deal, if you push before me (which I assume will happen) I'll merge. But look at my changes: http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/test/hotspot/jtreg/runtime/CompressedOops/CompressedClassPointers.java.udiff.html What I did was to explicitly disable placement checks on selected platforms - windows and, for us, ppc. So the tests run in two modes, one only testing that the VM does come up with the given parameters, one more strict where we test ccs placement. Feel free to integrate that into your change. However, that said, I am fine with your change as it is. I would merge your changes into my change then. Cheers Thomas On Tue, May 12, 2020 at 4:10 PM Erik ?sterlund wrote: > Hi, > > I ran into some merge conflicts in the tests and the new tests behaved a > bit differently > on different platforms. The new tests naturally do not behave the same > way on the platforms that do > not yet support compressed class pointers without compressed oops, so I > just filtered away those > platforms from running the new tests. Will update the filter when they > start supporting it. > > Another interesting thing is that on macOS, sometimes we did not get the > expected compressed class > pointer encoding without compressed oops. The reason is that without > compressed oops, we allocate > the heap at some random address where the OS thinks is suitable. > Sometimes that is where we would > ideally have put the compressed class space. This forces metaspace to be > mapped in somewhere else > with different characteristics. Ideally, the bootstrapping logic should > be disentangled so that > when you run without compressed oops, metaspace gets reserved before the > heap. But I think that > warrants a follow-up RFE, as I do not want to complicate this patch any > further. Disentangling that > seems like it could become quite tricky on its own. > > So here is another webrev with the test issues resolved: > http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ > > Incremental: > http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ > > Thanks, > /Erik > > On 2020-04-29 09:39, Erik ?sterlund wrote: > > Hi Thomas, > > > > On 2020-04-29 08:27, Thomas St?fe wrote: > >> Hi Erik, > >> > >> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund > >> > wrote: > >> > >> Hi Thomas, > >> > >> Thanks for reviewing this. > >> > >> On 2020-04-28 13:14, Thomas St?fe wrote: > >> > >>> With compressed oops we continue to squeeze ccs in after the heap > >>> in the hope that it all still fits into the lower 4G or 32G. (Do > >>> we ever bother to check if heap is actually reserved in low area? > >>> It may have given up and reserved upstate instead, leaving free > >>> room below...). > >> > >> Right. And yes I'm pretty sure we do check. The heap reserves > >> memory, and sets the CompressedOops::end() based on that, which is > >> fed as input to the metaspace/CDS placement. > >> > >> > >> No, what I meant was: since we reserve at CompressedOops::end(), if > >> the heap was too large to place it in the lower address regions, the > >> class space will not live there either even if it could, since it is > >> smaller and might fit. > >> > >> For example: > >> > >> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ > >> ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace -Xshare:off > >> -Xmx31g > >> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: > >> reserved=1048576.00 KB, committed=0.00 KB ( 0%), used=0.00 KB ( 0%) > >> [0.003s][trace][gc,metaspace] [0x00000017c0400000, > >> 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) > >> [0.003s][trace][gc,metaspace] Narrow klass base: 0x00000017c0400000, > >> Narrow klass shift: 0 > >> [0.003s][trace][gc,metaspace] Compressed class space size: 1073741824 > >> Address: 0x00000017c0400000 Req Addr: 0x00000017c0400000 > >> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: > >> reserved=8192.00 KB, committed=0.00 KB ( 0%), used=0.00 KB ( 0%) > >> [0.003s][trace][gc,metaspace] [0x00007fc9cb800000, > >> 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) > >> > >> (large heap and no CDS, since it prevents zero based mode). > >> > >> In that case narrow klass base is not zero based even it if probably > >> could be, if we were to allocate class space in the lower regions, > >> which are now unoccupied since the large heap does not fit there. A > >> missed opportunity to go zero based. Not a big deal, since we mostly > >> run with CDS now anyway. > > > > I see. Yeah, there is room for improvement there. Although, as you > > say, people mostly run with CDS on, so it probably won't matter even > > if that is improved. > > > >>> Now with your patch without compressed oops we attempt to map at > >>> HeapBaseMinAddress since this is the lowest address we are > >>> allowed to map at and the heap won't obviously need it, right? > >> > >> Exactly. > >> > >>> My only misgiving would be that HeapBaseMinAddress is even more > >>> of a misnomer (its already now a bit, witness my year long > >>> confusion :-), since it now controls not only heap placement but > >>> ccs too. Actually if it is that important to avoid mapping to > >>> addresses below HeapBaseMinAddress we should probably prevent it > >>> somewhere below in the os layer but I really do not want to open > >>> that can of worms. > >> > >> Yes agreed. Ideally it would be called something like > >> MappingMinAddress instead, because it really isn't tied to the > >> heap really, it just happened to be the primary user of mapping > >> things at low addresses. But I also think changing that, and > >> enforcing it more widely in the os layer sounds like a different > >> can of worms. > >> > >>> -- > >>> > >>> Some more remarks about arguments.cpp: > >>> > >>> - In Arguments::set_use_compressed_klass_ptrs(), if now > >>> compressed class pointers are independent from compressed oops, > >>> checking CompressedClassSpaceSize for validity should now always > >>> happen for UseCompressedClassPointers=1. > >> > >> I think it looks like I do that. > >> > >>> - If HeapBaseMinAddress is now always used, we should check it > >>> always too (see Arguments::set_heap_size) > >> > >> Good idea. Fixed. > >> > >>> > >>>> Also, could we have some tests which exercise cds/metaspace > >>>> initialization without coops and with ccps? Easiest way > >>>> would be to extend > >>>> "CompressedOops/CompressedClassPointers.java" by some cases. > >>> > >>> > >>> Thank you. I think this makes sense especially since compressed > >>> class pointer encoding is restricted to one mode if CDS is on > >>> (base = shared address and shift=3), and since CDS=on is the > >>> default zerobased encoding may not happen that often. > >> > >> Yup. I added some explicit test cases for -XX:-UseCompressedOops > >> -XX:+UseCompressedClassPointers as suggested. This will also be > >> the default configuration for ZGC, so that should give that > >> configuration more test coverage in the various tests we have. > >> > >> Here is a new webrev: > >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ > >> > >> Incremental: > >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ > >> > >> I hope I fixed everything you wanted in this patch. I also fixed > >> what Frederic and Coleen wanted. > >> > >> > >> arguments.cpp, the metaspace stuff and the test look good to me now. > >> Thank you for your work! > > > > Thanks for the review Thomas! > > > > /Erik > > From thomas.stuefe at gmail.com Wed May 13 05:42:23 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 13 May 2020 07:42:23 +0200 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: References: <56a0cf85-9698-64b2-95b4-e5b123c3e9e2@oracle.com> <9a314ebe-5499-ecb6-3c5b-0c8ce7f9c8d2@oracle.com> <44febab6-5703-1c8b-d4ad-0f4c12517afa@oracle.com> Message-ID: p.s. I also think that we can rework this test and ccs placement in general at a later follow up rfe. ccs placement strategy is not optimal but that would be out of scope for both your change and my cds rework patch, which is supposed to be mainly code shuffling around. ..Thomas On Wed, May 13, 2020 at 7:40 AM Thomas St?fe wrote: > Hi Erik, > > note that I am changing the CompressClassPointers too as part of "8243392: > Remodel CDS/Metaspace storage reservation" which is out for review. That is > not a big deal, if you push before me (which I assume will happen) I'll > merge. But look at my changes: > > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/test/hotspot/jtreg/runtime/CompressedOops/CompressedClassPointers.java.udiff.html > > What I did was to explicitly disable placement checks on selected > platforms - windows and, for us, ppc. So the tests run in two modes, one > only testing that the VM does come up with the given parameters, one more > strict where we test ccs placement. Feel free to integrate that into your > change. > > However, that said, I am fine with your change as it is. I would merge > your changes into my change then. > > Cheers Thomas > > > > On Tue, May 12, 2020 at 4:10 PM Erik ?sterlund > wrote: > >> Hi, >> >> I ran into some merge conflicts in the tests and the new tests behaved a >> bit differently >> on different platforms. The new tests naturally do not behave the same >> way on the platforms that do >> not yet support compressed class pointers without compressed oops, so I >> just filtered away those >> platforms from running the new tests. Will update the filter when they >> start supporting it. >> >> Another interesting thing is that on macOS, sometimes we did not get the >> expected compressed class >> pointer encoding without compressed oops. The reason is that without >> compressed oops, we allocate >> the heap at some random address where the OS thinks is suitable. >> Sometimes that is where we would >> ideally have put the compressed class space. This forces metaspace to be >> mapped in somewhere else >> with different characteristics. Ideally, the bootstrapping logic should >> be disentangled so that >> when you run without compressed oops, metaspace gets reserved before the >> heap. But I think that >> warrants a follow-up RFE, as I do not want to complicate this patch any >> further. Disentangling that >> seems like it could become quite tricky on its own. >> >> So here is another webrev with the test issues resolved: >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ >> >> Incremental: >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ >> >> Thanks, >> /Erik >> >> On 2020-04-29 09:39, Erik ?sterlund wrote: >> > Hi Thomas, >> > >> > On 2020-04-29 08:27, Thomas St?fe wrote: >> >> Hi Erik, >> >> >> >> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund >> >> > wrote: >> >> >> >> Hi Thomas, >> >> >> >> Thanks for reviewing this. >> >> >> >> On 2020-04-28 13:14, Thomas St?fe wrote: >> >> >> >>> With compressed oops we continue to squeeze ccs in after the heap >> >>> in the hope that it all still fits into the lower 4G or 32G. (Do >> >>> we ever bother to check if heap is actually reserved in low area? >> >>> It may have given up and reserved upstate instead, leaving free >> >>> room below...). >> >> >> >> Right. And yes I'm pretty sure we do check. The heap reserves >> >> memory, and sets the CompressedOops::end() based on that, which is >> >> fed as input to the metaspace/CDS placement. >> >> >> >> >> >> No, what I meant was: since we reserve at CompressedOops::end(), if >> >> the heap was too large to place it in the lower address regions, the >> >> class space will not live there either even if it could, since it is >> >> smaller and might fit. >> >> >> >> For example: >> >> >> >> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ >> >> ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace -Xshare:off >> >> -Xmx31g >> >> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: >> >> reserved=1048576.00 KB, committed=0.00 KB ( 0%), used=0.00 KB ( 0%) >> >> [0.003s][trace][gc,metaspace] [0x00000017c0400000, >> >> 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) >> >> [0.003s][trace][gc,metaspace] Narrow klass base: 0x00000017c0400000, >> >> Narrow klass shift: 0 >> >> [0.003s][trace][gc,metaspace] Compressed class space size: 1073741824 >> >> Address: 0x00000017c0400000 Req Addr: 0x00000017c0400000 >> >> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: >> >> reserved=8192.00 KB, committed=0.00 KB ( 0%), used=0.00 KB ( 0%) >> >> [0.003s][trace][gc,metaspace] [0x00007fc9cb800000, >> >> 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) >> >> >> >> (large heap and no CDS, since it prevents zero based mode). >> >> >> >> In that case narrow klass base is not zero based even it if probably >> >> could be, if we were to allocate class space in the lower regions, >> >> which are now unoccupied since the large heap does not fit there. A >> >> missed opportunity to go zero based. Not a big deal, since we mostly >> >> run with CDS now anyway. >> > >> > I see. Yeah, there is room for improvement there. Although, as you >> > say, people mostly run with CDS on, so it probably won't matter even >> > if that is improved. >> > >> >>> Now with your patch without compressed oops we attempt to map at >> >>> HeapBaseMinAddress since this is the lowest address we are >> >>> allowed to map at and the heap won't obviously need it, right? >> >> >> >> Exactly. >> >> >> >>> My only misgiving would be that HeapBaseMinAddress is even more >> >>> of a misnomer (its already now a bit, witness my year long >> >>> confusion :-), since it now controls not only heap placement but >> >>> ccs too. Actually if it is that important to avoid mapping to >> >>> addresses below HeapBaseMinAddress we should probably prevent it >> >>> somewhere below in the os layer but I really do not want to open >> >>> that can of worms. >> >> >> >> Yes agreed. Ideally it would be called something like >> >> MappingMinAddress instead, because it really isn't tied to the >> >> heap really, it just happened to be the primary user of mapping >> >> things at low addresses. But I also think changing that, and >> >> enforcing it more widely in the os layer sounds like a different >> >> can of worms. >> >> >> >>> -- >> >>> >> >>> Some more remarks about arguments.cpp: >> >>> >> >>> - In Arguments::set_use_compressed_klass_ptrs(), if now >> >>> compressed class pointers are independent from compressed oops, >> >>> checking CompressedClassSpaceSize for validity should now always >> >>> happen for UseCompressedClassPointers=1. >> >> >> >> I think it looks like I do that. >> >> >> >>> - If HeapBaseMinAddress is now always used, we should check it >> >>> always too (see Arguments::set_heap_size) >> >> >> >> Good idea. Fixed. >> >> >> >>> >> >>>> Also, could we have some tests which exercise cds/metaspace >> >>>> initialization without coops and with ccps? Easiest way >> >>>> would be to extend >> >>>> "CompressedOops/CompressedClassPointers.java" by some cases. >> >>> >> >>> >> >>> Thank you. I think this makes sense especially since compressed >> >>> class pointer encoding is restricted to one mode if CDS is on >> >>> (base = shared address and shift=3), and since CDS=on is the >> >>> default zerobased encoding may not happen that often. >> >> >> >> Yup. I added some explicit test cases for -XX:-UseCompressedOops >> >> -XX:+UseCompressedClassPointers as suggested. This will also be >> >> the default configuration for ZGC, so that should give that >> >> configuration more test coverage in the various tests we have. >> >> >> >> Here is a new webrev: >> >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ >> >> >> >> Incremental: >> >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ >> >> >> >> I hope I fixed everything you wanted in this patch. I also fixed >> >> what Frederic and Coleen wanted. >> >> >> >> >> >> arguments.cpp, the metaspace stuff and the test look good to me now. >> >> Thank you for your work! >> > >> > Thanks for the review Thomas! >> > >> > /Erik >> >> From david.holmes at oracle.com Wed May 13 05:42:50 2020 From: david.holmes at oracle.com (David Holmes) Date: Wed, 13 May 2020 15:42:50 +1000 Subject: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant In-Reply-To: References: <410eed04-e2ef-0f4f-1c56-19e6734a10f6@oracle.com> <81d7caa8-4244-85f3-4d4e-78117fe5e25b@oss.nttdata.com> <550b95ac-8b29-1eb8-a507-533e81d02322@oracle.com> <9c49ea2d-e3b8-b576-1d17-d18ad87cd6ed@oracle.com> Message-ID: <36d5e2c0-c724-7ff7-d37e-decb5cc0005b@oracle.com> On 4/05/2020 8:33 pm, Reingruber, Richard wrote: > // Trimmed the list of recipients. If the list gets too long then the message needs to be approved > // by a moderator. Yes I noticed that too :) In general if you send to hotspot-dev you shouldn't need to also send to hotspot-X-dev. > Hi David, Hi Richard, >> On 28/04/2020 12:09 am, Reingruber, Richard wrote: >>> Hi David, >>> >>>> Not a review but some general commentary ... >>> >>> That's welcome. > >> Having had to take an even closer look now I have a review comment too :) > >> src/hotspot/share/prims/jvmtiThreadState.cpp > >> void JvmtiThreadState::invalidate_cur_stack_depth() { >> ! assert(SafepointSynchronize::is_at_safepoint() || >> ! (Thread::current()->is_VM_thread() && >> get_thread()->is_vmthread_processing_handshake()) || >> (JavaThread *)Thread::current() == get_thread(), >> "must be current thread or at safepoint"); > > You're looking at an outdated webrev, I'm afraid. > > This would be the post with the current webrev.1 > > http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-April/031245.html Sorry I missed that update. Okay so this is working with direct handshakes now. One style nit in jvmtiThreadState.cpp: assert(SafepointSynchronize::is_at_safepoint() || ! (JavaThread *)Thread::current() == get_thread() || ! Thread::current() == get_thread()->active_handshaker(), ! "bad synchronization with owner thread"); the ! lines should ident as follows assert(SafepointSynchronize::is_at_safepoint() || (JavaThread *)Thread::current() == get_thread() || Thread::current() == get_thread()->active_handshaker(), ! "bad synchronization with owner thread"); Lets see how this plays out. Cheers, David > Thanks, Richard. > > -----Original Message----- > From: David Holmes > Sent: Montag, 4. Mai 2020 08:51 > To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net > Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant > > Hi Richard, > > On 28/04/2020 12:09 am, Reingruber, Richard wrote: >> Hi David, >> >>> Not a review but some general commentary ... >> >> That's welcome. > > Having had to take an even closer look now I have a review comment too :) > > src/hotspot/share/prims/jvmtiThreadState.cpp > > void JvmtiThreadState::invalidate_cur_stack_depth() { > ! assert(SafepointSynchronize::is_at_safepoint() || > ! (Thread::current()->is_VM_thread() && > get_thread()->is_vmthread_processing_handshake()) || > (JavaThread *)Thread::current() == get_thread(), > "must be current thread or at safepoint"); > > The message needs updating to include handshakes. > > More below ... > >>> On 25/04/2020 2:08 am, Reingruber, Richard wrote: >>>> Hi Yasumasa, Patricio, >>>> >>>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>>> Does it help you? I think it gives you to remove workaround. >>>>>> >>>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>>> >>>>> Thanks for your information. >>>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>>> I will modify and will test it after yours. >>>> >>>> Thanks :) >>>> >>>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>>> to me, how this has to be handled. >>>> >>>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>>> >>>> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >>>> also I'm unsure if a thread should do safepoint checks while executing a handshake. >> >>> I'm growing increasingly concerned that use of direct handshakes to >>> replace VM operations needs a much greater examination for correctness >>> than might initially be thought. I see a number of issues: >> >> I agree. I'll address your concerns in the context of this review thread for JDK-8238585 below. >> >> In addition I would suggest to take the general part of the discussion to a dedicated thread or to >> the review thread for JDK-8242427. I would like to keep this thread closer to its subject. > > I will focus on the issues in the context of this particular change > then, though the issues themselves are applicable to all handshake > situations (and more so with direct handshakes). This is mostly just > discussion. > >>> First, the VMThread executes (most) VM operations with a clean stack in >>> a clean state, so it has lots of room to work. If we now execute the >>> same logic in a JavaThread then we risk hitting stackoverflows if >>> nothing else. But we are also now executing code in a JavaThread and so >>> we have to be sure that code is not going to act differently (in a bad >>> way) if executed by a JavaThread rather than the VMThread. For example, >>> may it be possible that if executing in the VMThread we defer some >>> activity that might require execution of Java code, or else hand it off >>> to one of the service threads? If we execute that code directly in the >>> current JavaThread instead we may not be in a valid state (e.g. consider >>> re-entrancy to various subsystems that is not allowed). >> >> It is not too complex, what EnterInterpOnlyModeClosure::do_thread() is doing. I already added a >> paragraph to the JBS-Item [1] explaining why the direct handshake is sufficient from a >> synchronization point of view. > > Just to be clear, your proposed change is not using a direct handshake. > >> Furthermore the stack is walked and the return pc of compiled frames is replaced with the address of >> the deopt handler. >> >> I can't see why this cannot be done with a direct handshake. Something very similar is already done >> in JavaThread::deoptimize_marked_methods() which is executed as part of an ordinary handshake. > > Note that existing non-direct handshakes may also have issues that not > have been fully investigated. > >> The demand on stack-space should be very modest. I would not expect a higher risk for stackoverflow. > > For the target thread if you use more stack than would be used stopping > at a safepoint then you are at risk. For the thread initiating the > direct handshake if you use more stack than would be used enqueuing a VM > operation, then you are at risk. As we have not quantified these > numbers, nor have any easy way to establish the stack use of the actual > code to be executed, we're really just hoping for the best. This is a > general problem with handshakes that needs to be investigated more > deeply. As a simple, general, example just imagine if the code involves > logging that might utilise an on-stack buffer. > >>> Second, we have this question mark over what happens if the operation >>> hits further safepoint or handshake polls/checks? Are there constraints >>> on what is allowed here? How can we recognise this problem may exist and >>> so deal with it? >> >> The thread in EnterInterpOnlyModeClosure::do_thread() can't become safepoint/handshake safe. I >> tested locally test/hotspot/jtreg:vmTestbase_nsk_jvmti with a NoSafepointVerifier. > > That's good to hear but such tests are not exhaustive, they will detect > if you do reach a safepoint/handshake but they can't prove that you > cannot reach one. What you have done is necessary but may not be > sufficient. Plus you didn't actually add the NSV to the code - is there > a reason we can't actually keep it in do_thread? (I'm not sure if the > NSV also acts as a NoHandshakeVerifier?) > >>> Third, while we are generally considering what appear to be >>> single-thread operations, which should be amenable to a direct >>> handshake, we also have to be careful that some of the code involved >>> doesn't already expect/assume we are at a safepoint - e.g. a VM op may >>> not need to take a lock where a direct handshake might! >> >> See again my arguments in the JBS item [1]. > > Yes I see the reasoning and that is good. My point is a general one as > it may not be obvious when such assumptions exist in the current code. > > Thanks, > David > >> Thanks, >> Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8238585 >> >> -----Original Message----- >> From: David Holmes >> Sent: Montag, 27. April 2020 07:16 >> To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >> >> Hi all, >> >> Not a review but some general commentary ... >> >> On 25/04/2020 2:08 am, Reingruber, Richard wrote: >>> Hi Yasumasa, Patricio, >>> >>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>> Does it help you? I think it gives you to remove workaround. >>>>> >>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>> >>>> Thanks for your information. >>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>> I will modify and will test it after yours. >>> >>> Thanks :) >>> >>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>> to me, how this has to be handled. >>> >>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>> >>> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >>> also I'm unsure if a thread should do safepoint checks while executing a handshake. >> >> I'm growing increasingly concerned that use of direct handshakes to >> replace VM operations needs a much greater examination for correctness >> than might initially be thought. I see a number of issues: >> >> First, the VMThread executes (most) VM operations with a clean stack in >> a clean state, so it has lots of room to work. If we now execute the >> same logic in a JavaThread then we risk hitting stackoverflows if >> nothing else. But we are also now executing code in a JavaThread and so >> we have to be sure that code is not going to act differently (in a bad >> way) if executed by a JavaThread rather than the VMThread. For example, >> may it be possible that if executing in the VMThread we defer some >> activity that might require execution of Java code, or else hand it off >> to one of the service threads? If we execute that code directly in the >> current JavaThread instead we may not be in a valid state (e.g. consider >> re-entrancy to various subsystems that is not allowed). >> >> Second, we have this question mark over what happens if the operation >> hits further safepoint or handshake polls/checks? Are there constraints >> on what is allowed here? How can we recognise this problem may exist and >> so deal with it? >> >> Third, while we are generally considering what appear to be >> single-thread operations, which should be amenable to a direct >> handshake, we also have to be careful that some of the code involved >> doesn't already expect/assume we are at a safepoint - e.g. a VM op may >> not need to take a lock where a direct handshake might! >> >> Cheers, >> David >> ----- >> >>> @Patricio, coming back to my question [1]: >>> >>> In the example you gave in your answer [2]: the java thread would execute a vm operation during a >>> direct handshake operation, while the VMThread is actually in the middle of a VM_HandshakeAllThreads >>> operation, waiting to handshake the same handshakee: why can't the VMThread just proceed? The >>> handshakee would be safepoint safe, wouldn't it? >>> >>> Thanks, Richard. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301677&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301677 >>> >>> [2] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301763&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301763 >>> >>> -----Original Message----- >>> From: Yasumasa Suenaga >>> Sent: Freitag, 24. April 2020 17:23 >>> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>> >>> Hi Richard, >>> >>> On 2020/04/24 23:44, Reingruber, Richard wrote: >>>> Hi Yasumasa, >>>> >>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>> Does it help you? I think it gives you to remove workaround. >>>> >>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>> >>> Thanks for your information. >>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>> I will modify and will test it after yours. >>> >>> >>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>> to me, how this has to be handled. >>> >>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>>> So it appears to me that it would be easier to push JDK-8242427 after this (JDK-8238585). >>>> >>>>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>>> >>>> Would be interesting to see how you handled the issues above :) >>>> >>>> Thanks, Richard. >>>> >>>> [1] See question in comment https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14302030&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14302030 >>>> >>>> -----Original Message----- >>>> From: Yasumasa Suenaga >>>> Sent: Freitag, 24. April 2020 13:34 >>>> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>> >>>> Hi Richard, >>>> >>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>> Does it help you? I think it gives you to remove workaround. >>>> >>>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> On 2020/04/24 17:18, Reingruber, Richard wrote: >>>>> Hi Patricio, Vladimir, and Serguei, >>>>> >>>>> now that direct handshakes are available, I've updated the patch to make use of them. >>>>> >>>>> In addition I have done some clean-up changes I missed in the first webrev. >>>>> >>>>> Finally I have implemented the workaround suggested by Patricio to avoid nesting the handshake >>>>> into the vm operation VM_SetFramePop [1] >>>>> >>>>> Kindly review again: >>>>> >>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1/ >>>>> Webrev(delta): http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1.inc/ >>>>> >>>>> I updated the JBS item explaining why the vm operation VM_EnterInterpOnlyMode can be replaced with a >>>>> direct handshake: >>>>> >>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>>> >>>>> Testing: >>>>> >>>>> * JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>>> >>>>> * Submit-repo: mach5-one-rrich-JDK-8238585-20200423-1436-10441737 >>>>> >>>>> Thanks, >>>>> Richard. >>>>> >>>>> [1] An assertion in Handshake::execute_direct() fails, if called be VMThread, because it is no JavaThread. >>>>> >>>>> -----Original Message----- >>>>> From: hotspot-dev On Behalf Of Reingruber, Richard >>>>> Sent: Freitag, 14. Februar 2020 19:47 >>>>> To: Patricio Chilano ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>> Subject: RE: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>> >>>>> Hi Patricio, >>>>> >>>>> > > I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>>> > > handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>>> > > Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>>> > > >>>>> > > > Alternatively I think you could do something similar to what we do in >>>>> > > > Deoptimization::deoptimize_all_marked(): >>>>> > > > >>>>> > > > EnterInterpOnlyModeClosure hs; >>>>> > > > if (SafepointSynchronize::is_at_safepoint()) { >>>>> > > > hs.do_thread(state->get_thread()); >>>>> > > > } else { >>>>> > > > Handshake::execute(&hs, state->get_thread()); >>>>> > > > } >>>>> > > > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>> > > > HandshakeClosure() constructor) >>>>> > > >>>>> > > Maybe this could be used also in the Handshake::execute() methods as general solution? >>>>> > Right, we could also do that. Avoiding to clear the polling page in >>>>> > HandshakeState::clear_handshake() should be enough to fix this issue and >>>>> > execute a handshake inside a safepoint, but adding that "if" statement >>>>> > in Hanshake::execute() sounds good to avoid all the extra code that we >>>>> > go through when executing a handshake. I filed 8239084 to make that change. >>>>> >>>>> Thanks for taking care of this and creating the RFE. >>>>> >>>>> > >>>>> > > > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>> > > > always called in a nested operation or just sometimes. >>>>> > > >>>>> > > At least one execution path without vm operation exists: >>>>> > > >>>>> > > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>>> > > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>>> > > JvmtiEventControllerPrivate::recompute_enabled() : void >>>>> > > JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>>> > > JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>>> > > JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>>> > > jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>>> > > >>>>> > > I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>>> > > handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>>> > > encouraged to do it with a handshake :) >>>>> > Ah! I think you can still do it with a handshake with the >>>>> > Deoptimization::deoptimize_all_marked() like solution. I can change the >>>>> > if-else statement with just the Handshake::execute() call in 8239084. >>>>> > But up to you. : ) >>>>> >>>>> Well, I think that's enough encouragement :) >>>>> I'll wait for 8239084 and try then again. >>>>> (no urgency and all) >>>>> >>>>> Thanks, >>>>> Richard. >>>>> >>>>> -----Original Message----- >>>>> From: Patricio Chilano >>>>> Sent: Freitag, 14. Februar 2020 15:54 >>>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>> >>>>> Hi Richard, >>>>> >>>>> On 2/14/20 9:58 AM, Reingruber, Richard wrote: >>>>>> Hi Patricio, >>>>>> >>>>>> thanks for having a look. >>>>>> >>>>>> > I?m only commenting on the handshake changes. >>>>>> > I see that operation VM_EnterInterpOnlyMode can be called inside >>>>>> > operation VM_SetFramePop which also allows nested operations. Here is a >>>>>> > comment in VM_SetFramePop definition: >>>>>> > >>>>>> > // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>>> > // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>>> > >>>>>> > So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>>> > could have a handshake inside a safepoint operation. The issue I see >>>>>> > there is that at the end of the handshake the polling page of the target >>>>>> > thread could be disarmed. So if the target thread happens to be in a >>>>>> > blocked state just transiently and wakes up then it will not stop for >>>>>> > the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>>> > polling page is armed at the beginning of disarm_safepoint(). >>>>>> >>>>>> I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>>>> handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>>>> Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>>>> >>>>>> > Alternatively I think you could do something similar to what we do in >>>>>> > Deoptimization::deoptimize_all_marked(): >>>>>> > >>>>>> > EnterInterpOnlyModeClosure hs; >>>>>> > if (SafepointSynchronize::is_at_safepoint()) { >>>>>> > hs.do_thread(state->get_thread()); >>>>>> > } else { >>>>>> > Handshake::execute(&hs, state->get_thread()); >>>>>> > } >>>>>> > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>>> > HandshakeClosure() constructor) >>>>>> >>>>>> Maybe this could be used also in the Handshake::execute() methods as general solution? >>>>> Right, we could also do that. Avoiding to clear the polling page in >>>>> HandshakeState::clear_handshake() should be enough to fix this issue and >>>>> execute a handshake inside a safepoint, but adding that "if" statement >>>>> in Hanshake::execute() sounds good to avoid all the extra code that we >>>>> go through when executing a handshake. I filed 8239084 to make that change. >>>>> >>>>>> > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>>> > always called in a nested operation or just sometimes. >>>>>> >>>>>> At least one execution path without vm operation exists: >>>>>> >>>>>> JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>>>> JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>>>> JvmtiEventControllerPrivate::recompute_enabled() : void >>>>>> JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>>>> JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>>>> JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>>>> jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>>>> >>>>>> I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>>>> handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>>>> encouraged to do it with a handshake :) >>>>> Ah! I think you can still do it with a handshake with the >>>>> Deoptimization::deoptimize_all_marked() like solution. I can change the >>>>> if-else statement with just the Handshake::execute() call in 8239084. >>>>> But up to you.? : ) >>>>> >>>>> Thanks, >>>>> Patricio >>>>>> Thanks again, >>>>>> Richard. >>>>>> >>>>>> -----Original Message----- >>>>>> From: Patricio Chilano >>>>>> Sent: Donnerstag, 13. Februar 2020 18:47 >>>>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>>> >>>>>> Hi Richard, >>>>>> >>>>>> I?m only commenting on the handshake changes. >>>>>> I see that operation VM_EnterInterpOnlyMode can be called inside >>>>>> operation VM_SetFramePop which also allows nested operations. Here is a >>>>>> comment in VM_SetFramePop definition: >>>>>> >>>>>> // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>>> // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>>> >>>>>> So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>>> could have a handshake inside a safepoint operation. The issue I see >>>>>> there is that at the end of the handshake the polling page of the target >>>>>> thread could be disarmed. So if the target thread happens to be in a >>>>>> blocked state just transiently and wakes up then it will not stop for >>>>>> the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>>> polling page is armed at the beginning of disarm_safepoint(). >>>>>> >>>>>> I think one option could be to remove >>>>>> SafepointMechanism::disarm_if_needed() in >>>>>> HandshakeState::clear_handshake() and let each JavaThread disarm itself >>>>>> for the handshake case. >>>>>> >>>>>> Alternatively I think you could do something similar to what we do in >>>>>> Deoptimization::deoptimize_all_marked(): >>>>>> >>>>>> ? EnterInterpOnlyModeClosure hs; >>>>>> ? if (SafepointSynchronize::is_at_safepoint()) { >>>>>> ??? hs.do_thread(state->get_thread()); >>>>>> ? } else { >>>>>> ??? Handshake::execute(&hs, state->get_thread()); >>>>>> ? } >>>>>> (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>>> HandshakeClosure() constructor) >>>>>> >>>>>> I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>>> always called in a nested operation or just sometimes. >>>>>> >>>>>> Thanks, >>>>>> Patricio >>>>>> >>>>>> On 2/12/20 7:23 AM, Reingruber, Richard wrote: >>>>>>> // Repost including hotspot runtime and gc lists. >>>>>>> // Dean Long suggested to do so, because the enhancement replaces a vm operation >>>>>>> // with a handshake. >>>>>>> // Original thread: http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-February/030359.html >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> could I please get reviews for this small enhancement in hotspot's jvmti implementation: >>>>>>> >>>>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.0/ >>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>>>>> >>>>>>> The change avoids making all compiled methods on stack not_entrant when switching a java thread to >>>>>>> interpreter only execution for jvmti purposes. It is sufficient to deoptimize the compiled frames on stack. >>>>>>> >>>>>>> Additionally a handshake is used instead of a vm operation to walk the stack and do the deoptimizations. >>>>>>> >>>>>>> Testing: JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>>>>> >>>>>>> Thanks, Richard. >>>>>>> >>>>>>> See also my question if anyone knows a reason for making the compiled methods not_entrant: >>>>>>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-January/030339.html >>>>> From thomas.stuefe at gmail.com Wed May 13 05:49:12 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 13 May 2020 07:49:12 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> Message-ID: (trimmed down the list of recipients) > If you have a running foreground JVM in your Linux terminal, you can type > "Ctrl-\" and see this. E.g., > > Oh, backslash. I tried backspace. My error. > $ java -cp . SleepForever > <...> > Heap > garbage-first heap total 1048576K, used 824K [0x0000000412000000, > 0x0000000800000000) > region size 8192K, 1 young (8192K), 0 survivors (0K) > Metaspace used 143K, capacity 4486K, committed 4864K, reserved > 1056768K > class space used 6K, capacity 386K, committed 512K, reserved 1048576K > > Note that this output is not from NMT and so the numbers are not affected by the NMT bug. Only the jcmd VM.native_memory output is affected. > Actually I see that the current output is not correct -- class space > doesn't include the CDS sizes :-( > > I think I'like to keep CDS and CCS separate in the output for a number of reasons. One is that CCSSize only applies to CCS. But more important, CDS can be shared and CCS not. But we may print out CDS too in this output, in a separate line. Cheers, Thomas From matthias.baesken at sap.com Wed May 13 08:11:29 2020 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Wed, 13 May 2020 08:11:29 +0000 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: <81b1d335-d898-5bc9-8aed-b105f8d73a2e@oracle.com> References: <81b1d335-d898-5bc9-8aed-b105f8d73a2e@oracle.com> Message-ID: >But according to the comment immediately before your changes: > >// in case of warnings like : "Your kernel does not support swap limit >capabilities or the cgroup is not mounted. Memory limited without swap." >// the getTotalSwapSpaceSize does not return the expected result, but 0 > >we should be returning zero. So this seems to be a bug in the >implementation not in the test. Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values that are normally expected. Previously we indeed got "0" . Probably the comment should be deleted . Best regards, Matthias From erik.osterlund at oracle.com Wed May 13 09:13:23 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 13 May 2020 11:13:23 +0200 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: References: <56a0cf85-9698-64b2-95b4-e5b123c3e9e2@oracle.com> <9a314ebe-5499-ecb6-3c5b-0c8ce7f9c8d2@oracle.com> <44febab6-5703-1c8b-d4ad-0f4c12517afa@oracle.com> Message-ID: <7a326841-02f8-3f1d-c891-1138b7997f91@oracle.com> Hi Thomas, On 2020-05-13 07:42, Thomas St?fe wrote: > p.s. I also think that we can rework this test and ccs placement in > general at a later follow up rfe. ccs placement strategy is not > optimal but that would be out of scope for both your change and my cds > rework patch, which is supposed to be mainly code shuffling around. Yes, agreed. > ..Thomas > > On Wed, May 13, 2020 at 7:40 AM Thomas St?fe > wrote: > > Hi Erik, > > note that I am changing the CompressClassPointers too as part of > "8243392: Remodel CDS/Metaspace storage reservation" which is out > for review. That is not a big deal, if you push before me (which I > assume will happen) I'll merge. But look at my changes: > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/test/hotspot/jtreg/runtime/CompressedOops/CompressedClassPointers.java.udiff.html > > What I did was to explicitly?disable placement checks on selected > platforms - windows and, for us, ppc. So the tests run in two > modes, one only testing that the VM does come up with the given > parameters, one more strict where we test ccs placement. Feel free > to integrate that into your change. > > However, that said, I am fine with your change as it is. I would > merge your changes into my change then. > Ah, okay. Thanks for the pointer. Then I shall push very soon. Thank you for dealing with the merge Thomas. And thanks for your review and helpful comments. /Erik > Cheers Thomas > > > > On Tue, May 12, 2020 at 4:10 PM Erik ?sterlund > > wrote: > > Hi, > > I ran into some merge conflicts in the tests and the new tests > behaved a > bit differently > on different platforms. The new tests naturally do not behave > the same > way on the platforms that do > not yet support compressed class pointers without compressed > oops, so I > just filtered away those > platforms from running the new tests. Will update the filter > when they > start supporting it. > > Another interesting thing is that on macOS, sometimes we did > not get the > expected compressed class > pointer encoding without compressed oops. The reason is that > without > compressed oops, we allocate > the heap at some random address where the OS thinks is suitable. > Sometimes that is where we would > ideally have put the compressed class space. This forces > metaspace to be > mapped in somewhere else > with different characteristics. Ideally, the bootstrapping > logic should > be disentangled so that > when you run without compressed oops, metaspace gets reserved > before the > heap. But I think that > warrants a follow-up RFE, as I do not want to complicate this > patch any > further. Disentangling that > seems like it could become quite tricky on its own. > > So here is another webrev with the test issues resolved: > http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ > > Incremental: > http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ > > Thanks, > /Erik > > On 2020-04-29 09:39, Erik ?sterlund wrote: > > Hi Thomas, > > > > On 2020-04-29 08:27, Thomas St?fe wrote: > >> Hi Erik, > >> > >> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund > >> >> wrote: > >> > >> ??? Hi Thomas, > >> > >> ??? Thanks for reviewing this. > >> > >> ??? On 2020-04-28 13:14, Thomas St?fe wrote: > >> > >>> ??? With compressed oops we continue to squeeze ccs in > after the heap > >>> ??? in the hope that it all still fits into the lower 4G > or 32G. (Do > >>> ??? we ever bother to check if heap is actually reserved > in low area? > >>> ??? It may have given up and reserved upstate instead, > leaving free > >>> ??? room below...). > >> > >> ??? Right. And yes I'm pretty sure we do check. The heap > reserves > >> ??? memory, and sets the CompressedOops::end() based on > that, which is > >> ??? fed as input to the metaspace/CDS placement. > >> > >> > >> No, what I meant was: since we reserve at > CompressedOops::end(), if > >> the heap was too large to place it in the lower address > regions, the > >> class space will not live there either even if it could, > since it is > >> smaller and might fit. > >> > >> For example: > >> > >> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ > >> ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace > -Xshare:off > >> -Xmx31g > >> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: > >> reserved=1048576.00 KB, committed=0.00 KB ( ?0%), used=0.00 > KB ( ?0%) > >> [0.003s][trace][gc,metaspace] ?[0x00000017c0400000, > >> 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) > >> [0.003s][trace][gc,metaspace] Narrow klass base: > 0x00000017c0400000, > >> Narrow klass shift: 0 > >> [0.003s][trace][gc,metaspace] Compressed class space size: > 1073741824 > >> Address: 0x00000017c0400000 Req Addr: 0x00000017c0400000 > >> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: > >> reserved=8192.00 KB, committed=0.00 KB ( ?0%), used=0.00 KB > ( ?0%) > >> [0.003s][trace][gc,metaspace] ?[0x00007fc9cb800000, > >> 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) > >> > >> (large heap and no CDS, since it prevents zero based mode). > >> > >> In that case narrow klass base is not zero based even it if > probably > >> could be, if we were to allocate class space in the lower > regions, > >> which are now unoccupied since the large heap does not fit > there. A > >> missed opportunity to go zero based. Not a big deal, since > we mostly > >> run with CDS now anyway. > > > > I see. Yeah, there is room for improvement there. Although, > as you > > say, people mostly run with CDS on, so it probably won't > matter even > > if that is improved. > > > >>> ??? Now with your patch without compressed oops we attempt > to map at > >>> ??? HeapBaseMinAddress since this is the lowest address we are > >>> ??? allowed to map at and the heap won't obviously need > it,?right? > >> > >> ??? Exactly. > >> > >>> ??? My only misgiving would be that?HeapBaseMinAddress is > even more > >>> ??? of a misnomer (its already now a?bit, witness my year long > >>> ??? confusion :-), since it now controls not only heap > placement but > >>> ??? ccs too. Actually if it is that important to avoid > mapping to > >>> ??? addresses below?HeapBaseMinAddress we should probably > prevent it > >>> ??? somewhere below in the os layer but I really do not > want to open > >>> ??? that can of worms. > >> > >> ??? Yes agreed. Ideally it would be called something like > >> ??? MappingMinAddress instead, because it really isn't tied > to the > >> ??? heap really, it just happened to be the primary user of > mapping > >> ??? things at low addresses. But I also think changing > that, and > >> ??? enforcing it more widely in the os layer sounds like a > different > >> ??? can of worms. > >> > >>> ??? -- > >>> > >>> ??? Some more remarks about arguments.cpp: > >>> > >>> ??? - In Arguments::set_use_compressed_klass_ptrs(), if now > >>> ??? compressed class pointers are independent from > compressed oops, > >>> ??? checking CompressedClassSpaceSize for validity should > now always > >>> ??? happen for UseCompressedClassPointers=1. > >> > >> ??? I think it looks like I do that. > >> > >>> ??? - If HeapBaseMinAddress is now always used, we should > check it > >>> ??? always too (see Arguments::set_heap_size) > >> > >> ??? Good idea. Fixed. > >> > >>> > >>>> ??????? Also, could we have some?tests which exercise > cds/metaspace > >>>> ??????? initialization without coops and with ccps? > Easiest way > >>>> ??????? would be to extend > >>>> "CompressedOops/CompressedClassPointers.java" by some cases. > >>> > >>> > >>> ??? Thank you. I think this makes sense especially since > compressed > >>> ??? class pointer encoding is restricted to one mode if > CDS is on > >>> ??? (base = shared address and shift=3), and since CDS=on > is the > >>> ??? default zerobased encoding may not happen that often. > >> > >> ??? Yup. I added some explicit test cases for > -XX:-UseCompressedOops > >> ??? -XX:+UseCompressedClassPointers as suggested. This will > also be > >> ??? the default configuration for ZGC, so that should give that > >> ??? configuration more test coverage in the various tests > we have. > >> > >> ??? Here is a new webrev: > >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ > >> > >> ??? Incremental: > >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ > >> > >> ??? I hope I fixed everything you wanted in this patch. I > also fixed > >> ??? what Frederic and Coleen wanted. > >> > >> > >> arguments.cpp, the metaspace stuff and the test look good > to me now. > >> Thank you for your work! > > > > Thanks for the review Thomas! > > > > /Erik > From bob.vandette at oracle.com Wed May 13 13:13:06 2020 From: bob.vandette at oracle.com (Bob Vandette) Date: Wed, 13 May 2020 09:13:06 -0400 Subject: RFR: 8244852 GraalVM native-image fails after JDK-8238048 change In-Reply-To: References: Message-ID: Thanks. I did confirm that the value returned prior to the regression was indeed a null. Bob. > On May 13, 2020, at 12:37 AM, David Holmes wrote: > > Hi Bob, > > On 13/05/2020 4:37 am, Bob Vandette wrote: >> Please review this fix for a regression in JDK 15 causing native-image to fail. >> https://bugs.openjdk.java.net/browse/JDK-8244852 >> diff --git a/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java b/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java >> --- a/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java >> +++ b/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotResolvedObjectTypeImpl.java >> @@ -818,7 +818,7 @@ >> @Override >> public String getSourceFileName() { >> if (isArray()) { >> - throw new JVMCIError("Cannot call getSourceFileName() on an array klass type: %s", this); >> + return (String)null; > > Fix seems reasonable. While JavaType.getSourceFileName() seems a little underspecified, JavaType includes array types so the call should be valid. > > Nit: No cast needed on the null. > > Thanks, > David > >> } >> return getConstantPool().getSourceFileName(); >> } >> Bob. From thomas.stuefe at gmail.com Wed May 13 13:39:04 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 13 May 2020 15:39:04 +0200 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: <7a326841-02f8-3f1d-c891-1138b7997f91@oracle.com> References: <56a0cf85-9698-64b2-95b4-e5b123c3e9e2@oracle.com> <9a314ebe-5499-ecb6-3c5b-0c8ce7f9c8d2@oracle.com> <44febab6-5703-1c8b-d4ad-0f4c12517afa@oracle.com> <7a326841-02f8-3f1d-c891-1138b7997f91@oracle.com> Message-ID: Hi Erik, I filed https://bugs.openjdk.java.net/browse/JDK-8244943 as a follow up for the improved ccs placement. But lets get the cds remodel go through first... Cheers, Thomas On Wed, May 13, 2020 at 11:13 AM Erik ?sterlund wrote: > Hi Thomas, > > On 2020-05-13 07:42, Thomas St?fe wrote: > > p.s. I also think that we can rework this test and ccs placement in > general at a later follow up rfe. ccs placement strategy is not optimal but > that would be out of scope for both your change and my cds rework patch, > which is supposed to be mainly code shuffling around. > > > Yes, agreed. > > ..Thomas > > On Wed, May 13, 2020 at 7:40 AM Thomas St?fe > wrote: > >> Hi Erik, >> >> note that I am changing the CompressClassPointers too as part of >> "8243392: Remodel CDS/Metaspace storage reservation" which is out for >> review. That is not a big deal, if you push before me (which I assume will >> happen) I'll merge. But look at my changes: >> >> >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/test/hotspot/jtreg/runtime/CompressedOops/CompressedClassPointers.java.udiff.html >> >> What I did was to explicitly disable placement checks on selected >> platforms - windows and, for us, ppc. So the tests run in two modes, one >> only testing that the VM does come up with the given parameters, one more >> strict where we test ccs placement. Feel free to integrate that into your >> change. >> >> However, that said, I am fine with your change as it is. I would merge >> your changes into my change then. >> > > Ah, okay. Thanks for the pointer. Then I shall push very soon. Thank you > for dealing with the merge Thomas. And thanks for your review and helpful > comments. > > /Erik > > Cheers Thomas >> >> >> >> On Tue, May 12, 2020 at 4:10 PM Erik ?sterlund >> wrote: >> >>> Hi, >>> >>> I ran into some merge conflicts in the tests and the new tests behaved a >>> bit differently >>> on different platforms. The new tests naturally do not behave the same >>> way on the platforms that do >>> not yet support compressed class pointers without compressed oops, so I >>> just filtered away those >>> platforms from running the new tests. Will update the filter when they >>> start supporting it. >>> >>> Another interesting thing is that on macOS, sometimes we did not get the >>> expected compressed class >>> pointer encoding without compressed oops. The reason is that without >>> compressed oops, we allocate >>> the heap at some random address where the OS thinks is suitable. >>> Sometimes that is where we would >>> ideally have put the compressed class space. This forces metaspace to be >>> mapped in somewhere else >>> with different characteristics. Ideally, the bootstrapping logic should >>> be disentangled so that >>> when you run without compressed oops, metaspace gets reserved before the >>> heap. But I think that >>> warrants a follow-up RFE, as I do not want to complicate this patch any >>> further. Disentangling that >>> seems like it could become quite tricky on its own. >>> >>> So here is another webrev with the test issues resolved: >>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ >>> >>> Incremental: >>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ >>> >>> Thanks, >>> /Erik >>> >>> On 2020-04-29 09:39, Erik ?sterlund wrote: >>> > Hi Thomas, >>> > >>> > On 2020-04-29 08:27, Thomas St?fe wrote: >>> >> Hi Erik, >>> >> >>> >> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund >>> >> > wrote: >>> >> >>> >> Hi Thomas, >>> >> >>> >> Thanks for reviewing this. >>> >> >>> >> On 2020-04-28 13:14, Thomas St?fe wrote: >>> >> >>> >>> With compressed oops we continue to squeeze ccs in after the heap >>> >>> in the hope that it all still fits into the lower 4G or 32G. (Do >>> >>> we ever bother to check if heap is actually reserved in low area? >>> >>> It may have given up and reserved upstate instead, leaving free >>> >>> room below...). >>> >> >>> >> Right. And yes I'm pretty sure we do check. The heap reserves >>> >> memory, and sets the CompressedOops::end() based on that, which is >>> >> fed as input to the metaspace/CDS placement. >>> >> >>> >> >>> >> No, what I meant was: since we reserve at CompressedOops::end(), if >>> >> the heap was too large to place it in the lower address regions, the >>> >> class space will not live there either even if it could, since it is >>> >> smaller and might fit. >>> >> >>> >> For example: >>> >> >>> >> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ >>> >> ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace -Xshare:off >>> >> -Xmx31g >>> >> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: >>> >> reserved=1048576.00 KB, committed=0.00 KB ( 0%), used=0.00 KB ( 0%) >>> >> [0.003s][trace][gc,metaspace] [0x00000017c0400000, >>> >> 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) >>> >> [0.003s][trace][gc,metaspace] Narrow klass base: 0x00000017c0400000, >>> >> Narrow klass shift: 0 >>> >> [0.003s][trace][gc,metaspace] Compressed class space size: 1073741824 >>> >> Address: 0x00000017c0400000 Req Addr: 0x00000017c0400000 >>> >> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: >>> >> reserved=8192.00 KB, committed=0.00 KB ( 0%), used=0.00 KB ( 0%) >>> >> [0.003s][trace][gc,metaspace] [0x00007fc9cb800000, >>> >> 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) >>> >> >>> >> (large heap and no CDS, since it prevents zero based mode). >>> >> >>> >> In that case narrow klass base is not zero based even it if probably >>> >> could be, if we were to allocate class space in the lower regions, >>> >> which are now unoccupied since the large heap does not fit there. A >>> >> missed opportunity to go zero based. Not a big deal, since we mostly >>> >> run with CDS now anyway. >>> > >>> > I see. Yeah, there is room for improvement there. Although, as you >>> > say, people mostly run with CDS on, so it probably won't matter even >>> > if that is improved. >>> > >>> >>> Now with your patch without compressed oops we attempt to map at >>> >>> HeapBaseMinAddress since this is the lowest address we are >>> >>> allowed to map at and the heap won't obviously need it, right? >>> >> >>> >> Exactly. >>> >> >>> >>> My only misgiving would be that HeapBaseMinAddress is even more >>> >>> of a misnomer (its already now a bit, witness my year long >>> >>> confusion :-), since it now controls not only heap placement but >>> >>> ccs too. Actually if it is that important to avoid mapping to >>> >>> addresses below HeapBaseMinAddress we should probably prevent it >>> >>> somewhere below in the os layer but I really do not want to open >>> >>> that can of worms. >>> >> >>> >> Yes agreed. Ideally it would be called something like >>> >> MappingMinAddress instead, because it really isn't tied to the >>> >> heap really, it just happened to be the primary user of mapping >>> >> things at low addresses. But I also think changing that, and >>> >> enforcing it more widely in the os layer sounds like a different >>> >> can of worms. >>> >> >>> >>> -- >>> >>> >>> >>> Some more remarks about arguments.cpp: >>> >>> >>> >>> - In Arguments::set_use_compressed_klass_ptrs(), if now >>> >>> compressed class pointers are independent from compressed oops, >>> >>> checking CompressedClassSpaceSize for validity should now always >>> >>> happen for UseCompressedClassPointers=1. >>> >> >>> >> I think it looks like I do that. >>> >> >>> >>> - If HeapBaseMinAddress is now always used, we should check it >>> >>> always too (see Arguments::set_heap_size) >>> >> >>> >> Good idea. Fixed. >>> >> >>> >>> >>> >>>> Also, could we have some tests which exercise cds/metaspace >>> >>>> initialization without coops and with ccps? Easiest way >>> >>>> would be to extend >>> >>>> "CompressedOops/CompressedClassPointers.java" by some cases. >>> >>> >>> >>> >>> >>> Thank you. I think this makes sense especially since compressed >>> >>> class pointer encoding is restricted to one mode if CDS is on >>> >>> (base = shared address and shift=3), and since CDS=on is the >>> >>> default zerobased encoding may not happen that often. >>> >> >>> >> Yup. I added some explicit test cases for -XX:-UseCompressedOops >>> >> -XX:+UseCompressedClassPointers as suggested. This will also be >>> >> the default configuration for ZGC, so that should give that >>> >> configuration more test coverage in the various tests we have. >>> >> >>> >> Here is a new webrev: >>> >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ >>> >> >>> >> Incremental: >>> >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ >>> >> >>> >> I hope I fixed everything you wanted in this patch. I also fixed >>> >> what Frederic and Coleen wanted. >>> >> >>> >> >>> >> arguments.cpp, the metaspace stuff and the test look good to me now. >>> >> Thank you for your work! >>> > >>> > Thanks for the review Thomas! >>> > >>> > /Erik >>> >>> > From erik.osterlund at oracle.com Wed May 13 13:45:14 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 13 May 2020 15:45:14 +0200 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: References: <56a0cf85-9698-64b2-95b4-e5b123c3e9e2@oracle.com> <9a314ebe-5499-ecb6-3c5b-0c8ce7f9c8d2@oracle.com> <44febab6-5703-1c8b-d4ad-0f4c12517afa@oracle.com> <7a326841-02f8-3f1d-c891-1138b7997f91@oracle.com> Message-ID: <52edaad9-9cde-abff-8a45-bc26ba5086aa@oracle.com> Hi Thomas, Excellent, thanks for filing that Thomas. Thanks, /Erik On 2020-05-13 15:39, Thomas St?fe wrote: > Hi Erik, > > I filed https://bugs.openjdk.java.net/browse/JDK-8244943?as a follow > up for the improved ccs placement. But lets get the cds remodel go > through first... > > Cheers, Thomas > > > On Wed, May 13, 2020 at 11:13 AM Erik ?sterlund > > wrote: > > Hi Thomas, > > On 2020-05-13 07:42, Thomas St?fe wrote: >> p.s. I also think that we can rework this test and ccs placement >> in general at a later follow up rfe. ccs placement strategy is >> not optimal but that would be out of scope for both your change >> and my cds rework patch, which is supposed to be mainly code >> shuffling around. > > Yes, agreed. > >> ..Thomas >> >> On Wed, May 13, 2020 at 7:40 AM Thomas St?fe >> > wrote: >> >> Hi Erik, >> >> note that I am changing the CompressClassPointers too as part >> of "8243392: Remodel CDS/Metaspace storage reservation" which >> is out for review. That is not a big deal, if you push before >> me (which I assume will happen) I'll merge. But look at my >> changes: >> >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/test/hotspot/jtreg/runtime/CompressedOops/CompressedClassPointers.java.udiff.html >> >> What I did was to explicitly?disable placement checks on >> selected platforms - windows and, for us, ppc. So the tests >> run in two modes, one only testing that the VM does come up >> with the given parameters, one more strict where we test ccs >> placement. Feel free to integrate that into your change. >> >> However, that said, I am fine with your change as it is. I >> would merge your changes into my change then. >> > > Ah, okay. Thanks for the pointer. Then I shall push very soon. > Thank you for dealing with the merge Thomas. And thanks for your > review and helpful comments. > > /Erik > >> Cheers Thomas >> >> >> >> On Tue, May 12, 2020 at 4:10 PM Erik ?sterlund >> > > wrote: >> >> Hi, >> >> I ran into some merge conflicts in the tests and the new >> tests behaved a >> bit differently >> on different platforms. The new tests naturally do not >> behave the same >> way on the platforms that do >> not yet support compressed class pointers without >> compressed oops, so I >> just filtered away those >> platforms from running the new tests. Will update the >> filter when they >> start supporting it. >> >> Another interesting thing is that on macOS, sometimes we >> did not get the >> expected compressed class >> pointer encoding without compressed oops. The reason is >> that without >> compressed oops, we allocate >> the heap at some random address where the OS thinks is >> suitable. >> Sometimes that is where we would >> ideally have put the compressed class space. This forces >> metaspace to be >> mapped in somewhere else >> with different characteristics. Ideally, the >> bootstrapping logic should >> be disentangled so that >> when you run without compressed oops, metaspace gets >> reserved before the >> heap. But I think that >> warrants a follow-up RFE, as I do not want to complicate >> this patch any >> further. Disentangling that >> seems like it could become quite tricky on its own. >> >> So here is another webrev with the test issues resolved: >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ >> >> Incremental: >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ >> >> Thanks, >> /Erik >> >> On 2020-04-29 09:39, Erik ?sterlund wrote: >> > Hi Thomas, >> > >> > On 2020-04-29 08:27, Thomas St?fe wrote: >> >> Hi Erik, >> >> >> >> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund >> >> > > >> wrote: >> >> >> >> ??? Hi Thomas, >> >> >> >> ??? Thanks for reviewing this. >> >> >> >> ??? On 2020-04-28 13:14, Thomas St?fe wrote: >> >> >> >>> ??? With compressed oops we continue to squeeze ccs >> in after the heap >> >>> ??? in the hope that it all still fits into the lower >> 4G or 32G. (Do >> >>> ??? we ever bother to check if heap is actually >> reserved in low area? >> >>> ??? It may have given up and reserved upstate >> instead, leaving free >> >>> ??? room below...). >> >> >> >> ??? Right. And yes I'm pretty sure we do check. The >> heap reserves >> >> ??? memory, and sets the CompressedOops::end() based >> on that, which is >> >> ??? fed as input to the metaspace/CDS placement. >> >> >> >> >> >> No, what I meant was: since we reserve at >> CompressedOops::end(), if >> >> the heap was too large to place it in the lower >> address regions, the >> >> class space will not live there either even if it >> could, since it is >> >> smaller and might fit. >> >> >> >> For example: >> >> >> >> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ >> >> ../../jdks/sapmachine14/bin/java >> -Xlog:gc+metaspace=trace -Xshare:off >> >> -Xmx31g >> >> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: >> >> reserved=1048576.00 KB, committed=0.00 KB ( ?0%), >> used=0.00 KB ( ?0%) >> >> [0.003s][trace][gc,metaspace] ?[0x00000017c0400000, >> >> 0x00000017c0400000, 0x00000017c0400000, >> 0x0000001800400000) >> >> [0.003s][trace][gc,metaspace] Narrow klass base: >> 0x00000017c0400000, >> >> Narrow klass shift: 0 >> >> [0.003s][trace][gc,metaspace] Compressed class space >> size: 1073741824 >> >> Address: 0x00000017c0400000 Req Addr: 0x00000017c0400000 >> >> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: >> >> reserved=8192.00 KB, committed=0.00 KB ( ?0%), >> used=0.00 KB ( ?0%) >> >> [0.003s][trace][gc,metaspace] ?[0x00007fc9cb800000, >> >> 0x00007fc9cb800000, 0x00007fc9cb800000, >> 0x00007fc9cc000000) >> >> >> >> (large heap and no CDS, since it prevents zero based >> mode). >> >> >> >> In that case narrow klass base is not zero based even >> it if probably >> >> could be, if we were to allocate class space in the >> lower regions, >> >> which are now unoccupied since the large heap does not >> fit there. A >> >> missed opportunity to go zero based. Not a big deal, >> since we mostly >> >> run with CDS now anyway. >> > >> > I see. Yeah, there is room for improvement there. >> Although, as you >> > say, people mostly run with CDS on, so it probably >> won't matter even >> > if that is improved. >> > >> >>> ??? Now with your patch without compressed oops we >> attempt to map at >> >>> ??? HeapBaseMinAddress since this is the lowest >> address we are >> >>> ??? allowed to map at and the heap won't obviously >> need it,?right? >> >> >> >> ??? Exactly. >> >> >> >>> ??? My only misgiving would be >> that?HeapBaseMinAddress is even more >> >>> ??? of a misnomer (its already now a?bit, witness my >> year long >> >>> ??? confusion :-), since it now controls not only >> heap placement but >> >>> ??? ccs too. Actually if it is that important to >> avoid mapping to >> >>> ??? addresses below?HeapBaseMinAddress we should >> probably prevent it >> >>> ??? somewhere below in the os layer but I really do >> not want to open >> >>> ??? that can of worms. >> >> >> >> ??? Yes agreed. Ideally it would be called something like >> >> ??? MappingMinAddress instead, because it really isn't >> tied to the >> >> ??? heap really, it just happened to be the primary >> user of mapping >> >> ??? things at low addresses. But I also think changing >> that, and >> >> ??? enforcing it more widely in the os layer sounds >> like a different >> >> ??? can of worms. >> >> >> >>> ??? -- >> >>> >> >>> ??? Some more remarks about arguments.cpp: >> >>> >> >>> ??? - In Arguments::set_use_compressed_klass_ptrs(), >> if now >> >>> ??? compressed class pointers are independent from >> compressed oops, >> >>> ??? checking CompressedClassSpaceSize for validity >> should now always >> >>> ??? happen for UseCompressedClassPointers=1. >> >> >> >> ??? I think it looks like I do that. >> >> >> >>> ??? - If HeapBaseMinAddress is now always used, we >> should check it >> >>> ??? always too (see Arguments::set_heap_size) >> >> >> >> ??? Good idea. Fixed. >> >> >> >>> >> >>>> ??????? Also, could we have some?tests which >> exercise cds/metaspace >> >>>> ??????? initialization without coops and with ccps? >> Easiest way >> >>>> ??????? would be to extend >> >>>> "CompressedOops/CompressedClassPointers.java" by >> some cases. >> >>> >> >>> >> >>> ??? Thank you. I think this makes sense especially >> since compressed >> >>> ??? class pointer encoding is restricted to one mode >> if CDS is on >> >>> ??? (base = shared address and shift=3), and since >> CDS=on is the >> >>> ??? default zerobased encoding may not happen that often. >> >> >> >> ??? Yup. I added some explicit test cases for >> -XX:-UseCompressedOops >> >> -XX:+UseCompressedClassPointers as suggested. This >> will also be >> >> ??? the default configuration for ZGC, so that should >> give that >> >> ??? configuration more test coverage in the various >> tests we have. >> >> >> >> ??? Here is a new webrev: >> >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ >> >> >> >> ??? Incremental: >> >> >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ >> >> >> >> ??? I hope I fixed everything you wanted in this >> patch. I also fixed >> >> ??? what Frederic and Coleen wanted. >> >> >> >> >> >> arguments.cpp, the metaspace stuff and the test look >> good to me now. >> >> Thank you for your work! >> > >> > Thanks for the review Thomas! >> > >> > /Erik >> > From matthias.baesken at sap.com Wed May 13 14:28:23 2020 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Wed, 13 May 2020 14:28:23 +0000 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java References: <81b1d335-d898-5bc9-8aed-b105f8d73a2e@oracle.com> Message-ID: Hello, here is a new webrev removing the comment : http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.1/ It removes the comment , and it also handles 0 for getFreeSwapSpaceSize ( this can occur after JDK-8242480 : Negative value may be returned by getFreeSwapSpaceSize() in the docker was pushed recently ). Best regards, Matthias -----Original Message----- From: Baesken, Matthias Sent: Mittwoch, 13. Mai 2020 10:11 To: 'David Holmes' ; 'hotspot-dev at openjdk.java.net' Subject: RE: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java >But according to the comment immediately before your changes: > >// in case of warnings like : "Your kernel does not support swap limit >capabilities or the cgroup is not mounted. Memory limited without swap." >// the getTotalSwapSpaceSize does not return the expected result, but 0 > >we should be returning zero. So this seems to be a bug in the >implementation not in the test. Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values that are normally expected. Previously we indeed got "0" . Probably the comment should be deleted . Best regards, Matthias From christian.hagedorn at oracle.com Wed May 13 15:10:40 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Wed, 13 May 2020 17:10:40 +0200 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles Message-ID: Hi Please review the following patch: https://bugs.openjdk.java.net/browse/JDK-8244946 http://cr.openjdk.java.net/~chagedorn/8244946/webrev.00/ This simply fixes a missing ResourceMark required to call Klass::oop_print_on(). The reason for the missing ResourceMark in oop_print_on() is stated in the RFR [1] for JDK-8222893 [2] which removed the ResourceMark due to a bug: "print functions with outputStream should not have a ResourceMark because you could call it with a LogStream or stream that was allocated outside the resource mark. The caller needs the ResourceMark." There are other places like Klass::print_on(outputStream*) where this is still the case. Should we file an RFE to clean these up? Thank you! Best regards, Christian [1] https://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2019-May/034095.html [2] https://bugs.openjdk.java.net/browse/JDK-8222893 From gnu.andrew at redhat.com Wed May 13 15:11:02 2020 From: gnu.andrew at redhat.com (Andrew Hughes) Date: Wed, 13 May 2020 16:11:02 +0100 Subject: [8u] RFR(S): 8244548: JDK 8u: sun.misc.Version.jdkUpdateVersion() returns wrong result In-Reply-To: <15a2fa41f80935d8c7d3893aad7d080c1f67aa2a.camel@redhat.com> References: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> <808419e3-316b-381d-f636-606aadff39b6@redhat.com> <15a2fa41f80935d8c7d3893aad7d080c1f67aa2a.camel@redhat.com> Message-ID: On 12/05/2020 15:14, Severin Gehwolf wrote: > Hi Andrew, > > Thanks for looking at this. > > On Tue, 2020-05-12 at 14:07 +0100, Andrew Haley wrote: >> On 5/7/20 4:09 PM, Severin Gehwolf wrote: >>> jvm_version_info.jvm_version currently holds this quadruplet: >>> >>> Most significant 8 bits => major version, followed by 8 bits => minor >>> version, followed by 8 bits => micro version, followed by 8 bits => >>> build version. Note that JVM minor version represents the update >>> version as passed in via configure and the micro version is currently >>> not used (always 0). See vm_version.cpp lines 100-102 where only major, >>> minor and build number are ever been set. Knowing this, we can still >>> preserve the same behavior after patch by defining JVM_VERSION_MICRO to >>> 0 for any version. >> >> This is tricky. JVM_GetVersionInfo is a function exported by >> libjvm.so, and the version is simply encoded as >> >> unsigned int Abstract_VM_Version::jvm_version() { >> return ((Abstract_VM_Version::vm_major_version() & 0xFF) << 24) | >> ((Abstract_VM_Version::vm_minor_version() & 0xFF) << 16) | >> (Abstract_VM_Version::vm_build_number() & 0xFF); >> } >> >> I guess we could argue that this is for JVM-JDK internal use only, and >> no-one else cares. >> >> Or we could encode it in a different way such that at least we have a >> jvm_version that is monotonically increasing, perhaps by (ab)using the >> lower 8 bits of the version, the vm_build_number. But I guess I'm >> being paranoid, and no tools are going to care about minor versions >> anyway,even if they do call JVM_GetVersionInfo. > > Yes, this is indeed tricky. The trouble is that > Abstract_VM_Version::vm_build_number() actually holds the configured > JDK build number, we can't really use it. It's already being used. > > (gdb) > 99 > 100 _vm_major_version = atoi(vm_major_ver); > 101 _vm_minor_version = atoi(vm_minor_ver); > 102 _vm_build_number = atoi(vm_build_num); > 103 > 104 os::free(vm_version); > 105 _initialized = true; > 106 } > 107 > 108 #if defined(_LP64) > (gdb) p _vm_build_number > $1 = 2 > > The only bits which seem unused are bits 8 through 16 of this unsigned > int. So even if JVM_GetVersionInfo *is* being used somewhere in the > wild, it *should* continue to work. Hence, the proposal to now also use > those unused bits for the minor version. > > Thoughts? > > Thanks, > Severin > The April release of OpenJDK 7u has the values: Major version: 24, minor version: 5, micro version: 0, build number: 2 when it should be 24.261-b02. So we already have a situation that could be potentially be misinterpreted as an old JDK version, 7u5 b02, using the current interpretation of the structure. 8u will see similar issues if we make no change in this cycle. As the micro version appears to have always been zero, it seems pretty safe to just hard-code this value and instead use all 16 bits of the interpreted value for the micro version. In terms of backwards compatibility, if a micro version greater than zero is read using the old method, this will be an indicator that it needs to be re-interpreted as a 16-bit value. The current value for 7 is 402980866 (0001 1000 0000 0101 0000 0000 0000 0010) which thus gets interpreted as above. The new method encodes the same version numbers as 402720002 (0001 1000 0000 0001 0000 0101 0000 0010) which can be read back as major 24, minor 261, micro 0 and build 2 using the updated decoding. If the old decoding is used, we get major 24, minor 1, micro 5 and build 2. As micro is usually set to zero, updated code should be able to use this to instead use the new interpretation, while, in the worst case, stale code will get a unique, if incorrect, version. As 7u releases are oddly numbered and increment in tens, we won't hit a case where the second byte (the old micro version) is zero. With 8u, this will happen where the update version is a multiple of 256 (e.g. 8u512, 8u1792, ...) Regarding the contract of this API, it is generally used only for interactions between the JVM and the class library. jvm.h is not included in the JDK image. Attempts to follow the API when CACAO & JamVM were still trying to work with OpenJDK suggest it has been unstable even across update releases [0] [1]. [0] https://icedtea.classpath.org/hg/icedtea7/rev/810d698ffdfb [1] https://icedtea.classpath.org/hg/icedtea7/rev/139f2a8f085d -- Andrew :) Senior Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net) Fingerprint = 5132 579D D154 0ED2 3E04 C5A0 CFDA 0F9B 3596 4222 From erik.osterlund at oracle.com Wed May 13 15:26:46 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 13 May 2020 17:26:46 +0200 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot Message-ID: Hi, Let's remove hardcoded field offsets. Main trick to get this done is to split interpreter initialization into two parts: 1. reserve the interpreter memory, 2. generate the interpreter. This allows reserving it before loading methods (which generates i2c adapters that assert things are "in the interpreter", requiring knowledge about its address range), and then load the methods, and then generate the interpreter. This moving of interpreter code generation until after methods have been loaded allows offsets to be looked up, needed by code generation. That way, we no longer need to hardcode field offsets, which makes it less annoying to change the field layout code in the future. Thanks to Coleen for pre-reviewing this. Bug: https://bugs.openjdk.java.net/browse/JDK-8243996 Webrev: http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ Thanks, /Erik From richard.reingruber at sap.com Wed May 13 15:37:59 2020 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Wed, 13 May 2020 15:37:59 +0000 Subject: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant In-Reply-To: <36d5e2c0-c724-7ff7-d37e-decb5cc0005b@oracle.com> References: <410eed04-e2ef-0f4f-1c56-19e6734a10f6@oracle.com> <81d7caa8-4244-85f3-4d4e-78117fe5e25b@oss.nttdata.com> <550b95ac-8b29-1eb8-a507-533e81d02322@oracle.com> <9c49ea2d-e3b8-b576-1d17-d18ad87cd6ed@oracle.com> <36d5e2c0-c724-7ff7-d37e-decb5cc0005b@oracle.com> Message-ID: Hi David, > On 4/05/2020 8:33 pm, Reingruber, Richard wrote: > > // Trimmed the list of recipients. If the list gets too long then the message needs to be approved > > // by a moderator. > Yes I noticed that too :) In general if you send to hotspot-dev you > shouldn't need to also send to hotspot-X-dev. Makes sense. Will do so next time. > > > > This would be the post with the current webrev.1 > > > > http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-April/031245.html > Sorry I missed that update. Okay so this is working with direct > handshakes now. > One style nit in jvmtiThreadState.cpp: > assert(SafepointSynchronize::is_at_safepoint() || > ! (JavaThread *)Thread::current() == get_thread() || > ! Thread::current() == get_thread()->active_handshaker(), > ! "bad synchronization with owner thread"); > the ! lines should ident as follows > assert(SafepointSynchronize::is_at_safepoint() || > (JavaThread *)Thread::current() == get_thread() || > Thread::current() == get_thread()->active_handshaker(), > ! "bad synchronization with owner thread"); Sure. > Lets see how this plays out. Hopefully not too bad... :) >> Not a review but some general commentary ... Still not a review, or is it now? Thanks, Richard. -----Original Message----- From: David Holmes Sent: Mittwoch, 13. Mai 2020 07:43 To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant On 4/05/2020 8:33 pm, Reingruber, Richard wrote: > // Trimmed the list of recipients. If the list gets too long then the message needs to be approved > // by a moderator. Yes I noticed that too :) In general if you send to hotspot-dev you shouldn't need to also send to hotspot-X-dev. > Hi David, Hi Richard, >> On 28/04/2020 12:09 am, Reingruber, Richard wrote: >>> Hi David, >>> >>>> Not a review but some general commentary ... >>> >>> That's welcome. > >> Having had to take an even closer look now I have a review comment too :) > >> src/hotspot/share/prims/jvmtiThreadState.cpp > >> void JvmtiThreadState::invalidate_cur_stack_depth() { >> ! assert(SafepointSynchronize::is_at_safepoint() || >> ! (Thread::current()->is_VM_thread() && >> get_thread()->is_vmthread_processing_handshake()) || >> (JavaThread *)Thread::current() == get_thread(), >> "must be current thread or at safepoint"); > > You're looking at an outdated webrev, I'm afraid. > > This would be the post with the current webrev.1 > > http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-April/031245.html Sorry I missed that update. Okay so this is working with direct handshakes now. One style nit in jvmtiThreadState.cpp: assert(SafepointSynchronize::is_at_safepoint() || ! (JavaThread *)Thread::current() == get_thread() || ! Thread::current() == get_thread()->active_handshaker(), ! "bad synchronization with owner thread"); the ! lines should ident as follows assert(SafepointSynchronize::is_at_safepoint() || (JavaThread *)Thread::current() == get_thread() || Thread::current() == get_thread()->active_handshaker(), ! "bad synchronization with owner thread"); Lets see how this plays out. Cheers, David > Thanks, Richard. > > -----Original Message----- > From: David Holmes > Sent: Montag, 4. Mai 2020 08:51 > To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net > Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant > > Hi Richard, > > On 28/04/2020 12:09 am, Reingruber, Richard wrote: >> Hi David, >> >>> Not a review but some general commentary ... >> >> That's welcome. > > Having had to take an even closer look now I have a review comment too :) > > src/hotspot/share/prims/jvmtiThreadState.cpp > > void JvmtiThreadState::invalidate_cur_stack_depth() { > ! assert(SafepointSynchronize::is_at_safepoint() || > ! (Thread::current()->is_VM_thread() && > get_thread()->is_vmthread_processing_handshake()) || > (JavaThread *)Thread::current() == get_thread(), > "must be current thread or at safepoint"); > > The message needs updating to include handshakes. > > More below ... > >>> On 25/04/2020 2:08 am, Reingruber, Richard wrote: >>>> Hi Yasumasa, Patricio, >>>> >>>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>>> Does it help you? I think it gives you to remove workaround. >>>>>> >>>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>>> >>>>> Thanks for your information. >>>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>>> I will modify and will test it after yours. >>>> >>>> Thanks :) >>>> >>>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>>> to me, how this has to be handled. >>>> >>>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>>> >>>> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >>>> also I'm unsure if a thread should do safepoint checks while executing a handshake. >> >>> I'm growing increasingly concerned that use of direct handshakes to >>> replace VM operations needs a much greater examination for correctness >>> than might initially be thought. I see a number of issues: >> >> I agree. I'll address your concerns in the context of this review thread for JDK-8238585 below. >> >> In addition I would suggest to take the general part of the discussion to a dedicated thread or to >> the review thread for JDK-8242427. I would like to keep this thread closer to its subject. > > I will focus on the issues in the context of this particular change > then, though the issues themselves are applicable to all handshake > situations (and more so with direct handshakes). This is mostly just > discussion. > >>> First, the VMThread executes (most) VM operations with a clean stack in >>> a clean state, so it has lots of room to work. If we now execute the >>> same logic in a JavaThread then we risk hitting stackoverflows if >>> nothing else. But we are also now executing code in a JavaThread and so >>> we have to be sure that code is not going to act differently (in a bad >>> way) if executed by a JavaThread rather than the VMThread. For example, >>> may it be possible that if executing in the VMThread we defer some >>> activity that might require execution of Java code, or else hand it off >>> to one of the service threads? If we execute that code directly in the >>> current JavaThread instead we may not be in a valid state (e.g. consider >>> re-entrancy to various subsystems that is not allowed). >> >> It is not too complex, what EnterInterpOnlyModeClosure::do_thread() is doing. I already added a >> paragraph to the JBS-Item [1] explaining why the direct handshake is sufficient from a >> synchronization point of view. > > Just to be clear, your proposed change is not using a direct handshake. > >> Furthermore the stack is walked and the return pc of compiled frames is replaced with the address of >> the deopt handler. >> >> I can't see why this cannot be done with a direct handshake. Something very similar is already done >> in JavaThread::deoptimize_marked_methods() which is executed as part of an ordinary handshake. > > Note that existing non-direct handshakes may also have issues that not > have been fully investigated. > >> The demand on stack-space should be very modest. I would not expect a higher risk for stackoverflow. > > For the target thread if you use more stack than would be used stopping > at a safepoint then you are at risk. For the thread initiating the > direct handshake if you use more stack than would be used enqueuing a VM > operation, then you are at risk. As we have not quantified these > numbers, nor have any easy way to establish the stack use of the actual > code to be executed, we're really just hoping for the best. This is a > general problem with handshakes that needs to be investigated more > deeply. As a simple, general, example just imagine if the code involves > logging that might utilise an on-stack buffer. > >>> Second, we have this question mark over what happens if the operation >>> hits further safepoint or handshake polls/checks? Are there constraints >>> on what is allowed here? How can we recognise this problem may exist and >>> so deal with it? >> >> The thread in EnterInterpOnlyModeClosure::do_thread() can't become safepoint/handshake safe. I >> tested locally test/hotspot/jtreg:vmTestbase_nsk_jvmti with a NoSafepointVerifier. > > That's good to hear but such tests are not exhaustive, they will detect > if you do reach a safepoint/handshake but they can't prove that you > cannot reach one. What you have done is necessary but may not be > sufficient. Plus you didn't actually add the NSV to the code - is there > a reason we can't actually keep it in do_thread? (I'm not sure if the > NSV also acts as a NoHandshakeVerifier?) > >>> Third, while we are generally considering what appear to be >>> single-thread operations, which should be amenable to a direct >>> handshake, we also have to be careful that some of the code involved >>> doesn't already expect/assume we are at a safepoint - e.g. a VM op may >>> not need to take a lock where a direct handshake might! >> >> See again my arguments in the JBS item [1]. > > Yes I see the reasoning and that is good. My point is a general one as > it may not be obvious when such assumptions exist in the current code. > > Thanks, > David > >> Thanks, >> Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8238585 >> >> -----Original Message----- >> From: David Holmes >> Sent: Montag, 27. April 2020 07:16 >> To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >> >> Hi all, >> >> Not a review but some general commentary ... >> >> On 25/04/2020 2:08 am, Reingruber, Richard wrote: >>> Hi Yasumasa, Patricio, >>> >>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>> Does it help you? I think it gives you to remove workaround. >>>>> >>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>> >>>> Thanks for your information. >>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>> I will modify and will test it after yours. >>> >>> Thanks :) >>> >>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>> to me, how this has to be handled. >>> >>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>> >>> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >>> also I'm unsure if a thread should do safepoint checks while executing a handshake. >> >> I'm growing increasingly concerned that use of direct handshakes to >> replace VM operations needs a much greater examination for correctness >> than might initially be thought. I see a number of issues: >> >> First, the VMThread executes (most) VM operations with a clean stack in >> a clean state, so it has lots of room to work. If we now execute the >> same logic in a JavaThread then we risk hitting stackoverflows if >> nothing else. But we are also now executing code in a JavaThread and so >> we have to be sure that code is not going to act differently (in a bad >> way) if executed by a JavaThread rather than the VMThread. For example, >> may it be possible that if executing in the VMThread we defer some >> activity that might require execution of Java code, or else hand it off >> to one of the service threads? If we execute that code directly in the >> current JavaThread instead we may not be in a valid state (e.g. consider >> re-entrancy to various subsystems that is not allowed). >> >> Second, we have this question mark over what happens if the operation >> hits further safepoint or handshake polls/checks? Are there constraints >> on what is allowed here? How can we recognise this problem may exist and >> so deal with it? >> >> Third, while we are generally considering what appear to be >> single-thread operations, which should be amenable to a direct >> handshake, we also have to be careful that some of the code involved >> doesn't already expect/assume we are at a safepoint - e.g. a VM op may >> not need to take a lock where a direct handshake might! >> >> Cheers, >> David >> ----- >> >>> @Patricio, coming back to my question [1]: >>> >>> In the example you gave in your answer [2]: the java thread would execute a vm operation during a >>> direct handshake operation, while the VMThread is actually in the middle of a VM_HandshakeAllThreads >>> operation, waiting to handshake the same handshakee: why can't the VMThread just proceed? The >>> handshakee would be safepoint safe, wouldn't it? >>> >>> Thanks, Richard. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301677&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301677 >>> >>> [2] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301763&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301763 >>> >>> -----Original Message----- >>> From: Yasumasa Suenaga >>> Sent: Freitag, 24. April 2020 17:23 >>> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>> >>> Hi Richard, >>> >>> On 2020/04/24 23:44, Reingruber, Richard wrote: >>>> Hi Yasumasa, >>>> >>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>> Does it help you? I think it gives you to remove workaround. >>>> >>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>> >>> Thanks for your information. >>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>> I will modify and will test it after yours. >>> >>> >>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>> to me, how this has to be handled. >>> >>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>>> So it appears to me that it would be easier to push JDK-8242427 after this (JDK-8238585). >>>> >>>>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>>> >>>> Would be interesting to see how you handled the issues above :) >>>> >>>> Thanks, Richard. >>>> >>>> [1] See question in comment https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14302030&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14302030 >>>> >>>> -----Original Message----- >>>> From: Yasumasa Suenaga >>>> Sent: Freitag, 24. April 2020 13:34 >>>> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>> >>>> Hi Richard, >>>> >>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>> Does it help you? I think it gives you to remove workaround. >>>> >>>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>> On 2020/04/24 17:18, Reingruber, Richard wrote: >>>>> Hi Patricio, Vladimir, and Serguei, >>>>> >>>>> now that direct handshakes are available, I've updated the patch to make use of them. >>>>> >>>>> In addition I have done some clean-up changes I missed in the first webrev. >>>>> >>>>> Finally I have implemented the workaround suggested by Patricio to avoid nesting the handshake >>>>> into the vm operation VM_SetFramePop [1] >>>>> >>>>> Kindly review again: >>>>> >>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1/ >>>>> Webrev(delta): http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1.inc/ >>>>> >>>>> I updated the JBS item explaining why the vm operation VM_EnterInterpOnlyMode can be replaced with a >>>>> direct handshake: >>>>> >>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>>> >>>>> Testing: >>>>> >>>>> * JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>>> >>>>> * Submit-repo: mach5-one-rrich-JDK-8238585-20200423-1436-10441737 >>>>> >>>>> Thanks, >>>>> Richard. >>>>> >>>>> [1] An assertion in Handshake::execute_direct() fails, if called be VMThread, because it is no JavaThread. >>>>> >>>>> -----Original Message----- >>>>> From: hotspot-dev On Behalf Of Reingruber, Richard >>>>> Sent: Freitag, 14. Februar 2020 19:47 >>>>> To: Patricio Chilano ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>> Subject: RE: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>> >>>>> Hi Patricio, >>>>> >>>>> > > I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>>> > > handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>>> > > Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>>> > > >>>>> > > > Alternatively I think you could do something similar to what we do in >>>>> > > > Deoptimization::deoptimize_all_marked(): >>>>> > > > >>>>> > > > EnterInterpOnlyModeClosure hs; >>>>> > > > if (SafepointSynchronize::is_at_safepoint()) { >>>>> > > > hs.do_thread(state->get_thread()); >>>>> > > > } else { >>>>> > > > Handshake::execute(&hs, state->get_thread()); >>>>> > > > } >>>>> > > > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>> > > > HandshakeClosure() constructor) >>>>> > > >>>>> > > Maybe this could be used also in the Handshake::execute() methods as general solution? >>>>> > Right, we could also do that. Avoiding to clear the polling page in >>>>> > HandshakeState::clear_handshake() should be enough to fix this issue and >>>>> > execute a handshake inside a safepoint, but adding that "if" statement >>>>> > in Hanshake::execute() sounds good to avoid all the extra code that we >>>>> > go through when executing a handshake. I filed 8239084 to make that change. >>>>> >>>>> Thanks for taking care of this and creating the RFE. >>>>> >>>>> > >>>>> > > > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>> > > > always called in a nested operation or just sometimes. >>>>> > > >>>>> > > At least one execution path without vm operation exists: >>>>> > > >>>>> > > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>>> > > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>>> > > JvmtiEventControllerPrivate::recompute_enabled() : void >>>>> > > JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>>> > > JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>>> > > JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>>> > > jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>>> > > >>>>> > > I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>>> > > handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>>> > > encouraged to do it with a handshake :) >>>>> > Ah! I think you can still do it with a handshake with the >>>>> > Deoptimization::deoptimize_all_marked() like solution. I can change the >>>>> > if-else statement with just the Handshake::execute() call in 8239084. >>>>> > But up to you. : ) >>>>> >>>>> Well, I think that's enough encouragement :) >>>>> I'll wait for 8239084 and try then again. >>>>> (no urgency and all) >>>>> >>>>> Thanks, >>>>> Richard. >>>>> >>>>> -----Original Message----- >>>>> From: Patricio Chilano >>>>> Sent: Freitag, 14. Februar 2020 15:54 >>>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>> >>>>> Hi Richard, >>>>> >>>>> On 2/14/20 9:58 AM, Reingruber, Richard wrote: >>>>>> Hi Patricio, >>>>>> >>>>>> thanks for having a look. >>>>>> >>>>>> > I?m only commenting on the handshake changes. >>>>>> > I see that operation VM_EnterInterpOnlyMode can be called inside >>>>>> > operation VM_SetFramePop which also allows nested operations. Here is a >>>>>> > comment in VM_SetFramePop definition: >>>>>> > >>>>>> > // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>>> > // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>>> > >>>>>> > So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>>> > could have a handshake inside a safepoint operation. The issue I see >>>>>> > there is that at the end of the handshake the polling page of the target >>>>>> > thread could be disarmed. So if the target thread happens to be in a >>>>>> > blocked state just transiently and wakes up then it will not stop for >>>>>> > the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>>> > polling page is armed at the beginning of disarm_safepoint(). >>>>>> >>>>>> I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>>>> handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>>>> Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>>>> >>>>>> > Alternatively I think you could do something similar to what we do in >>>>>> > Deoptimization::deoptimize_all_marked(): >>>>>> > >>>>>> > EnterInterpOnlyModeClosure hs; >>>>>> > if (SafepointSynchronize::is_at_safepoint()) { >>>>>> > hs.do_thread(state->get_thread()); >>>>>> > } else { >>>>>> > Handshake::execute(&hs, state->get_thread()); >>>>>> > } >>>>>> > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>>> > HandshakeClosure() constructor) >>>>>> >>>>>> Maybe this could be used also in the Handshake::execute() methods as general solution? >>>>> Right, we could also do that. Avoiding to clear the polling page in >>>>> HandshakeState::clear_handshake() should be enough to fix this issue and >>>>> execute a handshake inside a safepoint, but adding that "if" statement >>>>> in Hanshake::execute() sounds good to avoid all the extra code that we >>>>> go through when executing a handshake. I filed 8239084 to make that change. >>>>> >>>>>> > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>>> > always called in a nested operation or just sometimes. >>>>>> >>>>>> At least one execution path without vm operation exists: >>>>>> >>>>>> JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>>>> JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>>>> JvmtiEventControllerPrivate::recompute_enabled() : void >>>>>> JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>>>> JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>>>> JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>>>> jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>>>> >>>>>> I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>>>> handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>>>> encouraged to do it with a handshake :) >>>>> Ah! I think you can still do it with a handshake with the >>>>> Deoptimization::deoptimize_all_marked() like solution. I can change the >>>>> if-else statement with just the Handshake::execute() call in 8239084. >>>>> But up to you.? : ) >>>>> >>>>> Thanks, >>>>> Patricio >>>>>> Thanks again, >>>>>> Richard. >>>>>> >>>>>> -----Original Message----- >>>>>> From: Patricio Chilano >>>>>> Sent: Donnerstag, 13. Februar 2020 18:47 >>>>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>>> >>>>>> Hi Richard, >>>>>> >>>>>> I?m only commenting on the handshake changes. >>>>>> I see that operation VM_EnterInterpOnlyMode can be called inside >>>>>> operation VM_SetFramePop which also allows nested operations. Here is a >>>>>> comment in VM_SetFramePop definition: >>>>>> >>>>>> // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>>> // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>>> >>>>>> So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>>> could have a handshake inside a safepoint operation. The issue I see >>>>>> there is that at the end of the handshake the polling page of the target >>>>>> thread could be disarmed. So if the target thread happens to be in a >>>>>> blocked state just transiently and wakes up then it will not stop for >>>>>> the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>>> polling page is armed at the beginning of disarm_safepoint(). >>>>>> >>>>>> I think one option could be to remove >>>>>> SafepointMechanism::disarm_if_needed() in >>>>>> HandshakeState::clear_handshake() and let each JavaThread disarm itself >>>>>> for the handshake case. >>>>>> >>>>>> Alternatively I think you could do something similar to what we do in >>>>>> Deoptimization::deoptimize_all_marked(): >>>>>> >>>>>> ? EnterInterpOnlyModeClosure hs; >>>>>> ? if (SafepointSynchronize::is_at_safepoint()) { >>>>>> ??? hs.do_thread(state->get_thread()); >>>>>> ? } else { >>>>>> ??? Handshake::execute(&hs, state->get_thread()); >>>>>> ? } >>>>>> (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>>> HandshakeClosure() constructor) >>>>>> >>>>>> I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>>> always called in a nested operation or just sometimes. >>>>>> >>>>>> Thanks, >>>>>> Patricio >>>>>> >>>>>> On 2/12/20 7:23 AM, Reingruber, Richard wrote: >>>>>>> // Repost including hotspot runtime and gc lists. >>>>>>> // Dean Long suggested to do so, because the enhancement replaces a vm operation >>>>>>> // with a handshake. >>>>>>> // Original thread: http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-February/030359.html >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> could I please get reviews for this small enhancement in hotspot's jvmti implementation: >>>>>>> >>>>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.0/ >>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>>>>> >>>>>>> The change avoids making all compiled methods on stack not_entrant when switching a java thread to >>>>>>> interpreter only execution for jvmti purposes. It is sufficient to deoptimize the compiled frames on stack. >>>>>>> >>>>>>> Additionally a handshake is used instead of a vm operation to walk the stack and do the deoptimizations. >>>>>>> >>>>>>> Testing: JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>>>>> >>>>>>> Thanks, Richard. >>>>>>> >>>>>>> See also my question if anyone knows a reason for making the compiled methods not_entrant: >>>>>>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-January/030339.html >>>>> From daniel.daugherty at oracle.com Wed May 13 15:54:18 2020 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Wed, 13 May 2020 11:54:18 -0400 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: Message-ID: On 5/13/20 11:10 AM, Christian Hagedorn wrote: > Hi > > Please review the following patch: > https://bugs.openjdk.java.net/browse/JDK-8244946 > http://cr.openjdk.java.net/~chagedorn/8244946/webrev.00/ src/hotspot/cpu/x86/methodHandles_x86.cpp ??? I was looking at your expanded scope and I happened to see: ??? ? ? L525: ????? PRESERVE_EXCEPTION_MARK; // may not be needed by safer and unexpensive here ??? Not your bug, but the comment has two typos: ??????? s/by safer/but safer/ ??????? s/unexpensive/inexpensive/ ??? So your expanded scope now extends the PRESERVE_EXCEPTION_MARK over ??? this block: ??????? L571: ????? if (has_mh && oopDesc::is_oop(mh)) { ??????? L572: ??????? mh->print(); ??????? L573: ??????? if (java_lang_invoke_MethodHandle::is_instance(mh)) { ??????? L574: ????????? if (java_lang_invoke_MethodHandle::form_offset_in_bytes() != 0) { ??????? L575: java_lang_invoke_MethodHandle::form(mh)->print(); ??????? L576: ????????? } ??????? L577: ??????? } ??????? L578: ????? } ??? I don't see anything worrisome about extending the ??? PRESERVE_EXCEPTION_MARK, but I'm not really an invoke/MethodHandle ??? expert here. :-) src/hotspot/share/oops/klass.cpp ??? No comments. test/hotspot/jtreg/runtime/logging/TestMethodHandlesVerbose.java ??? Thanks for including a test. Thumbs up. Dan > > This simply fixes a missing ResourceMark required to call > Klass::oop_print_on(). > > The reason for the missing ResourceMark in oop_print_on() is stated in > the RFR [1] for JDK-8222893 [2] which removed the ResourceMark due to > a bug: > > "print functions with outputStream should not have a ResourceMark > because you could call it with a LogStream or stream that was > allocated outside the resource mark. The caller needs the ResourceMark." > > There are other places like Klass::print_on(outputStream*) where this > is still the case. Should we file an RFE to clean these up? > > Thank you! > > Best regards, > Christian > > > [1] > https://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2019-May/034095.html > [2] https://bugs.openjdk.java.net/browse/JDK-8222893 From john.r.rose at oracle.com Wed May 13 16:00:11 2020 From: john.r.rose at oracle.com (John Rose) Date: Wed, 13 May 2020 09:00:11 -0700 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: References: Message-ID: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> I like this cleanup! There are new tricky dependencies between the interpreter and other runtime bits, and class loading; they are adequately documented in new comments. A few comments of my own: The whitespace on the new line in vmSymbols.hpp needs to be adjusted to resemble whitespace on neighboring lines. Hotspot is a jungle of old code, and our new code needs to blend into that jungle, not make spots of new color (unless of course there is a purpose to some new color). I don?t want to gripe needlessly about identifier choice, but? maybe I do in this case: ?interpreter_init_code? and ?interpreter_init_code_stub? seem overly cryptic to me; they don?t help me remember what they do. I suggest that?interpreter_init_stubs? and ?interpreter_init_code? or ?interpreter_init_stub_code? and ?interpreter_init_main_code? would be easier on maintainers. But if it?s just me, leave it as is. We coders eat cryptic identifiers for breakfast. Can someone remind me (a) which platforms use the C++ interpreter, and (b) how well they are tested? I spot-checked for queries to the hard-coded offsets, and found enough ?guarantee(this_offset != 0?)? to make me feel secure. If there?s a bug related to this change, it?s quite possibly a zero offset being used on some path in the JVM which is rarely executed. (Such a bug could be introduced in the future, right?) Usually we avoid such bugs by encapsulating variable state into C++ access functions which perform their own assertions, rather than requiring users of the variable to make their own checks. Did you look into making all those public variables private, and wrapping access functions around them? I think that would make for a more complete cleanup. It would touch more places in the interpreter, including platform code, which I think would be a good thing. It would (a) allow review now of dependency checks in the interpreter, and (b) force downstream porters to similarly review their use of hard-coded offsets. ? John > On May 13, 2020, at 8:26 AM, Erik ?sterlund wrote: > > Hi, > > Let's remove hardcoded field offsets. Main trick to get this done is to split interpreter initialization > into two parts: 1. reserve the interpreter memory, 2. generate the interpreter. This allows reserving > it before loading methods (which generates i2c adapters that assert things are "in the interpreter", > requiring knowledge about its address range), and then load the methods, and then generate the interpreter. > This moving of interpreter code generation until after methods have been loaded allows offsets to be > looked up, needed by code generation. That way, we no longer need to hardcode field offsets, which > makes it less annoying to change the field layout code in the future. > > Thanks to Coleen for pre-reviewing this. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8243996 > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ > > Thanks, > /Erik From coleen.phillimore at oracle.com Wed May 13 18:07:39 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Wed, 13 May 2020 14:07:39 -0400 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> References: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> Message-ID: <1585c97d-3f3f-b39d-42a8-5d79d29fe8af@oracle.com> On 5/13/20 12:00 PM, John Rose wrote: > I like this cleanup! > > There are new tricky dependencies between the interpreter and other > runtime bits, and class loading; they are adequately documented in new > comments. > > A few comments of my own: > > The whitespace on the new line in vmSymbols.hpp needs to be adjusted to > resemble whitespace on neighboring lines. Hotspot is a jungle of old code, > and our new code needs to blend into that jungle, not make spots of new > color (unless of course there is a purpose to some new color). > > I don?t want to gripe needlessly about identifier choice, but? maybe I > do in this case: ?interpreter_init_code? and ?interpreter_init_code_stub? > seem overly cryptic to me; they don?t help me remember what they do. > I suggest that?interpreter_init_stubs? and ?interpreter_init_code? or > ?interpreter_init_stub_code? and ?interpreter_init_main_code? would > be easier on maintainers. But if it?s just me, leave it as is. We coders eat > cryptic identifiers for breakfast. Erik and I came up with these names earlier today as they were more descriptive than some original names.? "interpreter_init_stubs" and "interpreter_init_code" seem okay also. > > Can someone remind me (a) which platforms use the C++ interpreter, > and (b) how well they are tested? > > I spot-checked for queries to the hard-coded offsets, and found enough > ?guarantee(this_offset != 0?)? to make me feel secure. If there?s a bug > related to this change, it?s quite possibly a zero offset being used on some > path in the JVM which is rarely executed. (Such a bug could be introduced > in the future, right?) Usually we avoid such bugs by encapsulating variable > state into C++ access functions which perform their own assertions, rather > than requiring users of the variable to make their own checks. > > Did you look into making all those public variables private, and wrapping > access functions around them? I think that would make for a more complete > cleanup. It would touch more places in the interpreter, including platform > code, which I think would be a good thing. It would (a) allow review now > of dependency checks in the interpreter, and (b) force downstream porters > to similarly review their use of hard-coded offsets. This seems like a good suggestion for these classes.? For the most part, javaClasses foo_offset are private. I was going to reply to this thread "Looks good to me" and thanks for adding the comments. Coleen > > ? John > >> On May 13, 2020, at 8:26 AM, Erik ?sterlund wrote: >> >> Hi, >> >> Let's remove hardcoded field offsets. Main trick to get this done is to split interpreter initialization >> into two parts: 1. reserve the interpreter memory, 2. generate the interpreter. This allows reserving >> it before loading methods (which generates i2c adapters that assert things are "in the interpreter", >> requiring knowledge about its address range), and then load the methods, and then generate the interpreter. >> This moving of interpreter code generation until after methods have been loaded allows offsets to be >> looked up, needed by code generation. That way, we no longer need to hardcode field offsets, which >> makes it less annoying to change the field layout code in the future. >> >> Thanks to Coleen for pre-reviewing this. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8243996 >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ >> >> Thanks, >> /Erik From frederic.parain at oracle.com Wed May 13 18:23:12 2020 From: frederic.parain at oracle.com (Frederic Parain) Date: Wed, 13 May 2020 14:23:12 -0400 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: References: Message-ID: Erik, Thank you for fixing this long standing technical debt. Looks good to me, comments are very helpful. If you change accessibility as suggested by John, let me know if you need another review. Regards, Fred > On May 13, 2020, at 11:26, Erik ?sterlund wrote: > > Hi, > > Let's remove hardcoded field offsets. Main trick to get this done is to split interpreter initialization > into two parts: 1. reserve the interpreter memory, 2. generate the interpreter. This allows reserving > it before loading methods (which generates i2c adapters that assert things are "in the interpreter", > requiring knowledge about its address range), and then load the methods, and then generate the interpreter. > This moving of interpreter code generation until after methods have been loaded allows offsets to be > looked up, needed by code generation. That way, we no longer need to hardcode field offsets, which > makes it less annoying to change the field layout code in the future. > > Thanks to Coleen for pre-reviewing this. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8243996 > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ > > Thanks, > /Erik From vladimir.kozlov at oracle.com Wed May 13 18:44:16 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Wed, 13 May 2020 11:44:16 -0700 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: References: <56a0cf85-9698-64b2-95b4-e5b123c3e9e2@oracle.com> <9a314ebe-5499-ecb6-3c5b-0c8ce7f9c8d2@oracle.com> <44febab6-5703-1c8b-d4ad-0f4c12517afa@oracle.com> Message-ID: Hi Erik x86 changes looks fine to me (assembler, C1, C2). How src/hotspot/share/classfile/* changes are related to this? I don't get changes in metaspace.cpp. Why use HeapBaseMinAddress when COOP is off (looks like you want klass heap at the beginning of Javva heap)? Can you add comment? arguments.cpp - should you also check UseNewFieldLayout flag as you do in instanceOop.hpp? lcm.cpp - I concern about this change. This code is for compressed oops only - implicit null check is for oops. Thanks, Vladimir On 5/12/20 7:09 AM, Erik ?sterlund wrote: > Hi, > > I ran into some merge conflicts in the tests and the new tests behaved a bit differently > on different platforms. The new tests naturally do not behave the same way on the platforms that do > not yet support compressed class pointers without compressed oops, so I just filtered away those > platforms from running the new tests. Will update the filter when they start supporting it. > > Another interesting thing is that on macOS, sometimes we did not get the expected compressed class > pointer encoding without compressed oops. The reason is that without compressed oops, we allocate > the heap at some random address where the OS thinks is suitable. Sometimes that is where we would > ideally have put the compressed class space. This forces metaspace to be mapped in somewhere else > with different characteristics. Ideally, the bootstrapping logic should be disentangled so that > when you run without compressed oops, metaspace gets reserved before the heap. But I think that > warrants a follow-up RFE, as I do not want to complicate this patch any further. Disentangling that > seems like it could become quite tricky on its own. > > So here is another webrev with the test issues resolved: > http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ > > Incremental: > http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ > > Thanks, > /Erik > > On 2020-04-29 09:39, Erik ?sterlund wrote: >> Hi Thomas, >> >> On 2020-04-29 08:27, Thomas St?fe wrote: >>> Hi Erik, >>> >>> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund > wrote: >>> >>> ??? Hi Thomas, >>> >>> ??? Thanks for reviewing this. >>> >>> ??? On 2020-04-28 13:14, Thomas St?fe wrote: >>> >>>> ??? With compressed oops we continue to squeeze ccs in after the heap >>>> ??? in the hope that it all still fits into the lower 4G or 32G. (Do >>>> ??? we ever bother to check if heap is actually reserved in low area? >>>> ??? It may have given up and reserved upstate instead, leaving free >>>> ??? room below...). >>> >>> ??? Right. And yes I'm pretty sure we do check. The heap reserves >>> ??? memory, and sets the CompressedOops::end() based on that, which is >>> ??? fed as input to the metaspace/CDS placement. >>> >>> >>> No, what I meant was: since we reserve at CompressedOops::end(), if the heap was too large to place it in the lower >>> address regions, the class space will not live there either even if it could, since it is smaller and might fit. >>> >>> For example: >>> >>> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace >>> -Xshare:off -Xmx31g >>> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: reserved=1048576.00 KB, committed=0.00 KB ( ?0%), used=0.00 >>> KB ( ?0%) >>> [0.003s][trace][gc,metaspace] ?[0x00000017c0400000, 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) >>> [0.003s][trace][gc,metaspace] Narrow klass base: 0x00000017c0400000, Narrow klass shift: 0 >>> [0.003s][trace][gc,metaspace] Compressed class space size: 1073741824 Address: 0x00000017c0400000 Req Addr: >>> 0x00000017c0400000 >>> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: reserved=8192.00 KB, committed=0.00 KB ( ?0%), used=0.00 KB ( >>> ?0%) >>> [0.003s][trace][gc,metaspace] ?[0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) >>> >>> (large heap and no CDS, since it prevents zero based mode). >>> >>> In that case narrow klass base is not zero based even it if probably could be, if we were to allocate class space in >>> the lower regions, which are now unoccupied since the large heap does not fit there. A missed opportunity to go zero >>> based. Not a big deal, since we mostly run with CDS now anyway. >> >> I see. Yeah, there is room for improvement there. Although, as you say, people mostly run with CDS on, so it probably >> won't matter even if that is improved. >> >>>> ??? Now with your patch without compressed oops we attempt to map at >>>> ??? HeapBaseMinAddress since this is the lowest address we are >>>> ??? allowed to map at and the heap won't obviously need it,?right? >>> >>> ??? Exactly. >>> >>>> ??? My only misgiving would be that?HeapBaseMinAddress is even more >>>> ??? of a misnomer (its already now a?bit, witness my year long >>>> ??? confusion :-), since it now controls not only heap placement but >>>> ??? ccs too. Actually if it is that important to avoid mapping to >>>> ??? addresses below?HeapBaseMinAddress we should probably prevent it >>>> ??? somewhere below in the os layer but I really do not want to open >>>> ??? that can of worms. >>> >>> ??? Yes agreed. Ideally it would be called something like >>> ??? MappingMinAddress instead, because it really isn't tied to the >>> ??? heap really, it just happened to be the primary user of mapping >>> ??? things at low addresses. But I also think changing that, and >>> ??? enforcing it more widely in the os layer sounds like a different >>> ??? can of worms. >>> >>>> ??? -- >>>> >>>> ??? Some more remarks about arguments.cpp: >>>> >>>> ??? - In Arguments::set_use_compressed_klass_ptrs(), if now >>>> ??? compressed class pointers are independent from compressed oops, >>>> ??? checking CompressedClassSpaceSize for validity should now always >>>> ??? happen for UseCompressedClassPointers=1. >>> >>> ??? I think it looks like I do that. >>> >>>> ??? - If HeapBaseMinAddress is now always used, we should check it >>>> ??? always too (see Arguments::set_heap_size) >>> >>> ??? Good idea. Fixed. >>> >>>> >>>>> ??????? Also, could we have some?tests which exercise cds/metaspace >>>>> ??????? initialization without coops and with ccps? Easiest way >>>>> ??????? would be to extend >>>>> ??????? "CompressedOops/CompressedClassPointers.java" by some cases. >>>> >>>> >>>> ??? Thank you. I think this makes sense especially since compressed >>>> ??? class pointer encoding is restricted to one mode if CDS is on >>>> ??? (base = shared address and shift=3), and since CDS=on is the >>>> ??? default zerobased encoding may not happen that often. >>> >>> ??? Yup. I added some explicit test cases for -XX:-UseCompressedOops >>> ??? -XX:+UseCompressedClassPointers as suggested. This will also be >>> ??? the default configuration for ZGC, so that should give that >>> ??? configuration more test coverage in the various tests we have. >>> >>> ??? Here is a new webrev: >>> ??? http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ >>> >>> ??? Incremental: >>> ??? http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ >>> >>> ??? I hope I fixed everything you wanted in this patch. I also fixed >>> ??? what Frederic and Coleen wanted. >>> >>> >>> arguments.cpp, the metaspace stuff and the test look good to me now. Thank you for your work! >> >> Thanks for the review Thomas! >> >> /Erik > From vladimir.kozlov at oracle.com Wed May 13 19:35:16 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Wed, 13 May 2020 12:35:16 -0700 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: References: <56a0cf85-9698-64b2-95b4-e5b123c3e9e2@oracle.com> <9a314ebe-5499-ecb6-3c5b-0c8ce7f9c8d2@oracle.com> <44febab6-5703-1c8b-d4ad-0f4c12517afa@oracle.com> Message-ID: <8a4c0b9a-c96f-2c39-f9bf-7c0fdc54b2d3@oracle.com> At least lcm.cpp changes should be something like next because compressed oops and classes are independent now: if ((UseCompressedOops && CompressedOops::shift() == 0) || (UseCompressedClassPointers && CompressedKlassPointers::shift() == 0)) { Vladimir On 5/13/20 11:44 AM, Vladimir Kozlov wrote: > Hi Erik > > x86 changes looks fine to me (assembler, C1, C2). > > How src/hotspot/share/classfile/* changes are related to this? > > I don't get changes in metaspace.cpp. Why use HeapBaseMinAddress when COOP is off (looks like you want klass heap at the > beginning of Javva heap)? Can you add comment? > > arguments.cpp - should you also check UseNewFieldLayout flag as you do in instanceOop.hpp? > > lcm.cpp - I concern about this change. This code is for compressed oops only - implicit null check is for oops. > > Thanks, > Vladimir > > On 5/12/20 7:09 AM, Erik ?sterlund wrote: >> Hi, >> >> I ran into some merge conflicts in the tests and the new tests behaved a bit differently >> on different platforms. The new tests naturally do not behave the same way on the platforms that do >> not yet support compressed class pointers without compressed oops, so I just filtered away those >> platforms from running the new tests. Will update the filter when they start supporting it. >> >> Another interesting thing is that on macOS, sometimes we did not get the expected compressed class >> pointer encoding without compressed oops. The reason is that without compressed oops, we allocate >> the heap at some random address where the OS thinks is suitable. Sometimes that is where we would >> ideally have put the compressed class space. This forces metaspace to be mapped in somewhere else >> with different characteristics. Ideally, the bootstrapping logic should be disentangled so that >> when you run without compressed oops, metaspace gets reserved before the heap. But I think that >> warrants a follow-up RFE, as I do not want to complicate this patch any further. Disentangling that >> seems like it could become quite tricky on its own. >> >> So here is another webrev with the test issues resolved: >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ >> >> Incremental: >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ >> >> Thanks, >> /Erik >> >> On 2020-04-29 09:39, Erik ?sterlund wrote: >>> Hi Thomas, >>> >>> On 2020-04-29 08:27, Thomas St?fe wrote: >>>> Hi Erik, >>>> >>>> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund > wrote: >>>> >>>> ??? Hi Thomas, >>>> >>>> ??? Thanks for reviewing this. >>>> >>>> ??? On 2020-04-28 13:14, Thomas St?fe wrote: >>>> >>>>> ??? With compressed oops we continue to squeeze ccs in after the heap >>>>> ??? in the hope that it all still fits into the lower 4G or 32G. (Do >>>>> ??? we ever bother to check if heap is actually reserved in low area? >>>>> ??? It may have given up and reserved upstate instead, leaving free >>>>> ??? room below...). >>>> >>>> ??? Right. And yes I'm pretty sure we do check. The heap reserves >>>> ??? memory, and sets the CompressedOops::end() based on that, which is >>>> ??? fed as input to the metaspace/CDS placement. >>>> >>>> >>>> No, what I meant was: since we reserve at CompressedOops::end(), if the heap was too large to place it in the lower >>>> address regions, the class space will not live there either even if it could, since it is smaller and might fit. >>>> >>>> For example: >>>> >>>> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace >>>> -Xshare:off -Xmx31g >>>> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: reserved=1048576.00 KB, committed=0.00 KB ( ?0%), used=0.00 >>>> KB ( ?0%) >>>> [0.003s][trace][gc,metaspace] ?[0x00000017c0400000, 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) >>>> [0.003s][trace][gc,metaspace] Narrow klass base: 0x00000017c0400000, Narrow klass shift: 0 >>>> [0.003s][trace][gc,metaspace] Compressed class space size: 1073741824 Address: 0x00000017c0400000 Req Addr: >>>> 0x00000017c0400000 >>>> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: reserved=8192.00 KB, committed=0.00 KB ( ?0%), used=0.00 KB >>>> ( ?0%) >>>> [0.003s][trace][gc,metaspace] ?[0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) >>>> >>>> (large heap and no CDS, since it prevents zero based mode). >>>> >>>> In that case narrow klass base is not zero based even it if probably could be, if we were to allocate class space in >>>> the lower regions, which are now unoccupied since the large heap does not fit there. A missed opportunity to go zero >>>> based. Not a big deal, since we mostly run with CDS now anyway. >>> >>> I see. Yeah, there is room for improvement there. Although, as you say, people mostly run with CDS on, so it probably >>> won't matter even if that is improved. >>> >>>>> ??? Now with your patch without compressed oops we attempt to map at >>>>> ??? HeapBaseMinAddress since this is the lowest address we are >>>>> ??? allowed to map at and the heap won't obviously need it,?right? >>>> >>>> ??? Exactly. >>>> >>>>> ??? My only misgiving would be that?HeapBaseMinAddress is even more >>>>> ??? of a misnomer (its already now a?bit, witness my year long >>>>> ??? confusion :-), since it now controls not only heap placement but >>>>> ??? ccs too. Actually if it is that important to avoid mapping to >>>>> ??? addresses below?HeapBaseMinAddress we should probably prevent it >>>>> ??? somewhere below in the os layer but I really do not want to open >>>>> ??? that can of worms. >>>> >>>> ??? Yes agreed. Ideally it would be called something like >>>> ??? MappingMinAddress instead, because it really isn't tied to the >>>> ??? heap really, it just happened to be the primary user of mapping >>>> ??? things at low addresses. But I also think changing that, and >>>> ??? enforcing it more widely in the os layer sounds like a different >>>> ??? can of worms. >>>> >>>>> ??? -- >>>>> >>>>> ??? Some more remarks about arguments.cpp: >>>>> >>>>> ??? - In Arguments::set_use_compressed_klass_ptrs(), if now >>>>> ??? compressed class pointers are independent from compressed oops, >>>>> ??? checking CompressedClassSpaceSize for validity should now always >>>>> ??? happen for UseCompressedClassPointers=1. >>>> >>>> ??? I think it looks like I do that. >>>> >>>>> ??? - If HeapBaseMinAddress is now always used, we should check it >>>>> ??? always too (see Arguments::set_heap_size) >>>> >>>> ??? Good idea. Fixed. >>>> >>>>> >>>>>> ??????? Also, could we have some?tests which exercise cds/metaspace >>>>>> ??????? initialization without coops and with ccps? Easiest way >>>>>> ??????? would be to extend >>>>>> ??????? "CompressedOops/CompressedClassPointers.java" by some cases. >>>>> >>>>> >>>>> ??? Thank you. I think this makes sense especially since compressed >>>>> ??? class pointer encoding is restricted to one mode if CDS is on >>>>> ??? (base = shared address and shift=3), and since CDS=on is the >>>>> ??? default zerobased encoding may not happen that often. >>>> >>>> ??? Yup. I added some explicit test cases for -XX:-UseCompressedOops >>>> ??? -XX:+UseCompressedClassPointers as suggested. This will also be >>>> ??? the default configuration for ZGC, so that should give that >>>> ??? configuration more test coverage in the various tests we have. >>>> >>>> ??? Here is a new webrev: >>>> ??? http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ >>>> >>>> ??? Incremental: >>>> ??? http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ >>>> >>>> ??? I hope I fixed everything you wanted in this patch. I also fixed >>>> ??? what Frederic and Coleen wanted. >>>> >>>> >>>> arguments.cpp, the metaspace stuff and the test look good to me now. Thank you for your work! >>> >>> Thanks for the review Thomas! >>> >>> /Erik >> From serguei.spitsyn at oracle.com Wed May 13 22:31:47 2020 From: serguei.spitsyn at oracle.com (serguei.spitsyn at oracle.com) Date: Wed, 13 May 2020 15:31:47 -0700 Subject: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant In-Reply-To: References: <3c59b9f9-ec38-18c9-8f24-e1186a08a04a@oracle.com> <410eed04-e2ef-0f4f-1c56-19e6734a10f6@oracle.com> Message-ID: <03c9a0ce-8f78-00e7-9db3-70d6f6cb8156@oracle.com> Hi Richard, Thank you for the bug report update - it is helpful. The fix/update looks good in general but I need more time to check some points. I'm thinking it would be more safe to run full tier5. I can do it after you get all thumbs ups. Thanks, Serguei On 4/24/20 01:18, Reingruber, Richard wrote: > Hi Patricio, Vladimir, and Serguei, > > now that direct handshakes are available, I've updated the patch to make use of them. > > In addition I have done some clean-up changes I missed in the first webrev. > > Finally I have implemented the workaround suggested by Patricio to avoid nesting the handshake > into the vm operation VM_SetFramePop [1] > > Kindly review again: > > Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1/ > Webrev(delta): http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1.inc/ > > I updated the JBS item explaining why the vm operation VM_EnterInterpOnlyMode can be replaced with a > direct handshake: > > JBS: https://bugs.openjdk.java.net/browse/JDK-8238585 > > Testing: > > * JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. > > * Submit-repo: mach5-one-rrich-JDK-8238585-20200423-1436-10441737 > > Thanks, > Richard. > > [1] An assertion in Handshake::execute_direct() fails, if called be VMThread, because it is no JavaThread. > > -----Original Message----- > From: hotspot-dev On Behalf Of Reingruber, Richard > Sent: Freitag, 14. Februar 2020 19:47 > To: Patricio Chilano ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net > Subject: RE: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant > > Hi Patricio, > > > > I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a > > > handshake cannot be nested in a vm operation. Maybe it should be asserted in the > > > Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? > > > > > > > Alternatively I think you could do something similar to what we do in > > > > Deoptimization::deoptimize_all_marked(): > > > > > > > > EnterInterpOnlyModeClosure hs; > > > > if (SafepointSynchronize::is_at_safepoint()) { > > > > hs.do_thread(state->get_thread()); > > > > } else { > > > > Handshake::execute(&hs, state->get_thread()); > > > > } > > > > (you could pass ?EnterInterpOnlyModeClosure? directly to the > > > > HandshakeClosure() constructor) > > > > > > Maybe this could be used also in the Handshake::execute() methods as general solution? > > Right, we could also do that. Avoiding to clear the polling page in > > HandshakeState::clear_handshake() should be enough to fix this issue and > > execute a handshake inside a safepoint, but adding that "if" statement > > in Hanshake::execute() sounds good to avoid all the extra code that we > > go through when executing a handshake. I filed 8239084 to make that change. > > Thanks for taking care of this and creating the RFE. > > > > > > > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is > > > > always called in a nested operation or just sometimes. > > > > > > At least one execution path without vm operation exists: > > > > > > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void > > > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong > > > JvmtiEventControllerPrivate::recompute_enabled() : void > > > JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) > > > JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void > > > JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError > > > jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError > > > > > > I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a > > > handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further > > > encouraged to do it with a handshake :) > > Ah! I think you can still do it with a handshake with the > > Deoptimization::deoptimize_all_marked() like solution. I can change the > > if-else statement with just the Handshake::execute() call in 8239084. > > But up to you. : ) > > Well, I think that's enough encouragement :) > I'll wait for 8239084 and try then again. > (no urgency and all) > > Thanks, > Richard. > > -----Original Message----- > From: Patricio Chilano > Sent: Freitag, 14. Februar 2020 15:54 > To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net > Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant > > Hi Richard, > > On 2/14/20 9:58 AM, Reingruber, Richard wrote: >> Hi Patricio, >> >> thanks for having a look. >> >> > I?m only commenting on the handshake changes. >> > I see that operation VM_EnterInterpOnlyMode can be called inside >> > operation VM_SetFramePop which also allows nested operations. Here is a >> > comment in VM_SetFramePop definition: >> > >> > // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >> > // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >> > >> > So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >> > could have a handshake inside a safepoint operation. The issue I see >> > there is that at the end of the handshake the polling page of the target >> > thread could be disarmed. So if the target thread happens to be in a >> > blocked state just transiently and wakes up then it will not stop for >> > the ongoing safepoint. Maybe I can file an RFE to assert that the >> > polling page is armed at the beginning of disarm_safepoint(). >> >> I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >> handshake cannot be nested in a vm operation. Maybe it should be asserted in the >> Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >> >> > Alternatively I think you could do something similar to what we do in >> > Deoptimization::deoptimize_all_marked(): >> > >> > EnterInterpOnlyModeClosure hs; >> > if (SafepointSynchronize::is_at_safepoint()) { >> > hs.do_thread(state->get_thread()); >> > } else { >> > Handshake::execute(&hs, state->get_thread()); >> > } >> > (you could pass ?EnterInterpOnlyModeClosure? directly to the >> > HandshakeClosure() constructor) >> >> Maybe this could be used also in the Handshake::execute() methods as general solution? > Right, we could also do that. Avoiding to clear the polling page in > HandshakeState::clear_handshake() should be enough to fix this issue and > execute a handshake inside a safepoint, but adding that "if" statement > in Hanshake::execute() sounds good to avoid all the extra code that we > go through when executing a handshake. I filed 8239084 to make that change. > >> > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >> > always called in a nested operation or just sometimes. >> >> At least one execution path without vm operation exists: >> >> JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >> JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >> JvmtiEventControllerPrivate::recompute_enabled() : void >> JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >> JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >> JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >> jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >> >> I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >> handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >> encouraged to do it with a handshake :) > Ah! I think you can still do it with a handshake with the > Deoptimization::deoptimize_all_marked() like solution. I can change the > if-else statement with just the Handshake::execute() call in 8239084. > But up to you.? : ) > > Thanks, > Patricio >> Thanks again, >> Richard. >> >> -----Original Message----- >> From: Patricio Chilano >> Sent: Donnerstag, 13. Februar 2020 18:47 >> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >> >> Hi Richard, >> >> I?m only commenting on the handshake changes. >> I see that operation VM_EnterInterpOnlyMode can be called inside >> operation VM_SetFramePop which also allows nested operations. Here is a >> comment in VM_SetFramePop definition: >> >> // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >> // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >> >> So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >> could have a handshake inside a safepoint operation. The issue I see >> there is that at the end of the handshake the polling page of the target >> thread could be disarmed. So if the target thread happens to be in a >> blocked state just transiently and wakes up then it will not stop for >> the ongoing safepoint. Maybe I can file an RFE to assert that the >> polling page is armed at the beginning of disarm_safepoint(). >> >> I think one option could be to remove >> SafepointMechanism::disarm_if_needed() in >> HandshakeState::clear_handshake() and let each JavaThread disarm itself >> for the handshake case. >> >> Alternatively I think you could do something similar to what we do in >> Deoptimization::deoptimize_all_marked(): >> >> ? EnterInterpOnlyModeClosure hs; >> ? if (SafepointSynchronize::is_at_safepoint()) { >> ??? hs.do_thread(state->get_thread()); >> ? } else { >> ??? Handshake::execute(&hs, state->get_thread()); >> ? } >> (you could pass ?EnterInterpOnlyModeClosure? directly to the >> HandshakeClosure() constructor) >> >> I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >> always called in a nested operation or just sometimes. >> >> Thanks, >> Patricio >> >> On 2/12/20 7:23 AM, Reingruber, Richard wrote: >>> // Repost including hotspot runtime and gc lists. >>> // Dean Long suggested to do so, because the enhancement replaces a vm operation >>> // with a handshake. >>> // Original thread: http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-February/030359.html >>> >>> Hi, >>> >>> could I please get reviews for this small enhancement in hotspot's jvmti implementation: >>> >>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.0/ >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8238585 >>> >>> The change avoids making all compiled methods on stack not_entrant when switching a java thread to >>> interpreter only execution for jvmti purposes. It is sufficient to deoptimize the compiled frames on stack. >>> >>> Additionally a handshake is used instead of a vm operation to walk the stack and do the deoptimizations. >>> >>> Testing: JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>> >>> Thanks, Richard. >>> >>> See also my question if anyone knows a reason for making the compiled methods not_entrant: >>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-January/030339.html From david.holmes at oracle.com Wed May 13 23:51:55 2020 From: david.holmes at oracle.com (David Holmes) Date: Thu, 14 May 2020 09:51:55 +1000 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: Message-ID: Hi Christian, On 14/05/2020 1:10 am, Christian Hagedorn wrote: > Hi > > Please review the following patch: > https://bugs.openjdk.java.net/browse/JDK-8244946 > http://cr.openjdk.java.net/~chagedorn/8244946/webrev.00/ > > This simply fixes a missing ResourceMark required to call > Klass::oop_print_on(). > > The reason for the missing ResourceMark in oop_print_on() is stated in > the RFR [1] for JDK-8222893 [2] which removed the ResourceMark due to a > bug: > > "print functions with outputStream should not have a ResourceMark > because you could call it with a LogStream or stream that was allocated > outside the resource mark. The caller needs the ResourceMark." src/hotspot/cpu/x86/methodHandles_x86.cpp Given all of the code is inside: 504 if (Verbose) { I would just move the ResourceMark to the top of that block. We are going to use it unconditionally, and we are not releasing it until the end of the method regardless. That avoids the potential concern Dan raised about extending the scope of the PRESERVE_EXCEPTION_MARK. > There are other places like Klass::print_on(outputStream*) where this is > still the case. Should we file an RFE to clean these up? Obviously we don't have any tests running in a mode that causes this code to be executed, so yes it is probably a good idea to take a closer look at other call-sites and file a RFE if needed. Thanks, David ----- > Thank you! > > Best regards, > Christian > > > [1] > https://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2019-May/034095.html > > [2] https://bugs.openjdk.java.net/browse/JDK-8222893 From jiefu at tencent.com Thu May 14 02:03:44 2020 From: jiefu at tencent.com (=?utf-8?B?amllZnUo5YKF5p2wKQ==?=) Date: Thu, 14 May 2020 02:03:44 +0000 Subject: FW: RFR: 8244971: Zero VM is broken after JDK-8241825 (COMPRESSED_CLASS_POINTERS_DEPENDS_ON_COMPRESSED_OOPS not defined) Message-ID: <23C8FCCB-1218-458F-92D5-A5812A57A654@tencent.com> Forward to hotspot-dev. Thanks Yumin for your help. Best regards, Jie ?On 2020/5/14, 9:59 AM, "Yumin Qi" wrote: Hi, Jie The original codereview of 8241825 was posted on hotspot-dev at openjdk.java.net. You should post yours to that mailing alias too. Yumin On 5/13/20 6:34 PM, jiefu(??) wrote: > Hi all, > > May I get reviews for this fix? > > JBS: https://bugs.openjdk.java.net/browse/JDK-8244971 > Webrev: http://cr.openjdk.java.net/~jiefu/8244971/webrev.00/ > > Thanks a lot. > Best regards, > Jie From david.holmes at oracle.com Thu May 14 02:16:28 2020 From: david.holmes at oracle.com (David Holmes) Date: Thu, 14 May 2020 12:16:28 +1000 Subject: FW: RFR: 8244971: Zero VM is broken after JDK-8241825 (COMPRESSED_CLASS_POINTERS_DEPENDS_ON_COMPRESSED_OOPS not defined) In-Reply-To: <23C8FCCB-1218-458F-92D5-A5812A57A654@tencent.com> References: <23C8FCCB-1218-458F-92D5-A5812A57A654@tencent.com> Message-ID: Looks good and trivial. Thanks, David On 14/05/2020 12:03 pm, jiefu(??) wrote: > Forward to hotspot-dev. > > Thanks Yumin for your help. > Best regards, > Jie > > ?On 2020/5/14, 9:59 AM, "Yumin Qi" wrote: > > Hi, Jie > > The original codereview of 8241825 was posted on > hotspot-dev at openjdk.java.net. You should post yours to that mailing > alias too. > > Yumin > > On 5/13/20 6:34 PM, jiefu(??) wrote: > > Hi all, > > > > May I get reviews for this fix? > > > > JBS: https://bugs.openjdk.java.net/browse/JDK-8244971 > > Webrev: http://cr.openjdk.java.net/~jiefu/8244971/webrev.00/ > > > > Thanks a lot. > > Best regards, > > Jie > > > > From jiefu at tencent.com Thu May 14 02:44:58 2020 From: jiefu at tencent.com (=?utf-8?B?amllZnUo5YKF5p2wKQ==?=) Date: Thu, 14 May 2020 02:44:58 +0000 Subject: 8244971: Zero VM is broken after JDK-8241825 (COMPRESSED_CLASS_POINTERS_DEPENDS_ON_COMPRESSED_OOPS not defined) Message-ID: Thanks David for your review. Pushed. Best regards, Jie ?On 2020/5/14, 10:17 AM, "David Holmes" wrote: Looks good and trivial. Thanks, David On 14/05/2020 12:03 pm, jiefu(??) wrote: > Forward to hotspot-dev. > > Thanks Yumin for your help. > Best regards, > Jie > > On 2020/5/14, 9:59 AM, "Yumin Qi" wrote: > > Hi, Jie > > The original codereview of 8241825 was posted on > hotspot-dev at openjdk.java.net. You should post yours to that mailing > alias too. > > Yumin > > On 5/13/20 6:34 PM, jiefu(??) wrote: > > Hi all, > > > > May I get reviews for this fix? > > > > JBS: https://bugs.openjdk.java.net/browse/JDK-8244971 > > Webrev: http://cr.openjdk.java.net/~jiefu/8244971/webrev.00/ > > > > Thanks a lot. > > Best regards, > > Jie > > > > From david.holmes at oracle.com Thu May 14 05:02:04 2020 From: david.holmes at oracle.com (David Holmes) Date: Thu, 14 May 2020 15:02:04 +1000 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: References: <81b1d335-d898-5bc9-8aed-b105f8d73a2e@oracle.com> Message-ID: On 14/05/2020 12:28 am, Baesken, Matthias wrote: > Hello, here is a new webrev removing the comment : > > http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.1/ > > It removes the comment , and it also handles 0 for getFreeSwapSpaceSize > ( this can occur after JDK-8242480 : Negative value may be returned by getFreeSwapSpaceSize() in the docker was pushed recently ). Sorry Matthias but I still see this partly as a bug in the underlying container support code. If you get the warning then what does getTotalSwapSpaceSize() return? Surely it must be specified to return 0 or some such sentinel in such a case? Other comments: ! try { ! out.shouldMatch("OperatingSystemMXBean\\.getFreeSwapSpaceSize: [1-9][0-9]+"); ! } catch(RuntimeException ex) { // allow 0, see 8242480 ! out.shouldContain("OperatingSystemMXBean.getFreeSwapSpaceSize: 0"); ! } can't you just change the original regex to match >= 0? Style nit: avoid historical comments like "see 8242480" and "since 8231111 ...". Thanks, David > Best regards, Matthias > > -----Original Message----- > From: Baesken, Matthias > Sent: Mittwoch, 13. Mai 2020 10:11 > To: 'David Holmes' ; 'hotspot-dev at openjdk.java.net' > Subject: RE: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > >> But according to the comment immediately before your changes: >> >> // in case of warnings like : "Your kernel does not support swap limit >> capabilities or the cgroup is not mounted. Memory limited without swap." >> // the getTotalSwapSpaceSize does not return the expected result, but 0 >> >> we should be returning zero. So this seems to be a bug in the >> implementation not in the test. > > > Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values > that are normally expected. > Previously we indeed got "0" . > Probably the comment should be deleted . > > Best regards, Matthias > From david.holmes at oracle.com Thu May 14 05:19:36 2020 From: david.holmes at oracle.com (David Holmes) Date: Thu, 14 May 2020 15:19:36 +1000 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: References: Message-ID: Hi Erik, On 14/05/2020 1:26 am, Erik ?sterlund wrote: > Hi, > > Let's remove hardcoded field offsets. Main trick to get this done is to > split interpreter initialization > into two parts: 1. reserve the interpreter memory, 2. generate the > interpreter. This allows reserving > it before loading methods (which generates i2c adapters that assert > things are "in the interpreter", > requiring knowledge about its address range), and then load the methods, > and then generate the interpreter. > This moving of interpreter code generation until after methods have been > loaded allows offsets to be > looked up, needed by code generation. That way, we no longer need to > hardcode field offsets, which > makes it less annoying to change the field layout code in the future. > > Thanks to Coleen for pre-reviewing this. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8243996 > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ This seems like a good simplification, and allows for future flexibility. Initialization order changes are always tricky as the dependencies are not always obvious - there may be logging or error path related code that eventually assumes the interpreter has been generated - but we can deal with such issues if they arise. Generally all seems fine to me - I won't comment on the colour of the bikeshed. :) One small comment: src/hotspot/share/runtime/init.cpp ! void interpreter_init_code(); // after methods loaded given the later comment: + interpreter_init_code(); // after javaClasses_init and before any method gets linked (not loaded) should the first comment be expanded to: void interpreter_init_code(); // after methods loaded, but before they are linked ? Also IIUC the parenthetical "(not loaded)" alludes to the currently incorrect comment: void invocationCounter_init(); // before any methods loaded but once you've fixed that there's no context to know what the "(not loaded)" relates to so it can just be dropped. Thanks, David ----- > Thanks, > /Erik From david.holmes at oracle.com Thu May 14 05:29:22 2020 From: david.holmes at oracle.com (David Holmes) Date: Thu, 14 May 2020 15:29:22 +1000 Subject: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant In-Reply-To: References: <81d7caa8-4244-85f3-4d4e-78117fe5e25b@oss.nttdata.com> <550b95ac-8b29-1eb8-a507-533e81d02322@oracle.com> <9c49ea2d-e3b8-b576-1d17-d18ad87cd6ed@oracle.com> <36d5e2c0-c724-7ff7-d37e-decb5cc0005b@oracle.com> Message-ID: <9e64b51d-8ac8-9e9a-1f89-52ca897932a4@oracle.com> > Still not a review, or is it now? I'd say still not a review as I'm only looking at the general structure. Cheers, David On 14/05/2020 1:37 am, Reingruber, Richard wrote: > Hi David, > >> On 4/05/2020 8:33 pm, Reingruber, Richard wrote: >>> // Trimmed the list of recipients. If the list gets too long then the message needs to be approved >>> // by a moderator. > >> Yes I noticed that too :) In general if you send to hotspot-dev you >> shouldn't need to also send to hotspot-X-dev. > > Makes sense. Will do so next time. > >>> >>> This would be the post with the current webrev.1 >>> >>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-April/031245.html > >> Sorry I missed that update. Okay so this is working with direct >> handshakes now. > >> One style nit in jvmtiThreadState.cpp: > >> assert(SafepointSynchronize::is_at_safepoint() || >> ! (JavaThread *)Thread::current() == get_thread() || >> ! Thread::current() == get_thread()->active_handshaker(), >> ! "bad synchronization with owner thread"); > >> the ! lines should ident as follows > >> assert(SafepointSynchronize::is_at_safepoint() || >> (JavaThread *)Thread::current() == get_thread() || >> Thread::current() == get_thread()->active_handshaker(), >> ! "bad synchronization with owner thread"); > > Sure. > >> Lets see how this plays out. > > Hopefully not too bad... :) > >>> Not a review but some general commentary ... > > Still not a review, or is it now? > > Thanks, Richard. > > -----Original Message----- > From: David Holmes > Sent: Mittwoch, 13. Mai 2020 07:43 > To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net > Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant > > On 4/05/2020 8:33 pm, Reingruber, Richard wrote: >> // Trimmed the list of recipients. If the list gets too long then the message needs to be approved >> // by a moderator. > > Yes I noticed that too :) In general if you send to hotspot-dev you > shouldn't need to also send to hotspot-X-dev. > >> Hi David, > > Hi Richard, > >>> On 28/04/2020 12:09 am, Reingruber, Richard wrote: >>>> Hi David, >>>> >>>>> Not a review but some general commentary ... >>>> >>>> That's welcome. >> >>> Having had to take an even closer look now I have a review comment too :) >> >>> src/hotspot/share/prims/jvmtiThreadState.cpp >> >>> void JvmtiThreadState::invalidate_cur_stack_depth() { >>> ! assert(SafepointSynchronize::is_at_safepoint() || >>> ! (Thread::current()->is_VM_thread() && >>> get_thread()->is_vmthread_processing_handshake()) || >>> (JavaThread *)Thread::current() == get_thread(), >>> "must be current thread or at safepoint"); >> >> You're looking at an outdated webrev, I'm afraid. >> >> This would be the post with the current webrev.1 >> >> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-April/031245.html > > Sorry I missed that update. Okay so this is working with direct > handshakes now. > > One style nit in jvmtiThreadState.cpp: > > assert(SafepointSynchronize::is_at_safepoint() || > ! (JavaThread *)Thread::current() == get_thread() || > ! Thread::current() == get_thread()->active_handshaker(), > ! "bad synchronization with owner thread"); > > the ! lines should ident as follows > > assert(SafepointSynchronize::is_at_safepoint() || > (JavaThread *)Thread::current() == get_thread() || > Thread::current() == get_thread()->active_handshaker(), > ! "bad synchronization with owner thread"); > > Lets see how this plays out. > > Cheers, > David > >> Thanks, Richard. >> >> -----Original Message----- >> From: David Holmes >> Sent: Montag, 4. Mai 2020 08:51 >> To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >> >> Hi Richard, >> >> On 28/04/2020 12:09 am, Reingruber, Richard wrote: >>> Hi David, >>> >>>> Not a review but some general commentary ... >>> >>> That's welcome. >> >> Having had to take an even closer look now I have a review comment too :) >> >> src/hotspot/share/prims/jvmtiThreadState.cpp >> >> void JvmtiThreadState::invalidate_cur_stack_depth() { >> ! assert(SafepointSynchronize::is_at_safepoint() || >> ! (Thread::current()->is_VM_thread() && >> get_thread()->is_vmthread_processing_handshake()) || >> (JavaThread *)Thread::current() == get_thread(), >> "must be current thread or at safepoint"); >> >> The message needs updating to include handshakes. >> >> More below ... >> >>>> On 25/04/2020 2:08 am, Reingruber, Richard wrote: >>>>> Hi Yasumasa, Patricio, >>>>> >>>>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>>>> Does it help you? I think it gives you to remove workaround. >>>>>>> >>>>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>>>> >>>>>> Thanks for your information. >>>>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>>>> I will modify and will test it after yours. >>>>> >>>>> Thanks :) >>>>> >>>>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>>>> to me, how this has to be handled. >>>>> >>>>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>>>> >>>>> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >>>>> also I'm unsure if a thread should do safepoint checks while executing a handshake. >>> >>>> I'm growing increasingly concerned that use of direct handshakes to >>>> replace VM operations needs a much greater examination for correctness >>>> than might initially be thought. I see a number of issues: >>> >>> I agree. I'll address your concerns in the context of this review thread for JDK-8238585 below. >>> >>> In addition I would suggest to take the general part of the discussion to a dedicated thread or to >>> the review thread for JDK-8242427. I would like to keep this thread closer to its subject. >> >> I will focus on the issues in the context of this particular change >> then, though the issues themselves are applicable to all handshake >> situations (and more so with direct handshakes). This is mostly just >> discussion. >> >>>> First, the VMThread executes (most) VM operations with a clean stack in >>>> a clean state, so it has lots of room to work. If we now execute the >>>> same logic in a JavaThread then we risk hitting stackoverflows if >>>> nothing else. But we are also now executing code in a JavaThread and so >>>> we have to be sure that code is not going to act differently (in a bad >>>> way) if executed by a JavaThread rather than the VMThread. For example, >>>> may it be possible that if executing in the VMThread we defer some >>>> activity that might require execution of Java code, or else hand it off >>>> to one of the service threads? If we execute that code directly in the >>>> current JavaThread instead we may not be in a valid state (e.g. consider >>>> re-entrancy to various subsystems that is not allowed). >>> >>> It is not too complex, what EnterInterpOnlyModeClosure::do_thread() is doing. I already added a >>> paragraph to the JBS-Item [1] explaining why the direct handshake is sufficient from a >>> synchronization point of view. >> >> Just to be clear, your proposed change is not using a direct handshake. >> >>> Furthermore the stack is walked and the return pc of compiled frames is replaced with the address of >>> the deopt handler. >>> >>> I can't see why this cannot be done with a direct handshake. Something very similar is already done >>> in JavaThread::deoptimize_marked_methods() which is executed as part of an ordinary handshake. >> >> Note that existing non-direct handshakes may also have issues that not >> have been fully investigated. >> >>> The demand on stack-space should be very modest. I would not expect a higher risk for stackoverflow. >> >> For the target thread if you use more stack than would be used stopping >> at a safepoint then you are at risk. For the thread initiating the >> direct handshake if you use more stack than would be used enqueuing a VM >> operation, then you are at risk. As we have not quantified these >> numbers, nor have any easy way to establish the stack use of the actual >> code to be executed, we're really just hoping for the best. This is a >> general problem with handshakes that needs to be investigated more >> deeply. As a simple, general, example just imagine if the code involves >> logging that might utilise an on-stack buffer. >> >>>> Second, we have this question mark over what happens if the operation >>>> hits further safepoint or handshake polls/checks? Are there constraints >>>> on what is allowed here? How can we recognise this problem may exist and >>>> so deal with it? >>> >>> The thread in EnterInterpOnlyModeClosure::do_thread() can't become safepoint/handshake safe. I >>> tested locally test/hotspot/jtreg:vmTestbase_nsk_jvmti with a NoSafepointVerifier. >> >> That's good to hear but such tests are not exhaustive, they will detect >> if you do reach a safepoint/handshake but they can't prove that you >> cannot reach one. What you have done is necessary but may not be >> sufficient. Plus you didn't actually add the NSV to the code - is there >> a reason we can't actually keep it in do_thread? (I'm not sure if the >> NSV also acts as a NoHandshakeVerifier?) >> >>>> Third, while we are generally considering what appear to be >>>> single-thread operations, which should be amenable to a direct >>>> handshake, we also have to be careful that some of the code involved >>>> doesn't already expect/assume we are at a safepoint - e.g. a VM op may >>>> not need to take a lock where a direct handshake might! >>> >>> See again my arguments in the JBS item [1]. >> >> Yes I see the reasoning and that is good. My point is a general one as >> it may not be obvious when such assumptions exist in the current code. >> >> Thanks, >> David >> >>> Thanks, >>> Richard. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8238585 >>> >>> -----Original Message----- >>> From: David Holmes >>> Sent: Montag, 27. April 2020 07:16 >>> To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>> >>> Hi all, >>> >>> Not a review but some general commentary ... >>> >>> On 25/04/2020 2:08 am, Reingruber, Richard wrote: >>>> Hi Yasumasa, Patricio, >>>> >>>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>>> Does it help you? I think it gives you to remove workaround. >>>>>> >>>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>>> >>>>> Thanks for your information. >>>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>>> I will modify and will test it after yours. >>>> >>>> Thanks :) >>>> >>>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>>> to me, how this has to be handled. >>>> >>>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>>> >>>> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >>>> also I'm unsure if a thread should do safepoint checks while executing a handshake. >>> >>> I'm growing increasingly concerned that use of direct handshakes to >>> replace VM operations needs a much greater examination for correctness >>> than might initially be thought. I see a number of issues: >>> >>> First, the VMThread executes (most) VM operations with a clean stack in >>> a clean state, so it has lots of room to work. If we now execute the >>> same logic in a JavaThread then we risk hitting stackoverflows if >>> nothing else. But we are also now executing code in a JavaThread and so >>> we have to be sure that code is not going to act differently (in a bad >>> way) if executed by a JavaThread rather than the VMThread. For example, >>> may it be possible that if executing in the VMThread we defer some >>> activity that might require execution of Java code, or else hand it off >>> to one of the service threads? If we execute that code directly in the >>> current JavaThread instead we may not be in a valid state (e.g. consider >>> re-entrancy to various subsystems that is not allowed). >>> >>> Second, we have this question mark over what happens if the operation >>> hits further safepoint or handshake polls/checks? Are there constraints >>> on what is allowed here? How can we recognise this problem may exist and >>> so deal with it? >>> >>> Third, while we are generally considering what appear to be >>> single-thread operations, which should be amenable to a direct >>> handshake, we also have to be careful that some of the code involved >>> doesn't already expect/assume we are at a safepoint - e.g. a VM op may >>> not need to take a lock where a direct handshake might! >>> >>> Cheers, >>> David >>> ----- >>> >>>> @Patricio, coming back to my question [1]: >>>> >>>> In the example you gave in your answer [2]: the java thread would execute a vm operation during a >>>> direct handshake operation, while the VMThread is actually in the middle of a VM_HandshakeAllThreads >>>> operation, waiting to handshake the same handshakee: why can't the VMThread just proceed? The >>>> handshakee would be safepoint safe, wouldn't it? >>>> >>>> Thanks, Richard. >>>> >>>> [1] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301677&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301677 >>>> >>>> [2] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301763&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301763 >>>> >>>> -----Original Message----- >>>> From: Yasumasa Suenaga >>>> Sent: Freitag, 24. April 2020 17:23 >>>> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>> >>>> Hi Richard, >>>> >>>> On 2020/04/24 23:44, Reingruber, Richard wrote: >>>>> Hi Yasumasa, >>>>> >>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>> Does it help you? I think it gives you to remove workaround. >>>>> >>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>>> >>>> Thanks for your information. >>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>> I will modify and will test it after yours. >>>> >>>> >>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>> to me, how this has to be handled. >>>> >>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>>> So it appears to me that it would be easier to push JDK-8242427 after this (JDK-8238585). >>>>> >>>>>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>>>> >>>>> Would be interesting to see how you handled the issues above :) >>>>> >>>>> Thanks, Richard. >>>>> >>>>> [1] See question in comment https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14302030&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14302030 >>>>> >>>>> -----Original Message----- >>>>> From: Yasumasa Suenaga >>>>> Sent: Freitag, 24. April 2020 13:34 >>>>> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>> >>>>> Hi Richard, >>>>> >>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>> Does it help you? I think it gives you to remove workaround. >>>>> >>>>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> On 2020/04/24 17:18, Reingruber, Richard wrote: >>>>>> Hi Patricio, Vladimir, and Serguei, >>>>>> >>>>>> now that direct handshakes are available, I've updated the patch to make use of them. >>>>>> >>>>>> In addition I have done some clean-up changes I missed in the first webrev. >>>>>> >>>>>> Finally I have implemented the workaround suggested by Patricio to avoid nesting the handshake >>>>>> into the vm operation VM_SetFramePop [1] >>>>>> >>>>>> Kindly review again: >>>>>> >>>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1/ >>>>>> Webrev(delta): http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1.inc/ >>>>>> >>>>>> I updated the JBS item explaining why the vm operation VM_EnterInterpOnlyMode can be replaced with a >>>>>> direct handshake: >>>>>> >>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>>>> >>>>>> Testing: >>>>>> >>>>>> * JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>>>> >>>>>> * Submit-repo: mach5-one-rrich-JDK-8238585-20200423-1436-10441737 >>>>>> >>>>>> Thanks, >>>>>> Richard. >>>>>> >>>>>> [1] An assertion in Handshake::execute_direct() fails, if called be VMThread, because it is no JavaThread. >>>>>> >>>>>> -----Original Message----- >>>>>> From: hotspot-dev On Behalf Of Reingruber, Richard >>>>>> Sent: Freitag, 14. Februar 2020 19:47 >>>>>> To: Patricio Chilano ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>>> Subject: RE: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>>> >>>>>> Hi Patricio, >>>>>> >>>>>> > > I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>>>> > > handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>>>> > > Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>>>> > > >>>>>> > > > Alternatively I think you could do something similar to what we do in >>>>>> > > > Deoptimization::deoptimize_all_marked(): >>>>>> > > > >>>>>> > > > EnterInterpOnlyModeClosure hs; >>>>>> > > > if (SafepointSynchronize::is_at_safepoint()) { >>>>>> > > > hs.do_thread(state->get_thread()); >>>>>> > > > } else { >>>>>> > > > Handshake::execute(&hs, state->get_thread()); >>>>>> > > > } >>>>>> > > > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>>> > > > HandshakeClosure() constructor) >>>>>> > > >>>>>> > > Maybe this could be used also in the Handshake::execute() methods as general solution? >>>>>> > Right, we could also do that. Avoiding to clear the polling page in >>>>>> > HandshakeState::clear_handshake() should be enough to fix this issue and >>>>>> > execute a handshake inside a safepoint, but adding that "if" statement >>>>>> > in Hanshake::execute() sounds good to avoid all the extra code that we >>>>>> > go through when executing a handshake. I filed 8239084 to make that change. >>>>>> >>>>>> Thanks for taking care of this and creating the RFE. >>>>>> >>>>>> > >>>>>> > > > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>>> > > > always called in a nested operation or just sometimes. >>>>>> > > >>>>>> > > At least one execution path without vm operation exists: >>>>>> > > >>>>>> > > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>>>> > > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>>>> > > JvmtiEventControllerPrivate::recompute_enabled() : void >>>>>> > > JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>>>> > > JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>>>> > > JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>>>> > > jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>>>> > > >>>>>> > > I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>>>> > > handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>>>> > > encouraged to do it with a handshake :) >>>>>> > Ah! I think you can still do it with a handshake with the >>>>>> > Deoptimization::deoptimize_all_marked() like solution. I can change the >>>>>> > if-else statement with just the Handshake::execute() call in 8239084. >>>>>> > But up to you. : ) >>>>>> >>>>>> Well, I think that's enough encouragement :) >>>>>> I'll wait for 8239084 and try then again. >>>>>> (no urgency and all) >>>>>> >>>>>> Thanks, >>>>>> Richard. >>>>>> >>>>>> -----Original Message----- >>>>>> From: Patricio Chilano >>>>>> Sent: Freitag, 14. Februar 2020 15:54 >>>>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>>> >>>>>> Hi Richard, >>>>>> >>>>>> On 2/14/20 9:58 AM, Reingruber, Richard wrote: >>>>>>> Hi Patricio, >>>>>>> >>>>>>> thanks for having a look. >>>>>>> >>>>>>> > I?m only commenting on the handshake changes. >>>>>>> > I see that operation VM_EnterInterpOnlyMode can be called inside >>>>>>> > operation VM_SetFramePop which also allows nested operations. Here is a >>>>>>> > comment in VM_SetFramePop definition: >>>>>>> > >>>>>>> > // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>>>> > // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>>>> > >>>>>>> > So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>>>> > could have a handshake inside a safepoint operation. The issue I see >>>>>>> > there is that at the end of the handshake the polling page of the target >>>>>>> > thread could be disarmed. So if the target thread happens to be in a >>>>>>> > blocked state just transiently and wakes up then it will not stop for >>>>>>> > the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>>>> > polling page is armed at the beginning of disarm_safepoint(). >>>>>>> >>>>>>> I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>>>>> handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>>>>> Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>>>>> >>>>>>> > Alternatively I think you could do something similar to what we do in >>>>>>> > Deoptimization::deoptimize_all_marked(): >>>>>>> > >>>>>>> > EnterInterpOnlyModeClosure hs; >>>>>>> > if (SafepointSynchronize::is_at_safepoint()) { >>>>>>> > hs.do_thread(state->get_thread()); >>>>>>> > } else { >>>>>>> > Handshake::execute(&hs, state->get_thread()); >>>>>>> > } >>>>>>> > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>>>> > HandshakeClosure() constructor) >>>>>>> >>>>>>> Maybe this could be used also in the Handshake::execute() methods as general solution? >>>>>> Right, we could also do that. Avoiding to clear the polling page in >>>>>> HandshakeState::clear_handshake() should be enough to fix this issue and >>>>>> execute a handshake inside a safepoint, but adding that "if" statement >>>>>> in Hanshake::execute() sounds good to avoid all the extra code that we >>>>>> go through when executing a handshake. I filed 8239084 to make that change. >>>>>> >>>>>>> > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>>>> > always called in a nested operation or just sometimes. >>>>>>> >>>>>>> At least one execution path without vm operation exists: >>>>>>> >>>>>>> JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>>>>> JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>>>>> JvmtiEventControllerPrivate::recompute_enabled() : void >>>>>>> JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>>>>> JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>>>>> JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>>>>> jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>>>>> >>>>>>> I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>>>>> handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>>>>> encouraged to do it with a handshake :) >>>>>> Ah! I think you can still do it with a handshake with the >>>>>> Deoptimization::deoptimize_all_marked() like solution. I can change the >>>>>> if-else statement with just the Handshake::execute() call in 8239084. >>>>>> But up to you.? : ) >>>>>> >>>>>> Thanks, >>>>>> Patricio >>>>>>> Thanks again, >>>>>>> Richard. >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Patricio Chilano >>>>>>> Sent: Donnerstag, 13. Februar 2020 18:47 >>>>>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>>>> >>>>>>> Hi Richard, >>>>>>> >>>>>>> I?m only commenting on the handshake changes. >>>>>>> I see that operation VM_EnterInterpOnlyMode can be called inside >>>>>>> operation VM_SetFramePop which also allows nested operations. Here is a >>>>>>> comment in VM_SetFramePop definition: >>>>>>> >>>>>>> // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>>>> // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>>>> >>>>>>> So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>>>> could have a handshake inside a safepoint operation. The issue I see >>>>>>> there is that at the end of the handshake the polling page of the target >>>>>>> thread could be disarmed. So if the target thread happens to be in a >>>>>>> blocked state just transiently and wakes up then it will not stop for >>>>>>> the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>>>> polling page is armed at the beginning of disarm_safepoint(). >>>>>>> >>>>>>> I think one option could be to remove >>>>>>> SafepointMechanism::disarm_if_needed() in >>>>>>> HandshakeState::clear_handshake() and let each JavaThread disarm itself >>>>>>> for the handshake case. >>>>>>> >>>>>>> Alternatively I think you could do something similar to what we do in >>>>>>> Deoptimization::deoptimize_all_marked(): >>>>>>> >>>>>>> ? EnterInterpOnlyModeClosure hs; >>>>>>> ? if (SafepointSynchronize::is_at_safepoint()) { >>>>>>> ??? hs.do_thread(state->get_thread()); >>>>>>> ? } else { >>>>>>> ??? Handshake::execute(&hs, state->get_thread()); >>>>>>> ? } >>>>>>> (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>>>> HandshakeClosure() constructor) >>>>>>> >>>>>>> I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>>>> always called in a nested operation or just sometimes. >>>>>>> >>>>>>> Thanks, >>>>>>> Patricio >>>>>>> >>>>>>> On 2/12/20 7:23 AM, Reingruber, Richard wrote: >>>>>>>> // Repost including hotspot runtime and gc lists. >>>>>>>> // Dean Long suggested to do so, because the enhancement replaces a vm operation >>>>>>>> // with a handshake. >>>>>>>> // Original thread: http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-February/030359.html >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> could I please get reviews for this small enhancement in hotspot's jvmti implementation: >>>>>>>> >>>>>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.0/ >>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>>>>>> >>>>>>>> The change avoids making all compiled methods on stack not_entrant when switching a java thread to >>>>>>>> interpreter only execution for jvmti purposes. It is sufficient to deoptimize the compiled frames on stack. >>>>>>>> >>>>>>>> Additionally a handshake is used instead of a vm operation to walk the stack and do the deoptimizations. >>>>>>>> >>>>>>>> Testing: JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>>>>>> >>>>>>>> Thanks, Richard. >>>>>>>> >>>>>>>> See also my question if anyone knows a reason for making the compiled methods not_entrant: >>>>>>>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-January/030339.html >>>>>> From erik.osterlund at oracle.com Thu May 14 07:31:49 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 14 May 2020 09:31:49 +0200 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: References: <56a0cf85-9698-64b2-95b4-e5b123c3e9e2@oracle.com> <9a314ebe-5499-ecb6-3c5b-0c8ce7f9c8d2@oracle.com> <44febab6-5703-1c8b-d4ad-0f4c12517afa@oracle.com> Message-ID: <1742cad7-15ac-ad6e-e4ba-4cdb3596fcb4@oracle.com> Hi Kozlov, Thanks for the review. I pushed this before the review though, so any comments I will deal with in a follow-up RFR. Let'swork out first what needs changing. On 2020-05-13 20:44, Vladimir Kozlov wrote: > Hi Erik > > x86 changes looks fine to me (assembler, C1, C2). Thanks. > How src/hotspot/share/classfile/* changes are related to this? Just enabling -coop +ccp gives us back 4 bytes in the klass gap. But the layout engine for InstanceKlass doesn't putany fields there, it aligns up to the next word by default. In other words, a lot of changes for almostno gain (except arrays that utilize this), unless the klass gap is given back to the layout code in this configuration. So I gave the klass gap to the fieldlayout code, which caused some fan out. Those files are the result of that. > I don't get changes in metaspace.cpp. Why use HeapBaseMinAddress when > COOP is off (looks like you want klass heap at the beginning of Javva > heap)? Can you add comment? I can add a comment. It's because HeapBaseMinAddress is the JVM switch for the lowest address we can map in memory without getting dangerously close to bad sbrk interactions, e.g. due to poor malloc implementations. Had a conversation with Thomas about this, and the conclusion is that the name of the flag is unfortunate but it is the value I should be using. > arguments.cpp - should you also check UseNewFieldLayout flag as you do > in instanceOop.hpp? That is not necessary. The -coop +ccp combination with -XX:-UseNewFieldLayout still works fine, it just happens to be that it is unable to fully benefit from the fact that class pointers are being used for instanceOops, but it still does benefit for arrays, which is not as great, but not wrong. > lcm.cpp - I concern about this change. This code is for compressed > oops only - implicit null check is for oops. Right. I wasn't quite sure about this one. It seemed like the if statement captured the condition that there *could* be a compressed pointer without shift, but could also include cases where this is impossible, i.e. allowing false positives (but not false negatives, crucial for correctness). For example, if you previously ran with +coop (shift 3) and -ccp (shift 0), then this if would take the "then" path, despite it being impossible for a compressed pointer with shift 0 being present. So I continued to model it as having false positives but importantly no false negatives, like before. Having said that, perhaps we could increase the accuracy of that check, getting rid of false negatives. I don't believe that is a correctness fixthough, but rather a way of having the JIT spend a few cycles less in some configurations. The then clausealways works fine, but the else clause does not, but is faster to evaluate. Thanks, /Erik > Thanks, > Vladimir > > On 5/12/20 7:09 AM, Erik ?sterlund wrote: >> Hi, >> >> I ran into some merge conflicts in the tests and the new tests >> behaved a bit differently >> on different platforms. The new tests naturally do not behave the >> same way on the platforms that do >> not yet support compressed class pointers without compressed oops, so >> I just filtered away those >> platforms from running the new tests. Will update the filter when >> they start supporting it. >> >> Another interesting thing is that on macOS, sometimes we did not get >> the expected compressed class >> pointer encoding without compressed oops. The reason is that without >> compressed oops, we allocate >> the heap at some random address where the OS thinks is suitable. >> Sometimes that is where we would >> ideally have put the compressed class space. This forces metaspace to >> be mapped in somewhere else >> with different characteristics. Ideally, the bootstrapping logic >> should be disentangled so that >> when you run without compressed oops, metaspace gets reserved before >> the heap. But I think that >> warrants a follow-up RFE, as I do not want to complicate this patch >> any further. Disentangling that >> seems like it could become quite tricky on its own. >> >> So here is another webrev with the test issues resolved: >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ >> >> Incremental: >> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ >> >> Thanks, >> /Erik >> >> On 2020-04-29 09:39, Erik ?sterlund wrote: >>> Hi Thomas, >>> >>> On 2020-04-29 08:27, Thomas St?fe wrote: >>>> Hi Erik, >>>> >>>> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund >>>> > wrote: >>>> >>>> ??? Hi Thomas, >>>> >>>> ??? Thanks for reviewing this. >>>> >>>> ??? On 2020-04-28 13:14, Thomas St?fe wrote: >>>> >>>>> ??? With compressed oops we continue to squeeze ccs in after the heap >>>>> ??? in the hope that it all still fits into the lower 4G or 32G. (Do >>>>> ??? we ever bother to check if heap is actually reserved in low area? >>>>> ??? It may have given up and reserved upstate instead, leaving free >>>>> ??? room below...). >>>> >>>> ??? Right. And yes I'm pretty sure we do check. The heap reserves >>>> ??? memory, and sets the CompressedOops::end() based on that, which is >>>> ??? fed as input to the metaspace/CDS placement. >>>> >>>> >>>> No, what I meant was: since we reserve at CompressedOops::end(), if >>>> the heap was too large to place it in the lower address regions, >>>> the class space will not live there either even if it could, since >>>> it is smaller and might fit. >>>> >>>> For example: >>>> >>>> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ >>>> ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace >>>> -Xshare:off -Xmx31g >>>> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: >>>> reserved=1048576.00 KB, committed=0.00 KB ( ?0%), used=0.00 KB ( ?0%) >>>> [0.003s][trace][gc,metaspace] ?[0x00000017c0400000, >>>> 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) >>>> [0.003s][trace][gc,metaspace] Narrow klass base: >>>> 0x00000017c0400000, Narrow klass shift: 0 >>>> [0.003s][trace][gc,metaspace] Compressed class space size: >>>> 1073741824 Address: 0x00000017c0400000 Req Addr: 0x00000017c0400000 >>>> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: >>>> reserved=8192.00 KB, committed=0.00 KB ( ?0%), used=0.00 KB ( ?0%) >>>> [0.003s][trace][gc,metaspace] ?[0x00007fc9cb800000, >>>> 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) >>>> >>>> (large heap and no CDS, since it prevents zero based mode). >>>> >>>> In that case narrow klass base is not zero based even it if >>>> probably could be, if we were to allocate class space in the lower >>>> regions, which are now unoccupied since the large heap does not fit >>>> there. A missed opportunity to go zero based. Not a big deal, since >>>> we mostly run with CDS now anyway. >>> >>> I see. Yeah, there is room for improvement there. Although, as you >>> say, people mostly run with CDS on, so it probably won't matter even >>> if that is improved. >>> >>>>> ??? Now with your patch without compressed oops we attempt to map at >>>>> ??? HeapBaseMinAddress since this is the lowest address we are >>>>> ??? allowed to map at and the heap won't obviously need it,?right? >>>> >>>> ??? Exactly. >>>> >>>>> ??? My only misgiving would be that?HeapBaseMinAddress is even more >>>>> ??? of a misnomer (its already now a?bit, witness my year long >>>>> ??? confusion :-), since it now controls not only heap placement but >>>>> ??? ccs too. Actually if it is that important to avoid mapping to >>>>> ??? addresses below?HeapBaseMinAddress we should probably prevent it >>>>> ??? somewhere below in the os layer but I really do not want to open >>>>> ??? that can of worms. >>>> >>>> ??? Yes agreed. Ideally it would be called something like >>>> ??? MappingMinAddress instead, because it really isn't tied to the >>>> ??? heap really, it just happened to be the primary user of mapping >>>> ??? things at low addresses. But I also think changing that, and >>>> ??? enforcing it more widely in the os layer sounds like a different >>>> ??? can of worms. >>>> >>>>> ??? -- >>>>> >>>>> ??? Some more remarks about arguments.cpp: >>>>> >>>>> ??? - In Arguments::set_use_compressed_klass_ptrs(), if now >>>>> ??? compressed class pointers are independent from compressed oops, >>>>> ??? checking CompressedClassSpaceSize for validity should now always >>>>> ??? happen for UseCompressedClassPointers=1. >>>> >>>> ??? I think it looks like I do that. >>>> >>>>> ??? - If HeapBaseMinAddress is now always used, we should check it >>>>> ??? always too (see Arguments::set_heap_size) >>>> >>>> ??? Good idea. Fixed. >>>> >>>>> >>>>>> ??????? Also, could we have some?tests which exercise cds/metaspace >>>>>> ??????? initialization without coops and with ccps? Easiest way >>>>>> ??????? would be to extend >>>>>> "CompressedOops/CompressedClassPointers.java" by some cases. >>>>> >>>>> >>>>> ??? Thank you. I think this makes sense especially since compressed >>>>> ??? class pointer encoding is restricted to one mode if CDS is on >>>>> ??? (base = shared address and shift=3), and since CDS=on is the >>>>> ??? default zerobased encoding may not happen that often. >>>> >>>> ??? Yup. I added some explicit test cases for -XX:-UseCompressedOops >>>> ??? -XX:+UseCompressedClassPointers as suggested. This will also be >>>> ??? the default configuration for ZGC, so that should give that >>>> ??? configuration more test coverage in the various tests we have. >>>> >>>> ??? Here is a new webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ >>>> >>>> ??? Incremental: >>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ >>>> >>>> ??? I hope I fixed everything you wanted in this patch. I also fixed >>>> ??? what Frederic and Coleen wanted. >>>> >>>> >>>> arguments.cpp, the metaspace stuff and the test look good to me >>>> now. Thank you for your work! >>> >>> Thanks for the review Thomas! >>> >>> /Erik >> From richard.reingruber at sap.com Thu May 14 07:38:30 2020 From: richard.reingruber at sap.com (Reingruber, Richard) Date: Thu, 14 May 2020 07:38:30 +0000 Subject: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant In-Reply-To: <9e64b51d-8ac8-9e9a-1f89-52ca897932a4@oracle.com> References: <81d7caa8-4244-85f3-4d4e-78117fe5e25b@oss.nttdata.com> <550b95ac-8b29-1eb8-a507-533e81d02322@oracle.com> <9c49ea2d-e3b8-b576-1d17-d18ad87cd6ed@oracle.com> <36d5e2c0-c724-7ff7-d37e-decb5cc0005b@oracle.com> <9e64b51d-8ac8-9e9a-1f89-52ca897932a4@oracle.com> Message-ID: Ok. Thanks for the feedback anyways. Cheers, Richard. -----Original Message----- From: David Holmes Sent: Donnerstag, 14. Mai 2020 07:29 To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant > Still not a review, or is it now? I'd say still not a review as I'm only looking at the general structure. Cheers, David On 14/05/2020 1:37 am, Reingruber, Richard wrote: > Hi David, > >> On 4/05/2020 8:33 pm, Reingruber, Richard wrote: >>> // Trimmed the list of recipients. If the list gets too long then the message needs to be approved >>> // by a moderator. > >> Yes I noticed that too :) In general if you send to hotspot-dev you >> shouldn't need to also send to hotspot-X-dev. > > Makes sense. Will do so next time. > >>> >>> This would be the post with the current webrev.1 >>> >>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-April/031245.html > >> Sorry I missed that update. Okay so this is working with direct >> handshakes now. > >> One style nit in jvmtiThreadState.cpp: > >> assert(SafepointSynchronize::is_at_safepoint() || >> ! (JavaThread *)Thread::current() == get_thread() || >> ! Thread::current() == get_thread()->active_handshaker(), >> ! "bad synchronization with owner thread"); > >> the ! lines should ident as follows > >> assert(SafepointSynchronize::is_at_safepoint() || >> (JavaThread *)Thread::current() == get_thread() || >> Thread::current() == get_thread()->active_handshaker(), >> ! "bad synchronization with owner thread"); > > Sure. > >> Lets see how this plays out. > > Hopefully not too bad... :) > >>> Not a review but some general commentary ... > > Still not a review, or is it now? > > Thanks, Richard. > > -----Original Message----- > From: David Holmes > Sent: Mittwoch, 13. Mai 2020 07:43 > To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net > Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant > > On 4/05/2020 8:33 pm, Reingruber, Richard wrote: >> // Trimmed the list of recipients. If the list gets too long then the message needs to be approved >> // by a moderator. > > Yes I noticed that too :) In general if you send to hotspot-dev you > shouldn't need to also send to hotspot-X-dev. > >> Hi David, > > Hi Richard, > >>> On 28/04/2020 12:09 am, Reingruber, Richard wrote: >>>> Hi David, >>>> >>>>> Not a review but some general commentary ... >>>> >>>> That's welcome. >> >>> Having had to take an even closer look now I have a review comment too :) >> >>> src/hotspot/share/prims/jvmtiThreadState.cpp >> >>> void JvmtiThreadState::invalidate_cur_stack_depth() { >>> ! assert(SafepointSynchronize::is_at_safepoint() || >>> ! (Thread::current()->is_VM_thread() && >>> get_thread()->is_vmthread_processing_handshake()) || >>> (JavaThread *)Thread::current() == get_thread(), >>> "must be current thread or at safepoint"); >> >> You're looking at an outdated webrev, I'm afraid. >> >> This would be the post with the current webrev.1 >> >> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-April/031245.html > > Sorry I missed that update. Okay so this is working with direct > handshakes now. > > One style nit in jvmtiThreadState.cpp: > > assert(SafepointSynchronize::is_at_safepoint() || > ! (JavaThread *)Thread::current() == get_thread() || > ! Thread::current() == get_thread()->active_handshaker(), > ! "bad synchronization with owner thread"); > > the ! lines should ident as follows > > assert(SafepointSynchronize::is_at_safepoint() || > (JavaThread *)Thread::current() == get_thread() || > Thread::current() == get_thread()->active_handshaker(), > ! "bad synchronization with owner thread"); > > Lets see how this plays out. > > Cheers, > David > >> Thanks, Richard. >> >> -----Original Message----- >> From: David Holmes >> Sent: Montag, 4. Mai 2020 08:51 >> To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >> >> Hi Richard, >> >> On 28/04/2020 12:09 am, Reingruber, Richard wrote: >>> Hi David, >>> >>>> Not a review but some general commentary ... >>> >>> That's welcome. >> >> Having had to take an even closer look now I have a review comment too :) >> >> src/hotspot/share/prims/jvmtiThreadState.cpp >> >> void JvmtiThreadState::invalidate_cur_stack_depth() { >> ! assert(SafepointSynchronize::is_at_safepoint() || >> ! (Thread::current()->is_VM_thread() && >> get_thread()->is_vmthread_processing_handshake()) || >> (JavaThread *)Thread::current() == get_thread(), >> "must be current thread or at safepoint"); >> >> The message needs updating to include handshakes. >> >> More below ... >> >>>> On 25/04/2020 2:08 am, Reingruber, Richard wrote: >>>>> Hi Yasumasa, Patricio, >>>>> >>>>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>>>> Does it help you? I think it gives you to remove workaround. >>>>>>> >>>>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>>>> >>>>>> Thanks for your information. >>>>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>>>> I will modify and will test it after yours. >>>>> >>>>> Thanks :) >>>>> >>>>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>>>> to me, how this has to be handled. >>>>> >>>>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>>>> >>>>> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >>>>> also I'm unsure if a thread should do safepoint checks while executing a handshake. >>> >>>> I'm growing increasingly concerned that use of direct handshakes to >>>> replace VM operations needs a much greater examination for correctness >>>> than might initially be thought. I see a number of issues: >>> >>> I agree. I'll address your concerns in the context of this review thread for JDK-8238585 below. >>> >>> In addition I would suggest to take the general part of the discussion to a dedicated thread or to >>> the review thread for JDK-8242427. I would like to keep this thread closer to its subject. >> >> I will focus on the issues in the context of this particular change >> then, though the issues themselves are applicable to all handshake >> situations (and more so with direct handshakes). This is mostly just >> discussion. >> >>>> First, the VMThread executes (most) VM operations with a clean stack in >>>> a clean state, so it has lots of room to work. If we now execute the >>>> same logic in a JavaThread then we risk hitting stackoverflows if >>>> nothing else. But we are also now executing code in a JavaThread and so >>>> we have to be sure that code is not going to act differently (in a bad >>>> way) if executed by a JavaThread rather than the VMThread. For example, >>>> may it be possible that if executing in the VMThread we defer some >>>> activity that might require execution of Java code, or else hand it off >>>> to one of the service threads? If we execute that code directly in the >>>> current JavaThread instead we may not be in a valid state (e.g. consider >>>> re-entrancy to various subsystems that is not allowed). >>> >>> It is not too complex, what EnterInterpOnlyModeClosure::do_thread() is doing. I already added a >>> paragraph to the JBS-Item [1] explaining why the direct handshake is sufficient from a >>> synchronization point of view. >> >> Just to be clear, your proposed change is not using a direct handshake. >> >>> Furthermore the stack is walked and the return pc of compiled frames is replaced with the address of >>> the deopt handler. >>> >>> I can't see why this cannot be done with a direct handshake. Something very similar is already done >>> in JavaThread::deoptimize_marked_methods() which is executed as part of an ordinary handshake. >> >> Note that existing non-direct handshakes may also have issues that not >> have been fully investigated. >> >>> The demand on stack-space should be very modest. I would not expect a higher risk for stackoverflow. >> >> For the target thread if you use more stack than would be used stopping >> at a safepoint then you are at risk. For the thread initiating the >> direct handshake if you use more stack than would be used enqueuing a VM >> operation, then you are at risk. As we have not quantified these >> numbers, nor have any easy way to establish the stack use of the actual >> code to be executed, we're really just hoping for the best. This is a >> general problem with handshakes that needs to be investigated more >> deeply. As a simple, general, example just imagine if the code involves >> logging that might utilise an on-stack buffer. >> >>>> Second, we have this question mark over what happens if the operation >>>> hits further safepoint or handshake polls/checks? Are there constraints >>>> on what is allowed here? How can we recognise this problem may exist and >>>> so deal with it? >>> >>> The thread in EnterInterpOnlyModeClosure::do_thread() can't become safepoint/handshake safe. I >>> tested locally test/hotspot/jtreg:vmTestbase_nsk_jvmti with a NoSafepointVerifier. >> >> That's good to hear but such tests are not exhaustive, they will detect >> if you do reach a safepoint/handshake but they can't prove that you >> cannot reach one. What you have done is necessary but may not be >> sufficient. Plus you didn't actually add the NSV to the code - is there >> a reason we can't actually keep it in do_thread? (I'm not sure if the >> NSV also acts as a NoHandshakeVerifier?) >> >>>> Third, while we are generally considering what appear to be >>>> single-thread operations, which should be amenable to a direct >>>> handshake, we also have to be careful that some of the code involved >>>> doesn't already expect/assume we are at a safepoint - e.g. a VM op may >>>> not need to take a lock where a direct handshake might! >>> >>> See again my arguments in the JBS item [1]. >> >> Yes I see the reasoning and that is good. My point is a general one as >> it may not be obvious when such assumptions exist in the current code. >> >> Thanks, >> David >> >>> Thanks, >>> Richard. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8238585 >>> >>> -----Original Message----- >>> From: David Holmes >>> Sent: Montag, 27. April 2020 07:16 >>> To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>> >>> Hi all, >>> >>> Not a review but some general commentary ... >>> >>> On 25/04/2020 2:08 am, Reingruber, Richard wrote: >>>> Hi Yasumasa, Patricio, >>>> >>>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>>> Does it help you? I think it gives you to remove workaround. >>>>>> >>>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>>> >>>>> Thanks for your information. >>>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>>> I will modify and will test it after yours. >>>> >>>> Thanks :) >>>> >>>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>>> to me, how this has to be handled. >>>> >>>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>>> >>>> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >>>> also I'm unsure if a thread should do safepoint checks while executing a handshake. >>> >>> I'm growing increasingly concerned that use of direct handshakes to >>> replace VM operations needs a much greater examination for correctness >>> than might initially be thought. I see a number of issues: >>> >>> First, the VMThread executes (most) VM operations with a clean stack in >>> a clean state, so it has lots of room to work. If we now execute the >>> same logic in a JavaThread then we risk hitting stackoverflows if >>> nothing else. But we are also now executing code in a JavaThread and so >>> we have to be sure that code is not going to act differently (in a bad >>> way) if executed by a JavaThread rather than the VMThread. For example, >>> may it be possible that if executing in the VMThread we defer some >>> activity that might require execution of Java code, or else hand it off >>> to one of the service threads? If we execute that code directly in the >>> current JavaThread instead we may not be in a valid state (e.g. consider >>> re-entrancy to various subsystems that is not allowed). >>> >>> Second, we have this question mark over what happens if the operation >>> hits further safepoint or handshake polls/checks? Are there constraints >>> on what is allowed here? How can we recognise this problem may exist and >>> so deal with it? >>> >>> Third, while we are generally considering what appear to be >>> single-thread operations, which should be amenable to a direct >>> handshake, we also have to be careful that some of the code involved >>> doesn't already expect/assume we are at a safepoint - e.g. a VM op may >>> not need to take a lock where a direct handshake might! >>> >>> Cheers, >>> David >>> ----- >>> >>>> @Patricio, coming back to my question [1]: >>>> >>>> In the example you gave in your answer [2]: the java thread would execute a vm operation during a >>>> direct handshake operation, while the VMThread is actually in the middle of a VM_HandshakeAllThreads >>>> operation, waiting to handshake the same handshakee: why can't the VMThread just proceed? The >>>> handshakee would be safepoint safe, wouldn't it? >>>> >>>> Thanks, Richard. >>>> >>>> [1] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301677&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301677 >>>> >>>> [2] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301763&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301763 >>>> >>>> -----Original Message----- >>>> From: Yasumasa Suenaga >>>> Sent: Freitag, 24. April 2020 17:23 >>>> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>> >>>> Hi Richard, >>>> >>>> On 2020/04/24 23:44, Reingruber, Richard wrote: >>>>> Hi Yasumasa, >>>>> >>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>> Does it help you? I think it gives you to remove workaround. >>>>> >>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>>> >>>> Thanks for your information. >>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>> I will modify and will test it after yours. >>>> >>>> >>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>> to me, how this has to be handled. >>>> >>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>>> >>>> >>>> Thanks, >>>> >>>> Yasumasa >>>> >>>> >>>>> So it appears to me that it would be easier to push JDK-8242427 after this (JDK-8238585). >>>>> >>>>>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>>>> >>>>> Would be interesting to see how you handled the issues above :) >>>>> >>>>> Thanks, Richard. >>>>> >>>>> [1] See question in comment https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14302030&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14302030 >>>>> >>>>> -----Original Message----- >>>>> From: Yasumasa Suenaga >>>>> Sent: Freitag, 24. April 2020 13:34 >>>>> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>> >>>>> Hi Richard, >>>>> >>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>> Does it help you? I think it gives you to remove workaround. >>>>> >>>>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Yasumasa >>>>> >>>>> >>>>> On 2020/04/24 17:18, Reingruber, Richard wrote: >>>>>> Hi Patricio, Vladimir, and Serguei, >>>>>> >>>>>> now that direct handshakes are available, I've updated the patch to make use of them. >>>>>> >>>>>> In addition I have done some clean-up changes I missed in the first webrev. >>>>>> >>>>>> Finally I have implemented the workaround suggested by Patricio to avoid nesting the handshake >>>>>> into the vm operation VM_SetFramePop [1] >>>>>> >>>>>> Kindly review again: >>>>>> >>>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1/ >>>>>> Webrev(delta): http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1.inc/ >>>>>> >>>>>> I updated the JBS item explaining why the vm operation VM_EnterInterpOnlyMode can be replaced with a >>>>>> direct handshake: >>>>>> >>>>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>>>> >>>>>> Testing: >>>>>> >>>>>> * JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>>>> >>>>>> * Submit-repo: mach5-one-rrich-JDK-8238585-20200423-1436-10441737 >>>>>> >>>>>> Thanks, >>>>>> Richard. >>>>>> >>>>>> [1] An assertion in Handshake::execute_direct() fails, if called be VMThread, because it is no JavaThread. >>>>>> >>>>>> -----Original Message----- >>>>>> From: hotspot-dev On Behalf Of Reingruber, Richard >>>>>> Sent: Freitag, 14. Februar 2020 19:47 >>>>>> To: Patricio Chilano ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>>> Subject: RE: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>>> >>>>>> Hi Patricio, >>>>>> >>>>>> > > I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>>>> > > handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>>>> > > Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>>>> > > >>>>>> > > > Alternatively I think you could do something similar to what we do in >>>>>> > > > Deoptimization::deoptimize_all_marked(): >>>>>> > > > >>>>>> > > > EnterInterpOnlyModeClosure hs; >>>>>> > > > if (SafepointSynchronize::is_at_safepoint()) { >>>>>> > > > hs.do_thread(state->get_thread()); >>>>>> > > > } else { >>>>>> > > > Handshake::execute(&hs, state->get_thread()); >>>>>> > > > } >>>>>> > > > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>>> > > > HandshakeClosure() constructor) >>>>>> > > >>>>>> > > Maybe this could be used also in the Handshake::execute() methods as general solution? >>>>>> > Right, we could also do that. Avoiding to clear the polling page in >>>>>> > HandshakeState::clear_handshake() should be enough to fix this issue and >>>>>> > execute a handshake inside a safepoint, but adding that "if" statement >>>>>> > in Hanshake::execute() sounds good to avoid all the extra code that we >>>>>> > go through when executing a handshake. I filed 8239084 to make that change. >>>>>> >>>>>> Thanks for taking care of this and creating the RFE. >>>>>> >>>>>> > >>>>>> > > > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>>> > > > always called in a nested operation or just sometimes. >>>>>> > > >>>>>> > > At least one execution path without vm operation exists: >>>>>> > > >>>>>> > > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>>>> > > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>>>> > > JvmtiEventControllerPrivate::recompute_enabled() : void >>>>>> > > JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>>>> > > JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>>>> > > JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>>>> > > jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>>>> > > >>>>>> > > I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>>>> > > handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>>>> > > encouraged to do it with a handshake :) >>>>>> > Ah! I think you can still do it with a handshake with the >>>>>> > Deoptimization::deoptimize_all_marked() like solution. I can change the >>>>>> > if-else statement with just the Handshake::execute() call in 8239084. >>>>>> > But up to you. : ) >>>>>> >>>>>> Well, I think that's enough encouragement :) >>>>>> I'll wait for 8239084 and try then again. >>>>>> (no urgency and all) >>>>>> >>>>>> Thanks, >>>>>> Richard. >>>>>> >>>>>> -----Original Message----- >>>>>> From: Patricio Chilano >>>>>> Sent: Freitag, 14. Februar 2020 15:54 >>>>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>>> >>>>>> Hi Richard, >>>>>> >>>>>> On 2/14/20 9:58 AM, Reingruber, Richard wrote: >>>>>>> Hi Patricio, >>>>>>> >>>>>>> thanks for having a look. >>>>>>> >>>>>>> > I?m only commenting on the handshake changes. >>>>>>> > I see that operation VM_EnterInterpOnlyMode can be called inside >>>>>>> > operation VM_SetFramePop which also allows nested operations. Here is a >>>>>>> > comment in VM_SetFramePop definition: >>>>>>> > >>>>>>> > // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>>>> > // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>>>> > >>>>>>> > So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>>>> > could have a handshake inside a safepoint operation. The issue I see >>>>>>> > there is that at the end of the handshake the polling page of the target >>>>>>> > thread could be disarmed. So if the target thread happens to be in a >>>>>>> > blocked state just transiently and wakes up then it will not stop for >>>>>>> > the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>>>> > polling page is armed at the beginning of disarm_safepoint(). >>>>>>> >>>>>>> I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>>>>> handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>>>>> Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>>>>> >>>>>>> > Alternatively I think you could do something similar to what we do in >>>>>>> > Deoptimization::deoptimize_all_marked(): >>>>>>> > >>>>>>> > EnterInterpOnlyModeClosure hs; >>>>>>> > if (SafepointSynchronize::is_at_safepoint()) { >>>>>>> > hs.do_thread(state->get_thread()); >>>>>>> > } else { >>>>>>> > Handshake::execute(&hs, state->get_thread()); >>>>>>> > } >>>>>>> > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>>>> > HandshakeClosure() constructor) >>>>>>> >>>>>>> Maybe this could be used also in the Handshake::execute() methods as general solution? >>>>>> Right, we could also do that. Avoiding to clear the polling page in >>>>>> HandshakeState::clear_handshake() should be enough to fix this issue and >>>>>> execute a handshake inside a safepoint, but adding that "if" statement >>>>>> in Hanshake::execute() sounds good to avoid all the extra code that we >>>>>> go through when executing a handshake. I filed 8239084 to make that change. >>>>>> >>>>>>> > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>>>> > always called in a nested operation or just sometimes. >>>>>>> >>>>>>> At least one execution path without vm operation exists: >>>>>>> >>>>>>> JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>>>>> JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>>>>> JvmtiEventControllerPrivate::recompute_enabled() : void >>>>>>> JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>>>>> JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>>>>> JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>>>>> jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>>>>> >>>>>>> I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>>>>> handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>>>>> encouraged to do it with a handshake :) >>>>>> Ah! I think you can still do it with a handshake with the >>>>>> Deoptimization::deoptimize_all_marked() like solution. I can change the >>>>>> if-else statement with just the Handshake::execute() call in 8239084. >>>>>> But up to you.? : ) >>>>>> >>>>>> Thanks, >>>>>> Patricio >>>>>>> Thanks again, >>>>>>> Richard. >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Patricio Chilano >>>>>>> Sent: Donnerstag, 13. Februar 2020 18:47 >>>>>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>>>> >>>>>>> Hi Richard, >>>>>>> >>>>>>> I?m only commenting on the handshake changes. >>>>>>> I see that operation VM_EnterInterpOnlyMode can be called inside >>>>>>> operation VM_SetFramePop which also allows nested operations. Here is a >>>>>>> comment in VM_SetFramePop definition: >>>>>>> >>>>>>> // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>>>> // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>>>> >>>>>>> So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>>>> could have a handshake inside a safepoint operation. The issue I see >>>>>>> there is that at the end of the handshake the polling page of the target >>>>>>> thread could be disarmed. So if the target thread happens to be in a >>>>>>> blocked state just transiently and wakes up then it will not stop for >>>>>>> the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>>>> polling page is armed at the beginning of disarm_safepoint(). >>>>>>> >>>>>>> I think one option could be to remove >>>>>>> SafepointMechanism::disarm_if_needed() in >>>>>>> HandshakeState::clear_handshake() and let each JavaThread disarm itself >>>>>>> for the handshake case. >>>>>>> >>>>>>> Alternatively I think you could do something similar to what we do in >>>>>>> Deoptimization::deoptimize_all_marked(): >>>>>>> >>>>>>> ? EnterInterpOnlyModeClosure hs; >>>>>>> ? if (SafepointSynchronize::is_at_safepoint()) { >>>>>>> ??? hs.do_thread(state->get_thread()); >>>>>>> ? } else { >>>>>>> ??? Handshake::execute(&hs, state->get_thread()); >>>>>>> ? } >>>>>>> (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>>>> HandshakeClosure() constructor) >>>>>>> >>>>>>> I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>>>> always called in a nested operation or just sometimes. >>>>>>> >>>>>>> Thanks, >>>>>>> Patricio >>>>>>> >>>>>>> On 2/12/20 7:23 AM, Reingruber, Richard wrote: >>>>>>>> // Repost including hotspot runtime and gc lists. >>>>>>>> // Dean Long suggested to do so, because the enhancement replaces a vm operation >>>>>>>> // with a handshake. >>>>>>>> // Original thread: http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-February/030359.html >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> could I please get reviews for this small enhancement in hotspot's jvmti implementation: >>>>>>>> >>>>>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.0/ >>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>>>>>> >>>>>>>> The change avoids making all compiled methods on stack not_entrant when switching a java thread to >>>>>>>> interpreter only execution for jvmti purposes. It is sufficient to deoptimize the compiled frames on stack. >>>>>>>> >>>>>>>> Additionally a handshake is used instead of a vm operation to walk the stack and do the deoptimizations. >>>>>>>> >>>>>>>> Testing: JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>>>>>> >>>>>>>> Thanks, Richard. >>>>>>>> >>>>>>>> See also my question if anyone knows a reason for making the compiled methods not_entrant: >>>>>>>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-January/030339.html >>>>>> From christian.hagedorn at oracle.com Thu May 14 08:00:47 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Thu, 14 May 2020 10:00:47 +0200 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: Message-ID: Hi David, hi Daniel Thank you for your reviews! On 13.05.20 17:54, Daniel D. Daugherty wrote: > Not your bug, but the comment has two typos: > > s/by safer/but safer/ > s/unexpensive/inexpensive/ Thanks, fixed it. On 14.05.20 01:51, David Holmes wrote: > src/hotspot/cpu/x86/methodHandles_x86.cpp > > Given all of the code is inside: > > ?504?? if (Verbose) { > > I would just move the ResourceMark to the top of that block. We are > going to use it unconditionally, and we are not releasing it until the > end of the method regardless. That avoids the potential concern Dan > raised about extending the scope of the PRESERVE_EXCEPTION_MARK. This sounds reasonable. I moved it to the top of that block. >> There are other places like Klass::print_on(outputStream*) where this >> is still the case. Should we file an RFE to clean these up? > > Obviously we don't have any tests running in a mode that causes this > code to be executed, so yes it is probably a good idea to take a closer > look at other call-sites and file a RFE if needed. Yes, I filed one to take a closer look at these [1]. I included the changes in a new webrev: http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ Best regards, Christian [1] https://bugs.openjdk.java.net/browse/JDK-8244999 From erik.osterlund at oracle.com Thu May 14 09:30:27 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 14 May 2020 11:30:27 +0200 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> References: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> Message-ID: Hi John, Thank you for reviewing this change and sharing your thoughts. On 2020-05-13 18:00, John Rose wrote: > I like this cleanup! > > There are new tricky dependencies between the interpreter and other > runtime bits, and class loading; they are adequately documented in new > comments. > > A few comments of my own: > > The whitespace on the new line in vmSymbols.hpp needs to be adjusted to > resemble whitespace on neighboring lines. Hotspot is a jungle of old code, > and our new code needs to blend into that jungle, not make spots of new > color (unless of course there is a purpose to some new color). Ah, missed that. Fixed. Emacs nearly dies every time I open that file; looks like I was too eager to get out of that file. > I don?t want to gripe needlessly about identifier choice, but? maybe I > do in this case: ?interpreter_init_code? and ?interpreter_init_code_stub? > seem overly cryptic to me; they don?t help me remember what they do. > I suggest that?interpreter_init_stubs? and ?interpreter_init_code? or > ?interpreter_init_stub_code? and ?interpreter_init_main_code? would > be easier on maintainers. But if it?s just me, leave it as is. We coders eat > cryptic identifiers for breakfast. Ah yes. Coleen and I have already been through a few rounds of naming for this. I started off with initialize1 and initialize2, similar to other initialization functions. But Coleen accurately pointed out those names do not reflect what it does. So I changed the name to initialize_code_stub and initialize_generate, but Coleen accurately pointed out that initialize_generate is grammatically incorrect. So I changed it to initialize_code to be grammatically correct yet still state what is being initialized. But I agree with you that unfortunately the difference between initializing a code stub and code is a bit unclear. I liked your first suggestion more than the second suggestion, so let's take that for a spin and see if anyone else wants a different name. > Can someone remind me (a) which platforms use the C++ interpreter, > and (b) how well they are tested? Unfortunately, I do not know. > I spot-checked for queries to the hard-coded offsets, and found enough > ?guarantee(this_offset != 0?)? to make me feel secure. If there?s a bug > related to this change, it?s quite possibly a zero offset being used on some > path in the JVM which is rarely executed. (Such a bug could be introduced > in the future, right?) Usually we avoid such bugs by encapsulating variable > state into C++ access functions which perform their own assertions, rather > than requiring users of the variable to make their own checks. Right. > Did you look into making all those public variables private, and wrapping > access functions around them? I think that would make for a more complete > cleanup. It would touch more places in the interpreter, including platform > code, which I think would be a good thing. It would (a) allow review now > of dependency checks in the interpreter, and (b) force downstream porters > to similarly review their use of hard-coded offsets. Indeed I noted that odd old colour, but was not sure I had a strong enough reason to change it to a new colour, given the fan out. ;) Among the public variables you suggest making private, It seems that only the referent is used in platform specific code (interpreter), and is used in the same way on all platforms. And the whole point of this exercise was to remove *all* dependencies from interpreter code to offset initialization order. In other words, before the interpreter code had to explicitly know exactly which offsets it could vs could not use, based on whether they were hardcoded or not. Now it can use all offsets as it pleases, by design. Therefore I don't think we achieve neither (a) nor (b) by changing this colour to a new one. Having said that, it certainly is nicer and good practice to make these variables private and use accessor functions for them, for stylistic, consistency and encapsulation reasons. I also noticed that half of the offset variables use a leading '_' and the others do not, making the code quite inconsistent. I could send a new RFR trying to normalize the style of the various offsets in JavaClasses to use a leading '_', be private and have an accessor function that asserts offset sanity. What do you think? Not sure how I got here implementing support for compressed class pointers in the absence of compressed oops. Oh dear... New webrev so far: http://cr.openjdk.java.net/~eosterlund/8243996/webrev.02/ Incremental: http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01..02/ Thanks, /Erik > ? John > >> On May 13, 2020, at 8:26 AM, Erik ?sterlund wrote: >> >> Hi, >> >> Let's remove hardcoded field offsets. Main trick to get this done is to split interpreter initialization >> into two parts: 1. reserve the interpreter memory, 2. generate the interpreter. This allows reserving >> it before loading methods (which generates i2c adapters that assert things are "in the interpreter", >> requiring knowledge about its address range), and then load the methods, and then generate the interpreter. >> This moving of interpreter code generation until after methods have been loaded allows offsets to be >> looked up, needed by code generation. That way, we no longer need to hardcode field offsets, which >> makes it less annoying to change the field layout code in the future. >> >> Thanks to Coleen for pre-reviewing this. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8243996 >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ >> >> Thanks, >> /Erik From erik.osterlund at oracle.com Thu May 14 09:32:44 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 14 May 2020 11:32:44 +0200 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: References: Message-ID: Hi Fred, Thanks for the review. I am hoping to move change of accessibility as well as some other random inconsistencies regarding the field offsets to a new RFR instead. Let's see how that goes. Thanks, /Erik On 2020-05-13 20:23, Frederic Parain wrote: > Erik, > > Thank you for fixing this long standing technical debt. > Looks good to me, comments are very helpful. > If you change accessibility as suggested by John, let > me know if you need another review. > > Regards, > > Fred > >> On May 13, 2020, at 11:26, Erik ?sterlund wrote: >> >> Hi, >> >> Let's remove hardcoded field offsets. Main trick to get this done is to split interpreter initialization >> into two parts: 1. reserve the interpreter memory, 2. generate the interpreter. This allows reserving >> it before loading methods (which generates i2c adapters that assert things are "in the interpreter", >> requiring knowledge about its address range), and then load the methods, and then generate the interpreter. >> This moving of interpreter code generation until after methods have been loaded allows offsets to be >> looked up, needed by code generation. That way, we no longer need to hardcode field offsets, which >> makes it less annoying to change the field layout code in the future. >> >> Thanks to Coleen for pre-reviewing this. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8243996 >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ >> >> Thanks, >> /Erik From erik.osterlund at oracle.com Thu May 14 09:37:00 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 14 May 2020 11:37:00 +0200 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: References: Message-ID: <1ea48e4a-3409-3f5d-7b3d-2c25af6ddc8e@oracle.com> Hi David, Thanks for having a look at this! On 2020-05-14 07:19, David Holmes wrote: > Hi Erik, > > On 14/05/2020 1:26 am, Erik ?sterlund wrote: >> Hi, >> >> Let's remove hardcoded field offsets. Main trick to get this done is >> to split interpreter initialization >> into two parts: 1. reserve the interpreter memory, 2. generate the >> interpreter. This allows reserving >> it before loading methods (which generates i2c adapters that assert >> things are "in the interpreter", >> requiring knowledge about its address range), and then load the >> methods, and then generate the interpreter. >> This moving of interpreter code generation until after methods have >> been loaded allows offsets to be >> looked up, needed by code generation. That way, we no longer need to >> hardcode field offsets, which >> makes it less annoying to change the field layout code in the future. >> >> Thanks to Coleen for pre-reviewing this. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8243996 >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ > > This seems like a good simplification, and allows for future > flexibility. Initialization order changes are always tricky as the > dependencies are not always obvious - there may be logging or error > path related code that eventually assumes the interpreter has been > generated - but we can deal with such issues if they arise. > > Generally all seems fine to me - I won't comment on the colour of the > bikeshed. :) Phew! ;) > One small comment: > > src/hotspot/share/runtime/init.cpp > > ! void interpreter_init_code();????? // after methods loaded > > given the later comment: > > +?? interpreter_init_code(); // after javaClasses_init and before any > method gets linked (not loaded) > > should the first comment be expanded to: > > void interpreter_init_code();????? // after methods loaded, but before > they are linked > > ? > > Also IIUC the parenthetical "(not loaded)" alludes to the currently > incorrect comment: > > void invocationCounter_init(); // before any methods loaded > > but once you've fixed that there's no context to know what the "(not > loaded)" relates to so it can just be dropped. I think I hopefully fixed the comments there the way you envision it in the latest webrev: http://cr.openjdk.java.net/~eosterlund/8243996/webrev.02/ Incremental: http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01..02/ Although about loading vs linking it's not strictly true always. With CDS, the loading also links, but that linking is a bit different and unproblematic for these initialization purposes. I tried to keep it simple enough to get the gist of it in these one liner comments, instead of expanding the (for this context) unnecessarily complicated whole truth. Hope that is okay. Thanks, /Erik > Thanks, > David > ----- > >> Thanks, >> /Erik From christian.hagedorn at oracle.com Thu May 14 09:38:55 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Thu, 14 May 2020 11:38:55 +0200 Subject: [15] RFR(S): 8244433: Remove saving of RSP in Assembler::pusha_uncached() Message-ID: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> Hi Please review the following enhancement for x86: https://bugs.openjdk.java.net/browse/JDK-8244433 http://cr.openjdk.java.net/~chagedorn/8244433/webrev.00/ This removes the move instruction for saving the actual value of RSP in Assembler::pusha_uncached()/pusha(). The original value of RSP is normally not used on the stack as the value of RSP will automatically be restored after popa to the same value before doing pusha. There are two locations, however, where we need to know the original value of RSP in order to print it. But these places can also compute the correct value of RSP by using the new value of RSP after pusha and adding 16 * wordSize to it. I fixed those. We still keep the same alignment by subtracting 16 * wordSize from RSP in pusha. Does anybody see any potential problems by not saving the value of RSP on the stack with pusha? Either way, as Erik ?. has pointed out, the Windows x64 ABI does not specify a red zone of 128 bytes. If we are unlucky and get an interrupt between saving RSP and decrementing RSP in the current code, we could end up with a corrupt value for RSP on Windows. Therefore, we do need to fix pusha_uncached() if we still want to save the old value of RSP. For example, we could first subtract 16 * wordSize and then calculate the correct value: subq(rsp, 16 * wordSize); movq(Address(rsp, 11 * wordSize), rsp); addq(Address(rsp, 11 * wordSize), 16 * wordSize); Thank you! Best regards, Christian From erik.osterlund at oracle.com Thu May 14 09:41:34 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 14 May 2020 11:41:34 +0200 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: <1585c97d-3f3f-b39d-42a8-5d79d29fe8af@oracle.com> References: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> <1585c97d-3f3f-b39d-42a8-5d79d29fe8af@oracle.com> Message-ID: <8ae45a9f-f4e8-9a32-ae91-38d8e34905d8@oracle.com> Hi Coleen, Thanks for the review. On 2020-05-13 20:07, coleen.phillimore at oracle.com wrote: > > > On 5/13/20 12:00 PM, John Rose wrote: >> I like this cleanup! >> >> There are new tricky dependencies between the interpreter and other >> runtime bits, and class loading; they are adequately documented in new >> comments. >> >> A few comments of my own: >> >> The whitespace on the new line in vmSymbols.hpp needs to be adjusted to >> resemble whitespace on neighboring lines.? Hotspot is a jungle of old >> code, >> and our new code needs to blend into that jungle, not make spots of new >> color (unless of course there is a purpose to some new color). >> >> I don?t want to gripe needlessly about identifier choice, but? maybe I >> do in this case:? ?interpreter_init_code? and >> ?interpreter_init_code_stub? >> seem overly cryptic to me; they don?t help me remember what they do. >> I suggest that?interpreter_init_stubs? and ?interpreter_init_code? or >> ?interpreter_init_stub_code? and ?interpreter_init_main_code? would >> be easier on maintainers.? But if it?s just me, leave it as is.? We >> coders eat >> cryptic identifiers for breakfast. > > Erik and I came up with these names earlier today as they were more > descriptive than some original names. "interpreter_init_stubs" and > "interpreter_init_code" seem okay also. >> >> Can someone remind me (a) which platforms use the C++ interpreter, >> and (b) how well they are tested? >> >> I spot-checked for queries to the hard-coded offsets, and found enough >> ?guarantee(this_offset != 0?)? to make me feel secure. If there?s a bug >> related to this change, it?s quite possibly a zero offset being used >> on some >> path in the JVM which is rarely executed.? (Such a bug could be >> introduced >> in the future, right?)? Usually we avoid such bugs by encapsulating >> variable >> state into C++ access functions which perform their own assertions, >> rather >> than requiring users of the variable to make their own checks. >> >> Did you look into making all those public variables private, and >> wrapping >> access functions around them?? I think that would make for a more >> complete >> cleanup.? It would touch more places in the interpreter, including >> platform >> code, which I think would be a good thing.? It would (a) allow review >> now >> of dependency checks in the interpreter, and (b) force downstream >> porters >> to similarly review their use of hard-coded offsets. > > This seems like a good suggestion for these classes.? For the most > part, javaClasses foo_offset are private. I'm trying to move the pre-existing various offset style inconsistencies of JavaClasses (50-50 use of leading '_', sometimes using accessors, sometimes being public, sometimes asserting offsets are initialized, etc) into a new RFR. I think there is plenty of weirdness there and don't think fixing it belongs to this change. > I was going to reply to this thread "Looks good to me" and thanks for > adding the comments. Thanks! /Erik > Coleen >> >> ? John >> >>> On May 13, 2020, at 8:26 AM, Erik ?sterlund >>> wrote: >>> >>> Hi, >>> >>> Let's remove hardcoded field offsets. Main trick to get this done is >>> to split interpreter initialization >>> into two parts: 1. reserve the interpreter memory, 2. generate the >>> interpreter. This allows reserving >>> it before loading methods (which generates i2c adapters that assert >>> things are "in the interpreter", >>> requiring knowledge about its address range), and then load the >>> methods, and then generate the interpreter. >>> This moving of interpreter code generation until after methods have >>> been loaded allows offsets to be >>> looked up, needed by code generation. That way, we no longer need to >>> hardcode field offsets, which >>> makes it less annoying to change the field layout code in the future. >>> >>> Thanks to Coleen for pre-reviewing this. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8243996 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ >>> >>> Thanks, >>> /Erik > From erik.osterlund at oracle.com Thu May 14 09:47:42 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 14 May 2020 11:47:42 +0200 Subject: [15] RFR(S): 8244433: Remove saving of RSP in Assembler::pusha_uncached() In-Reply-To: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> References: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> Message-ID: <5b5c091a-8ae7-2982-7d1b-0b1c101a5223@oracle.com> Hi Christian, Thanks for taking care of this. Looks good to me. Thanks, /Erik On 2020-05-14 11:38, Christian Hagedorn wrote: > Hi > > Please review the following enhancement for x86: > https://bugs.openjdk.java.net/browse/JDK-8244433 > http://cr.openjdk.java.net/~chagedorn/8244433/webrev.00/ > > This removes the move instruction for saving the actual value of RSP > in Assembler::pusha_uncached()/pusha(). The original value of RSP is > normally not used on the stack as the value of RSP will automatically > be restored after popa to the same value before doing pusha. There are > two locations, however, where we need to know the original value of > RSP in order to print it. But these places can also compute the > correct value of RSP by using the new value of RSP after pusha and > adding 16 * wordSize to it. I fixed those. > > We still keep the same alignment by subtracting 16 * wordSize from RSP > in pusha. Does anybody see any potential problems by not saving the > value of RSP on the stack with pusha? > > Either way, as Erik ?. has pointed out, the Windows x64 ABI does not > specify a red zone of 128 bytes. If we are unlucky and get an > interrupt between saving RSP and decrementing RSP in the current code, > we could end up with a corrupt value for RSP on Windows. Therefore, we > do need to fix pusha_uncached() if we still want to save the old value > of RSP. For example, we could first subtract 16 * wordSize and then > calculate the correct value: > > subq(rsp, 16 * wordSize); > movq(Address(rsp, 11 * wordSize), rsp); > addq(Address(rsp, 11 * wordSize), 16 * wordSize); > > Thank you! > > Best regards, > Christian From christian.hagedorn at oracle.com Thu May 14 09:49:21 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Thu, 14 May 2020 11:49:21 +0200 Subject: [15] RFR(S): 8244433: Remove saving of RSP in Assembler::pusha_uncached() In-Reply-To: <5b5c091a-8ae7-2982-7d1b-0b1c101a5223@oracle.com> References: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> <5b5c091a-8ae7-2982-7d1b-0b1c101a5223@oracle.com> Message-ID: Thank you Erik for your review! Best regards, Christian On 14.05.20 11:47, Erik ?sterlund wrote: > Hi Christian, > > Thanks for taking care of this. Looks good to me. > > Thanks, > /Erik > > On 2020-05-14 11:38, Christian Hagedorn wrote: >> Hi >> >> Please review the following enhancement for x86: >> https://bugs.openjdk.java.net/browse/JDK-8244433 >> http://cr.openjdk.java.net/~chagedorn/8244433/webrev.00/ >> >> This removes the move instruction for saving the actual value of RSP >> in Assembler::pusha_uncached()/pusha(). The original value of RSP is >> normally not used on the stack as the value of RSP will automatically >> be restored after popa to the same value before doing pusha. There are >> two locations, however, where we need to know the original value of >> RSP in order to print it. But these places can also compute the >> correct value of RSP by using the new value of RSP after pusha and >> adding 16 * wordSize to it. I fixed those. >> >> We still keep the same alignment by subtracting 16 * wordSize from RSP >> in pusha. Does anybody see any potential problems by not saving the >> value of RSP on the stack with pusha? >> >> Either way, as Erik ?. has pointed out, the Windows x64 ABI does not >> specify a red zone of 128 bytes. If we are unlucky and get an >> interrupt between saving RSP and decrementing RSP in the current code, >> we could end up with a corrupt value for RSP on Windows. Therefore, we >> do need to fix pusha_uncached() if we still want to save the old value >> of RSP. For example, we could first subtract 16 * wordSize and then >> calculate the correct value: >> >> subq(rsp, 16 * wordSize); >> movq(Address(rsp, 11 * wordSize), rsp); >> addq(Address(rsp, 11 * wordSize), 16 * wordSize); >> >> Thank you! >> >> Best regards, >> Christian > From coleen.phillimore at oracle.com Thu May 14 10:51:38 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 14 May 2020 06:51:38 -0400 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: <8ae45a9f-f4e8-9a32-ae91-38d8e34905d8@oracle.com> References: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> <1585c97d-3f3f-b39d-42a8-5d79d29fe8af@oracle.com> <8ae45a9f-f4e8-9a32-ae91-38d8e34905d8@oracle.com> Message-ID: <26cb0ba8-c9ae-9db9-4398-ba6a26902807@oracle.com> On 5/14/20 5:41 AM, Erik ?sterlund wrote: > Hi Coleen, > > Thanks for the review. > > On 2020-05-13 20:07, coleen.phillimore at oracle.com wrote: >> >> >> On 5/13/20 12:00 PM, John Rose wrote: >>> I like this cleanup! >>> >>> There are new tricky dependencies between the interpreter and other >>> runtime bits, and class loading; they are adequately documented in new >>> comments. >>> >>> A few comments of my own: >>> >>> The whitespace on the new line in vmSymbols.hpp needs to be adjusted to >>> resemble whitespace on neighboring lines.? Hotspot is a jungle of >>> old code, >>> and our new code needs to blend into that jungle, not make spots of new >>> color (unless of course there is a purpose to some new color). >>> >>> I don?t want to gripe needlessly about identifier choice, but? maybe I >>> do in this case:? ?interpreter_init_code? and >>> ?interpreter_init_code_stub? >>> seem overly cryptic to me; they don?t help me remember what they do. >>> I suggest that?interpreter_init_stubs? and ?interpreter_init_code? or >>> ?interpreter_init_stub_code? and ?interpreter_init_main_code? would >>> be easier on maintainers.? But if it?s just me, leave it as is.? We >>> coders eat >>> cryptic identifiers for breakfast. >> >> Erik and I came up with these names earlier today as they were more >> descriptive than some original names. "interpreter_init_stubs" and >> "interpreter_init_code" seem okay also. >>> >>> Can someone remind me (a) which platforms use the C++ interpreter, >>> and (b) how well they are tested? >>> >>> I spot-checked for queries to the hard-coded offsets, and found enough >>> ?guarantee(this_offset != 0?)? to make me feel secure. If there?s a bug >>> related to this change, it?s quite possibly a zero offset being used >>> on some >>> path in the JVM which is rarely executed.? (Such a bug could be >>> introduced >>> in the future, right?)? Usually we avoid such bugs by encapsulating >>> variable >>> state into C++ access functions which perform their own assertions, >>> rather >>> than requiring users of the variable to make their own checks. >>> >>> Did you look into making all those public variables private, and >>> wrapping >>> access functions around them?? I think that would make for a more >>> complete >>> cleanup.? It would touch more places in the interpreter, including >>> platform >>> code, which I think would be a good thing.? It would (a) allow >>> review now >>> of dependency checks in the interpreter, and (b) force downstream >>> porters >>> to similarly review their use of hard-coded offsets. >> >> This seems like a good suggestion for these classes.? For the most >> part, javaClasses foo_offset are private. > > I'm trying to move the pre-existing various offset style > inconsistencies of JavaClasses > (50-50 use of leading '_', sometimes using accessors, sometimes being > public, sometimes > asserting offsets are initialized, etc) into a new RFR. I think there > is plenty of weirdness there > and don't think fixing it belongs to this change. Oh I thought the additional change should only apply to the fields of java_lang_ref_Reference class which are public, but I'm fine with a future RFE. Coleen > >> I was going to reply to this thread "Looks good to me" and thanks for >> adding the comments. > > Thanks! > > /Erik > >> Coleen >>> >>> ? John >>> >>>> On May 13, 2020, at 8:26 AM, Erik ?sterlund >>>> wrote: >>>> >>>> Hi, >>>> >>>> Let's remove hardcoded field offsets. Main trick to get this done >>>> is to split interpreter initialization >>>> into two parts: 1. reserve the interpreter memory, 2. generate the >>>> interpreter. This allows reserving >>>> it before loading methods (which generates i2c adapters that assert >>>> things are "in the interpreter", >>>> requiring knowledge about its address range), and then load the >>>> methods, and then generate the interpreter. >>>> This moving of interpreter code generation until after methods have >>>> been loaded allows offsets to be >>>> looked up, needed by code generation. That way, we no longer need >>>> to hardcode field offsets, which >>>> makes it less annoying to change the field layout code in the future. >>>> >>>> Thanks to Coleen for pre-reviewing this. >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8243996 >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ >>>> >>>> Thanks, >>>> /Erik >> > From coleen.phillimore at oracle.com Thu May 14 10:53:26 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 14 May 2020 06:53:26 -0400 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: References: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> Message-ID: <762a288c-87cc-ac84-bae0-5ebc36527d37@oracle.com> On 5/14/20 5:30 AM, Erik ?sterlund wrote: > Hi John, > > Thank you for reviewing this change and sharing your thoughts. > > On 2020-05-13 18:00, John Rose wrote: >> I like this cleanup! >> >> There are new tricky dependencies between the interpreter and other >> runtime bits, and class loading; they are adequately documented in new >> comments. >> >> A few comments of my own: >> >> The whitespace on the new line in vmSymbols.hpp needs to be adjusted to >> resemble whitespace on neighboring lines.? Hotspot is a jungle of old >> code, >> and our new code needs to blend into that jungle, not make spots of new >> color (unless of course there is a purpose to some new color). > > Ah, missed that. Fixed. Emacs nearly dies every time I open that file; > looks like I > was too eager to get out of that file. > >> I don?t want to gripe needlessly about identifier choice, but? maybe I >> do in this case:? ?interpreter_init_code? and >> ?interpreter_init_code_stub? >> seem overly cryptic to me; they don?t help me remember what they do. >> I suggest that?interpreter_init_stubs? and ?interpreter_init_code? or >> ?interpreter_init_stub_code? and ?interpreter_init_main_code? would >> be easier on maintainers.? But if it?s just me, leave it as is. We >> coders eat >> cryptic identifiers for breakfast. > > Ah yes. Coleen and I have already been through a few rounds of naming > for this. > I started off with initialize1 and initialize2, similar to other > initialization functions. > But Coleen accurately pointed out those names do not reflect what it > does. So I > changed the name to initialize_code_stub and initialize_generate, but > Coleen > accurately pointed out that initialize_generate is grammatically > incorrect. So I changed > it to initialize_code to be grammatically correct yet still state what > is being initialized. > But I agree with you that unfortunately the difference between > initializing a code stub > and code is a bit unclear. I liked your first suggestion more than the > second suggestion, > so let's take that for a spin and see if anyone else wants a different > name. > >> Can someone remind me (a) which platforms use the C++ interpreter, >> and (b) how well they are tested? > > Unfortunately, I do not know. Zero uses the C++ interpreter and it's tested by OpenJDK developers.? We *should* test it ourselves in mach5 but apparently we don't. Coleen > >> I spot-checked for queries to the hard-coded offsets, and found enough >> ?guarantee(this_offset != 0?)? to make me feel secure.? If there?s a bug >> related to this change, it?s quite possibly a zero offset being used >> on some >> path in the JVM which is rarely executed.? (Such a bug could be >> introduced >> in the future, right?)? Usually we avoid such bugs by encapsulating >> variable >> state into C++ access functions which perform their own assertions, >> rather >> than requiring users of the variable to make their own checks. > > Right. > >> Did you look into making all those public variables private, and >> wrapping >> access functions around them?? I think that would make for a more >> complete >> cleanup.? It would touch more places in the interpreter, including >> platform >> code, which I think would be a good thing.? It would (a) allow review >> now >> of dependency checks in the interpreter, and (b) force downstream >> porters >> to similarly review their use of hard-coded offsets. > > Indeed I noted that odd old colour, but was not sure I had a strong > enough reason to change it to a > new colour, given the fan out. ;) > > Among the public variables you suggest making private, It seems that > only the referent is used > in platform specific code (interpreter), and is used in the same way > on all platforms. And the whole point > of this exercise was to remove *all* dependencies from interpreter > code to offset initialization order. > In other words, before the interpreter code had to explicitly know > exactly which offsets it could vs could > not use, based on whether they were hardcoded or not. Now it can use > all offsets as it pleases, by design. > Therefore I don't think we achieve neither (a) nor (b) by changing > this colour to a new one. > > Having said that, it certainly is nicer and good practice to make > these variables private and use accessor > functions for them, for stylistic, consistency and encapsulation > reasons. I also noticed that half of the > offset variables use a leading '_' and the others do not, making the > code quite inconsistent. I could send > a new RFR trying to normalize the style of the various offsets in > JavaClasses to use a leading '_', be private > and have an accessor function that asserts offset sanity. What do you > think? > > Not sure how I got here implementing support for compressed class > pointers in the absence of compressed > oops. Oh dear... > > New webrev so far: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.02/ > > Incremental: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01..02/ > > Thanks, > /Erik > >> ? John >> >>> On May 13, 2020, at 8:26 AM, Erik ?sterlund >>> wrote: >>> >>> Hi, >>> >>> Let's remove hardcoded field offsets. Main trick to get this done is >>> to split interpreter initialization >>> into two parts: 1. reserve the interpreter memory, 2. generate the >>> interpreter. This allows reserving >>> it before loading methods (which generates i2c adapters that assert >>> things are "in the interpreter", >>> requiring knowledge about its address range), and then load the >>> methods, and then generate the interpreter. >>> This moving of interpreter code generation until after methods have >>> been loaded allows offsets to be >>> looked up, needed by code generation. That way, we no longer need to >>> hardcode field offsets, which >>> makes it less annoying to change the field layout code in the future. >>> >>> Thanks to Coleen for pre-reviewing this. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8243996 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ >>> >>> Thanks, >>> /Erik > From coleen.phillimore at oracle.com Thu May 14 11:03:04 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 14 May 2020 07:03:04 -0400 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: <1ea48e4a-3409-3f5d-7b3d-2c25af6ddc8e@oracle.com> References: <1ea48e4a-3409-3f5d-7b3d-2c25af6ddc8e@oracle.com> Message-ID: <306556a6-cea5-1b71-2f7f-d41eeede8dbd@oracle.com> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01..02/src/hotspot/share/runtime/init.cpp.udiff.html It seems really annoying to me that the top part, which are forward declarations and not in any particular order have duplicate comments as the bottom part of the change that does need to have the comments about ordering. And that they have to be kept in sync.? I don't think you should change these but I don't know why anyone would read the comments on the forward declarations. Coleen On 5/14/20 5:37 AM, Erik ?sterlund wrote: > Hi David, > > Thanks for having a look at this! > > On 2020-05-14 07:19, David Holmes wrote: >> Hi Erik, >> >> On 14/05/2020 1:26 am, Erik ?sterlund wrote: >>> Hi, >>> >>> Let's remove hardcoded field offsets. Main trick to get this done is >>> to split interpreter initialization >>> into two parts: 1. reserve the interpreter memory, 2. generate the >>> interpreter. This allows reserving >>> it before loading methods (which generates i2c adapters that assert >>> things are "in the interpreter", >>> requiring knowledge about its address range), and then load the >>> methods, and then generate the interpreter. >>> This moving of interpreter code generation until after methods have >>> been loaded allows offsets to be >>> looked up, needed by code generation. That way, we no longer need to >>> hardcode field offsets, which >>> makes it less annoying to change the field layout code in the future. >>> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01..02/src/hotspot/share/runtime/init.cpp.udiff.html >>> Thanks to Coleen for pre-reviewing this. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8243996 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ >> >> This seems like a good simplification, and allows for future >> flexibility. Initialization order changes are always tricky as the >> dependencies are not always obvious - there may be logging or error >> path related code that eventually assumes the interpreter has been >> generated - but we can deal with such issues if they arise. >> >> Generally all seems fine to me - I won't comment on the colour of the >> bikeshed. :) > > Phew! ;) > >> One small comment: >> >> src/hotspot/share/runtime/init.cpp >> >> ! void interpreter_init_code();????? // after methods loaded >> >> given the later comment: >> >> +?? interpreter_init_code(); // after javaClasses_init and before any >> method gets linked (not loaded) >> >> should the first comment be expanded to: >> >> void interpreter_init_code();????? // after methods loaded, but >> before they are linked >> >> ? >> >> Also IIUC the parenthetical "(not loaded)" alludes to the currently >> incorrect comment: >> >> void invocationCounter_init(); // before any methods loaded >> >> but once you've fixed that there's no context to know what the "(not >> loaded)" relates to so it can just be dropped. > > I think I hopefully fixed the comments there the way you envision it > in the latest webrev: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.02/ > > Incremental: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01..02/ > > Although about loading vs linking it's not strictly true always. With > CDS, the loading also links, but that linking is > a bit different and unproblematic for these initialization purposes. I > tried to keep it simple enough to get the gist > of it in these one liner comments, instead of expanding the (for this > context) unnecessarily complicated whole truth. > Hope that is okay. > > Thanks, > /Erik > >> Thanks, >> David >> ----- >> >>> Thanks, >>> /Erik > From christian.hagedorn at oracle.com Thu May 14 12:01:23 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Thu, 14 May 2020 14:01:23 +0200 Subject: [15] RFR(S): 8244433: Remove saving of RSP in Assembler::pusha_uncached() In-Reply-To: References: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> <5b5c091a-8ae7-2982-7d1b-0b1c101a5223@oracle.com> Message-ID: <94c7cd63-2a81-cf85-fbfc-c8702397f5a1@oracle.com> Forgot to adjust the rsp variable in print_state64() on line 912. I updated my webrev in place. Best regards, Christian On 14.05.20 11:49, Christian Hagedorn wrote: > Thank you Erik for your review! > > Best regards, > Christian > > On 14.05.20 11:47, Erik ?sterlund wrote: >> Hi Christian, >> >> Thanks for taking care of this. Looks good to me. >> >> Thanks, >> /Erik >> >> On 2020-05-14 11:38, Christian Hagedorn wrote: >>> Hi >>> >>> Please review the following enhancement for x86: >>> https://bugs.openjdk.java.net/browse/JDK-8244433 >>> http://cr.openjdk.java.net/~chagedorn/8244433/webrev.00/ >>> >>> This removes the move instruction for saving the actual value of RSP >>> in Assembler::pusha_uncached()/pusha(). The original value of RSP is >>> normally not used on the stack as the value of RSP will automatically >>> be restored after popa to the same value before doing pusha. There >>> are two locations, however, where we need to know the original value >>> of RSP in order to print it. But these places can also compute the >>> correct value of RSP by using the new value of RSP after pusha and >>> adding 16 * wordSize to it. I fixed those. >>> >>> We still keep the same alignment by subtracting 16 * wordSize from >>> RSP in pusha. Does anybody see any potential problems by not saving >>> the value of RSP on the stack with pusha? >>> >>> Either way, as Erik ?. has pointed out, the Windows x64 ABI does not >>> specify a red zone of 128 bytes. If we are unlucky and get an >>> interrupt between saving RSP and decrementing RSP in the current >>> code, we could end up with a corrupt value for RSP on Windows. >>> Therefore, we do need to fix pusha_uncached() if we still want to >>> save the old value of RSP. For example, we could first subtract 16 * >>> wordSize and then calculate the correct value: >>> >>> subq(rsp, 16 * wordSize); >>> movq(Address(rsp, 11 * wordSize), rsp); >>> addq(Address(rsp, 11 * wordSize), 16 * wordSize); >>> >>> Thank you! >>> >>> Best regards, >>> Christian >> From erik.osterlund at oracle.com Thu May 14 12:03:09 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 14 May 2020 14:03:09 +0200 Subject: [15] RFR(S): 8244433: Remove saving of RSP in Assembler::pusha_uncached() In-Reply-To: <94c7cd63-2a81-cf85-fbfc-c8702397f5a1@oracle.com> References: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> <5b5c091a-8ae7-2982-7d1b-0b1c101a5223@oracle.com> <94c7cd63-2a81-cf85-fbfc-c8702397f5a1@oracle.com> Message-ID: <44a9e36d-297d-bbe7-830d-1b01efdc5f2b@oracle.com> Oups. Looks good. Thanks, /Erik On 2020-05-14 14:01, Christian Hagedorn wrote: > Forgot to adjust the rsp variable in print_state64() on line 912. I > updated my webrev in place. > > Best regards, > Christian > > On 14.05.20 11:49, Christian Hagedorn wrote: >> Thank you Erik for your review! >> >> Best regards, >> Christian >> >> On 14.05.20 11:47, Erik ?sterlund wrote: >>> Hi Christian, >>> >>> Thanks for taking care of this. Looks good to me. >>> >>> Thanks, >>> /Erik >>> >>> On 2020-05-14 11:38, Christian Hagedorn wrote: >>>> Hi >>>> >>>> Please review the following enhancement for x86: >>>> https://bugs.openjdk.java.net/browse/JDK-8244433 >>>> http://cr.openjdk.java.net/~chagedorn/8244433/webrev.00/ >>>> >>>> This removes the move instruction for saving the actual value of >>>> RSP in Assembler::pusha_uncached()/pusha(). The original value of >>>> RSP is normally not used on the stack as the value of RSP will >>>> automatically be restored after popa to the same value before doing >>>> pusha. There are two locations, however, where we need to know the >>>> original value of RSP in order to print it. But these places can >>>> also compute the correct value of RSP by using the new value of RSP >>>> after pusha and adding 16 * wordSize to it. I fixed those. >>>> >>>> We still keep the same alignment by subtracting 16 * wordSize from >>>> RSP in pusha. Does anybody see any potential problems by not saving >>>> the value of RSP on the stack with pusha? >>>> >>>> Either way, as Erik ?. has pointed out, the Windows x64 ABI does >>>> not specify a red zone of 128 bytes. If we are unlucky and get an >>>> interrupt between saving RSP and decrementing RSP in the current >>>> code, we could end up with a corrupt value for RSP on Windows. >>>> Therefore, we do need to fix pusha_uncached() if we still want to >>>> save the old value of RSP. For example, we could first subtract 16 >>>> * wordSize and then calculate the correct value: >>>> >>>> subq(rsp, 16 * wordSize); >>>> movq(Address(rsp, 11 * wordSize), rsp); >>>> addq(Address(rsp, 11 * wordSize), 16 * wordSize); >>>> >>>> Thank you! >>>> >>>> Best regards, >>>> Christian >>> From christian.hagedorn at oracle.com Thu May 14 12:04:57 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Thu, 14 May 2020 14:04:57 +0200 Subject: [15] RFR(S): 8244433: Remove saving of RSP in Assembler::pusha_uncached() In-Reply-To: <44a9e36d-297d-bbe7-830d-1b01efdc5f2b@oracle.com> References: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> <5b5c091a-8ae7-2982-7d1b-0b1c101a5223@oracle.com> <94c7cd63-2a81-cf85-fbfc-c8702397f5a1@oracle.com> <44a9e36d-297d-bbe7-830d-1b01efdc5f2b@oracle.com> Message-ID: <11b1f691-c7a2-3fec-4cb7-3fb1212855a4@oracle.com> Thanks Erik for having a look at it again! Best regards, Christian On 14.05.20 14:03, Erik ?sterlund wrote: > Oups. Looks good. > > Thanks, > /Erik > > On 2020-05-14 14:01, Christian Hagedorn wrote: >> Forgot to adjust the rsp variable in print_state64() on line 912. I >> updated my webrev in place. >> >> Best regards, >> Christian >> >> On 14.05.20 11:49, Christian Hagedorn wrote: >>> Thank you Erik for your review! >>> >>> Best regards, >>> Christian >>> >>> On 14.05.20 11:47, Erik ?sterlund wrote: >>>> Hi Christian, >>>> >>>> Thanks for taking care of this. Looks good to me. >>>> >>>> Thanks, >>>> /Erik >>>> >>>> On 2020-05-14 11:38, Christian Hagedorn wrote: >>>>> Hi >>>>> >>>>> Please review the following enhancement for x86: >>>>> https://bugs.openjdk.java.net/browse/JDK-8244433 >>>>> http://cr.openjdk.java.net/~chagedorn/8244433/webrev.00/ >>>>> >>>>> This removes the move instruction for saving the actual value of >>>>> RSP in Assembler::pusha_uncached()/pusha(). The original value of >>>>> RSP is normally not used on the stack as the value of RSP will >>>>> automatically be restored after popa to the same value before doing >>>>> pusha. There are two locations, however, where we need to know the >>>>> original value of RSP in order to print it. But these places can >>>>> also compute the correct value of RSP by using the new value of RSP >>>>> after pusha and adding 16 * wordSize to it. I fixed those. >>>>> >>>>> We still keep the same alignment by subtracting 16 * wordSize from >>>>> RSP in pusha. Does anybody see any potential problems by not saving >>>>> the value of RSP on the stack with pusha? >>>>> >>>>> Either way, as Erik ?. has pointed out, the Windows x64 ABI does >>>>> not specify a red zone of 128 bytes. If we are unlucky and get an >>>>> interrupt between saving RSP and decrementing RSP in the current >>>>> code, we could end up with a corrupt value for RSP on Windows. >>>>> Therefore, we do need to fix pusha_uncached() if we still want to >>>>> save the old value of RSP. For example, we could first subtract 16 >>>>> * wordSize and then calculate the correct value: >>>>> >>>>> subq(rsp, 16 * wordSize); >>>>> movq(Address(rsp, 11 * wordSize), rsp); >>>>> addq(Address(rsp, 11 * wordSize), 16 * wordSize); >>>>> >>>>> Thank you! >>>>> >>>>> Best regards, >>>>> Christian >>>> > From erik.osterlund at oracle.com Thu May 14 12:07:46 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 14 May 2020 14:07:46 +0200 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: <26cb0ba8-c9ae-9db9-4398-ba6a26902807@oracle.com> References: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> <1585c97d-3f3f-b39d-42a8-5d79d29fe8af@oracle.com> <8ae45a9f-f4e8-9a32-ae91-38d8e34905d8@oracle.com> <26cb0ba8-c9ae-9db9-4398-ba6a26902807@oracle.com> Message-ID: <1bf33422-22e2-1fac-7a37-f3ee9791a951@oracle.com> >> I'm trying to move the pre-existing various offset style >> inconsistencies of JavaClasses >> (50-50 use of leading '_', sometimes using accessors, sometimes being >> public, sometimes >> asserting offsets are initialized, etc) into a new RFR. I think there >> is plenty of weirdness there >> and don't think fixing it belongs to this change. > > Oh I thought the additional change should only apply to the fields of > java_lang_ref_Reference class which are public, but I'm fine with a > future RFE. Thanks Coleen! /Erik > Coleen >> >>> I was going to reply to this thread "Looks good to me" and thanks >>> for adding the comments. >> >> Thanks! >> >> /Erik >> >>> Coleen >>>> >>>> ? John >>>> >>>>> On May 13, 2020, at 8:26 AM, Erik ?sterlund >>>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> Let's remove hardcoded field offsets. Main trick to get this done >>>>> is to split interpreter initialization >>>>> into two parts: 1. reserve the interpreter memory, 2. generate the >>>>> interpreter. This allows reserving >>>>> it before loading methods (which generates i2c adapters that >>>>> assert things are "in the interpreter", >>>>> requiring knowledge about its address range), and then load the >>>>> methods, and then generate the interpreter. >>>>> This moving of interpreter code generation until after methods >>>>> have been loaded allows offsets to be >>>>> looked up, needed by code generation. That way, we no longer need >>>>> to hardcode field offsets, which >>>>> makes it less annoying to change the field layout code in the future. >>>>> >>>>> Thanks to Coleen for pre-reviewing this. >>>>> >>>>> Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8243996 >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ >>>>> >>>>> Thanks, >>>>> /Erik >>> >> > From tobias.hartmann at oracle.com Thu May 14 12:31:04 2020 From: tobias.hartmann at oracle.com (Tobias Hartmann) Date: Thu, 14 May 2020 14:31:04 +0200 Subject: [15] RFR(S): 8244433: Remove saving of RSP in Assembler::pusha_uncached() In-Reply-To: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> References: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> Message-ID: <58ae3976-4ab1-49b3-670b-15370e8e018b@oracle.com> Hi Christian, this looks good to me. Best regards, Tobias On 14.05.20 11:38, Christian Hagedorn wrote: > Hi > > Please review the following enhancement for x86: > https://bugs.openjdk.java.net/browse/JDK-8244433 > http://cr.openjdk.java.net/~chagedorn/8244433/webrev.00/ > > This removes the move instruction for saving the actual value of RSP in > Assembler::pusha_uncached()/pusha(). The original value of RSP is normally not used on the stack as > the value of RSP will automatically be restored after popa to the same value before doing pusha. > There are two locations, however, where we need to know the original value of RSP in order to print > it. But these places can also compute the correct value of RSP by using the new value of RSP after > pusha and adding 16 * wordSize to it. I fixed those. > > We still keep the same alignment by subtracting 16 * wordSize from RSP in pusha. Does anybody see > any potential problems by not saving the value of RSP on the stack with pusha? > > Either way, as Erik ?. has pointed out, the Windows x64 ABI does not specify a red zone of 128 > bytes. If we are unlucky and get an interrupt between saving RSP and decrementing RSP in the current > code, we could end up with a corrupt value for RSP on Windows. Therefore, we do need to fix > pusha_uncached() if we still want to save the old value of RSP. For example, we could first subtract > 16 * wordSize and then calculate the correct value: > > subq(rsp, 16 * wordSize); > movq(Address(rsp, 11 * wordSize), rsp); > addq(Address(rsp, 11 * wordSize), 16 * wordSize); > > Thank you! > > Best regards, > Christian From christian.hagedorn at oracle.com Thu May 14 12:35:42 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Thu, 14 May 2020 14:35:42 +0200 Subject: [15] RFR(S): 8244433: Remove saving of RSP in Assembler::pusha_uncached() In-Reply-To: <58ae3976-4ab1-49b3-670b-15370e8e018b@oracle.com> References: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> <58ae3976-4ab1-49b3-670b-15370e8e018b@oracle.com> Message-ID: <5a2a07ed-2d37-fa7c-4171-af6f03dafd38@oracle.com> Thank you Tobias for your review! Best regards, Christian On 14.05.20 14:31, Tobias Hartmann wrote: > Hi Christian, > > this looks good to me. > > Best regards, > Tobias > > On 14.05.20 11:38, Christian Hagedorn wrote: >> Hi >> >> Please review the following enhancement for x86: >> https://bugs.openjdk.java.net/browse/JDK-8244433 >> http://cr.openjdk.java.net/~chagedorn/8244433/webrev.00/ >> >> This removes the move instruction for saving the actual value of RSP in >> Assembler::pusha_uncached()/pusha(). The original value of RSP is normally not used on the stack as >> the value of RSP will automatically be restored after popa to the same value before doing pusha. >> There are two locations, however, where we need to know the original value of RSP in order to print >> it. But these places can also compute the correct value of RSP by using the new value of RSP after >> pusha and adding 16 * wordSize to it. I fixed those. >> >> We still keep the same alignment by subtracting 16 * wordSize from RSP in pusha. Does anybody see >> any potential problems by not saving the value of RSP on the stack with pusha? >> >> Either way, as Erik ?. has pointed out, the Windows x64 ABI does not specify a red zone of 128 >> bytes. If we are unlucky and get an interrupt between saving RSP and decrementing RSP in the current >> code, we could end up with a corrupt value for RSP on Windows. Therefore, we do need to fix >> pusha_uncached() if we still want to save the old value of RSP. For example, we could first subtract >> 16 * wordSize and then calculate the correct value: >> >> subq(rsp, 16 * wordSize); >> movq(Address(rsp, 11 * wordSize), rsp); >> addq(Address(rsp, 11 * wordSize), 16 * wordSize); >> >> Thank you! >> >> Best regards, >> Christian From erik.osterlund at oracle.com Thu May 14 13:07:45 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 14 May 2020 15:07:45 +0200 Subject: RFR: 8244416: Remove incorrect assert during inline cache cleaning Message-ID: <5a4574d6-0d7d-9d52-19b0-cbacebb1fba5@oracle.com> Hi, Months of stress testing triggered the following assert in inline cache cleaning: assert(!from->is_zombie(), "should not clean inline caches on zombies"); It used to be assert(!from->is_alive()), but after finding a race that resulted in false positives for unloaded nmethods, the assert was relaxed to check only that we shouldn't have zombies here, with a fat comment explaining why. Unfortunately, the relaxed assert also gets false positives, so I have chosen to remove it, and further increase the size of that fat comment, explaning why we can't assert that it isn't a zombie either. What it boils down to is a race where inline cache cleaning is performed from a concurrent GC thread, concurrent to the sweeper making an nmethod zombie. The order of the race ingredients making the nmethod zombie from the sweeper is the following: 1. Transition state to zombie 2. Unregister nmethod form GC (importantly triggers a wait for concurrent code cache unloading to finish) 3. clear_ic_callsites() importantly releasing all IC stubs The race happens when the GC finds an is_alive && !is_unloading nmethod and triggers inline cache cleaning on it before 1, but then when racingly checking again in the assert, we are past 1. Since the assert will spuriously fail even though there is no actual problem, and we are at a point where there isn't much else it could assert, I chose to remove it. Other than triggering the assert, it is worth noting that this racingly zombiefied nmethod will have its inline caches cleaned *with* IC stubs from the concurrent GC thread. Subsequent IC finalization in safepoint cleanup requires live IC stubs to belong to is_alive() nmethods, so the IC stubs must be released before the next safepoint for this race to not be dangerous. That is ensured, because the racing sweeper will get stuck waiting for concurrent code cache unloading to finish, and when it wakes up, it will clear_ic_callsites(), releasing the spurious ICStubs, before finishing the make_zombie() function. Then it will eventually hit a safepoint poll, when the nmethod has been purged from IC stubs. Bug: https://bugs.openjdk.java.net/browse/JDK-8244416 Webrev: http://cr.openjdk.java.net/~eosterlund/8244416/webrev.00/ Thanks, /Erik From daniel.daugherty at oracle.com Thu May 14 13:28:10 2020 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Thu, 14 May 2020 09:28:10 -0400 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: Message-ID: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> On 5/14/20 4:00 AM, Christian Hagedorn wrote: > Hi David, hi Daniel > > Thank you for your reviews! > > On 13.05.20 17:54, Daniel D. Daugherty wrote: >> ???? Not your bug, but the comment has two typos: >> >> ???????? s/by safer/but safer/ >> ???????? s/unexpensive/inexpensive/ > > Thanks, fixed it. > > On 14.05.20 01:51, David Holmes wrote: >> src/hotspot/cpu/x86/methodHandles_x86.cpp >> >> Given all of the code is inside: >> >> ??504?? if (Verbose) { >> >> I would just move the ResourceMark to the top of that block. We are >> going to use it unconditionally, and we are not releasing it until >> the end of the method regardless. That avoids the potential concern >> Dan raised about extending the scope of the PRESERVE_EXCEPTION_MARK. > > This sounds reasonable. I moved it to the top of that block. > >>> There are other places like Klass::print_on(outputStream*) where >>> this is still the case. Should we file an RFE to clean these up? >> >> Obviously we don't have any tests running in a mode that causes this >> code to be executed, so yes it is probably a good idea to take a >> closer look at other call-sites and file a RFE if needed. > > Yes, I filed one to take a closer look at these [1]. > > I included the changes in a new webrev: > http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ Thumbs up. Dan > > Best regards, > Christian > > > [1] https://bugs.openjdk.java.net/browse/JDK-8244999 > From christian.hagedorn at oracle.com Thu May 14 13:31:37 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Thu, 14 May 2020 15:31:37 +0200 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> Message-ID: <6612082f-d98e-ab03-d430-c714404e5b14@oracle.com> Thank you Dan for reviewing it again! Best regards, Christian On 14.05.20 15:28, Daniel D. Daugherty wrote: > On 5/14/20 4:00 AM, Christian Hagedorn wrote: >> Hi David, hi Daniel >> >> Thank you for your reviews! >> >> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>> ???? Not your bug, but the comment has two typos: >>> >>> ???????? s/by safer/but safer/ >>> ???????? s/unexpensive/inexpensive/ >> >> Thanks, fixed it. >> >> On 14.05.20 01:51, David Holmes wrote: >>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>> >>> Given all of the code is inside: >>> >>> ??504?? if (Verbose) { >>> >>> I would just move the ResourceMark to the top of that block. We are >>> going to use it unconditionally, and we are not releasing it until >>> the end of the method regardless. That avoids the potential concern >>> Dan raised about extending the scope of the PRESERVE_EXCEPTION_MARK. >> >> This sounds reasonable. I moved it to the top of that block. >> >>>> There are other places like Klass::print_on(outputStream*) where >>>> this is still the case. Should we file an RFE to clean these up? >>> >>> Obviously we don't have any tests running in a mode that causes this >>> code to be executed, so yes it is probably a good idea to take a >>> closer look at other call-sites and file a RFE if needed. >> >> Yes, I filed one to take a closer look at these [1]. >> >> I included the changes in a new webrev: >> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ > > Thumbs up. > > Dan > >> >> Best regards, >> Christian >> >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >> > From david.holmes at oracle.com Thu May 14 13:50:20 2020 From: david.holmes at oracle.com (David Holmes) Date: Thu, 14 May 2020 23:50:20 +1000 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> Message-ID: <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> +1 But I'd be inclined to generalize the test to run: -Xlog:all=trace -XX:+Verbose Curiously I couldn't reproduce the crash that this bug is about. But the hs_err file speaks for itself. Cheers, David On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: > On 5/14/20 4:00 AM, Christian Hagedorn wrote: >> Hi David, hi Daniel >> >> Thank you for your reviews! >> >> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>> ???? Not your bug, but the comment has two typos: >>> >>> ???????? s/by safer/but safer/ >>> ???????? s/unexpensive/inexpensive/ >> >> Thanks, fixed it. >> >> On 14.05.20 01:51, David Holmes wrote: >>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>> >>> Given all of the code is inside: >>> >>> ??504?? if (Verbose) { >>> >>> I would just move the ResourceMark to the top of that block. We are >>> going to use it unconditionally, and we are not releasing it until >>> the end of the method regardless. That avoids the potential concern >>> Dan raised about extending the scope of the PRESERVE_EXCEPTION_MARK. >> >> This sounds reasonable. I moved it to the top of that block. >> >>>> There are other places like Klass::print_on(outputStream*) where >>>> this is still the case. Should we file an RFE to clean these up? >>> >>> Obviously we don't have any tests running in a mode that causes this >>> code to be executed, so yes it is probably a good idea to take a >>> closer look at other call-sites and file a RFE if needed. >> >> Yes, I filed one to take a closer look at these [1]. >> >> I included the changes in a new webrev: >> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ > > Thumbs up. > > Dan > >> >> Best regards, >> Christian >> >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >> > From coleen.phillimore at oracle.com Thu May 14 14:02:27 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 14 May 2020 10:02:27 -0400 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> Message-ID: I thought that this code under Verbose should have been changed to use logging with log_level = verbose.? We were trying to minimize/remove the Verbose flag. On 5/14/20 9:50 AM, David Holmes wrote: > +1 > > But I'd be inclined to generalize the test to run: > > -Xlog:all=trace -XX:+Verbose Don't do this, it would be tons of output and make the test unusable if it fails. Coleen > > Curiously I couldn't reproduce the crash that this bug is about. But > the hs_err file speaks for itself. > > Cheers, > David > > On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>> Hi David, hi Daniel >>> >>> Thank you for your reviews! >>> >>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>> ???? Not your bug, but the comment has two typos: >>>> >>>> ???????? s/by safer/but safer/ >>>> ???????? s/unexpensive/inexpensive/ >>> >>> Thanks, fixed it. >>> >>> On 14.05.20 01:51, David Holmes wrote: >>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>> >>>> Given all of the code is inside: >>>> >>>> ??504?? if (Verbose) { >>>> >>>> I would just move the ResourceMark to the top of that block. We are >>>> going to use it unconditionally, and we are not releasing it until >>>> the end of the method regardless. That avoids the potential concern >>>> Dan raised about extending the scope of the PRESERVE_EXCEPTION_MARK. >>> >>> This sounds reasonable. I moved it to the top of that block. >>> >>>>> There are other places like Klass::print_on(outputStream*) where >>>>> this is still the case. Should we file an RFE to clean these up? >>>> >>>> Obviously we don't have any tests running in a mode that causes >>>> this code to be executed, so yes it is probably a good idea to take >>>> a closer look at other call-sites and file a RFE if needed. >>> >>> Yes, I filed one to take a closer look at these [1]. >>> >>> I included the changes in a new webrev: >>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >> >> Thumbs up. >> >> Dan >> >>> >>> Best regards, >>> Christian >>> >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>> >> From vladimir.kozlov at oracle.com Thu May 14 18:52:12 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Thu, 14 May 2020 11:52:12 -0700 Subject: [15] RFR(S): 8244433: Remove saving of RSP in Assembler::pusha_uncached() In-Reply-To: <5a2a07ed-2d37-fa7c-4171-af6f03dafd38@oracle.com> References: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> <58ae3976-4ab1-49b3-670b-15370e8e018b@oracle.com> <5a2a07ed-2d37-fa7c-4171-af6f03dafd38@oracle.com> Message-ID: <34f07323-acc9-ad05-2d81-2d82dfcbcd14@oracle.com> Hi, Christian Changes looks good. I looked and this code existed from day one. And I don't remember we ever had an issue with possible RSP corrupted values. I don't think you need to worry about this after you fixed print code. Thanks, Vladimir On 5/14/20 5:35 AM, Christian Hagedorn wrote: > Thank you Tobias for your review! > > Best regards, > Christian > > On 14.05.20 14:31, Tobias Hartmann wrote: >> Hi Christian, >> >> this looks good to me. >> >> Best regards, >> Tobias >> >> On 14.05.20 11:38, Christian Hagedorn wrote: >>> Hi >>> >>> Please review the following enhancement for x86: >>> https://bugs.openjdk.java.net/browse/JDK-8244433 >>> http://cr.openjdk.java.net/~chagedorn/8244433/webrev.00/ >>> >>> This removes the move instruction for saving the actual value of RSP in >>> Assembler::pusha_uncached()/pusha(). The original value of RSP is normally not used on the stack as >>> the value of RSP will automatically be restored after popa to the same value before doing pusha. >>> There are two locations, however, where we need to know the original value of RSP in order to print >>> it. But these places can also compute the correct value of RSP by using the new value of RSP after >>> pusha and adding 16 * wordSize to it. I fixed those. >>> >>> We still keep the same alignment by subtracting 16 * wordSize from RSP in pusha. Does anybody see >>> any potential problems by not saving the value of RSP on the stack with pusha? >>> >>> Either way, as Erik ?. has pointed out, the Windows x64 ABI does not specify a red zone of 128 >>> bytes. If we are unlucky and get an interrupt between saving RSP and decrementing RSP in the current >>> code, we could end up with a corrupt value for RSP on Windows. Therefore, we do need to fix >>> pusha_uncached() if we still want to save the old value of RSP. For example, we could first subtract >>> 16 * wordSize and then calculate the correct value: >>> >>> subq(rsp, 16 * wordSize); >>> movq(Address(rsp, 11 * wordSize), rsp); >>> addq(Address(rsp, 11 * wordSize), 16 * wordSize); >>> >>> Thank you! >>> >>> Best regards, >>> Christian From john.r.rose at oracle.com Thu May 14 19:12:03 2020 From: john.r.rose at oracle.com (John Rose) Date: Thu, 14 May 2020 12:12:03 -0700 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: References: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> Message-ID: On May 14, 2020, at 2:30 AM, Erik ?sterlund wrote: > ... > I could send > a new RFR trying to normalize the style of the various offsets in JavaClasses to use a leading '_', be private > and have an accessor function that asserts offset sanity. What do you think? That is a fine idea; thanks for suggesting it. > Not sure how I got here implementing support for compressed class pointers in the absence of compressed > oops. Oh dear? Indeed. That?s how it goes with this code base. Sometimes it?s as if you pick at a little thread on your sleeve, and then it animates and starts to wrap itself around your finger. Suddenly you realize your sleeve is full of swaying fibers? :-) ? John P.S. To deal with Emacs macro seizures, I have this in my Preferences.el: (push '("/vmSymbols\\.hpp\\'" . fundamental-mode) auto-mode-alist) (push '("/globals\\.hpp\\'" . fundamental-mode) auto-mode-alist) (push '("/vmStructs\\.cpp\\'" . fundamental-mode) auto-mode-alist) From frederic.parain at oracle.com Thu May 14 19:24:43 2020 From: frederic.parain at oracle.com (Frederic Parain) Date: Thu, 14 May 2020 15:24:43 -0400 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: References: <929ECBF7-19CA-425D-88B7-994F510160ED@oracle.com> Message-ID: <946BB4ED-2A54-4AB8-87D8-AEF30DD04667@oracle.com> Still looks good. Fred > On May 14, 2020, at 05:30, Erik ?sterlund wrote: > > New webrev so far: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.02/ > > Incremental: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01..02/ From vladimir.kozlov at oracle.com Thu May 14 20:19:58 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Thu, 14 May 2020 13:19:58 -0700 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: <1742cad7-15ac-ad6e-e4ba-4cdb3596fcb4@oracle.com> References: <56a0cf85-9698-64b2-95b4-e5b123c3e9e2@oracle.com> <9a314ebe-5499-ecb6-3c5b-0c8ce7f9c8d2@oracle.com> <44febab6-5703-1c8b-d4ad-0f4c12517afa@oracle.com> <1742cad7-15ac-ad6e-e4ba-4cdb3596fcb4@oracle.com> Message-ID: On 5/14/20 12:31 AM, Erik ?sterlund wrote: > Hi Kozlov, > > Thanks for the review. I pushed this before the review though, so any comments I will deal with in a follow-up RFR. > Let'swork out first what needs changing. Okay. > > On 2020-05-13 20:44, Vladimir Kozlov wrote: >> Hi Erik >> >> x86 changes looks fine to me (assembler, C1, C2). > > Thanks. > >> How src/hotspot/share/classfile/* changes are related to this? > > Just enabling -coop +ccp gives us back 4 bytes in the klass gap. But the layout engine for InstanceKlass > doesn't putany fields there, it aligns up to the next word by default. In other words, a lot of changes for > almostno gain (except arrays that utilize this), unless the klass gap is given back to the layout code in this > configuration. So I gave the klass gap to the fieldlayout code, which caused some fan out. Those files are > the result of that. Good to know. > >> I don't get changes in metaspace.cpp. Why use HeapBaseMinAddress when COOP is off (looks like you want klass heap at >> the beginning of Javva heap)? Can you add comment? > > I can add a comment. It's because HeapBaseMinAddress is the JVM switch for the lowest address we can map > in memory without getting dangerously close to bad sbrk interactions, e.g. due to poor malloc implementations. > Had a conversation with Thomas about this, and the conclusion is that the name of the flag is unfortunate > but it is the value I should be using. My concern is how class metaspace mapping will interact with Java heap mapping in this case. I assume (may be wrong) that both allocations will use HeapBaseMinAddress with your change. I always thought that for metaspace we will use somthing like HeapBaseMinAddress + java_heap_size address. May be I am totally wrong here. Please, explain. > >> arguments.cpp - should you also check UseNewFieldLayout flag as you do in instanceOop.hpp? > > That is not necessary. The -coop +ccp combination with -XX:-UseNewFieldLayout still works fine, it just happens > to be that it is unable to fully benefit from the fact that class pointers are being used for instanceOops, but > it still does benefit for arrays, which is not as great, but not wrong. okay > >> lcm.cpp - I concern about this change. This code is for compressed oops only - implicit null check is for oops. > > Right. I wasn't quite sure about this one. It seemed like the if statement captured the condition that there *could* > be a compressed pointer without shift, but could also include cases where this is impossible, i.e. allowing false > positives (but not false negatives, crucial for correctness). For example, if you previously ran with > +coop (shift 3) and -ccp (shift 0), then this if would take the "then" path, despite it being impossible for a > compressed pointer with shift 0 being present. So I continued to model it as having false positives but importantly > no false negatives, like before. For +coop (shift 3) and +ccp(shift 3) cases we would get base == NULL (NULL+ptr<<3+heap_base) and execute code before this code. Yes, we may execute this code for normal class pointer. Which is fine because make_ptr() which is called by get_ptr_type() handles both cases, narrow and normal pointers. > > Having said that, perhaps we could increase the accuracy of that check, getting rid of false negatives. I don't > believe that is a correctness fix though, but rather a way of having the JIT spend a few cycles less in some > configurations. The then clause always works fine, but the else clause does not, but is faster to evaluate. I think we can simplify this code because make_ptr() can handle all cases and get_ptr_type() check for null in debug build as is_ptr() does. @@ -264,15 +263,8 @@ continue; // cannot reason about it; is probably not implicit null exception } else { - const TypePtr* tptr; - if (UseCompressedOops && (CompressedOops::shift() == 0 || - CompressedKlassPointers::shift() == 0)) { - // 32-bits narrow oop can be the base of address expressions - tptr = base->get_ptr_type(); - } else { - // only regular oops are expected here - tptr = base->bottom_type()->is_ptr(); - } + // This handles both normal and compressed 32-bit pointer cases (when shift is 0). + const TypePtr* tptr = base->get_ptr_type(); // Give up if offset is not a compile-time constant. if (offset == Type::OffsetBot || tptr->_offset == Type::OffsetBot) continue; Thanks, Vladimir > > Thanks, > /Erik > >> Thanks, >> Vladimir >> >> On 5/12/20 7:09 AM, Erik ?sterlund wrote: >>> Hi, >>> >>> I ran into some merge conflicts in the tests and the new tests behaved a bit differently >>> on different platforms. The new tests naturally do not behave the same way on the platforms that do >>> not yet support compressed class pointers without compressed oops, so I just filtered away those >>> platforms from running the new tests. Will update the filter when they start supporting it. >>> >>> Another interesting thing is that on macOS, sometimes we did not get the expected compressed class >>> pointer encoding without compressed oops. The reason is that without compressed oops, we allocate >>> the heap at some random address where the OS thinks is suitable. Sometimes that is where we would >>> ideally have put the compressed class space. This forces metaspace to be mapped in somewhere else >>> with different characteristics. Ideally, the bootstrapping logic should be disentangled so that >>> when you run without compressed oops, metaspace gets reserved before the heap. But I think that >>> warrants a follow-up RFE, as I do not want to complicate this patch any further. Disentangling that >>> seems like it could become quite tricky on its own. >>> >>> So here is another webrev with the test issues resolved: >>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ >>> >>> Incremental: >>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ >>> >>> Thanks, >>> /Erik >>> >>> On 2020-04-29 09:39, Erik ?sterlund wrote: >>>> Hi Thomas, >>>> >>>> On 2020-04-29 08:27, Thomas St?fe wrote: >>>>> Hi Erik, >>>>> >>>>> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund > wrote: >>>>> >>>>> ??? Hi Thomas, >>>>> >>>>> ??? Thanks for reviewing this. >>>>> >>>>> ??? On 2020-04-28 13:14, Thomas St?fe wrote: >>>>> >>>>>> ??? With compressed oops we continue to squeeze ccs in after the heap >>>>>> ??? in the hope that it all still fits into the lower 4G or 32G. (Do >>>>>> ??? we ever bother to check if heap is actually reserved in low area? >>>>>> ??? It may have given up and reserved upstate instead, leaving free >>>>>> ??? room below...). >>>>> >>>>> ??? Right. And yes I'm pretty sure we do check. The heap reserves >>>>> ??? memory, and sets the CompressedOops::end() based on that, which is >>>>> ??? fed as input to the metaspace/CDS placement. >>>>> >>>>> >>>>> No, what I meant was: since we reserve at CompressedOops::end(), if the heap was too large to place it in the lower >>>>> address regions, the class space will not live there either even if it could, since it is smaller and might fit. >>>>> >>>>> For example: >>>>> >>>>> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace >>>>> -Xshare:off -Xmx31g >>>>> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: reserved=1048576.00 KB, committed=0.00 KB ( ?0%), used=0.00 >>>>> KB ( ?0%) >>>>> [0.003s][trace][gc,metaspace] ?[0x00000017c0400000, 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) >>>>> [0.003s][trace][gc,metaspace] Narrow klass base: 0x00000017c0400000, Narrow klass shift: 0 >>>>> [0.003s][trace][gc,metaspace] Compressed class space size: 1073741824 Address: 0x00000017c0400000 Req Addr: >>>>> 0x00000017c0400000 >>>>> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: reserved=8192.00 KB, committed=0.00 KB ( ?0%), used=0.00 KB >>>>> ( ?0%) >>>>> [0.003s][trace][gc,metaspace] ?[0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) >>>>> >>>>> (large heap and no CDS, since it prevents zero based mode). >>>>> >>>>> In that case narrow klass base is not zero based even it if probably could be, if we were to allocate class space >>>>> in the lower regions, which are now unoccupied since the large heap does not fit there. A missed opportunity to go >>>>> zero based. Not a big deal, since we mostly run with CDS now anyway. >>>> >>>> I see. Yeah, there is room for improvement there. Although, as you say, people mostly run with CDS on, so it >>>> probably won't matter even if that is improved. >>>> >>>>>> ??? Now with your patch without compressed oops we attempt to map at >>>>>> ??? HeapBaseMinAddress since this is the lowest address we are >>>>>> ??? allowed to map at and the heap won't obviously need it,?right? >>>>> >>>>> ??? Exactly. >>>>> >>>>>> ??? My only misgiving would be that?HeapBaseMinAddress is even more >>>>>> ??? of a misnomer (its already now a?bit, witness my year long >>>>>> ??? confusion :-), since it now controls not only heap placement but >>>>>> ??? ccs too. Actually if it is that important to avoid mapping to >>>>>> ??? addresses below?HeapBaseMinAddress we should probably prevent it >>>>>> ??? somewhere below in the os layer but I really do not want to open >>>>>> ??? that can of worms. >>>>> >>>>> ??? Yes agreed. Ideally it would be called something like >>>>> ??? MappingMinAddress instead, because it really isn't tied to the >>>>> ??? heap really, it just happened to be the primary user of mapping >>>>> ??? things at low addresses. But I also think changing that, and >>>>> ??? enforcing it more widely in the os layer sounds like a different >>>>> ??? can of worms. >>>>> >>>>>> ??? -- >>>>>> >>>>>> ??? Some more remarks about arguments.cpp: >>>>>> >>>>>> ??? - In Arguments::set_use_compressed_klass_ptrs(), if now >>>>>> ??? compressed class pointers are independent from compressed oops, >>>>>> ??? checking CompressedClassSpaceSize for validity should now always >>>>>> ??? happen for UseCompressedClassPointers=1. >>>>> >>>>> ??? I think it looks like I do that. >>>>> >>>>>> ??? - If HeapBaseMinAddress is now always used, we should check it >>>>>> ??? always too (see Arguments::set_heap_size) >>>>> >>>>> ??? Good idea. Fixed. >>>>> >>>>>> >>>>>>> ??????? Also, could we have some?tests which exercise cds/metaspace >>>>>>> ??????? initialization without coops and with ccps? Easiest way >>>>>>> ??????? would be to extend >>>>>>> "CompressedOops/CompressedClassPointers.java" by some cases. >>>>>> >>>>>> >>>>>> ??? Thank you. I think this makes sense especially since compressed >>>>>> ??? class pointer encoding is restricted to one mode if CDS is on >>>>>> ??? (base = shared address and shift=3), and since CDS=on is the >>>>>> ??? default zerobased encoding may not happen that often. >>>>> >>>>> ??? Yup. I added some explicit test cases for -XX:-UseCompressedOops >>>>> ??? -XX:+UseCompressedClassPointers as suggested. This will also be >>>>> ??? the default configuration for ZGC, so that should give that >>>>> ??? configuration more test coverage in the various tests we have. >>>>> >>>>> ??? Here is a new webrev: >>>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ >>>>> >>>>> ??? Incremental: >>>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ >>>>> >>>>> ??? I hope I fixed everything you wanted in this patch. I also fixed >>>>> ??? what Frederic and Coleen wanted. >>>>> >>>>> >>>>> arguments.cpp, the metaspace stuff and the test look good to me now. Thank you for your work! >>>> >>>> Thanks for the review Thomas! >>>> >>>> /Erik >>> > From erik.osterlund at oracle.com Thu May 14 21:00:29 2020 From: erik.osterlund at oracle.com (=?utf-8?Q?Erik_=C3=96sterlund?=) Date: Thu, 14 May 2020 23:00:29 +0200 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: References: Message-ID: Hi John, > On 14 May 2020, at 21:12, John Rose wrote: > > ? >> On May 14, 2020, at 2:30 AM, Erik ?sterlund wrote: > >> ... >> I could send >> a new RFR trying to normalize the style of the various offsets in JavaClasses to use a leading '_', be private >> and have an accessor function that asserts offset sanity. What do you think? > > That is a fine idea; thanks for suggesting it. Great. Will file. > >> Not sure how I got here implementing support for compressed class pointers in the absence of compressed >> oops. Oh dear? > > Indeed. That?s how it goes with this code base. > > Sometimes it?s as if you pick at a little thread on your sleeve, and > then it animates and starts to wrap itself around your finger. > Suddenly you realize your sleeve is full of swaying fibers? :-) Yup! :) > ? John > > P.S. To deal with Emacs macro seizures, I have this in my Preferences.el: > > (push '("/vmSymbols\\.hpp\\'" . fundamental-mode) auto-mode-alist) > (push '("/globals\\.hpp\\'" . fundamental-mode) auto-mode-alist) > (push '("/vmStructs\\.cpp\\'" . fundamental-mode) auto-mode-alist) Ah cool trick. Will try that one... Thanks, /Erik From erik.osterlund at oracle.com Thu May 14 21:00:47 2020 From: erik.osterlund at oracle.com (=?utf-8?Q?Erik_=C3=96sterlund?=) Date: Thu, 14 May 2020 23:00:47 +0200 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: <946BB4ED-2A54-4AB8-87D8-AEF30DD04667@oracle.com> References: <946BB4ED-2A54-4AB8-87D8-AEF30DD04667@oracle.com> Message-ID: <552A856C-7B8B-4509-BE79-EF60E7D71275@oracle.com> Hi Fred, Thanks for the review. /Erik > On 14 May 2020, at 21:24, Frederic Parain wrote: > > ?Still looks good. > > Fred > >> On May 14, 2020, at 05:30, Erik ?sterlund wrote: >> >> New webrev so far: >> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.02/ >> >> Incremental: >> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01..02/ > From erik.osterlund at oracle.com Thu May 14 21:04:51 2020 From: erik.osterlund at oracle.com (=?utf-8?Q?Erik_=C3=96sterlund?=) Date: Thu, 14 May 2020 23:04:51 +0200 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: References: Message-ID: <55586260-2D52-4804-9502-B2E07D1B00B7@oracle.com> > On 14 May 2020, at 22:20, Vladimir Kozlov wrote: > > ?On 5/14/20 12:31 AM, Erik ?sterlund wrote: >> Hi Kozlov, >> Thanks for the review. I pushed this before the review though, so any comments I will deal with in a follow-up RFR. >> Let'swork out first what needs changing. > > Okay. > >>> On 2020-05-13 20:44, Vladimir Kozlov wrote: >>> Hi Erik >>> >>> x86 changes looks fine to me (assembler, C1, C2). >> Thanks. >>> How src/hotspot/share/classfile/* changes are related to this? >> Just enabling -coop +ccp gives us back 4 bytes in the klass gap. But the layout engine for InstanceKlass >> doesn't putany fields there, it aligns up to the next word by default. In other words, a lot of changes for >> almostno gain (except arrays that utilize this), unless the klass gap is given back to the layout code in this >> configuration. So I gave the klass gap to the fieldlayout code, which caused some fan out. Those files are >> the result of that. > > Good to know. > >>> I don't get changes in metaspace.cpp. Why use HeapBaseMinAddress when COOP is off (looks like you want klass heap at the beginning of Javva heap)? Can you add comment? >> I can add a comment. It's because HeapBaseMinAddress is the JVM switch for the lowest address we can map >> in memory without getting dangerously close to bad sbrk interactions, e.g. due to poor malloc implementations. >> Had a conversation with Thomas about this, and the conclusion is that the name of the flag is unfortunate >> but it is the value I should be using. > > My concern is how class metaspace mapping will interact with Java heap mapping in this case. I assume (may be wrong) that both allocations will use HeapBaseMinAddress with your change. I always thought that for metaspace we will use somthing like HeapBaseMinAddress + java_heap_size address. May be I am totally wrong here. Please, explain. It will only try to put it in that low VA range if coop is disabled. Then the heap will be mapped in higher addresses. Sometimes on macOS though, the random address turns out to be quite low. Deferred fixing that to a separate RFE though that Thomas filed. > >>> arguments.cpp - should you also check UseNewFieldLayout flag as you do in instanceOop.hpp? >> That is not necessary. The -coop +ccp combination with -XX:-UseNewFieldLayout still works fine, it just happens >> to be that it is unable to fully benefit from the fact that class pointers are being used for instanceOops, but >> it still does benefit for arrays, which is not as great, but not wrong. > > okay > >>> lcm.cpp - I concern about this change. This code is for compressed oops only - implicit null check is for oops. >> Right. I wasn't quite sure about this one. It seemed like the if statement captured the condition that there *could* >> be a compressed pointer without shift, but could also include cases where this is impossible, i.e. allowing false >> positives (but not false negatives, crucial for correctness). For example, if you previously ran with >> +coop (shift 3) and -ccp (shift 0), then this if would take the "then" path, despite it being impossible for a >> compressed pointer with shift 0 being present. So I continued to model it as having false positives but importantly >> no false negatives, like before. > > For +coop (shift 3) and +ccp(shift 3) cases we would get base == NULL (NULL+ptr<<3+heap_base) and execute code before this code. > Yes, we may execute this code for normal class pointer. Which is fine because make_ptr() which is called by get_ptr_type() handles both cases, narrow and normal pointers. > >> Having said that, perhaps we could increase the accuracy of that check, getting rid of false negatives. I don't >> believe that is a correctness fix though, but rather a way of having the JIT spend a few cycles less in some >> configurations. The then clause always works fine, but the else clause does not, but is faster to evaluate. > > I think we can simplify this code because make_ptr() can handle all cases and get_ptr_type() check for null in debug build as is_ptr() does. > > @@ -264,15 +263,8 @@ > continue; > // cannot reason about it; is probably not implicit null exception > } else { > - const TypePtr* tptr; > - if (UseCompressedOops && (CompressedOops::shift() == 0 || > - CompressedKlassPointers::shift() == 0)) { > - // 32-bits narrow oop can be the base of address expressions > - tptr = base->get_ptr_type(); > - } else { > - // only regular oops are expected here > - tptr = base->bottom_type()->is_ptr(); > - } > + // This handles both normal and compressed 32-bit pointer cases (when shift is 0). > + const TypePtr* tptr = base->get_ptr_type(); > // Give up if offset is not a compile-time constant. > if (offset == Type::OffsetBot || tptr->_offset == Type::OffsetBot) > continue; That looks a lot simpler. Will file new RFE to clean that up. Thanks for your comments! /Erik > Thanks, > Vladimir > >> Thanks, >> /Erik >>> Thanks, >>> Vladimir >>> >>> On 5/12/20 7:09 AM, Erik ?sterlund wrote: >>>> Hi, >>>> >>>> I ran into some merge conflicts in the tests and the new tests behaved a bit differently >>>> on different platforms. The new tests naturally do not behave the same way on the platforms that do >>>> not yet support compressed class pointers without compressed oops, so I just filtered away those >>>> platforms from running the new tests. Will update the filter when they start supporting it. >>>> >>>> Another interesting thing is that on macOS, sometimes we did not get the expected compressed class >>>> pointer encoding without compressed oops. The reason is that without compressed oops, we allocate >>>> the heap at some random address where the OS thinks is suitable. Sometimes that is where we would >>>> ideally have put the compressed class space. This forces metaspace to be mapped in somewhere else >>>> with different characteristics. Ideally, the bootstrapping logic should be disentangled so that >>>> when you run without compressed oops, metaspace gets reserved before the heap. But I think that >>>> warrants a follow-up RFE, as I do not want to complicate this patch any further. Disentangling that >>>> seems like it could become quite tricky on its own. >>>> >>>> So here is another webrev with the test issues resolved: >>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ >>>> >>>> Incremental: >>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ >>>> >>>> Thanks, >>>> /Erik >>>> >>>> On 2020-04-29 09:39, Erik ?sterlund wrote: >>>>> Hi Thomas, >>>>> >>>>> On 2020-04-29 08:27, Thomas St?fe wrote: >>>>>> Hi Erik, >>>>>> >>>>>> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund > wrote: >>>>>> >>>>>> Hi Thomas, >>>>>> >>>>>> Thanks for reviewing this. >>>>>> >>>>>> On 2020-04-28 13:14, Thomas St?fe wrote: >>>>>> >>>>>>> With compressed oops we continue to squeeze ccs in after the heap >>>>>>> in the hope that it all still fits into the lower 4G or 32G. (Do >>>>>>> we ever bother to check if heap is actually reserved in low area? >>>>>>> It may have given up and reserved upstate instead, leaving free >>>>>>> room below...). >>>>>> >>>>>> Right. And yes I'm pretty sure we do check. The heap reserves >>>>>> memory, and sets the CompressedOops::end() based on that, which is >>>>>> fed as input to the metaspace/CDS placement. >>>>>> >>>>>> >>>>>> No, what I meant was: since we reserve at CompressedOops::end(), if the heap was too large to place it in the lower address regions, the class space will not live there either even if it could, since it is smaller and might fit. >>>>>> >>>>>> For example: >>>>>> >>>>>> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace -Xshare:off -Xmx31g >>>>>> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: reserved=1048576.00 KB, committed=0.00 KB ( 0%), used=0.00 KB ( 0%) >>>>>> [0.003s][trace][gc,metaspace] [0x00000017c0400000, 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) >>>>>> [0.003s][trace][gc,metaspace] Narrow klass base: 0x00000017c0400000, Narrow klass shift: 0 >>>>>> [0.003s][trace][gc,metaspace] Compressed class space size: 1073741824 Address: 0x00000017c0400000 Req Addr: 0x00000017c0400000 >>>>>> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: reserved=8192.00 KB, committed=0.00 KB ( 0%), used=0.00 KB ( 0%) >>>>>> [0.003s][trace][gc,metaspace] [0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) >>>>>> >>>>>> (large heap and no CDS, since it prevents zero based mode). >>>>>> >>>>>> In that case narrow klass base is not zero based even it if probably could be, if we were to allocate class space in the lower regions, which are now unoccupied since the large heap does not fit there. A missed opportunity to go zero based. Not a big deal, since we mostly run with CDS now anyway. >>>>> >>>>> I see. Yeah, there is room for improvement there. Although, as you say, people mostly run with CDS on, so it probably won't matter even if that is improved. >>>>> >>>>>>> Now with your patch without compressed oops we attempt to map at >>>>>>> HeapBaseMinAddress since this is the lowest address we are >>>>>>> allowed to map at and the heap won't obviously need it, right? >>>>>> >>>>>> Exactly. >>>>>> >>>>>>> My only misgiving would be that HeapBaseMinAddress is even more >>>>>>> of a misnomer (its already now a bit, witness my year long >>>>>>> confusion :-), since it now controls not only heap placement but >>>>>>> ccs too. Actually if it is that important to avoid mapping to >>>>>>> addresses below HeapBaseMinAddress we should probably prevent it >>>>>>> somewhere below in the os layer but I really do not want to open >>>>>>> that can of worms. >>>>>> >>>>>> Yes agreed. Ideally it would be called something like >>>>>> MappingMinAddress instead, because it really isn't tied to the >>>>>> heap really, it just happened to be the primary user of mapping >>>>>> things at low addresses. But I also think changing that, and >>>>>> enforcing it more widely in the os layer sounds like a different >>>>>> can of worms. >>>>>> >>>>>>> -- >>>>>>> >>>>>>> Some more remarks about arguments.cpp: >>>>>>> >>>>>>> - In Arguments::set_use_compressed_klass_ptrs(), if now >>>>>>> compressed class pointers are independent from compressed oops, >>>>>>> checking CompressedClassSpaceSize for validity should now always >>>>>>> happen for UseCompressedClassPointers=1. >>>>>> >>>>>> I think it looks like I do that. >>>>>> >>>>>>> - If HeapBaseMinAddress is now always used, we should check it >>>>>>> always too (see Arguments::set_heap_size) >>>>>> >>>>>> Good idea. Fixed. >>>>>> >>>>>>> >>>>>>>> Also, could we have some tests which exercise cds/metaspace >>>>>>>> initialization without coops and with ccps? Easiest way >>>>>>>> would be to extend >>>>>>>> "CompressedOops/CompressedClassPointers.java" by some cases. >>>>>>> >>>>>>> >>>>>>> Thank you. I think this makes sense especially since compressed >>>>>>> class pointer encoding is restricted to one mode if CDS is on >>>>>>> (base = shared address and shift=3), and since CDS=on is the >>>>>>> default zerobased encoding may not happen that often. >>>>>> >>>>>> Yup. I added some explicit test cases for -XX:-UseCompressedOops >>>>>> -XX:+UseCompressedClassPointers as suggested. This will also be >>>>>> the default configuration for ZGC, so that should give that >>>>>> configuration more test coverage in the various tests we have. >>>>>> >>>>>> Here is a new webrev: >>>>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ >>>>>> >>>>>> Incremental: >>>>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ >>>>>> >>>>>> I hope I fixed everything you wanted in this patch. I also fixed >>>>>> what Frederic and Coleen wanted. >>>>>> >>>>>> >>>>>> arguments.cpp, the metaspace stuff and the test look good to me now. Thank you for your work! >>>>> >>>>> Thanks for the review Thomas! >>>>> >>>>> /Erik >>>> From vladimir.kozlov at oracle.com Thu May 14 22:22:51 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Thu, 14 May 2020 15:22:51 -0700 Subject: RFR: 8241825: Make compressed oops and compressed class pointers independent on x86_64 In-Reply-To: <55586260-2D52-4804-9502-B2E07D1B00B7@oracle.com> References: <55586260-2D52-4804-9502-B2E07D1B00B7@oracle.com> Message-ID: <427a717c-5593-94f1-8ce1-5e62ba4d5782@oracle.com> On 5/14/20 2:04 PM, Erik ?sterlund wrote: > > >> On 14 May 2020, at 22:20, Vladimir Kozlov wrote: >> >> ?On 5/14/20 12:31 AM, Erik ?sterlund wrote: >>> Hi Kozlov, >>> Thanks for the review. I pushed this before the review though, so any comments I will deal with in a follow-up RFR. >>> Let'swork out first what needs changing. >> >> Okay. >> >>>> On 2020-05-13 20:44, Vladimir Kozlov wrote: >>>> Hi Erik >>>> >>>> x86 changes looks fine to me (assembler, C1, C2). >>> Thanks. >>>> How src/hotspot/share/classfile/* changes are related to this? >>> Just enabling -coop +ccp gives us back 4 bytes in the klass gap. But the layout engine for InstanceKlass >>> doesn't putany fields there, it aligns up to the next word by default. In other words, a lot of changes for >>> almostno gain (except arrays that utilize this), unless the klass gap is given back to the layout code in this >>> configuration. So I gave the klass gap to the fieldlayout code, which caused some fan out. Those files are >>> the result of that. >> >> Good to know. >> >>>> I don't get changes in metaspace.cpp. Why use HeapBaseMinAddress when COOP is off (looks like you want klass heap at the beginning of Javva heap)? Can you add comment? >>> I can add a comment. It's because HeapBaseMinAddress is the JVM switch for the lowest address we can map >>> in memory without getting dangerously close to bad sbrk interactions, e.g. due to poor malloc implementations. >>> Had a conversation with Thomas about this, and the conclusion is that the name of the flag is unfortunate >>> but it is the value I should be using. >> >> My concern is how class metaspace mapping will interact with Java heap mapping in this case. I assume (may be wrong) that both allocations will use HeapBaseMinAddress with your change. I always thought that for metaspace we will use somthing like HeapBaseMinAddress + java_heap_size address. May be I am totally wrong here. Please, explain. > > It will only try to put it in that low VA range if coop is disabled. Then the heap will be mapped in higher addresses. Sometimes on macOS though, the random address turns out to be quite low. Deferred fixing that to a separate RFE though that Thomas filed. Okay. > >> >>>> arguments.cpp - should you also check UseNewFieldLayout flag as you do in instanceOop.hpp? >>> That is not necessary. The -coop +ccp combination with -XX:-UseNewFieldLayout still works fine, it just happens >>> to be that it is unable to fully benefit from the fact that class pointers are being used for instanceOops, but >>> it still does benefit for arrays, which is not as great, but not wrong. >> >> okay >> >>>> lcm.cpp - I concern about this change. This code is for compressed oops only - implicit null check is for oops. >>> Right. I wasn't quite sure about this one. It seemed like the if statement captured the condition that there *could* >>> be a compressed pointer without shift, but could also include cases where this is impossible, i.e. allowing false >>> positives (but not false negatives, crucial for correctness). For example, if you previously ran with >>> +coop (shift 3) and -ccp (shift 0), then this if would take the "then" path, despite it being impossible for a >>> compressed pointer with shift 0 being present. So I continued to model it as having false positives but importantly >>> no false negatives, like before. >> >> For +coop (shift 3) and +ccp(shift 3) cases we would get base == NULL (NULL+ptr<<3+heap_base) and execute code before this code. >> Yes, we may execute this code for normal class pointer. Which is fine because make_ptr() which is called by get_ptr_type() handles both cases, narrow and normal pointers. >> >>> Having said that, perhaps we could increase the accuracy of that check, getting rid of false negatives. I don't >>> believe that is a correctness fix though, but rather a way of having the JIT spend a few cycles less in some >>> configurations. The then clause always works fine, but the else clause does not, but is faster to evaluate. >> >> I think we can simplify this code because make_ptr() can handle all cases and get_ptr_type() check for null in debug build as is_ptr() does. >> >> @@ -264,15 +263,8 @@ >> continue; >> // cannot reason about it; is probably not implicit null exception >> } else { >> - const TypePtr* tptr; >> - if (UseCompressedOops && (CompressedOops::shift() == 0 || >> - CompressedKlassPointers::shift() == 0)) { >> - // 32-bits narrow oop can be the base of address expressions >> - tptr = base->get_ptr_type(); >> - } else { >> - // only regular oops are expected here >> - tptr = base->bottom_type()->is_ptr(); >> - } >> + // This handles both normal and compressed 32-bit pointer cases (when shift is 0). >> + const TypePtr* tptr = base->get_ptr_type(); >> // Give up if offset is not a compile-time constant. >> if (offset == Type::OffsetBot || tptr->_offset == Type::OffsetBot) >> continue; > > That looks a lot simpler. Will file new RFE to clean that up. Good. Thanks, Vladimir > > Thanks for your comments! > > /Erik > >> Thanks, >> Vladimir >> >>> Thanks, >>> /Erik >>>> Thanks, >>>> Vladimir >>>> >>>> On 5/12/20 7:09 AM, Erik ?sterlund wrote: >>>>> Hi, >>>>> >>>>> I ran into some merge conflicts in the tests and the new tests behaved a bit differently >>>>> on different platforms. The new tests naturally do not behave the same way on the platforms that do >>>>> not yet support compressed class pointers without compressed oops, so I just filtered away those >>>>> platforms from running the new tests. Will update the filter when they start supporting it. >>>>> >>>>> Another interesting thing is that on macOS, sometimes we did not get the expected compressed class >>>>> pointer encoding without compressed oops. The reason is that without compressed oops, we allocate >>>>> the heap at some random address where the OS thinks is suitable. Sometimes that is where we would >>>>> ideally have put the compressed class space. This forces metaspace to be mapped in somewhere else >>>>> with different characteristics. Ideally, the bootstrapping logic should be disentangled so that >>>>> when you run without compressed oops, metaspace gets reserved before the heap. But I think that >>>>> warrants a follow-up RFE, as I do not want to complicate this patch any further. Disentangling that >>>>> seems like it could become quite tricky on its own. >>>>> >>>>> So here is another webrev with the test issues resolved: >>>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.02/ >>>>> >>>>> Incremental: >>>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01..02/ >>>>> >>>>> Thanks, >>>>> /Erik >>>>> >>>>> On 2020-04-29 09:39, Erik ?sterlund wrote: >>>>>> Hi Thomas, >>>>>> >>>>>> On 2020-04-29 08:27, Thomas St?fe wrote: >>>>>>> Hi Erik, >>>>>>> >>>>>>> On Tue, Apr 28, 2020 at 6:28 PM Erik ?sterlund > wrote: >>>>>>> >>>>>>> Hi Thomas, >>>>>>> >>>>>>> Thanks for reviewing this. >>>>>>> >>>>>>> On 2020-04-28 13:14, Thomas St?fe wrote: >>>>>>> >>>>>>>> With compressed oops we continue to squeeze ccs in after the heap >>>>>>>> in the hope that it all still fits into the lower 4G or 32G. (Do >>>>>>>> we ever bother to check if heap is actually reserved in low area? >>>>>>>> It may have given up and reserved upstate instead, leaving free >>>>>>>> room below...). >>>>>>> >>>>>>> Right. And yes I'm pretty sure we do check. The heap reserves >>>>>>> memory, and sets the CompressedOops::end() based on that, which is >>>>>>> fed as input to the metaspace/CDS placement. >>>>>>> >>>>>>> >>>>>>> No, what I meant was: since we reserve at CompressedOops::end(), if the heap was too large to place it in the lower address regions, the class space will not live there either even if it could, since it is smaller and might fit. >>>>>>> >>>>>>> For example: >>>>>>> >>>>>>> thomas at mainframe:/shared/projects/openjdk/jdk-jdk/source$ ../../jdks/sapmachine14/bin/java -Xlog:gc+metaspace=trace -Xshare:off -Xmx31g >>>>>>> [0.003s][trace][gc,metaspace] node @0x00007fca441b9f90: reserved=1048576.00 KB, committed=0.00 KB ( 0%), used=0.00 KB ( 0%) >>>>>>> [0.003s][trace][gc,metaspace] [0x00000017c0400000, 0x00000017c0400000, 0x00000017c0400000, 0x0000001800400000) >>>>>>> [0.003s][trace][gc,metaspace] Narrow klass base: 0x00000017c0400000, Narrow klass shift: 0 >>>>>>> [0.003s][trace][gc,metaspace] Compressed class space size: 1073741824 Address: 0x00000017c0400000 Req Addr: 0x00000017c0400000 >>>>>>> [0.003s][trace][gc,metaspace] node @0x00007fca441ba190: reserved=8192.00 KB, committed=0.00 KB ( 0%), used=0.00 KB ( 0%) >>>>>>> [0.003s][trace][gc,metaspace] [0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cb800000, 0x00007fc9cc000000) >>>>>>> >>>>>>> (large heap and no CDS, since it prevents zero based mode). >>>>>>> >>>>>>> In that case narrow klass base is not zero based even it if probably could be, if we were to allocate class space in the lower regions, which are now unoccupied since the large heap does not fit there. A missed opportunity to go zero based. Not a big deal, since we mostly run with CDS now anyway. >>>>>> >>>>>> I see. Yeah, there is room for improvement there. Although, as you say, people mostly run with CDS on, so it probably won't matter even if that is improved. >>>>>> >>>>>>>> Now with your patch without compressed oops we attempt to map at >>>>>>>> HeapBaseMinAddress since this is the lowest address we are >>>>>>>> allowed to map at and the heap won't obviously need it, right? >>>>>>> >>>>>>> Exactly. >>>>>>> >>>>>>>> My only misgiving would be that HeapBaseMinAddress is even more >>>>>>>> of a misnomer (its already now a bit, witness my year long >>>>>>>> confusion :-), since it now controls not only heap placement but >>>>>>>> ccs too. Actually if it is that important to avoid mapping to >>>>>>>> addresses below HeapBaseMinAddress we should probably prevent it >>>>>>>> somewhere below in the os layer but I really do not want to open >>>>>>>> that can of worms. >>>>>>> >>>>>>> Yes agreed. Ideally it would be called something like >>>>>>> MappingMinAddress instead, because it really isn't tied to the >>>>>>> heap really, it just happened to be the primary user of mapping >>>>>>> things at low addresses. But I also think changing that, and >>>>>>> enforcing it more widely in the os layer sounds like a different >>>>>>> can of worms. >>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Some more remarks about arguments.cpp: >>>>>>>> >>>>>>>> - In Arguments::set_use_compressed_klass_ptrs(), if now >>>>>>>> compressed class pointers are independent from compressed oops, >>>>>>>> checking CompressedClassSpaceSize for validity should now always >>>>>>>> happen for UseCompressedClassPointers=1. >>>>>>> >>>>>>> I think it looks like I do that. >>>>>>> >>>>>>>> - If HeapBaseMinAddress is now always used, we should check it >>>>>>>> always too (see Arguments::set_heap_size) >>>>>>> >>>>>>> Good idea. Fixed. >>>>>>> >>>>>>>> >>>>>>>>> Also, could we have some tests which exercise cds/metaspace >>>>>>>>> initialization without coops and with ccps? Easiest way >>>>>>>>> would be to extend >>>>>>>>> "CompressedOops/CompressedClassPointers.java" by some cases. >>>>>>>> >>>>>>>> >>>>>>>> Thank you. I think this makes sense especially since compressed >>>>>>>> class pointer encoding is restricted to one mode if CDS is on >>>>>>>> (base = shared address and shift=3), and since CDS=on is the >>>>>>>> default zerobased encoding may not happen that often. >>>>>>> >>>>>>> Yup. I added some explicit test cases for -XX:-UseCompressedOops >>>>>>> -XX:+UseCompressedClassPointers as suggested. This will also be >>>>>>> the default configuration for ZGC, so that should give that >>>>>>> configuration more test coverage in the various tests we have. >>>>>>> >>>>>>> Here is a new webrev: >>>>>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.01/ >>>>>>> >>>>>>> Incremental: >>>>>>> http://cr.openjdk.java.net/~eosterlund/8241825/webrev.00_01/ >>>>>>> >>>>>>> I hope I fixed everything you wanted in this patch. I also fixed >>>>>>> what Frederic and Coleen wanted. >>>>>>> >>>>>>> >>>>>>> arguments.cpp, the metaspace stuff and the test look good to me now. Thank you for your work! >>>>>> >>>>>> Thanks for the review Thomas! >>>>>> >>>>>> /Erik >>>>> > From david.holmes at oracle.com Thu May 14 22:38:42 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 15 May 2020 08:38:42 +1000 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> Message-ID: <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: > > I thought that this code under Verbose should have been changed to use > logging with log_level = verbose.? We were trying to minimize/remove the > Verbose flag. > > On 5/14/20 9:50 AM, David Holmes wrote: >> +1 >> >> But I'd be inclined to generalize the test to run: >> >> -Xlog:all=trace -XX:+Verbose > > Don't do this, it would be tons of output and make the test unusable if > it fails. The only way the test can fail is if it crashes. But we would want a level of indirection so that the test log doesn't contain the full output ~11000 lines of text. David ----- > Coleen > >> >> Curiously I couldn't reproduce the crash that this bug is about. But >> the hs_err file speaks for itself. >> >> Cheers, >> David >> >> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>> Hi David, hi Daniel >>>> >>>> Thank you for your reviews! >>>> >>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>> ???? Not your bug, but the comment has two typos: >>>>> >>>>> ???????? s/by safer/but safer/ >>>>> ???????? s/unexpensive/inexpensive/ >>>> >>>> Thanks, fixed it. >>>> >>>> On 14.05.20 01:51, David Holmes wrote: >>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>> >>>>> Given all of the code is inside: >>>>> >>>>> ??504?? if (Verbose) { >>>>> >>>>> I would just move the ResourceMark to the top of that block. We are >>>>> going to use it unconditionally, and we are not releasing it until >>>>> the end of the method regardless. That avoids the potential concern >>>>> Dan raised about extending the scope of the PRESERVE_EXCEPTION_MARK. >>>> >>>> This sounds reasonable. I moved it to the top of that block. >>>> >>>>>> There are other places like Klass::print_on(outputStream*) where >>>>>> this is still the case. Should we file an RFE to clean these up? >>>>> >>>>> Obviously we don't have any tests running in a mode that causes >>>>> this code to be executed, so yes it is probably a good idea to take >>>>> a closer look at other call-sites and file a RFE if needed. >>>> >>>> Yes, I filed one to take a closer look at these [1]. >>>> >>>> I included the changes in a new webrev: >>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>> >>> Thumbs up. >>> >>> Dan >>> >>>> >>>> Best regards, >>>> Christian >>>> >>>> >>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>> >>> > From david.holmes at oracle.com Thu May 14 22:49:20 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 15 May 2020 08:49:20 +1000 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: <1ea48e4a-3409-3f5d-7b3d-2c25af6ddc8e@oracle.com> References: <1ea48e4a-3409-3f5d-7b3d-2c25af6ddc8e@oracle.com> Message-ID: <6322afd3-bba4-ed87-2cd8-7cc866c29762@oracle.com> Nothing further from me. Thanks, David On 14/05/2020 7:37 pm, Erik ?sterlund wrote: > Hi David, > > Thanks for having a look at this! > > On 2020-05-14 07:19, David Holmes wrote: >> Hi Erik, >> >> On 14/05/2020 1:26 am, Erik ?sterlund wrote: >>> Hi, >>> >>> Let's remove hardcoded field offsets. Main trick to get this done is >>> to split interpreter initialization >>> into two parts: 1. reserve the interpreter memory, 2. generate the >>> interpreter. This allows reserving >>> it before loading methods (which generates i2c adapters that assert >>> things are "in the interpreter", >>> requiring knowledge about its address range), and then load the >>> methods, and then generate the interpreter. >>> This moving of interpreter code generation until after methods have >>> been loaded allows offsets to be >>> looked up, needed by code generation. That way, we no longer need to >>> hardcode field offsets, which >>> makes it less annoying to change the field layout code in the future. >>> >>> Thanks to Coleen for pre-reviewing this. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8243996 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ >> >> This seems like a good simplification, and allows for future >> flexibility. Initialization order changes are always tricky as the >> dependencies are not always obvious - there may be logging or error >> path related code that eventually assumes the interpreter has been >> generated - but we can deal with such issues if they arise. >> >> Generally all seems fine to me - I won't comment on the colour of the >> bikeshed. :) > > Phew! ;) > >> One small comment: >> >> src/hotspot/share/runtime/init.cpp >> >> ! void interpreter_init_code();????? // after methods loaded >> >> given the later comment: >> >> +?? interpreter_init_code(); // after javaClasses_init and before any >> method gets linked (not loaded) >> >> should the first comment be expanded to: >> >> void interpreter_init_code();????? // after methods loaded, but before >> they are linked >> >> ? >> >> Also IIUC the parenthetical "(not loaded)" alludes to the currently >> incorrect comment: >> >> void invocationCounter_init(); // before any methods loaded >> >> but once you've fixed that there's no context to know what the "(not >> loaded)" relates to so it can just be dropped. > > I think I hopefully fixed the comments there the way you envision it in > the latest webrev: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.02/ > > Incremental: > http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01..02/ > > Although about loading vs linking it's not strictly true always. With > CDS, the loading also links, but that linking is > a bit different and unproblematic for these initialization purposes. I > tried to keep it simple enough to get the gist > of it in these one liner comments, instead of expanding the (for this > context) unnecessarily complicated whole truth. > Hope that is okay. > > Thanks, > /Erik > >> Thanks, >> David >> ----- >> >>> Thanks, >>> /Erik > From vladimir.kozlov at oracle.com Thu May 14 23:20:14 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Thu, 14 May 2020 16:20:14 -0700 Subject: RFR: 8244416: Remove incorrect assert during inline cache cleaning In-Reply-To: <5a4574d6-0d7d-9d52-19b0-cbacebb1fba5@oracle.com> References: <5a4574d6-0d7d-9d52-19b0-cbacebb1fba5@oracle.com> Message-ID: <9ed9350f-a55b-33a2-6c36-be6fa9b4483a@oracle.com> Do we need to clean up IC in zombie nmethod 'from' here? Sweeper will do that later anyway as you said. Thanks, Vladimir On 5/14/20 6:07 AM, Erik ?sterlund wrote: > Hi, > > Months of stress testing triggered the following assert in inline cache cleaning: > > assert(!from->is_zombie(), "should not clean inline caches on zombies"); > > It used to be assert(!from->is_alive()), but after finding a race that resulted in > false positives for unloaded nmethods, the assert was relaxed to check only that > we shouldn't have zombies here, with a fat comment explaining why. > > Unfortunately, the relaxed assert also gets false positives, so I have chosen to > remove it, and further increase the size of that fat comment, explaning why we can't > assert that it isn't a zombie either. > > What it boils down to is a race where inline cache cleaning is performed from > a concurrent GC thread, concurrent to the sweeper making an nmethod zombie. > > The order of the race ingredients making the nmethod zombie from the sweeper is > the following: > > 1. Transition state to zombie > 2. Unregister nmethod form GC (importantly triggers a wait for concurrent code cache unloading to finish) > 3. clear_ic_callsites() importantly releasing all IC stubs > > The race happens when the GC finds an is_alive && !is_unloading nmethod and > triggers inline cache cleaning on it before 1, but then when racingly checking > again in the assert, we are past 1. > > Since the assert will spuriously fail even though there is no actual problem, and > we are at a point where there isn't much else it could assert, I chose to remove it. > > Other than triggering the assert, it is worth noting that this racingly zombiefied > nmethod will have its inline caches cleaned *with* IC stubs from the concurrent > GC thread. Subsequent IC finalization in safepoint cleanup requires live IC stubs > to belong to is_alive() nmethods, so the IC stubs must be released before the next > safepoint for this race to not be dangerous. That is ensured, because the racing > sweeper will get stuck waiting for concurrent code cache unloading to finish, > and when it wakes up, it will clear_ic_callsites(), releasing the spurious ICStubs, > before finishing the make_zombie() function. Then it will eventually hit a safepoint > poll, when the nmethod has been purged from IC stubs. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8244416 > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8244416/webrev.00/ > > Thanks, > /Erik From erik.osterlund at oracle.com Fri May 15 05:33:34 2020 From: erik.osterlund at oracle.com (=?utf-8?Q?Erik_=C3=96sterlund?=) Date: Fri, 15 May 2020 07:33:34 +0200 Subject: RFR: 8243996: Remove hardcoded field offsets from HotSpot In-Reply-To: <6322afd3-bba4-ed87-2cd8-7cc866c29762@oracle.com> References: <6322afd3-bba4-ed87-2cd8-7cc866c29762@oracle.com> Message-ID: Thanks David. /Erik > On 15 May 2020, at 00:49, David Holmes wrote: > > ?Nothing further from me. > > Thanks, > David > >> On 14/05/2020 7:37 pm, Erik ?sterlund wrote: >> Hi David, >> Thanks for having a look at this! >>> On 2020-05-14 07:19, David Holmes wrote: >>> Hi Erik, >>> >>> On 14/05/2020 1:26 am, Erik ?sterlund wrote: >>>> Hi, >>>> >>>> Let's remove hardcoded field offsets. Main trick to get this done is to split interpreter initialization >>>> into two parts: 1. reserve the interpreter memory, 2. generate the interpreter. This allows reserving >>>> it before loading methods (which generates i2c adapters that assert things are "in the interpreter", >>>> requiring knowledge about its address range), and then load the methods, and then generate the interpreter. >>>> This moving of interpreter code generation until after methods have been loaded allows offsets to be >>>> looked up, needed by code generation. That way, we no longer need to hardcode field offsets, which >>>> makes it less annoying to change the field layout code in the future. >>>> >>>> Thanks to Coleen for pre-reviewing this. >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8243996 >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01/ >>> >>> This seems like a good simplification, and allows for future flexibility. Initialization order changes are always tricky as the dependencies are not always obvious - there may be logging or error path related code that eventually assumes the interpreter has been generated - but we can deal with such issues if they arise. >>> >>> Generally all seems fine to me - I won't comment on the colour of the bikeshed. :) >> Phew! ;) >>> One small comment: >>> >>> src/hotspot/share/runtime/init.cpp >>> >>> ! void interpreter_init_code(); // after methods loaded >>> >>> given the later comment: >>> >>> + interpreter_init_code(); // after javaClasses_init and before any method gets linked (not loaded) >>> >>> should the first comment be expanded to: >>> >>> void interpreter_init_code(); // after methods loaded, but before they are linked >>> >>> ? >>> >>> Also IIUC the parenthetical "(not loaded)" alludes to the currently incorrect comment: >>> >>> void invocationCounter_init(); // before any methods loaded >>> >>> but once you've fixed that there's no context to know what the "(not loaded)" relates to so it can just be dropped. >> I think I hopefully fixed the comments there the way you envision it in the latest webrev: >> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.02/ >> Incremental: >> http://cr.openjdk.java.net/~eosterlund/8243996/webrev.01..02/ >> Although about loading vs linking it's not strictly true always. With CDS, the loading also links, but that linking is >> a bit different and unproblematic for these initialization purposes. I tried to keep it simple enough to get the gist >> of it in these one liner comments, instead of expanding the (for this context) unnecessarily complicated whole truth. >> Hope that is okay. >> Thanks, >> /Erik >>> Thanks, >>> David >>> ----- >>> >>>> Thanks, >>>> /Erik From christian.hagedorn at oracle.com Fri May 15 08:03:50 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Fri, 15 May 2020 10:03:50 +0200 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> Message-ID: On 15.05.20 00:38, David Holmes wrote: > On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >> >> I thought that this code under Verbose should have been changed to use >> logging with log_level = verbose.? We were trying to minimize/remove >> the Verbose flag. >> >> On 5/14/20 9:50 AM, David Holmes wrote: >>> +1 >>> >>> But I'd be inclined to generalize the test to run: >>> >>> -Xlog:all=trace -XX:+Verbose >> >> Don't do this, it would be tons of output and make the test unusable >> if it fails. > > The only way the test can fail is if it crashes. > > But we would want a level of indirection so that the test log doesn't > contain the full output ~11000 lines of text. Could we still generalize the test but redirect the output to a file with: -Xlog:all=trace:file=output.txt -XX:+Verbose ? When I just immediately return from trace_method_handle_stub() without printing anything, the above command will not print anything to the console. So, it would print the same amount of lines to the console as the current option in webrev.01 with -XX:+Verbose -Xlog:methodhandles. In a next step we could file an RFE to replace Verbose with unified logging in trace_method_handle_stub() which currently is only using tty. Even when using -Xlog:methodhandles alone without -XX:+Verbose, there is still some printing done in trace_method_handle_stub() to tty by 500 tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, 501 adaptername, mh_reg_name, 502 p2i(mh), p2i(entry_sp)); which is probably also not desired, as it is guarded by log_is_enabled(Info, methodhandles) earlier [1]. This could also be fixed in this follow-up RFE. What do you think? Best regards, Christian [1] http://hg.openjdk.java.net/jdk/jdk/file/748fedeb7cc1/src/hotspot/cpu/x86/methodHandles_x86.cpp#l598 > > David > ----- > >> Coleen >> >>> >>> Curiously I couldn't reproduce the crash that this bug is about. But >>> the hs_err file speaks for itself. >>> >>> Cheers, >>> David >>> >>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>> Hi David, hi Daniel >>>>> >>>>> Thank you for your reviews! >>>>> >>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>> ???? Not your bug, but the comment has two typos: >>>>>> >>>>>> ???????? s/by safer/but safer/ >>>>>> ???????? s/unexpensive/inexpensive/ >>>>> >>>>> Thanks, fixed it. >>>>> >>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>> >>>>>> Given all of the code is inside: >>>>>> >>>>>> ??504?? if (Verbose) { >>>>>> >>>>>> I would just move the ResourceMark to the top of that block. We >>>>>> are going to use it unconditionally, and we are not releasing it >>>>>> until the end of the method regardless. That avoids the potential >>>>>> concern Dan raised about extending the scope of the >>>>>> PRESERVE_EXCEPTION_MARK. >>>>> >>>>> This sounds reasonable. I moved it to the top of that block. >>>>> >>>>>>> There are other places like Klass::print_on(outputStream*) where >>>>>>> this is still the case. Should we file an RFE to clean these up? >>>>>> >>>>>> Obviously we don't have any tests running in a mode that causes >>>>>> this code to be executed, so yes it is probably a good idea to >>>>>> take a closer look at other call-sites and file a RFE if needed. >>>>> >>>>> Yes, I filed one to take a closer look at these [1]. >>>>> >>>>> I included the changes in a new webrev: >>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>> >>>> Thumbs up. >>>> >>>> Dan >>>> >>>>> >>>>> Best regards, >>>>> Christian >>>>> >>>>> >>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>> >>>> >> From christian.hagedorn at oracle.com Fri May 15 08:05:27 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Fri, 15 May 2020 10:05:27 +0200 Subject: [15] RFR(S): 8244433: Remove saving of RSP in Assembler::pusha_uncached() In-Reply-To: <34f07323-acc9-ad05-2d81-2d82dfcbcd14@oracle.com> References: <275960b8-a9f4-3469-b8aa-41c3f780fd19@oracle.com> <58ae3976-4ab1-49b3-670b-15370e8e018b@oracle.com> <5a2a07ed-2d37-fa7c-4171-af6f03dafd38@oracle.com> <34f07323-acc9-ad05-2d81-2d82dfcbcd14@oracle.com> Message-ID: <882b9eae-6949-40af-f796-4b21762f2f8b@oracle.com> Hi Vladimir On 14.05.20 20:52, Vladimir Kozlov wrote: > Changes looks good. > > I looked and this code existed from day one. And I don't remember we > ever had an issue with possible RSP corrupted values. I don't think you > need to worry about this after you fixed print code. That's good, thank you for checking and your review! Best regards, Christian > On 5/14/20 5:35 AM, Christian Hagedorn wrote: >> Thank you Tobias for your review! >> >> Best regards, >> Christian >> >> On 14.05.20 14:31, Tobias Hartmann wrote: >>> Hi Christian, >>> >>> this looks good to me. >>> >>> Best regards, >>> Tobias >>> >>> On 14.05.20 11:38, Christian Hagedorn wrote: >>>> Hi >>>> >>>> Please review the following enhancement for x86: >>>> https://bugs.openjdk.java.net/browse/JDK-8244433 >>>> http://cr.openjdk.java.net/~chagedorn/8244433/webrev.00/ >>>> >>>> This removes the move instruction for saving the actual value of RSP in >>>> Assembler::pusha_uncached()/pusha(). The original value of RSP is >>>> normally not used on the stack as >>>> the value of RSP will automatically be restored after popa to the >>>> same value before doing pusha. >>>> There are two locations, however, where we need to know the original >>>> value of RSP in order to print >>>> it. But these places can also compute the correct value of RSP by >>>> using the new value of RSP after >>>> pusha and adding 16 * wordSize to it. I fixed those. >>>> >>>> We still keep the same alignment by subtracting 16 * wordSize from >>>> RSP in pusha. Does anybody see >>>> any potential problems by not saving the value of RSP on the stack >>>> with pusha? >>>> >>>> Either way, as Erik ?. has pointed out, the Windows x64 ABI does not >>>> specify a red zone of 128 >>>> bytes. If we are unlucky and get an interrupt between saving RSP and >>>> decrementing RSP in the current >>>> code, we could end up with a corrupt value for RSP on Windows. >>>> Therefore, we do need to fix >>>> pusha_uncached() if we still want to save the old value of RSP. For >>>> example, we could first subtract >>>> 16 * wordSize and then calculate the correct value: >>>> >>>> subq(rsp, 16 * wordSize); >>>> movq(Address(rsp, 11 * wordSize), rsp); >>>> addq(Address(rsp, 11 * wordSize), 16 * wordSize); >>>> >>>> Thank you! >>>> >>>> Best regards, >>>> Christian From erik.osterlund at oracle.com Fri May 15 08:10:31 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Fri, 15 May 2020 10:10:31 +0200 Subject: RFR: 8244416: Remove incorrect assert during inline cache cleaning In-Reply-To: <9ed9350f-a55b-33a2-6c36-be6fa9b4483a@oracle.com> References: <5a4574d6-0d7d-9d52-19b0-cbacebb1fba5@oracle.com> <9ed9350f-a55b-33a2-6c36-be6fa9b4483a@oracle.com> Message-ID: <9774524d-ffe8-85eb-da98-4b6f128f472e@oracle.com> Hi Vladimir, On 2020-05-15 01:20, Vladimir Kozlov wrote: > Do we need to clean up IC in zombie nmethod 'from' here? Sweeper will > do that later anyway as you said. You are 100% right. We do not need to clean up IC in zombie nmethods. In fact we try to actively avoid it. The concurrent GC thread only initiates IC cleaning on is_alive() nmethods. The issue is that from checking if it is_alive() and then starting the IC cleaning, the state might flip over to zombie in a race with make_zombie from the sweeper. That is fine as I described, because the make_zombie() function will in this race wait for the GC to finish and then clear the ICs, releasing spurious IC stubs created by the GC mistakenly thinking it is_alive(), before IC finalization can run in a subsequent safepoint. The only issue is that we can't assert that from must not be zombie. It almost never is, but it is sometimes, and that is fine. Thanks, /Erik > Thanks, > Vladimir > > On 5/14/20 6:07 AM, Erik ?sterlund wrote: >> Hi, >> >> Months of stress testing triggered the following assert in inline >> cache cleaning: >> >> assert(!from->is_zombie(), "should not clean inline caches on zombies"); >> >> It used to be assert(!from->is_alive()), but after finding a race >> that resulted in >> false positives for unloaded nmethods, the assert was relaxed to >> check only that >> we shouldn't have zombies here, with a fat comment explaining why. >> >> Unfortunately, the relaxed assert also gets false positives, so I >> have chosen to >> remove it, and further increase the size of that fat comment, >> explaning why we can't >> assert that it isn't a zombie either. >> >> What it boils down to is a race where inline cache cleaning is >> performed from >> a concurrent GC thread, concurrent to the sweeper making an nmethod >> zombie. >> >> The order of the race ingredients making the nmethod zombie from the >> sweeper is >> the following: >> >> 1. Transition state to zombie >> 2. Unregister nmethod form GC (importantly triggers a wait for >> concurrent code cache unloading to finish) >> 3. clear_ic_callsites() importantly releasing all IC stubs > >> The race happens when the GC finds an is_alive && !is_unloading >> nmethod and >> triggers inline cache cleaning on it before 1, but then when racingly >> checking >> again in the assert, we are past 1. >> >> Since the assert will spuriously fail even though there is no actual >> problem, and >> we are at a point where there isn't much else it could assert, I >> chose to remove it. >> >> Other than triggering the assert, it is worth noting that this >> racingly zombiefied >> nmethod will have its inline caches cleaned *with* IC stubs from the >> concurrent >> GC thread. Subsequent IC finalization in safepoint cleanup requires >> live IC stubs >> to belong to is_alive() nmethods, so the IC stubs must be released >> before the next >> safepoint for this race to not be dangerous. That is ensured, because >> the racing >> sweeper will get stuck waiting for concurrent code cache unloading to >> finish, >> and when it wakes up, it will clear_ic_callsites(), releasing the >> spurious ICStubs, >> before finishing the make_zombie() function. Then it will eventually >> hit a safepoint >> poll, when the nmethod has been purged from IC stubs. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8244416 >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8244416/webrev.00/ >> >> Thanks, >> /Erik From sgehwolf at redhat.com Fri May 15 08:45:03 2020 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Fri, 15 May 2020 10:45:03 +0200 Subject: [8u] RFR(S): 8244548: JDK 8u: sun.misc.Version.jdkUpdateVersion() returns wrong result In-Reply-To: References: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> <808419e3-316b-381d-f636-606aadff39b6@redhat.com> <15a2fa41f80935d8c7d3893aad7d080c1f67aa2a.camel@redhat.com> Message-ID: <74e4d10641c8b6acca8ada68459a767f98e23585.camel@redhat.com> Hi, On Wed, 2020-05-13 at 16:11 +0100, Andrew Hughes wrote: > On 12/05/2020 15:14, Severin Gehwolf wrote: > > Hi Andrew, > > > > Thanks for looking at this. > > > > On Tue, 2020-05-12 at 14:07 +0100, Andrew Haley wrote: > > > On 5/7/20 4:09 PM, Severin Gehwolf wrote: > > > > jvm_version_info.jvm_version currently holds this quadruplet: > > > > > > > > Most significant 8 bits => major version, followed by 8 bits => minor > > > > version, followed by 8 bits => micro version, followed by 8 bits => > > > > build version. Note that JVM minor version represents the update > > > > version as passed in via configure and the micro version is currently > > > > not used (always 0). See vm_version.cpp lines 100-102 where only major, > > > > minor and build number are ever been set. Knowing this, we can still > > > > preserve the same behavior after patch by defining JVM_VERSION_MICRO to > > > > 0 for any version. > > > > > > This is tricky. JVM_GetVersionInfo is a function exported by > > > libjvm.so, and the version is simply encoded as > > > > > > unsigned int Abstract_VM_Version::jvm_version() { > > > return ((Abstract_VM_Version::vm_major_version() & 0xFF) << 24) | > > > ((Abstract_VM_Version::vm_minor_version() & 0xFF) << 16) | > > > (Abstract_VM_Version::vm_build_number() & 0xFF); > > > } > > > > > > I guess we could argue that this is for JVM-JDK internal use only, and > > > no-one else cares. > > > > > > Or we could encode it in a different way such that at least we have a > > > jvm_version that is monotonically increasing, perhaps by (ab)using the > > > lower 8 bits of the version, the vm_build_number. But I guess I'm > > > being paranoid, and no tools are going to care about minor versions > > > anyway,even if they do call JVM_GetVersionInfo. > > > > Yes, this is indeed tricky. The trouble is that > > Abstract_VM_Version::vm_build_number() actually holds the configured > > JDK build number, we can't really use it. It's already being used. > > > > (gdb) > > 99 > > 100 _vm_major_version = atoi(vm_major_ver); > > 101 _vm_minor_version = atoi(vm_minor_ver); > > 102 _vm_build_number = atoi(vm_build_num); > > 103 > > 104 os::free(vm_version); > > 105 _initialized = true; > > 106 } > > 107 > > 108 #if defined(_LP64) > > (gdb) p _vm_build_number > > $1 = 2 > > > > The only bits which seem unused are bits 8 through 16 of this unsigned > > int. So even if JVM_GetVersionInfo *is* being used somewhere in the > > wild, it *should* continue to work. Hence, the proposal to now also use > > those unused bits for the minor version. > > > > Thoughts? > > > > Thanks, > > Severin > > > > The April release of OpenJDK 7u has the values: > > Major version: 24, minor version: 5, micro version: 0, build number: 2 > > when it should be 24.261-b02. So we already have a situation that could > be potentially be misinterpreted as an old JDK version, 7u5 b02, using > the current interpretation of the structure. 8u will see similar issues > if we make no change in this cycle. Yes. I believe we need to get this fixed. The earlier in the 262 update cycle the better. > As the micro version appears to have always been zero, it seems pretty > safe to just hard-code this value and instead use all 16 bits of the > interpreted value for the micro version. In terms of backwards > compatibility, if a micro version greater than zero is read using the > old method, this will be an indicator that it needs to be > re-interpreted as a 16-bit value. As Andrew Hughes mentioned below, the only way to see the difference post-patch would be for somebody to use old jvm.h defines *and* use a supposedly a JDK private API (JVM_GetVersionInfo). It is an exported symbol, but as Andrew Hughes pointed out, for JDK <=> hotspot communication. On the other hand, the chance of somebody using forbidden (access restricted), yet available API via sun.misc.Version is more likely. Of which the following are broken (see my latest comment on the bug[1]): sun.misc.Version.jvmMinorVersion() sun.misc.Version.jdkUpdateVersion() Post patch the above are fixed, but somebody using JVM_GetVersionInfo directly with old jvm.h macros would get for 8u262-b03 (pseudo code): #define JVM_VERSION_MAJOR(version) ((version & 0xFF000000) >> 24) #define JVM_VERSION_MINOR(version) ((version & 0x00FF0000) >> 16) #define JVM_VERSION_MICRO(version) ((version & 0x0000FF00) >> 8) #define JVM_VERSION_BUILD(version) ((version & 0x000000FF)) jvm_version_info info; [ call JVM_GetVersionInfo, etc. ] JVM_VERSION_MAJOR(info.jvm_version) // => 25 JVM_VERSION_MINOR(info.jvm_version) // => 1 JVM_VERSION_MICRO(info.jvm_version) // => 6 JVM_VERSION_BUILD(info.jvm_version) // => 3 So, minor, micro would be wrong. Question is what's more likely in use today in the wild? JVM_GetVersionInfo or sun.misc.Version. > The current value for 7 is 402980866 > (0001 1000 0000 0101 0000 0000 0000 0010) > which thus gets interpreted as above. > > The new method encodes the same version numbers as 402720002 > (0001 1000 0000 0001 0000 0101 0000 0010) > > which can be read back as major 24, minor 261, micro 0 and build 2 using > the updated decoding. > > If the old decoding is used, we get major 24, minor 1, micro 5 and build > 2. As micro is usually set to zero, updated code should be able to use > this to instead use the new interpretation, while, in the worst case, > stale code will get a unique, if incorrect, version. Yes. So, Andrew Haley, what's the verdict on this one? There is a risk, yes. Not fixing the bug seems riskier, though. Thoughts? Thanks, Severin [1] https://bugs.openjdk.java.net/browse/JDK-8244548?focusedCommentId=14338519&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14338519 From coleen.phillimore at oracle.com Fri May 15 10:38:58 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 15 May 2020 06:38:58 -0400 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> Message-ID: On 5/15/20 4:03 AM, Christian Hagedorn wrote: > On 15.05.20 00:38, David Holmes wrote: >> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >>> >>> I thought that this code under Verbose should have been changed to >>> use logging with log_level = verbose.? We were trying to >>> minimize/remove the Verbose flag. >>> >>> On 5/14/20 9:50 AM, David Holmes wrote: >>>> +1 >>>> >>>> But I'd be inclined to generalize the test to run: >>>> >>>> -Xlog:all=trace -XX:+Verbose >>> >>> Don't do this, it would be tons of output and make the test unusable >>> if it fails. >> >> The only way the test can fail is if it crashes. >> >> But we would want a level of indirection so that the test log doesn't >> contain the full output ~11000 lines of text. > > Could we still generalize the test but redirect the output to a file > with: > > -Xlog:all=trace:file=output.txt -XX:+Verbose ? > > When I just immediately return from trace_method_handle_stub() without > printing anything, the above command will not print anything to the > console. So, it would print the same amount of lines to the console as > the current option in webrev.01 with -XX:+Verbose -Xlog:methodhandles. I'm confused.? Does the test crash with -Xlog:methodhandles -XX:+Verbose?? If so, that's all that should be needed.? If it needs -Xlog:all=trace, then don't add this test.? Even if it writes to an output file, it would be huge and affect test execution, particularly if more logging is added. > > In a next step we could file an RFE to replace Verbose with unified > logging in trace_method_handle_stub() which currently is only using > tty. Even when using -Xlog:methodhandles alone without -XX:+Verbose, > there is still some printing done in trace_method_handle_stub() to tty by > > ?500?? tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, > ?501???????????????? adaptername, mh_reg_name, > ?502???????????????? p2i(mh), p2i(entry_sp)); > > which is probably also not desired, as it is guarded by > log_is_enabled(Info, methodhandles) earlier [1]. This could also be > fixed in this follow-up RFE. Please, yes, this can be fixed with a follow-up RFE. Thanks, Coleen > > What do you think? > > Best regards, > Christian > > > [1] > http://hg.openjdk.java.net/jdk/jdk/file/748fedeb7cc1/src/hotspot/cpu/x86/methodHandles_x86.cpp#l598 > >> >> David >> ----- >> >>> Coleen >>> >>>> >>>> Curiously I couldn't reproduce the crash that this bug is about. >>>> But the hs_err file speaks for itself. >>>> >>>> Cheers, >>>> David >>>> >>>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>>> Hi David, hi Daniel >>>>>> >>>>>> Thank you for your reviews! >>>>>> >>>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>>> ???? Not your bug, but the comment has two typos: >>>>>>> >>>>>>> ???????? s/by safer/but safer/ >>>>>>> ???????? s/unexpensive/inexpensive/ >>>>>> >>>>>> Thanks, fixed it. >>>>>> >>>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>>> >>>>>>> Given all of the code is inside: >>>>>>> >>>>>>> ??504?? if (Verbose) { >>>>>>> >>>>>>> I would just move the ResourceMark to the top of that block. We >>>>>>> are going to use it unconditionally, and we are not releasing it >>>>>>> until the end of the method regardless. That avoids the >>>>>>> potential concern Dan raised about extending the scope of the >>>>>>> PRESERVE_EXCEPTION_MARK. >>>>>> >>>>>> This sounds reasonable. I moved it to the top of that block. >>>>>> >>>>>>>> There are other places like Klass::print_on(outputStream*) >>>>>>>> where this is still the case. Should we file an RFE to clean >>>>>>>> these up? >>>>>>> >>>>>>> Obviously we don't have any tests running in a mode that causes >>>>>>> this code to be executed, so yes it is probably a good idea to >>>>>>> take a closer look at other call-sites and file a RFE if needed. >>>>>> >>>>>> Yes, I filed one to take a closer look at these [1]. >>>>>> >>>>>> I included the changes in a new webrev: >>>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>>> >>>>> Thumbs up. >>>>> >>>>> Dan >>>>> >>>>>> >>>>>> Best regards, >>>>>> Christian >>>>>> >>>>>> >>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>>> >>>>> >>> From per.liden at oracle.com Fri May 15 12:03:57 2020 From: per.liden at oracle.com (Per Liden) Date: Fri, 15 May 2020 14:03:57 +0200 Subject: RFR: 8245098: Make SafeFetch32/N available earlier Message-ID: We have a need for SafeFetch32 during initializing of ZGC (see [1]). However, the SafeFetch32/N stubs are currently generated too late, in the stubRoutines_init2() phase, which is after the GC has been initialized. I propose we move the initialization of these stubs to the stubRoutines_init1() phase, which is before the GC has been initialized, to make them available during GC initialization. Please note that this patch touches platforms I can't easily test (x86-32, arm, ppc, s390). If you have access to one of those machines, I would appreciate if you could do some light verification. Bug: https://bugs.openjdk.java.net/browse/JDK-8245098 Webrev: http://cr.openjdk.java.net/~pliden/8245098/webrev.0 Testing: Tier1-3 on all Oracle supported platforms. /Per [1] https://bugs.openjdk.java.net/browse/JDK-8245106 From christian.hagedorn at oracle.com Fri May 15 12:06:44 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Fri, 15 May 2020 14:06:44 +0200 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> Message-ID: <412316cc-fb7e-f360-5a76-78e98ddf2890@oracle.com> Hi Coleen On 15.05.20 12:38, coleen.phillimore at oracle.com wrote: > On 5/15/20 4:03 AM, Christian Hagedorn wrote: >> On 15.05.20 00:38, David Holmes wrote: >>> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >>>> >>>> I thought that this code under Verbose should have been changed to >>>> use logging with log_level = verbose.? We were trying to >>>> minimize/remove the Verbose flag. >>>> >>>> On 5/14/20 9:50 AM, David Holmes wrote: >>>>> +1 >>>>> >>>>> But I'd be inclined to generalize the test to run: >>>>> >>>>> -Xlog:all=trace -XX:+Verbose >>>> >>>> Don't do this, it would be tons of output and make the test unusable >>>> if it fails. >>> >>> The only way the test can fail is if it crashes. >>> >>> But we would want a level of indirection so that the test log doesn't >>> contain the full output ~11000 lines of text. >> >> Could we still generalize the test but redirect the output to a file >> with: >> >> -Xlog:all=trace:file=output.txt -XX:+Verbose ? >> >> When I just immediately return from trace_method_handle_stub() without >> printing anything, the above command will not print anything to the >> console. So, it would print the same amount of lines to the console as >> the current option in webrev.01 with -XX:+Verbose -Xlog:methodhandles. > > I'm confused.? Does the test crash with -Xlog:methodhandles > -XX:+Verbose?? If so, that's all that should be needed.? If it needs > -Xlog:all=trace, then don't add this test.? Even if it writes to an > output file, it would be huge and affect test execution, particularly if > more logging is added. I could already reproduce it locally on my Linux machine with -Xlog:methodhandles -XX:+Verbose. From what I understood, I thought that David is just talking about having a more general log test that covers the entire logging together with -XX:+Verbose for a HelloWorld program. But as you said, that would have some bad effects on test execution. However, such a test is not required for this bug to be triggered. >> In a next step we could file an RFE to replace Verbose with unified >> logging in trace_method_handle_stub() which currently is only using >> tty. Even when using -Xlog:methodhandles alone without -XX:+Verbose, >> there is still some printing done in trace_method_handle_stub() to tty by >> >> ?500?? tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, >> ?501???????????????? adaptername, mh_reg_name, >> ?502???????????????? p2i(mh), p2i(entry_sp)); >> >> which is probably also not desired, as it is guarded by >> log_is_enabled(Info, methodhandles) earlier [1]. This could also be >> fixed in this follow-up RFE. > > Please, yes, this can be fixed with a follow-up RFE. Great, I filed an RFE for it [1]. Best regards, Christian [1] https://bugs.openjdk.java.net/browse/JDK-8245107 >>>>> Curiously I couldn't reproduce the crash that this bug is about. >>>>> But the hs_err file speaks for itself. >>>>> >>>>> Cheers, >>>>> David >>>>> >>>>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>>>> Hi David, hi Daniel >>>>>>> >>>>>>> Thank you for your reviews! >>>>>>> >>>>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>>>> ???? Not your bug, but the comment has two typos: >>>>>>>> >>>>>>>> ???????? s/by safer/but safer/ >>>>>>>> ???????? s/unexpensive/inexpensive/ >>>>>>> >>>>>>> Thanks, fixed it. >>>>>>> >>>>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>>>> >>>>>>>> Given all of the code is inside: >>>>>>>> >>>>>>>> ??504?? if (Verbose) { >>>>>>>> >>>>>>>> I would just move the ResourceMark to the top of that block. We >>>>>>>> are going to use it unconditionally, and we are not releasing it >>>>>>>> until the end of the method regardless. That avoids the >>>>>>>> potential concern Dan raised about extending the scope of the >>>>>>>> PRESERVE_EXCEPTION_MARK. >>>>>>> >>>>>>> This sounds reasonable. I moved it to the top of that block. >>>>>>> >>>>>>>>> There are other places like Klass::print_on(outputStream*) >>>>>>>>> where this is still the case. Should we file an RFE to clean >>>>>>>>> these up? >>>>>>>> >>>>>>>> Obviously we don't have any tests running in a mode that causes >>>>>>>> this code to be executed, so yes it is probably a good idea to >>>>>>>> take a closer look at other call-sites and file a RFE if needed. >>>>>>> >>>>>>> Yes, I filed one to take a closer look at these [1]. >>>>>>> >>>>>>> I included the changes in a new webrev: >>>>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>>>> >>>>>> Thumbs up. >>>>>> >>>>>> Dan >>>>>> >>>>>>> >>>>>>> Best regards, >>>>>>> Christian >>>>>>> >>>>>>> >>>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>>>> >>>>>> >>>> > From coleen.phillimore at oracle.com Fri May 15 12:56:59 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 15 May 2020 08:56:59 -0400 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: <412316cc-fb7e-f360-5a76-78e98ddf2890@oracle.com> References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> <412316cc-fb7e-f360-5a76-78e98ddf2890@oracle.com> Message-ID: On 5/15/20 8:06 AM, Christian Hagedorn wrote: > Hi Coleen > > On 15.05.20 12:38, coleen.phillimore at oracle.com wrote: >> On 5/15/20 4:03 AM, Christian Hagedorn wrote: >>> On 15.05.20 00:38, David Holmes wrote: >>>> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >>>>> >>>>> I thought that this code under Verbose should have been changed to >>>>> use logging with log_level = verbose.? We were trying to >>>>> minimize/remove the Verbose flag. >>>>> >>>>> On 5/14/20 9:50 AM, David Holmes wrote: >>>>>> +1 >>>>>> >>>>>> But I'd be inclined to generalize the test to run: >>>>>> >>>>>> -Xlog:all=trace -XX:+Verbose >>>>> >>>>> Don't do this, it would be tons of output and make the test >>>>> unusable if it fails. >>>> >>>> The only way the test can fail is if it crashes. >>>> >>>> But we would want a level of indirection so that the test log >>>> doesn't contain the full output ~11000 lines of text. >>> >>> Could we still generalize the test but redirect the output to a file >>> with: >>> >>> -Xlog:all=trace:file=output.txt -XX:+Verbose ? >>> >>> When I just immediately return from trace_method_handle_stub() >>> without printing anything, the above command will not print anything >>> to the console. So, it would print the same amount of lines to the >>> console as the current option in webrev.01 with -XX:+Verbose >>> -Xlog:methodhandles. >> >> I'm confused.? Does the test crash with -Xlog:methodhandles >> -XX:+Verbose?? If so, that's all that should be needed.? If it needs >> -Xlog:all=trace, then don't add this test.? Even if it writes to an >> output file, it would be huge and affect test execution, particularly >> if more logging is added. > > I could already reproduce it locally on my Linux machine with > -Xlog:methodhandles -XX:+Verbose. From what I understood, I thought > that David is just talking about having a more general log test that > covers the entire logging together with -XX:+Verbose for a HelloWorld > program. But as you said, that would have some bad effects on test > execution. However, such a test is not required for this bug to be > triggered. Good! This looks good to me then. Thank you for filing an RFE to do more logging with Unified Logging. Coleen > >>> In a next step we could file an RFE to replace Verbose with unified >>> logging in trace_method_handle_stub() which currently is only using >>> tty. Even when using -Xlog:methodhandles alone without -XX:+Verbose, >>> there is still some printing done in trace_method_handle_stub() to >>> tty by >>> >>> ?500?? tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, >>> ?501???????????????? adaptername, mh_reg_name, >>> ?502???????????????? p2i(mh), p2i(entry_sp)); >>> >>> which is probably also not desired, as it is guarded by >>> log_is_enabled(Info, methodhandles) earlier [1]. This could also be >>> fixed in this follow-up RFE. >> >> Please, yes, this can be fixed with a follow-up RFE. > > Great, I filed an RFE for it [1]. > > Best regards, > Christian > > > [1] https://bugs.openjdk.java.net/browse/JDK-8245107 > > >>>>>> Curiously I couldn't reproduce the crash that this bug is about. >>>>>> But the hs_err file speaks for itself. >>>>>> >>>>>> Cheers, >>>>>> David >>>>>> >>>>>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>>>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>>>>> Hi David, hi Daniel >>>>>>>> >>>>>>>> Thank you for your reviews! >>>>>>>> >>>>>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>>>>> ???? Not your bug, but the comment has two typos: >>>>>>>>> >>>>>>>>> ???????? s/by safer/but safer/ >>>>>>>>> ???????? s/unexpensive/inexpensive/ >>>>>>>> >>>>>>>> Thanks, fixed it. >>>>>>>> >>>>>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>>>>> >>>>>>>>> Given all of the code is inside: >>>>>>>>> >>>>>>>>> ??504?? if (Verbose) { >>>>>>>>> >>>>>>>>> I would just move the ResourceMark to the top of that block. >>>>>>>>> We are going to use it unconditionally, and we are not >>>>>>>>> releasing it until the end of the method regardless. That >>>>>>>>> avoids the potential concern Dan raised about extending the >>>>>>>>> scope of the PRESERVE_EXCEPTION_MARK. >>>>>>>> >>>>>>>> This sounds reasonable. I moved it to the top of that block. >>>>>>>> >>>>>>>>>> There are other places like Klass::print_on(outputStream*) >>>>>>>>>> where this is still the case. Should we file an RFE to clean >>>>>>>>>> these up? >>>>>>>>> >>>>>>>>> Obviously we don't have any tests running in a mode that >>>>>>>>> causes this code to be executed, so yes it is probably a good >>>>>>>>> idea to take a closer look at other call-sites and file a RFE >>>>>>>>> if needed. >>>>>>>> >>>>>>>> Yes, I filed one to take a closer look at these [1]. >>>>>>>> >>>>>>>> I included the changes in a new webrev: >>>>>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>>>>> >>>>>>> Thumbs up. >>>>>>> >>>>>>> Dan >>>>>>> >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Christian >>>>>>>> >>>>>>>> >>>>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>>>>> >>>>>>> >>>>> >> From christian.hagedorn at oracle.com Fri May 15 13:08:01 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Fri, 15 May 2020 15:08:01 +0200 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> <412316cc-fb7e-f360-5a76-78e98ddf2890@oracle.com> Message-ID: On 15.05.20 14:56, coleen.phillimore at oracle.com wrote: > On 5/15/20 8:06 AM, Christian Hagedorn wrote: >> Hi Coleen >> >> On 15.05.20 12:38, coleen.phillimore at oracle.com wrote: >>> On 5/15/20 4:03 AM, Christian Hagedorn wrote: >>>> On 15.05.20 00:38, David Holmes wrote: >>>>> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >>>>>> >>>>>> I thought that this code under Verbose should have been changed to >>>>>> use logging with log_level = verbose.? We were trying to >>>>>> minimize/remove the Verbose flag. >>>>>> >>>>>> On 5/14/20 9:50 AM, David Holmes wrote: >>>>>>> +1 >>>>>>> >>>>>>> But I'd be inclined to generalize the test to run: >>>>>>> >>>>>>> -Xlog:all=trace -XX:+Verbose >>>>>> >>>>>> Don't do this, it would be tons of output and make the test >>>>>> unusable if it fails. >>>>> >>>>> The only way the test can fail is if it crashes. >>>>> >>>>> But we would want a level of indirection so that the test log >>>>> doesn't contain the full output ~11000 lines of text. >>>> >>>> Could we still generalize the test but redirect the output to a file >>>> with: >>>> >>>> -Xlog:all=trace:file=output.txt -XX:+Verbose ? >>>> >>>> When I just immediately return from trace_method_handle_stub() >>>> without printing anything, the above command will not print anything >>>> to the console. So, it would print the same amount of lines to the >>>> console as the current option in webrev.01 with -XX:+Verbose >>>> -Xlog:methodhandles. >>> >>> I'm confused.? Does the test crash with -Xlog:methodhandles >>> -XX:+Verbose?? If so, that's all that should be needed.? If it needs >>> -Xlog:all=trace, then don't add this test.? Even if it writes to an >>> output file, it would be huge and affect test execution, particularly >>> if more logging is added. >> >> I could already reproduce it locally on my Linux machine with >> -Xlog:methodhandles -XX:+Verbose. From what I understood, I thought >> that David is just talking about having a more general log test that >> covers the entire logging together with -XX:+Verbose for a HelloWorld >> program. But as you said, that would have some bad effects on test >> execution. However, such a test is not required for this bug to be >> triggered. > > Good! This looks good to me then. Thank you Coleen for your review! > Thank you for filing an RFE to do more logging with Unified Logging. No problem, I will take a look at it next. Best regards, Christian >> >>>> In a next step we could file an RFE to replace Verbose with unified >>>> logging in trace_method_handle_stub() which currently is only using >>>> tty. Even when using -Xlog:methodhandles alone without -XX:+Verbose, >>>> there is still some printing done in trace_method_handle_stub() to >>>> tty by >>>> >>>> ?500?? tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, >>>> ?501???????????????? adaptername, mh_reg_name, >>>> ?502???????????????? p2i(mh), p2i(entry_sp)); >>>> >>>> which is probably also not desired, as it is guarded by >>>> log_is_enabled(Info, methodhandles) earlier [1]. This could also be >>>> fixed in this follow-up RFE. >>> >>> Please, yes, this can be fixed with a follow-up RFE. >> >> Great, I filed an RFE for it [1]. >> >> Best regards, >> Christian >> >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8245107 >> >> >>>>>>> Curiously I couldn't reproduce the crash that this bug is about. >>>>>>> But the hs_err file speaks for itself. >>>>>>> >>>>>>> Cheers, >>>>>>> David >>>>>>> >>>>>>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>>>>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>>>>>> Hi David, hi Daniel >>>>>>>>> >>>>>>>>> Thank you for your reviews! >>>>>>>>> >>>>>>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>>>>>> ???? Not your bug, but the comment has two typos: >>>>>>>>>> >>>>>>>>>> ???????? s/by safer/but safer/ >>>>>>>>>> ???????? s/unexpensive/inexpensive/ >>>>>>>>> >>>>>>>>> Thanks, fixed it. >>>>>>>>> >>>>>>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>>>>>> >>>>>>>>>> Given all of the code is inside: >>>>>>>>>> >>>>>>>>>> ??504?? if (Verbose) { >>>>>>>>>> >>>>>>>>>> I would just move the ResourceMark to the top of that block. >>>>>>>>>> We are going to use it unconditionally, and we are not >>>>>>>>>> releasing it until the end of the method regardless. That >>>>>>>>>> avoids the potential concern Dan raised about extending the >>>>>>>>>> scope of the PRESERVE_EXCEPTION_MARK. >>>>>>>>> >>>>>>>>> This sounds reasonable. I moved it to the top of that block. >>>>>>>>> >>>>>>>>>>> There are other places like Klass::print_on(outputStream*) >>>>>>>>>>> where this is still the case. Should we file an RFE to clean >>>>>>>>>>> these up? >>>>>>>>>> >>>>>>>>>> Obviously we don't have any tests running in a mode that >>>>>>>>>> causes this code to be executed, so yes it is probably a good >>>>>>>>>> idea to take a closer look at other call-sites and file a RFE >>>>>>>>>> if needed. >>>>>>>>> >>>>>>>>> Yes, I filed one to take a closer look at these [1]. >>>>>>>>> >>>>>>>>> I included the changes in a new webrev: >>>>>>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>>>>>> >>>>>>>> Thumbs up. >>>>>>>> >>>>>>>> Dan >>>>>>>> >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> Christian >>>>>>>>> >>>>>>>>> >>>>>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>>>>>> >>>>>>>> >>>>>> >>> > From aph at redhat.com Fri May 15 13:29:26 2020 From: aph at redhat.com (Andrew Haley) Date: Fri, 15 May 2020 14:29:26 +0100 Subject: [8u] RFR(S): 8244548: JDK 8u: sun.misc.Version.jdkUpdateVersion() returns wrong result In-Reply-To: <74e4d10641c8b6acca8ada68459a767f98e23585.camel@redhat.com> References: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> <808419e3-316b-381d-f636-606aadff39b6@redhat.com> <15a2fa41f80935d8c7d3893aad7d080c1f67aa2a.camel@redhat.com> <74e4d10641c8b6acca8ada68459a767f98e23585.camel@redhat.com> Message-ID: On 5/15/20 9:45 AM, Severin Gehwolf wrote: > So, Andrew Haley, what's the verdict on this one? There is a risk, yes. > Not fixing the bug seems riskier, though. Thoughts? Thanks for the detailed explanation. I think we have to keep sun.misc.Version working correctly but sacrifice binary consistency in JVM_GetVersionInfo. As long as stuff such as AOT compilation and class data sharing keep working I think we're OK. It's not great, but it's an almost-inevitable consequence of using a one-byte field for the minor version and bumping releases by 10. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From christian.hagedorn at oracle.com Fri May 15 14:06:36 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Fri, 15 May 2020 16:06:36 +0200 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> <412316cc-fb7e-f360-5a76-78e98ddf2890@oracle.com> Message-ID: <459a6a6f-5c40-c6d2-af3b-9c76b8a4f57e@oracle.com> While working on the replacement of Verbose by Unified Logging, I've seen that we should also fix this bug for ARM and S390 (PPC and SPARC already have the ResourceMark set correctly and Aarch64 is not implementing trace_method_handle_stub()). I included the necessary changes in a new webrev: http://cr.openjdk.java.net/~chagedorn/8244946/webrev.02/ Would be good if anybody could also have a quick look at these minor changes for ARM and S390 and verify them (I have only built and tested it on x86): Thank you! Best regards, Christian On 15.05.20 15:08, Christian Hagedorn wrote: > On 15.05.20 14:56, coleen.phillimore at oracle.com wrote: >> On 5/15/20 8:06 AM, Christian Hagedorn wrote: >>> Hi Coleen >>> >>> On 15.05.20 12:38, coleen.phillimore at oracle.com wrote: >>>> On 5/15/20 4:03 AM, Christian Hagedorn wrote: >>>>> On 15.05.20 00:38, David Holmes wrote: >>>>>> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >>>>>>> >>>>>>> I thought that this code under Verbose should have been changed >>>>>>> to use logging with log_level = verbose.? We were trying to >>>>>>> minimize/remove the Verbose flag. >>>>>>> >>>>>>> On 5/14/20 9:50 AM, David Holmes wrote: >>>>>>>> +1 >>>>>>>> >>>>>>>> But I'd be inclined to generalize the test to run: >>>>>>>> >>>>>>>> -Xlog:all=trace -XX:+Verbose >>>>>>> >>>>>>> Don't do this, it would be tons of output and make the test >>>>>>> unusable if it fails. >>>>>> >>>>>> The only way the test can fail is if it crashes. >>>>>> >>>>>> But we would want a level of indirection so that the test log >>>>>> doesn't contain the full output ~11000 lines of text. >>>>> >>>>> Could we still generalize the test but redirect the output to a >>>>> file with: >>>>> >>>>> -Xlog:all=trace:file=output.txt -XX:+Verbose ? >>>>> >>>>> When I just immediately return from trace_method_handle_stub() >>>>> without printing anything, the above command will not print >>>>> anything to the console. So, it would print the same amount of >>>>> lines to the console as the current option in webrev.01 with >>>>> -XX:+Verbose -Xlog:methodhandles. >>>> >>>> I'm confused.? Does the test crash with -Xlog:methodhandles >>>> -XX:+Verbose?? If so, that's all that should be needed.? If it needs >>>> -Xlog:all=trace, then don't add this test.? Even if it writes to an >>>> output file, it would be huge and affect test execution, >>>> particularly if more logging is added. >>> >>> I could already reproduce it locally on my Linux machine with >>> -Xlog:methodhandles -XX:+Verbose. From what I understood, I thought >>> that David is just talking about having a more general log test that >>> covers the entire logging together with -XX:+Verbose for a HelloWorld >>> program. But as you said, that would have some bad effects on test >>> execution. However, such a test is not required for this bug to be >>> triggered. >> >> Good! This looks good to me then. > > Thank you Coleen for your review! > >> Thank you for filing an RFE to do more logging with Unified Logging. > > No problem, I will take a look at it next. > > Best regards, > Christian > >>> >>>>> In a next step we could file an RFE to replace Verbose with unified >>>>> logging in trace_method_handle_stub() which currently is only using >>>>> tty. Even when using -Xlog:methodhandles alone without >>>>> -XX:+Verbose, there is still some printing done in >>>>> trace_method_handle_stub() to tty by >>>>> >>>>> ?500?? tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, >>>>> ?501???????????????? adaptername, mh_reg_name, >>>>> ?502???????????????? p2i(mh), p2i(entry_sp)); >>>>> >>>>> which is probably also not desired, as it is guarded by >>>>> log_is_enabled(Info, methodhandles) earlier [1]. This could also be >>>>> fixed in this follow-up RFE. >>>> >>>> Please, yes, this can be fixed with a follow-up RFE. >>> >>> Great, I filed an RFE for it [1]. >>> >>> Best regards, >>> Christian >>> >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8245107 >>> >>> >>>>>>>> Curiously I couldn't reproduce the crash that this bug is about. >>>>>>>> But the hs_err file speaks for itself. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> David >>>>>>>> >>>>>>>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>>>>>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>>>>>>> Hi David, hi Daniel >>>>>>>>>> >>>>>>>>>> Thank you for your reviews! >>>>>>>>>> >>>>>>>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>>>>>>> ???? Not your bug, but the comment has two typos: >>>>>>>>>>> >>>>>>>>>>> ???????? s/by safer/but safer/ >>>>>>>>>>> ???????? s/unexpensive/inexpensive/ >>>>>>>>>> >>>>>>>>>> Thanks, fixed it. >>>>>>>>>> >>>>>>>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>>>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>>>>>>> >>>>>>>>>>> Given all of the code is inside: >>>>>>>>>>> >>>>>>>>>>> ??504?? if (Verbose) { >>>>>>>>>>> >>>>>>>>>>> I would just move the ResourceMark to the top of that block. >>>>>>>>>>> We are going to use it unconditionally, and we are not >>>>>>>>>>> releasing it until the end of the method regardless. That >>>>>>>>>>> avoids the potential concern Dan raised about extending the >>>>>>>>>>> scope of the PRESERVE_EXCEPTION_MARK. >>>>>>>>>> >>>>>>>>>> This sounds reasonable. I moved it to the top of that block. >>>>>>>>>> >>>>>>>>>>>> There are other places like Klass::print_on(outputStream*) >>>>>>>>>>>> where this is still the case. Should we file an RFE to clean >>>>>>>>>>>> these up? >>>>>>>>>>> >>>>>>>>>>> Obviously we don't have any tests running in a mode that >>>>>>>>>>> causes this code to be executed, so yes it is probably a good >>>>>>>>>>> idea to take a closer look at other call-sites and file a RFE >>>>>>>>>>> if needed. >>>>>>>>>> >>>>>>>>>> Yes, I filed one to take a closer look at these [1]. >>>>>>>>>> >>>>>>>>>> I included the changes in a new webrev: >>>>>>>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>>>>>>> >>>>>>>>> Thumbs up. >>>>>>>>> >>>>>>>>> Dan >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> Christian >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>>>>>>> >>>>>>>>> >>>>>>> >>>> >> From sgehwolf at redhat.com Fri May 15 14:24:18 2020 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Fri, 15 May 2020 16:24:18 +0200 Subject: [8u] RFR(S): 8244548: JDK 8u: sun.misc.Version.jdkUpdateVersion() returns wrong result In-Reply-To: References: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> <808419e3-316b-381d-f636-606aadff39b6@redhat.com> <15a2fa41f80935d8c7d3893aad7d080c1f67aa2a.camel@redhat.com> <74e4d10641c8b6acca8ada68459a767f98e23585.camel@redhat.com> Message-ID: <6e023f46cec38601e6cd15356f971ab5d6cd99e2.camel@redhat.com> On Fri, 2020-05-15 at 14:29 +0100, Andrew Haley wrote: > On 5/15/20 9:45 AM, Severin Gehwolf wrote: > > So, Andrew Haley, what's the verdict on this one? There is a risk, yes. > > Not fixing the bug seems riskier, though. Thoughts? > > Thanks for the detailed explanation. > > I think we have to keep sun.misc.Version working correctly but > sacrifice binary consistency in JVM_GetVersionInfo. As long as stuff > such as AOT compilation and class data sharing keep working I think > we're OK. It's not great, but it's an almost-inevitable consequence of > using a one-byte field for the minor version and bumping releases by > 10. Agreed. Thanks for the review. I've requested 8u approval for this bug. Cheers, Severin From matthias.baesken at sap.com Fri May 15 14:59:29 2020 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Fri, 15 May 2020 14:59:29 +0000 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: References: <81b1d335-d898-5bc9-8aed-b105f8d73a2e@oracle.com> Message-ID: Hello David , I get the warning always on this linux machine (I think it depends on machine setup and/or kernel version). > Surely it must be specified to return 0 or some such sentinel in such a case? The doc just says this : 62 /** 63 * Returns the total amount of swap space in bytes. 64 * 65 * @return the total amount of swap space in bytes. 66 */ 67 public long getTotalSwapSpaceSize(); ( however it returned previously 0 before the latest changes in the container support code came in ). Regarding the getFreeSwapSpaceSize issue , Jie provided a change to src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java Because of this I could remove the try-construct . New webrev : http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.3/ Best regards, Matthias -----Original Message----- From: David Holmes Sent: Donnerstag, 14. Mai 2020 07:02 To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' Cc: jiefu(??) Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java On 14/05/2020 12:28 am, Baesken, Matthias wrote: > Hello, here is a new webrev removing the comment : > > http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.1/ > > It removes the comment , and it also handles 0 for getFreeSwapSpaceSize > ( this can occur after JDK-8242480 : Negative value may be returned by getFreeSwapSpaceSize() in the docker was pushed recently ). Sorry Matthias but I still see this partly as a bug in the underlying container support code. If you get the warning then what does getTotalSwapSpaceSize() return? Surely it must be specified to return 0 or some such sentinel in such a case? Other comments: ! try { ! out.shouldMatch("OperatingSystemMXBean\\.getFreeSwapSpaceSize: [1-9][0-9]+"); ! } catch(RuntimeException ex) { // allow 0, see 8242480 ! out.shouldContain("OperatingSystemMXBean.getFreeSwapSpaceSize: 0"); ! } can't you just change the original regex to match >= 0? Style nit: avoid historical comments like "see 8242480" and "since 8231111 ...". Thanks, David > Best regards, Matthias > > -----Original Message----- > From: Baesken, Matthias > Sent: Mittwoch, 13. Mai 2020 10:11 > To: 'David Holmes' ; 'hotspot-dev at openjdk.java.net' > Subject: RE: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > >> But according to the comment immediately before your changes: >> >> // in case of warnings like : "Your kernel does not support swap limit >> capabilities or the cgroup is not mounted. Memory limited without swap." >> // the getTotalSwapSpaceSize does not return the expected result, but 0 >> >> we should be returning zero. So this seems to be a bug in the >> implementation not in the test. > > > Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values > that are normally expected. > Previously we indeed got "0" . > Probably the comment should be deleted . > > Best regards, Matthias > From kim.barrett at oracle.com Fri May 15 16:07:27 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Fri, 15 May 2020 12:07:27 -0400 Subject: RFR: 8245098: Make SafeFetch32/N available earlier In-Reply-To: References: Message-ID: <1F383A58-BDF8-4197-BD58-1E7530E4411D@oracle.com> > On May 15, 2020, at 8:03 AM, Per Liden wrote: > > We have a need for SafeFetch32 during initializing of ZGC (see [1]). However, the SafeFetch32/N stubs are currently generated too late, in the stubRoutines_init2() phase, which is after the GC has been initialized. I propose we move the initialization of these stubs to the stubRoutines_init1() phase, which is before the GC has been initialized, to make them available during GC initialization. > > Please note that this patch touches platforms I can't easily test (x86-32, arm, ppc, s390). If you have access to one of those machines, I would appreciate if you could do some light verification. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8245098 > Webrev: http://cr.openjdk.java.net/~pliden/8245098/webrev.0 > Testing: Tier1-3 on all Oracle supported platforms. > > /Per > > [1] https://bugs.openjdk.java.net/browse/JDK-8245106 Looks good. From stefan.karlsson at oracle.com Fri May 15 17:46:43 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Fri, 15 May 2020 19:46:43 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving Message-ID: Hi all, Please review this patch to turn off -XX:+UseNUMAInterleaving as long as Windows GDI functions don't work with memory that crosses memory reservation boundaries. https://cr.openjdk.java.net/~stefank/8245002/webrev.00.cleanup https://cr.openjdk.java.net/~stefank/8245002/webrev.01.workaround https://cr.openjdk.java.net/~stefank/8245002/webrev.all https://bugs.openjdk.java.net/browse/JDK-8245002 During the investigation of JDK-8240654 it was found that some of the Windows GDI functions don't work with our NUMA interleaving implementation (-XX:+UseNUMAInterleaving). This causes severe repaint issues. The underlying issue is that these functions don't accept memory spanning over multiple memory reservations. The problem can easily be seen with the Java2D demos, where the drawn UI is grayed out instead of containing all relevant components. This can be reproduced by running building J2DBench and running: java -Xmx1g -Xms1g -XX:+UseNUMAInterleaving -Dsun.java2d.opengl=False -Dsun.java2d.d3d=False -jar dist/J2DBench.jar The -Dsun.java2d.d3d property is used to ensure that that java2d doesn't enable the alternative, D3D implementation instead of GDI. -Dsun.java2d.opengl is turned off for the same reason. The cleanup patch makes it slightly easier to insert the necessary check. It's not crucial for this fix, but it mimics the change I'm doing in large_page_init for JDK-8245002. The workaround patch reserves two adjacent memory areas, commit the memory, and then uses the address of last byte of the first memory region as input to one of the GDI functions. If the function returns a failure, we know that it doesn't work with memory spanning multiple reservations, and we turn off -XX:+UseNUMAInterleaving. I've chosen CreateBitmap as the GDI function to test, mainly because it has a very simple interface, without any extra dependencies. The intention is that as long as these function can't use multi-reservation memory, the flag will be turned off. If Microsoft changes the GDI code, so that it works with this kind of memory, then the proposed patch will automatically allow the flag to be enabled. I've run this through tier1-3 and also tested this manually. I temporarily turned of the checks to verify that it turns on the flag. Thanks, StefanK From stefan.karlsson at oracle.com Fri May 15 17:47:46 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Fri, 15 May 2020 19:47:46 +0200 Subject: RFR: 8245000: Windows GDI functions don't support large pages Message-ID: Hi all, Please review this patch to turn off -XX:+UseLargePages as long as Windows GDI functions don't work with large pages. https://cr.openjdk.java.net/~stefank/8245000/webrev.00.cleanup https://cr.openjdk.java.net/~stefank/8245000/webrev.01.workaround https://cr.openjdk.java.net/~stefank/8245000/webrev.all/ The patches build upon the changes in: 8245002: Windows GDI functions don't support NUMA interleaving https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-May/041798.html During the investigation of JDK-8240654 it was found that some of the Windows GDI functions don't work with large pages (-XX:+UseLargePages). This causes severe repaint issues. The problem can easily be seen with the Java2D demos, where the drawn UI is grayed out instead of containing all relevant components. This can be reproduced by running building J2DBench and running: java -Xmx1g -Xms1g -XX:+UseLargePages -Dsun.java2d.opengl=False -Dsun.java2d.d3d=False -jar dist/J2DBench.jar The -Dsun.java2d.d3d property is used to ensure that that java2d doesn't enable the alternative, D3D implementation instead of GDI. -Dsun.java2d.opengl is turned off for the same reason. The cleanup patch simplifies request_lock_memory_privilege and large_page_init, to make it easier to insert the necessary checks. The code first checks if memory allocated with large pages can be used by GDI functions. If this check fails it forcefully turns off -XX:+UseLargePages. Just like with JDK-8245002, the intention is that as long as the GDI functions can't use large pages, the -XX:+UseLargePages flag will be turned off. If Microsoft changes the GDI code, so that it works with large pages, then the proposed patch will automatically allow the flag to be enabled. There's a second case that we need to consider for large pages. If the GDI functions start to allow large pages, we have a flag (-XX:+UseLargePagesIndividualAllocation) that reserves and commits large pages in page-sized granules. This could cause the same multi-reservation problem described in JDK-8245002. I've also added a check for that, and turn off -XX:+UseLargePagesIndividualAllocation if it fails. I've run this through tier1-3 and also tested this manually. I've manually turned off the different checks to verify that it turns on the two flags. Thanks, StefanK From vladimir.kozlov at oracle.com Fri May 15 17:48:48 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Fri, 15 May 2020 10:48:48 -0700 Subject: RFR: 8244416: Remove incorrect assert during inline cache cleaning In-Reply-To: <9774524d-ffe8-85eb-da98-4b6f128f472e@oracle.com> References: <5a4574d6-0d7d-9d52-19b0-cbacebb1fba5@oracle.com> <9ed9350f-a55b-33a2-6c36-be6fa9b4483a@oracle.com> <9774524d-ffe8-85eb-da98-4b6f128f472e@oracle.com> Message-ID: <2bd67cdf-e878-d2e4-3f91-35cf022d3def@oracle.com> On 5/15/20 1:10 AM, Erik ?sterlund wrote: > Hi Vladimir, > > On 2020-05-15 01:20, Vladimir Kozlov wrote: >> Do we need to clean up IC in zombie nmethod 'from' here? Sweeper will do that later anyway as you said. > > You are 100% right. We do not need to clean up IC in zombie nmethods. In fact we try > to actively avoid it. The concurrent GC thread only initiates IC cleaning on is_alive() > nmethods. The issue is that from checking if it is_alive() and then starting the IC cleaning, > the state might flip over to zombie in a race with make_zombie from the sweeper. That is > fine as I described, because the make_zombie() function will in this race wait for the GC to > finish and then clear the ICs, releasing spurious IC stubs created by the GC mistakenly thinking > it is_alive(), before IC finalization can run in a subsequent safepoint. The only issue is that we > can't assert that from must not be zombie. It almost never is, but it is sometimes, and > that is fine. So you are saying that changing assert to from->is_zombie() check (to skip cleaning) will not help but only reduce concurrency window? Vladimir > > Thanks, > /Erik > >> Thanks, >> Vladimir >> >> On 5/14/20 6:07 AM, Erik ?sterlund wrote: >>> Hi, >>> >>> Months of stress testing triggered the following assert in inline cache cleaning: >>> >>> assert(!from->is_zombie(), "should not clean inline caches on zombies"); >>> >>> It used to be assert(!from->is_alive()), but after finding a race that resulted in >>> false positives for unloaded nmethods, the assert was relaxed to check only that >>> we shouldn't have zombies here, with a fat comment explaining why. >>> >>> Unfortunately, the relaxed assert also gets false positives, so I have chosen to >>> remove it, and further increase the size of that fat comment, explaning why we can't >>> assert that it isn't a zombie either. >>> >>> What it boils down to is a race where inline cache cleaning is performed from >>> a concurrent GC thread, concurrent to the sweeper making an nmethod zombie. >>> >>> The order of the race ingredients making the nmethod zombie from the sweeper is >>> the following: >>> >>> 1. Transition state to zombie >>> 2. Unregister nmethod form GC (importantly triggers a wait for concurrent code cache unloading to finish) >>> 3. clear_ic_callsites() importantly releasing all IC stubs > >>> The race happens when the GC finds an is_alive && !is_unloading nmethod and >>> triggers inline cache cleaning on it before 1, but then when racingly checking >>> again in the assert, we are past 1. >>> >>> Since the assert will spuriously fail even though there is no actual problem, and >>> we are at a point where there isn't much else it could assert, I chose to remove it. >>> >>> Other than triggering the assert, it is worth noting that this racingly zombiefied >>> nmethod will have its inline caches cleaned *with* IC stubs from the concurrent >>> GC thread. Subsequent IC finalization in safepoint cleanup requires live IC stubs >>> to belong to is_alive() nmethods, so the IC stubs must be released before the next >>> safepoint for this race to not be dangerous. That is ensured, because the racing >>> sweeper will get stuck waiting for concurrent code cache unloading to finish, >>> and when it wakes up, it will clear_ic_callsites(), releasing the spurious ICStubs, >>> before finishing the make_zombie() function. Then it will eventually hit a safepoint >>> poll, when the nmethod has been purged from IC stubs. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8244416 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8244416/webrev.00/ >>> >>> Thanks, >>> /Erik > From erik.osterlund at oracle.com Fri May 15 17:58:46 2020 From: erik.osterlund at oracle.com (=?utf-8?Q?Erik_=C3=96sterlund?=) Date: Fri, 15 May 2020 19:58:46 +0200 Subject: RFR: 8244416: Remove incorrect assert during inline cache cleaning In-Reply-To: <2bd67cdf-e878-d2e4-3f91-35cf022d3def@oracle.com> References: <2bd67cdf-e878-d2e4-3f91-35cf022d3def@oracle.com> Message-ID: Hi Vladimir, > On 15 May 2020, at 19:48, Vladimir Kozlov wrote: > > ?On 5/15/20 1:10 AM, Erik ?sterlund wrote: >> Hi Vladimir, >>> On 2020-05-15 01:20, Vladimir Kozlov wrote: >>> Do we need to clean up IC in zombie nmethod 'from' here? Sweeper will do that later anyway as you said. >> You are 100% right. We do not need to clean up IC in zombie nmethods. In fact we try >> to actively avoid it. The concurrent GC thread only initiates IC cleaning on is_alive() >> nmethods. The issue is that from checking if it is_alive() and then starting the IC cleaning, >> the state might flip over to zombie in a race with make_zombie from the sweeper. That is >> fine as I described, because the make_zombie() function will in this race wait for the GC to >> finish and then clear the ICs, releasing spurious IC stubs created by the GC mistakenly thinking >> it is_alive(), before IC finalization can run in a subsequent safepoint. The only issue is that we >> can't assert that from must not be zombie. It almost never is, but it is sometimes, and >> that is fine. > > So you are saying that changing assert to from->is_zombie() check (to skip cleaning) will not help but only reduce concurrency window? Exactly. /Erik > Vladimir > >> Thanks, >> /Erik >>> Thanks, >>> Vladimir >>> >>> On 5/14/20 6:07 AM, Erik ?sterlund wrote: >>>> Hi, >>>> >>>> Months of stress testing triggered the following assert in inline cache cleaning: >>>> >>>> assert(!from->is_zombie(), "should not clean inline caches on zombies"); >>>> >>>> It used to be assert(!from->is_alive()), but after finding a race that resulted in >>>> false positives for unloaded nmethods, the assert was relaxed to check only that >>>> we shouldn't have zombies here, with a fat comment explaining why. >>>> >>>> Unfortunately, the relaxed assert also gets false positives, so I have chosen to >>>> remove it, and further increase the size of that fat comment, explaning why we can't >>>> assert that it isn't a zombie either. >>>> >>>> What it boils down to is a race where inline cache cleaning is performed from >>>> a concurrent GC thread, concurrent to the sweeper making an nmethod zombie. >>>> >>>> The order of the race ingredients making the nmethod zombie from the sweeper is >>>> the following: >>>> >>>> 1. Transition state to zombie >>>> 2. Unregister nmethod form GC (importantly triggers a wait for concurrent code cache unloading to finish) >>>> 3. clear_ic_callsites() importantly releasing all IC stubs > >>>> The race happens when the GC finds an is_alive && !is_unloading nmethod and >>>> triggers inline cache cleaning on it before 1, but then when racingly checking >>>> again in the assert, we are past 1. >>>> >>>> Since the assert will spuriously fail even though there is no actual problem, and >>>> we are at a point where there isn't much else it could assert, I chose to remove it. >>>> >>>> Other than triggering the assert, it is worth noting that this racingly zombiefied >>>> nmethod will have its inline caches cleaned *with* IC stubs from the concurrent >>>> GC thread. Subsequent IC finalization in safepoint cleanup requires live IC stubs >>>> to belong to is_alive() nmethods, so the IC stubs must be released before the next >>>> safepoint for this race to not be dangerous. That is ensured, because the racing >>>> sweeper will get stuck waiting for concurrent code cache unloading to finish, >>>> and when it wakes up, it will clear_ic_callsites(), releasing the spurious ICStubs, >>>> before finishing the make_zombie() function. Then it will eventually hit a safepoint >>>> poll, when the nmethod has been purged from IC stubs. >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8244416 >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8244416/webrev.00/ >>>> >>>> Thanks, >>>> /Erik From vladimir.kozlov at oracle.com Fri May 15 18:00:43 2020 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Fri, 15 May 2020 11:00:43 -0700 Subject: RFR: 8244416: Remove incorrect assert during inline cache cleaning In-Reply-To: References: <2bd67cdf-e878-d2e4-3f91-35cf022d3def@oracle.com> Message-ID: <22770fc8-cd5b-a55a-7bfe-804f255d6f97@oracle.com> On 5/15/20 10:58 AM, Erik ?sterlund wrote: > Hi Vladimir, > >> On 15 May 2020, at 19:48, Vladimir Kozlov wrote: >> >> ?On 5/15/20 1:10 AM, Erik ?sterlund wrote: >>> Hi Vladimir, >>>> On 2020-05-15 01:20, Vladimir Kozlov wrote: >>>> Do we need to clean up IC in zombie nmethod 'from' here? Sweeper will do that later anyway as you said. >>> You are 100% right. We do not need to clean up IC in zombie nmethods. In fact we try >>> to actively avoid it. The concurrent GC thread only initiates IC cleaning on is_alive() >>> nmethods. The issue is that from checking if it is_alive() and then starting the IC cleaning, >>> the state might flip over to zombie in a race with make_zombie from the sweeper. That is >>> fine as I described, because the make_zombie() function will in this race wait for the GC to >>> finish and then clear the ICs, releasing spurious IC stubs created by the GC mistakenly thinking >>> it is_alive(), before IC finalization can run in a subsequent safepoint. The only issue is that we >>> can't assert that from must not be zombie. It almost never is, but it is sometimes, and >>> that is fine. >> >> So you are saying that changing assert to from->is_zombie() check (to skip cleaning) will not help but only reduce concurrency window? > > Exactly. Okay. Reviewed. Thanks, Vladimir > > /Erik > >> Vladimir >> >>> Thanks, >>> /Erik >>>> Thanks, >>>> Vladimir >>>> >>>> On 5/14/20 6:07 AM, Erik ?sterlund wrote: >>>>> Hi, >>>>> >>>>> Months of stress testing triggered the following assert in inline cache cleaning: >>>>> >>>>> assert(!from->is_zombie(), "should not clean inline caches on zombies"); >>>>> >>>>> It used to be assert(!from->is_alive()), but after finding a race that resulted in >>>>> false positives for unloaded nmethods, the assert was relaxed to check only that >>>>> we shouldn't have zombies here, with a fat comment explaining why. >>>>> >>>>> Unfortunately, the relaxed assert also gets false positives, so I have chosen to >>>>> remove it, and further increase the size of that fat comment, explaning why we can't >>>>> assert that it isn't a zombie either. >>>>> >>>>> What it boils down to is a race where inline cache cleaning is performed from >>>>> a concurrent GC thread, concurrent to the sweeper making an nmethod zombie. >>>>> >>>>> The order of the race ingredients making the nmethod zombie from the sweeper is >>>>> the following: >>>>> >>>>> 1. Transition state to zombie >>>>> 2. Unregister nmethod form GC (importantly triggers a wait for concurrent code cache unloading to finish) >>>>> 3. clear_ic_callsites() importantly releasing all IC stubs > >>>>> The race happens when the GC finds an is_alive && !is_unloading nmethod and >>>>> triggers inline cache cleaning on it before 1, but then when racingly checking >>>>> again in the assert, we are past 1. >>>>> >>>>> Since the assert will spuriously fail even though there is no actual problem, and >>>>> we are at a point where there isn't much else it could assert, I chose to remove it. >>>>> >>>>> Other than triggering the assert, it is worth noting that this racingly zombiefied >>>>> nmethod will have its inline caches cleaned *with* IC stubs from the concurrent >>>>> GC thread. Subsequent IC finalization in safepoint cleanup requires live IC stubs >>>>> to belong to is_alive() nmethods, so the IC stubs must be released before the next >>>>> safepoint for this race to not be dangerous. That is ensured, because the racing >>>>> sweeper will get stuck waiting for concurrent code cache unloading to finish, >>>>> and when it wakes up, it will clear_ic_callsites(), releasing the spurious ICStubs, >>>>> before finishing the make_zombie() function. Then it will eventually hit a safepoint >>>>> poll, when the nmethod has been purged from IC stubs. >>>>> >>>>> Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8244416 >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~eosterlund/8244416/webrev.00/ >>>>> >>>>> Thanks, >>>>> /Erik > From erik.osterlund at oracle.com Fri May 15 18:07:50 2020 From: erik.osterlund at oracle.com (=?utf-8?Q?Erik_=C3=96sterlund?=) Date: Fri, 15 May 2020 20:07:50 +0200 Subject: RFR: 8244416: Remove incorrect assert during inline cache cleaning In-Reply-To: <22770fc8-cd5b-a55a-7bfe-804f255d6f97@oracle.com> References: <22770fc8-cd5b-a55a-7bfe-804f255d6f97@oracle.com> Message-ID: <491024DA-C6A4-41E8-A7D9-A9B9AF16C479@oracle.com> Hi Vladimir, Thanks for the review. /Erik > On 15 May 2020, at 20:00, Vladimir Kozlov wrote: > > ?On 5/15/20 10:58 AM, Erik ?sterlund wrote: >> Hi Vladimir, >>>> On 15 May 2020, at 19:48, Vladimir Kozlov wrote: >>> >>> ?On 5/15/20 1:10 AM, Erik ?sterlund wrote: >>>> Hi Vladimir, >>>>> On 2020-05-15 01:20, Vladimir Kozlov wrote: >>>>> Do we need to clean up IC in zombie nmethod 'from' here? Sweeper will do that later anyway as you said. >>>> You are 100% right. We do not need to clean up IC in zombie nmethods. In fact we try >>>> to actively avoid it. The concurrent GC thread only initiates IC cleaning on is_alive() >>>> nmethods. The issue is that from checking if it is_alive() and then starting the IC cleaning, >>>> the state might flip over to zombie in a race with make_zombie from the sweeper. That is >>>> fine as I described, because the make_zombie() function will in this race wait for the GC to >>>> finish and then clear the ICs, releasing spurious IC stubs created by the GC mistakenly thinking >>>> it is_alive(), before IC finalization can run in a subsequent safepoint. The only issue is that we >>>> can't assert that from must not be zombie. It almost never is, but it is sometimes, and >>>> that is fine. >>> >>> So you are saying that changing assert to from->is_zombie() check (to skip cleaning) will not help but only reduce concurrency window? >> Exactly. > > Okay. Reviewed. > > Thanks, > Vladimir > >> /Erik >>> Vladimir >>> >>>> Thanks, >>>> /Erik >>>>> Thanks, >>>>> Vladimir >>>>> >>>>> On 5/14/20 6:07 AM, Erik ?sterlund wrote: >>>>>> Hi, >>>>>> >>>>>> Months of stress testing triggered the following assert in inline cache cleaning: >>>>>> >>>>>> assert(!from->is_zombie(), "should not clean inline caches on zombies"); >>>>>> >>>>>> It used to be assert(!from->is_alive()), but after finding a race that resulted in >>>>>> false positives for unloaded nmethods, the assert was relaxed to check only that >>>>>> we shouldn't have zombies here, with a fat comment explaining why. >>>>>> >>>>>> Unfortunately, the relaxed assert also gets false positives, so I have chosen to >>>>>> remove it, and further increase the size of that fat comment, explaning why we can't >>>>>> assert that it isn't a zombie either. >>>>>> >>>>>> What it boils down to is a race where inline cache cleaning is performed from >>>>>> a concurrent GC thread, concurrent to the sweeper making an nmethod zombie. >>>>>> >>>>>> The order of the race ingredients making the nmethod zombie from the sweeper is >>>>>> the following: >>>>>> >>>>>> 1. Transition state to zombie >>>>>> 2. Unregister nmethod form GC (importantly triggers a wait for concurrent code cache unloading to finish) >>>>>> 3. clear_ic_callsites() importantly releasing all IC stubs > >>>>>> The race happens when the GC finds an is_alive && !is_unloading nmethod and >>>>>> triggers inline cache cleaning on it before 1, but then when racingly checking >>>>>> again in the assert, we are past 1. >>>>>> >>>>>> Since the assert will spuriously fail even though there is no actual problem, and >>>>>> we are at a point where there isn't much else it could assert, I chose to remove it. >>>>>> >>>>>> Other than triggering the assert, it is worth noting that this racingly zombiefied >>>>>> nmethod will have its inline caches cleaned *with* IC stubs from the concurrent >>>>>> GC thread. Subsequent IC finalization in safepoint cleanup requires live IC stubs >>>>>> to belong to is_alive() nmethods, so the IC stubs must be released before the next >>>>>> safepoint for this race to not be dangerous. That is ensured, because the racing >>>>>> sweeper will get stuck waiting for concurrent code cache unloading to finish, >>>>>> and when it wakes up, it will clear_ic_callsites(), releasing the spurious ICStubs, >>>>>> before finishing the make_zombie() function. Then it will eventually hit a safepoint >>>>>> poll, when the nmethod has been purged from IC stubs. >>>>>> >>>>>> Bug: >>>>>> https://bugs.openjdk.java.net/browse/JDK-8244416 >>>>>> >>>>>> Webrev: >>>>>> http://cr.openjdk.java.net/~eosterlund/8244416/webrev.00/ >>>>>> >>>>>> Thanks, >>>>>> /Erik From kim.barrett at oracle.com Fri May 15 18:26:07 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Fri, 15 May 2020 14:26:07 -0400 Subject: RFR: 8245000: Windows GDI functions don't support large pages In-Reply-To: References: Message-ID: > On May 15, 2020, at 1:47 PM, Stefan Karlsson wrote: > > Hi all, > > Please review this patch to turn off -XX:+UseLargePages as long as Windows GDI functions don't work with large pages. > > https://cr.openjdk.java.net/~stefank/8245000/webrev.00.cleanup > https://cr.openjdk.java.net/~stefank/8245000/webrev.01.workaround > https://cr.openjdk.java.net/~stefank/8245000/webrev.all/ A couple of things that may or may not need to be addressed. Otherwise, looks good to me. ------------------------------------------------------------------------------ src/hotspot/os/windows/os_windows.cpp 3118 if (_large_page_size > default_page_size) { ... 3124 UseLargePages = _large_page_size != 0; Should UseLargePages be true if _large_page_size > 0 but < default_page_size? (Assuming that's actually possible, but I see nothing in this code that makes that impossible. The OS might...) The old code did the same thing, so if it's a bug, it's been there all along. ------------------------------------------------------------------------------ src/hotspot/os/windows/os_windows.cpp 2920 if (!gdi_can_use_split_reservation_memory(UseLargePages, min_interleave_granularity)) { 2921 WARN("Windows GDI cannot handle split reservations."); 2922 WARN("...Ignoring UseNUMAInterleaving flag."); If the failure is due to UseLargePages, it might be worth mentioning that in the warning. ------------------------------------------------------------------------------ From kim.barrett at oracle.com Fri May 15 18:39:13 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Fri, 15 May 2020 14:39:13 -0400 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: References: Message-ID: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> > On May 15, 2020, at 1:46 PM, Stefan Karlsson wrote: > > Hi all, > > Please review this patch to turn off -XX:+UseNUMAInterleaving as long as Windows GDI functions don't work with memory that crosses memory reservation boundaries. > > https://cr.openjdk.java.net/~stefank/8245002/webrev.00.cleanup > https://cr.openjdk.java.net/~stefank/8245002/webrev.01.workaround > https://cr.openjdk.java.net/~stefank/8245002/webrev.all > > https://bugs.openjdk.java.net/browse/JDK-8245002 ------------------------------------------------------------------------------ src/hotspot/os/windows/os_windows.cpp 2753 void* verify_mem = ::malloc(4 * 1024); There's no corresponding free. ------------------------------------------------------------------------------ src/hotspot/os/windows/os_windows.cpp 2776 assert(false, "Reservation failed"); 2791 assert(false, "Failed to commit memory"); 2802 assert(false, "Failed to commit memory"); These look like leftover debugging code? ------------------------------------------------------------------------------ From coleen.phillimore at oracle.com Fri May 15 22:14:21 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 15 May 2020 18:14:21 -0400 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> Message-ID: Hi I've reviewed this change and it looks good to me.? Some comments on the thread. On 5/13/20 1:49 AM, Thomas St?fe wrote: > (trimmed down the list of recipients) > > >> If you have a running foreground JVM in your Linux terminal, you can type >> "Ctrl-\" and see this. E.g., >> >> > Oh, backslash. I tried backspace. My error. > > >> $ java -cp . SleepForever >> <...> >> Heap >> garbage-first heap total 1048576K, used 824K [0x0000000412000000, >> 0x0000000800000000) >> region size 8192K, 1 young (8192K), 0 survivors (0K) >> Metaspace used 143K, capacity 4486K, committed 4864K, reserved >> 1056768K >> class space used 6K, capacity 386K, committed 512K, reserved 1048576K >> >> > Note that this output is not from NMT and so the numbers are not affected > by the NMT bug. Only the jcmd VM.native_memory output is affected. This info is printed by VM_PrintThreads operation (also kill -3). > >> Actually I see that the current output is not correct -- class space >> doesn't include the CDS sizes :-( >> >> > I think I'like to keep CDS and CCS separate in the output for a number of > reasons. One is that CCSSize only applies to CCS. But more important, CDS > can be shared and CCS not. > > But we may print out CDS too in this output, in a separate line. Yes, this would be good but I think the places that print the heap should also print CDS, and not add CDS printing to Metaspace::print_on() where the metaspace printing comes from. Coleen > > Cheers, Thomas From coleen.phillimore at oracle.com Fri May 15 22:26:25 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 15 May 2020 18:26:25 -0400 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: Message-ID: Hi, This looks good. The code motion makes sense so this is an improvement. On 5/7/20 10:21 AM, Thomas St?fe wrote: > Hi all, > > please take a look at the third iteration of this change: > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ > > Changes in this version: > > - at the request of Coleen (sorry I did not see your post earlier) I > removed all platform dependent?files and put the platform dependent > coding back to metaspace.cpp and compressedOops.cpp to ease the review > pain. I used plain platform defines though (#ifdef AARCH64) instead of > hiding them behind another macro to make things clearer. Note that I > still intent to put this code away into the platform corners but will > do so in a follow up RFE. Oh, please don't, especially since most platforms are just a ReservedSpace(size, align, false, NULL);? The AARCH64 code isn't a big intrusion in this code.? Thank you for putting it back. It is easier to follow how it is. > > - I reinstated, in a fashion, the special handling of reservations on > Windows. On all platforms we reserve address space to map the archive > files in with a subsequent mapping operation. However, on Windows, we > cannot use MapViewOfFile() into an existing mapping. So I remove the > mapping again before mapping the archives - see comment in code for > details. I missed this windows part of the change, where is it? http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/virtualSpaceList.cpp.udiff.html Why did you make this change?? Shouldn't the caller align it? Looks good to me. Coleen > > All CI tests at SAP run through without problems, including on Win64 > and aarch64, but I would be happy if others were to run test too. > > Thank you, Thomas > > On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe > wrote: > > Hi all, > > Could I have reviews for the following proposal of reworking > cds/class space reservation? > > Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 > > Webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ > > (Many thanks to Ioi Lam for so patiently explaining CDS internals > to me, and to Andrew Haley and Nick Gasson for help with aarch64!) > > Reservation of the compressed class space is needlessly > complicated and has some minor issues. It can be simplified and > made clearer. > > The complexity stems from the fact that this area lives at the > intersection of two to three sub systems, depending on how one > counts. Metaspace, CDS, and the platform which may or may not its > own view of how to reserve class space. And this code has been > growing organically over time. > > One small example: > > ReservedSpace Metaspace::reserve_preferred_space(size_t size, > size_t alignment, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?bool large_pages, > char *requested_addr, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?bool > use_requested_addr) > > which I spent hours decoding, resulting in a very confused mail to > hs-runtime and aarch64-port-dev [2]. > > This patch attempts to simplify cds and metaspace setup a bit; to > comment implicit knowledge which is not immediately clear; to > cleanly abstract platform concerns like optimized class space > placement; and to disentangle cds from metaspace to solve issues > which may bite us later with Elastic Metaspace [4]. > > --- > > The main change is the reworked reservation mechanism. This is > based on Ioi's proposal [5]. > > When reserving class space, three things must happen: > > 1) reservation of the space obviously. If cds is active that space > must be in the vicinity of cds archives to be covered by > compressed class pointer encoding. > 2) setting up the internal Metaspace structures atop of that space > 3) setting up compressed class pointer encoding. > > In its current form, Metaspace may or may not do some or all of > that in one function > (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace > metaspace_rs, char* requested_addr, address cds_base);) - if cds > is active, it will reserve the space for Metaspace and hand it in, > otherwise it will create it itself. > > When discussing this in [2], Ioi proposed to move the reservation > of the class space completely out of Metaspace and make it a > responsibility of the caller always. This would reduce some > complexity, and this patch follows the proposal. > > I removed > Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace > metaspace_rs, char* requested_addr, address cds_base); and all its > sub functions. > > (1) now has to always be done outside - a ReservedSpace for class > space has to be provided by the caller. However, Metaspace now > offers a utility function for reserving space at a "nice" > location, and explicitly doing nothing else: > > ReservedSpace > Metaspace::reserve_address_space_for_compressed_classes(size_t size); > > this function can be redefined on a platform level for platform > optimized reservation, see below for details. > > (2) is taken care of by a new function, > Metaspace::initialize_class_space(ReservedSpace rs) > > (3) is taken care of a new function > CompressedKlassPointers::initialize(), see below for details. > > > So, class space now is set up three explicit steps: > > - First, reserve a suitable space by however means you want. For > convenience you may use > Metaspace::reserve_address_space_for_compressed_classes(), or you > may roll your own reservation. > - Next, tell Metaspace to use that range as backing storage for > class space: Metaspace::initialize_class_space(ReservedSpace rs) > - Finally, set up encoding. Encoding is independent from the > concept of a ReservedSpace, it just gets an address range, see > below for details. > > Separating these steps and moving them out of the responsibility > of Metaspace makes this whole thing more flexible; it also removes > unnecessary knowledge (e.g. Metaspace does not need to know > anything about either ccp encoding or cds). > > --- > > How it comes together: > > If CDS is off, we just reserve a space using > Metaspace::reserve_address_space_for_compressed_classes(), > initialize it with Metaspace::initialize_class_space(ReservedSpace > rs), then set up compressed class pointer encoding covering the > range of this class space. > > If CDS is on (dump time), we reserve large 4G space, either at > SharedBaseAddress or using > Metaspace::reserve_address_space_for_compressed_classes(); we then > split that into 3G archive space and 1G class space; we set up > that space with Metaspace as class space; then we set > up?compressed class pointer encoding covering both archive space > and cds. > > If CDS is on (run time), we reserve a large space, split it into > archive space (large enough to hold both archives) and class > space, then basically proceed as above. > > Note that this is almost exactly how things worked before (modulo > some minor fixes, e.g. alignment issues), only the code is > reformed and made more explicit. > > --- > > I moved compressed class pointer setup over to > CompressedKlassPointers and changed the interface: > > -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace > metaspace_rs, address cds_base) > +void CompressedKlassPointers::initialize(address addr, size_t len); > > Instead of feeding it a single ReservedSpace, which is supposed to > represent class space, and an optional alternate base if cds is > on, now we give it just an numeric address range. That range marks > the limits to where Klass structures are to be expected, and is > the implicit promise that outside that range no Klass structures > will exist, so encoding has to cover only this range. > > This range may contain just the class space; or class space+cds; > or whatever allocation scheme we come up with in the future. > Encoding does not really care how the memory is organized as long > as the input range covers all possible Klass locations. That way > we remove knowledge about class space/cds from compressed class > pointer encoding. > > Moving it away from metaspace.cpp into the CompressedKlassPointers > class also mirrors CompressedOops::initialize(). > > --- > > I renamed _narrow_klass_range to just _range, because strictly > speaking this is the range un-narrow Klass pointers can have. > > As for the implementation of > CompressedKlassPointers::initialize(address addr, size_t len), I > mimicked very closely what happened before, so there should be > almost no differences. Since "almost no differences" sounds scary > :) here are the differences: > > - When CDS is active (dump or run time) we now always, > unconditionally, set the encoding range to 4G. This fixes a > theoretical bug discussed on aarch64-port-dev [1]. > > - When CDS is not active, we set the encoding range to the minimum > required length. Before, it was left at its default value of 4G. > > Both differences only affect aarch64, since they are currently the > only one using the range field in CompressedKlassPointers. > > I wanted to add an assert somewhere to test encoding of the very > last address of the CompressedKlassPointers range, again to > prevent errors like [3]. But I did not come up with a good place > for this assert which would cover also the encoding done by C1/C2. > > For the same reason I thought about introducing a mode where Klass > structures would be allocated in reverse order, starting at the > end of the ccs, but again left it out as too big a change. > > --- > > OS abstraction: platforms may have restrictions of what > constitutes a valid compressed class pointer encoding base. Or if > not, they may have at least preferences. There was logic like this > in metaspace.cpp, which I removed and cleanly factored out into > platform dependent files, giving each platform the option to add > special logic. > > These are two new methods: > > - bool CompressedKlassPointers::is_valid_base(address p) > > to let the platform tell you whether it considers p to be a valid > encoding base. The only platform having these restrictions > currently is aarch64. > > - ReservedSpace > Metaspace::reserve_address_space_for_compressed_classes(size_t size); > > this hands over the process of allocating a range suitable for > compressed class pointer encoding to the platform. Most platforms > will allocate just anywhere, but some platforms may have a better > strategy (e.g. trying low memory first, trying only correctly > aligned addresses and so on). > > Beforehand, this coding existed in a similar form in metaspace.cpp > for aarch64 and AIX. For now, I left the AIX part out - it seems > only half done, and I want to check further if we even need it, if > yes why not on Linux ppc, and C1 does not seem to support anything > other than base+offset with shift either, but I may be mistaken. > > These two methods should give the platform enough control to > implement their own scheme for optimized class space placement > without bothering any shared code about it. > > Note about the form, I introduced two new platform dependent > files, "metaspace_.cpp" and "compressedOops_.cpp". I am > not happy about this but this seems to be what we generally do in > hotspot, right? > > --- > > Metaspace reserve alignment vs cds alignment > > CDS was using Metaspace reserve alignment for CDS internal > purposes. I guess this was just a copy paste issue. It never > caused problems since Metaspace reserve alignment == page size, > but that is not true anymore in the upcoming Elastic Metaspace > where reserve alignment will be larger. This causes a number of > issues. > > I separated those two cleanly. CDS now uses > os::vm_allocation_granularity. Metaspace::reserve_alignment is > only used in those two places where it is needed, when CDS creates > the address space for class space on behalf of the Metaspace. > > --- > > Windows special handling in CDS > > To simplify coding I removed the windows specific handling which > left out reservation of the archive. This was needed because > windows cannot mmap files into reserved regions. But fallback code > exists in filemap.cpp for this case which just reads in the region > instead of mapping?it. > > Should that turn out to be a performance problem, I will reinstate > the feature. But a simpler way would be reserve the archive and > later just before mmapping?the archive file to release the archive > space. That would not only be simpler but give us the best > guarantee that that address space is actually available. But I'd > be happy to leave that part out completely if we do not see any > performance problems on windows x64. > > --- > > NMT cannot deal with spaces which are split. This problem > manifests in that bookkeeping for class space is done under > "Shared Classes", not "Classes" as it should. This problem exists > today too at dump?time and randomly at run time. But since I > simplified the reservation, this problem now shows up always, > whether or not we map at the SharedBaseAddress. > While I could work around this problem, I'd prefer this problem to > be solved at the core, and NMT to have an option to recognize > reservation splits. So I'd rather not put a workaround for this > into the patch but leave it for fixing as a separate issue. I > opened this issue to track it [6]. > > --- > > Jtreg tests: > > I expanded the CompressedOops/CompressedClassPointers.java. I also > extended them to Windows. The tests now optionally omit strict > class space placement tests, since these tests heavily depend on > ASLR and were the reason they were excluded on Windows. However I > think even without checking for class space placement they make > sense, just to see that the VM comes up and lives with the many > different settings we can run in. > > --- > > Tests: > > - I ran the patch through Oracles submit repo > - I ran tests manually for aarch64, zero, linux 32bit and windows x64 > - The whole battery of nightly tests at SAP, including ppc, ppcle > and aarch64, unfortunately excluding windows because of unrelated > errors. Windows x64 tests will be redone tonight. > > > Thank you, > > Thomas > > [1] > https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html > [2] > https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html > [3] https://bugs.openjdk.java.net/browse/JDK-8193266 > [4] https://bugs.openjdk.java.net/browse/JDK-8221173 > [5] > https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html > [6] https://bugs.openjdk.java.net/browse/JDK-8243535 > From coleen.phillimore at oracle.com Fri May 15 22:34:21 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 15 May 2020 18:34:21 -0400 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> Message-ID: <1108ac1a-f693-cfc7-9f86-1a84508b1487@oracle.com> One more in-context comment. On 5/8/20 3:33 AM, Thomas St?fe wrote: > Hi Ioi, > > thanks for taking the time to look at this! > > On Fri, May 8, 2020 at 7:37 AM Ioi Lam > wrote: > > Hi Thomas, > > I am running your patch in out CI pipeline now. Some comments: > > [1] Do we still need to use _class_space_list? It looks like we > just have a single range. Is > _class_space_list->current_virtual_space() always the same VS? > > ? if (_class_space_list != NULL) { > ??? address base = > (address)_class_space_list->current_virtual_space()->bottom(); > ??? address top = base + compressed_class_space_size(); > ??? st->print("Compressed class space mapped at: " PTR_FORMAT "-" > PTR_FORMAT ", size: " SIZE_FORMAT, > ?????????????? p2i(base), p2i(top), top - base); > > > Yes it is. Has been bugging me for years too. With ccs, the > VirtualSpaceList is degenerated and only contains one node. It still > makes some sense code wise. But maybe VirtualSpace*List* is a misnomer. > > It also ties in into the question whether Metaspace reservations > should be able to grow on demand. Oracle folks I talk to tend to say > yes. I personally think the "Metaspace is infinite don't worry" meme > was broken since ccs was a thing, since CompressedClassSpaceSize is an > absolute limit already. > > I may tweak and simplify the code a bit with Elastic Metaspace. Maybe, > as you say, with ccs we can get rid of VirtualSpaceList and deal with > the range directly. Have to check. Yes, this could be an improvement for a later time.? The compressed class space isn't a list.? I think that was the least amount of special code place to hook it in at the time.? It makes sense for it to be a single VirtualSpaceNode. CompressedClassSpace + Data (the rest of Metaspace) has a complicated relationship for determining high water mark for starting GC, and for reporting LowMemory thresholds.? I'd have to look up what was decided there. Data (the rest of Metaspace) does grow on demand unless someone uses -XX:MaxMetaspaceSize and then class metaspace size is factored in (maybe just committed space not the whole 1G). HTH. Coleen > > [2] Does JDK-8243535 exist with the current jdk/jdk repo? > > ? // This does currently not work because rs may be the result of > a split operation > ? // and NMT seems not to be able to handle splits. > ? // See JDK-8243535. > ? // MemTracker::record_virtual_memory_type((address)rs.base(), > mtClass); > > > Yes. It shows up more rarely. With this patch, it will show up always. > I had a conversation with Zhengyu about this. I think there may be > workarounds, like removing the original mapping in NMT and replacing > it with the two new ones for archive and ccs. But it seems iffy, I'd > rather have this fixed in NMT. > > I also think it is not such a big deal, but if you think otherwise, I > can integrate a workaround into this patch. > > > [3] I think this can be put in header file. > > bool Metaspace::class_space_is_initialized() { > ? return _class_space_list != NULL; > } > > > Will do. > > [4] Why does the CCS need to follow UseLargePagesInMetaspace? This > is the reason for the > ??? gap in the following chart: > > ??? // We do this by reserving space for the ccs behind the > archives. Note however that > ??? //? ccs follows a different alignment > (Metaspace::reserve_alignment), so there may > ??? //? be a gap between ccs and cds. > ??? // We use a similar layout at runtime, see > reserve_address_space_for_archives(). > ??? // > ??? //????????????????????????????? +-- SharedBaseAddress (default > = 0x800000000) > ??? //????????????????????????????? v > ??? // +-..---------+---------+ ... > +----+----+----+--------+--------------------+ > ??? // |??? Heap??? | Archive |???? | MC | RW | RO | [gap]? |??? > class space???? | > ??? // +-..---------+---------+ ... > +----+----+----+--------+--------------------+ > ??? // |<--?? MaxHeapSize? -->|???? |<-- UnscaledClassSpaceMax = > 4GB -->| > > > No, nothing to do with large pages. CCS does not have large pages. > > The gap can result from different alignment rules between metaspace > and cds. With Elastic Metaspace, metaspace reservations will be > aligned to 4M. So, ccs will have to start at a 4M aligned border. > > > _reserve_alignment is determined here: > > void Metaspace::ergo_initialize() { > ? if (DumpSharedSpaces) { > ??? // Using large pages when dumping the shared archive is > currently not implemented. > ??? FLAG_SET_ERGO(UseLargePagesInMetaspace, false); > ? } > > ? size_t page_size = os::vm_page_size(); > ? if (UseLargePages && UseLargePagesInMetaspace) { > ??? page_size = os::large_page_size(); > ? } > > ? _commit_alignment? = page_size; > ? _reserve_alignment = MAX2(page_size, > (size_t)os::vm_allocation_granularity()); > > But when CDS is enabled, the RS is reserved without large pages, > so it should be possible to incrementally commit the CCS using > just os::vm_allocation_granularity(). > > > Absolutely. CCS does not use large pages for that reason. As I wrote, > this is about the new reservation granularity of 4M. > > [5] I see the aarch64 code in > Metaspace::reserve_address_space_for_compressed_classes has been > changed. Is this necessary for this RFE, or is it a separate > improvement that can be done in a different RFE? > > > Well, it had to be changed somewhat, since the prototype differs. I > swapped Metaspace::reserve_preferred_space() for > Metaspace::reserve_address_space_for_compressed_classes(), which > follows a somewhat different protocol. > > That aside, the functionality for AARCH64 should be as unchanged?as > possible. I removed the AIX specific coding, since it was > subtly?wrong, but that's fine since we are AIX/ppc maintainers. > > > [6] For AARCH64, should we skip the part that reserves it right > above the heap? Or does AARCH64 always allocate the heap such that > it's a preferable address? > > > Yes and yes :) > > We do skip that for aarch64 (at least if the address is wrong): > > 1157 // Try right above the Java heap... 1158 address base = > align_up(CompressedOops::end(), Metaspace::reserve_alignment()); 1159 > assert(base != NULL, "Sanity"); 1160 1161 const size_t size = > align_up(CompressedClassSpaceSize, Metaspace::reserve_alignment()); > 1162 if (CompressedKlassPointers::is_valid_base(base)) { 1163 rs = > ReservedSpace(size, Metaspace::reserve_alignment(), false /* large */, > (char*)base); 1164 } > At 1162, we check if the attach address would be valid for this > platform. If aarch64 says no, we won't reserve there but go straight > to the "attach wherever you think is good" part. Which is implemented > by?Metaspace::reserve_address_space_for_compressed_classes(), which on > aarch64 does search a suitable address. > > ??? // case (b) > ??? ReservedSpace rs; > > ??? // Try right above the Java heap... > ??? address base = align_up(CompressedOops::end(), > Metaspace::reserve_alignment()); > ??? assert(base != NULL, "Sanity"); > > ??? const size_t size = align_up(CompressedClassSpaceSize, > Metaspace::reserve_alignment()); > ??? if (CompressedKlassPointers::is_valid_base(base)) { > ????? rs = ReservedSpace(size, Metaspace::reserve_alignment(), > false /* large */, (char*)base); > ??? } > > ??? // ...failing that, reserve anywhere, but let platform do > optimized placement: > ??? if (!rs.is_reserved()) { > ????? rs = > Metaspace::reserve_address_space_for_compressed_classes(size); > ??? } > > [7] argument.cpp checks this: > > ? if (UseSharedSpaces || DumpSharedSpaces) { > ??? if > (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) > { > ????? log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is > invalid for this platform, option will be ignored.", > ?????????????????????? p2i((address)SharedBaseAddress)); > ????? FLAG_SET_DEFAULT(SharedBaseAddress, > default_SharedBaseAddress()); > ??? } > ? } > > but initialize_dumptime_shared_and_meta_spaces aligns up the > SharedBaseAddress before doing the assert. > > ? void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { > ??? assert(DumpSharedSpaces, "should be called for dump time only"); > > ??? const size_t reserve_alignment = > MetaspaceShared::reserved_space_alignment(); > ??? char* shared_base = (char*)align_up((char*)SharedBaseAddress, > reserve_alignment); > > ? #ifdef _LP64 > assert(CompressedKlassPointers::is_valid_base((address)shared_base), > "Sanity"); > > So theoretically shared_base may no longer be is_valid_base after > the align_up. It probably won't happen, but I think the code will > be much clearer tofirst to align_up, then check for is_valid_base > and reset to default, and finally assert. > > > Okay, I'll change that. > > [8] In > MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the > following block can be simplified: > > ? if (!_shared_rs.is_reserved()) { > ??? // Get a reserved space anywhere if attaching at the > SharedBaseAddress fails: > ??? if (UseCompressedClassPointers) { > ????? // If we need a compressed class space too, let the platform > handle the reservation. > ????? _shared_rs = > Metaspace::reserve_address_space_for_compressed_classes(cds_total); > ??? } else { > ????? // anywhere is fine. > ????? _shared_rs = ReservedSpace(cds_total, reserve_alignment, > false /* large */, (char*)NULL); > ??? } > ? } > > ?? ... if you change the declaration to: > > ? static ReservedSpace > reserve_address_space_for_compressed_classes(size_t size) > NOT_LP64({ return ReservedSpace();}); > > [9] I think the #ifdef _LP64 is not necessary: > > #ifdef _LP64 > ??? // Some sanity checks after reserving address spaces for > archives and class space. > ??? assert(archive_space_rs.is_reserved(), "Sanity"); > ??? if (Metaspace::using_class_space()) { > ????? // Class space must closely follow the archive space. Both > spaces must be aligned correctly. > ????? assert(class_space_rs.is_reserved(), "A class space should > have been reserved"); > ????? assert(class_space_rs.base() >= archive_space_rs.end(), > "class space should follow the cds archive space"); > ????? assert(is_aligned(archive_space_rs.base(), > MetaspaceShared::reserved_space_alignment()), "Archive space > misaligned"); > ????? assert(is_aligned(class_space_rs.base(), > Metaspace::reserve_alignment()), "class space misaligned"); > ??? } > #endif > > > I'll take a look. > > The rest of the code looks OK to me, but I may take a look at it > again after getting more sleep :-) > > > Sure. Thanks man! :) > > ..Thomas > > Thanks > - Ioi > > > > > On 5/7/20 7:21 AM, Thomas St?fe wrote: >> Hi all, >> >> please take a look at the third iteration of this change: >> >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >> >> Changes in this version: >> >> - at the request of Coleen (sorry I did not see your post >> earlier) I removed all platform dependent?files and put the >> platform dependent coding back to metaspace.cpp and >> compressedOops.cpp to ease the review pain. I used plain platform >> defines though (#ifdef AARCH64) instead of hiding them behind >> another macro to make things clearer. Note that I still intent to >> put this code away into the platform corners but will do so in a >> follow up RFE. >> >> - I reinstated, in a fashion, the special handling of >> reservations on Windows. On all platforms we reserve address >> space to map the archive files in with a subsequent mapping >> operation. However, on Windows, we cannot use MapViewOfFile() >> into an existing mapping. So I remove the mapping again before >> mapping the archives - see comment in code for details. >> >> All CI tests at SAP run through without problems, including on >> Win64 and aarch64, but I would be happy if others were to run >> test too. >> >> Thank you, Thomas >> >> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >> > wrote: >> >> Hi all, >> >> Could I have reviews for the following proposal of reworking >> cds/class space reservation? >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >> >> Webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >> >> (Many thanks to Ioi Lam for so patiently explaining CDS >> internals to me, and to Andrew Haley and Nick Gasson for help >> with aarch64!) >> >> Reservation of the compressed class space is needlessly >> complicated and has some minor issues. It can be simplified >> and made clearer. >> >> The complexity stems from the fact that this area lives at >> the intersection of two to three sub systems, depending on >> how one counts. Metaspace, CDS, and the platform which may or >> may not its own view of how to reserve class space. And this >> code has been growing organically over time. >> >> One small example: >> >> ReservedSpace Metaspace::reserve_preferred_space(size_t size, >> size_t alignment, >> ?bool large_pages, char *requested_addr, >> ?bool use_requested_addr) >> >> which I spent hours decoding, resulting in a very confused >> mail to hs-runtime and aarch64-port-dev [2]. >> >> This patch attempts to simplify cds and metaspace setup a >> bit; to comment implicit knowledge which is not immediately >> clear; to cleanly abstract platform concerns like optimized >> class space placement; and to disentangle cds from metaspace >> to solve issues which may bite us later with Elastic >> Metaspace [4]. >> >> --- >> >> The main change is the reworked reservation mechanism. This >> is based on Ioi's proposal [5]. >> >> When reserving class space, three things must happen: >> >> 1) reservation of the space obviously. If cds is active that >> space must be in the vicinity of cds archives to be covered >> by compressed class pointer encoding. >> 2) setting up the internal Metaspace structures atop of that >> space >> 3) setting up compressed class pointer encoding. >> >> In its current form, Metaspace may or may not do some or all >> of that in one function >> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base);) - if >> cds is active, it will reserve the space for Metaspace and >> hand it in, otherwise it will create it itself. >> >> When discussing this in [2], Ioi proposed to move the >> reservation of the class space completely out of Metaspace >> and make it a responsibility of the caller always. This would >> reduce some complexity, and this patch follows the proposal. >> >> I removed >> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base); and >> all its sub functions. >> >> (1) now has to always be done outside - a ReservedSpace for >> class space has to be provided by the caller. However, >> Metaspace now offers a utility function for reserving space >> at a "nice" location, and explicitly doing nothing else: >> >> ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t >> size); >> >> this function can be redefined on a platform level for >> platform optimized reservation, see below for details. >> >> (2) is taken care of by a new function, >> Metaspace::initialize_class_space(ReservedSpace rs) >> >> (3) is taken care of a new function >> CompressedKlassPointers::initialize(), see below for details. >> >> >> So, class space now is set up three explicit steps: >> >> - First, reserve a suitable space by however means you want. >> For convenience you may use >> Metaspace::reserve_address_space_for_compressed_classes(), or >> you may roll your own reservation. >> - Next, tell Metaspace to use that range as backing storage >> for class space: >> Metaspace::initialize_class_space(ReservedSpace rs) >> - Finally, set up encoding. Encoding is independent from the >> concept of a ReservedSpace, it just gets an address range, >> see below for details. >> >> Separating these steps and moving them out of the >> responsibility of Metaspace makes this whole thing more >> flexible; it also removes unnecessary knowledge (e.g. >> Metaspace does not need to know anything about either ccp >> encoding or cds). >> >> --- >> >> How it comes together: >> >> If CDS is off, we just reserve a space using >> Metaspace::reserve_address_space_for_compressed_classes(), >> initialize it with >> Metaspace::initialize_class_space(ReservedSpace rs), then set >> up compressed class pointer encoding covering the range of >> this class space. >> >> If CDS is on (dump time), we reserve large 4G space, either >> at SharedBaseAddress or using >> Metaspace::reserve_address_space_for_compressed_classes(); we >> then split that into 3G archive space and 1G class space; we >> set up that space with Metaspace as class space; then we set >> up?compressed class pointer encoding covering both archive >> space and cds. >> >> If CDS is on (run time), we reserve a large space, split it >> into archive space (large enough to hold both archives) and >> class space, then basically proceed as above. >> >> Note that this is almost exactly how things worked before >> (modulo some minor fixes, e.g. alignment issues), only the >> code is reformed and made more explicit. >> >> --- >> >> I moved compressed class pointer setup over to >> CompressedKlassPointers and changed the interface: >> >> -void >> Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >> metaspace_rs, address cds_base) >> +void CompressedKlassPointers::initialize(address addr, >> size_t len); >> >> Instead of feeding it a single ReservedSpace, which is >> supposed to represent class space, and an optional alternate >> base if cds is on, now we give it just an numeric address >> range. That range marks the limits to where Klass structures >> are to be expected, and is the implicit promise that outside >> that range no Klass structures will exist, so encoding has to >> cover only this range. >> >> This range may contain just the class space; or class >> space+cds; or whatever allocation scheme we come up with in >> the future. Encoding does not really care how the memory is >> organized as long as the input range covers all possible >> Klass locations. That way we remove knowledge about class >> space/cds from compressed class pointer encoding. >> >> Moving it away from metaspace.cpp into the >> CompressedKlassPointers class also mirrors >> CompressedOops::initialize(). >> >> --- >> >> I renamed _narrow_klass_range to just _range, because >> strictly speaking this is the range un-narrow Klass pointers >> can have. >> >> As for the implementation of >> CompressedKlassPointers::initialize(address addr, size_t >> len), I mimicked very closely what happened before, so there >> should be almost no differences. Since "almost no >> differences" sounds scary :) here are the differences: >> >> - When CDS is active (dump or run time) we now always, >> unconditionally, set the encoding range to 4G. This fixes a >> theoretical bug discussed on aarch64-port-dev [1]. >> >> - When CDS is not active, we set the encoding range to the >> minimum required length. Before, it was left at its default >> value of 4G. >> >> Both differences only affect aarch64, since they are >> currently the only one using the range field in >> CompressedKlassPointers. >> >> I wanted to add an assert somewhere to test encoding of the >> very last address of the CompressedKlassPointers range, again >> to prevent errors like [3]. But I did not come up with a good >> place for this assert which would cover also the encoding >> done by C1/C2. >> >> For the same reason I thought about introducing a mode where >> Klass structures would be allocated in reverse order, >> starting at the end of the ccs, but again left it out as too >> big a change. >> >> --- >> >> OS abstraction: platforms may have restrictions of what >> constitutes a valid compressed class pointer encoding base. >> Or if not, they may have at least preferences. There was >> logic like this in metaspace.cpp, which I removed and cleanly >> factored out into platform dependent files, giving each >> platform the option to add special logic. >> >> These are two new methods: >> >> - bool CompressedKlassPointers::is_valid_base(address p) >> >> to let the platform tell you whether it considers p to be a >> valid encoding base. The only platform having these >> restrictions currently is aarch64. >> >> - ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t >> size); >> >> this hands over the process of allocating a range suitable >> for compressed class pointer encoding to the platform. Most >> platforms will allocate just anywhere, but some platforms may >> have a better strategy (e.g. trying low memory first, trying >> only correctly aligned addresses and so on). >> >> Beforehand, this coding existed in a similar form in >> metaspace.cpp for aarch64 and AIX. For now, I left the AIX >> part out - it seems only half done, and I want to check >> further if we even need it, if yes why not on Linux ppc, and >> C1 does not seem to support anything other than base+offset >> with shift either, but I may be mistaken. >> >> These two methods should give the platform enough control to >> implement their own scheme for optimized class space >> placement without bothering any shared code about it. >> >> Note about the form, I introduced two new platform dependent >> files, "metaspace_.cpp" and "compressedOops_.cpp". >> I am not happy about this but this seems to be what we >> generally do in hotspot, right? >> >> --- >> >> Metaspace reserve alignment vs cds alignment >> >> CDS was using Metaspace reserve alignment for CDS internal >> purposes. I guess this was just a copy paste issue. It never >> caused problems since Metaspace reserve alignment == page >> size, but that is not true anymore in the upcoming Elastic >> Metaspace where reserve alignment will be larger. This causes >> a number of issues. >> >> I separated those two cleanly. CDS now uses >> os::vm_allocation_granularity. Metaspace::reserve_alignment >> is only used in those two places where it is needed, when CDS >> creates the address space for class space on behalf of the >> Metaspace. >> >> --- >> >> Windows special handling in CDS >> >> To simplify coding I removed the windows specific handling >> which left out reservation of the archive. This was needed >> because windows cannot mmap files into reserved regions. But >> fallback code exists in filemap.cpp for this case which just >> reads in the region instead of mapping?it. >> >> Should that turn out to be a performance problem, I will >> reinstate the feature. But a simpler way would be reserve the >> archive and later just before mmapping?the archive file to >> release the archive space. That would not only be simpler but >> give us the best guarantee that that address space is >> actually available. But I'd be happy to leave that part out >> completely if we do not see any performance problems on >> windows x64. >> >> --- >> >> NMT cannot deal with spaces which are split. This problem >> manifests in that bookkeeping for class space is done under >> "Shared Classes", not "Classes" as it should. This problem >> exists today too at dump?time and randomly at run time. But >> since I simplified the reservation, this problem now shows up >> always, whether or not we map at the SharedBaseAddress. >> While I could work around this problem, I'd prefer this >> problem to be solved at the core, and NMT to have an option >> to recognize reservation splits. So I'd rather not put a >> workaround for this into the patch but leave it for fixing as >> a separate issue. I opened this issue to track it [6]. >> >> --- >> >> Jtreg tests: >> >> I expanded the CompressedOops/CompressedClassPointers.java. I >> also extended them to Windows. The tests now optionally omit >> strict class space placement tests, since these tests heavily >> depend on ASLR and were the reason they were excluded on >> Windows. However I think even without checking for class >> space placement they make sense, just to see that the VM >> comes up and lives with the many different settings we can >> run in. >> >> --- >> >> Tests: >> >> - I ran the patch through Oracles submit repo >> - I ran tests manually for aarch64, zero, linux 32bit and >> windows x64 >> - The whole battery of nightly tests at SAP, including ppc, >> ppcle and aarch64, unfortunately excluding windows because of >> unrelated errors. Windows x64 tests will be redone tonight. >> >> >> Thank you, >> >> Thomas >> >> [1] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >> [2] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >> [5] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >> > From coleen.phillimore at oracle.com Fri May 15 22:55:55 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 15 May 2020 18:55:55 -0400 Subject: RFR (S) 8242424: Deprecate InitialBootClassLoaderMetaspaceSize Message-ID: <00da20b6-59a4-67ec-9023-4adcd3154a22@oracle.com> Also: 8243147: Deprecate UseLargePagesInMetaspace Summary: mark these options for deprecation Tested with tier1-3. open webrev at http://cr.openjdk.java.net/~coleenp/2020/8242424.01/webrev bug link https://bugs.openjdk.java.net/browse/JDK-8242424 bug link https://bugs.openjdk.java.net/browse/JDK-8243147 Thanks, Coleen From david.holmes at oracle.com Sat May 16 03:52:22 2020 From: david.holmes at oracle.com (David Holmes) Date: Sat, 16 May 2020 13:52:22 +1000 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> Message-ID: <6f7cda55-305a-30f3-9169-bcfc2e7f4b7d@oracle.com> On 15/05/2020 8:38 pm, coleen.phillimore at oracle.com wrote: > On 5/15/20 4:03 AM, Christian Hagedorn wrote: >> On 15.05.20 00:38, David Holmes wrote: >>> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >>>> >>>> I thought that this code under Verbose should have been changed to >>>> use logging with log_level = verbose.? We were trying to >>>> minimize/remove the Verbose flag. >>>> >>>> On 5/14/20 9:50 AM, David Holmes wrote: >>>>> +1 >>>>> >>>>> But I'd be inclined to generalize the test to run: >>>>> >>>>> -Xlog:all=trace -XX:+Verbose >>>> >>>> Don't do this, it would be tons of output and make the test unusable >>>> if it fails. >>> >>> The only way the test can fail is if it crashes. >>> >>> But we would want a level of indirection so that the test log doesn't >>> contain the full output ~11000 lines of text. >> >> Could we still generalize the test but redirect the output to a file >> with: >> >> -Xlog:all=trace:file=output.txt -XX:+Verbose ? >> >> When I just immediately return from trace_method_handle_stub() without >> printing anything, the above command will not print anything to the >> console. So, it would print the same amount of lines to the console as >> the current option in webrev.01 with -XX:+Verbose -Xlog:methodhandles. > > I'm confused.? Does the test crash with -Xlog:methodhandles > -XX:+Verbose?? If so, that's all that should be needed.? If it needs > -Xlog:all=trace, then don't add this test.? Even if it writes to an > output file, it would be huge and affect test execution, particularly if > more logging is added. This issue was filed because combining -Xlog:methodhandles and -XX:+Verbose caused a crash, because we do not adequately test these things and so we can introduce bugs like the current one. Hence my suggestion to run a sanity test that enables all logging with Verbose to catch such regressions. The output file is a bit over 1MB - hardly huge. But I don't want to hold Christian up here, what he has is better than nothing. David ----- >> >> In a next step we could file an RFE to replace Verbose with unified >> logging in trace_method_handle_stub() which currently is only using >> tty. Even when using -Xlog:methodhandles alone without -XX:+Verbose, >> there is still some printing done in trace_method_handle_stub() to tty by >> >> ?500?? tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, >> ?501???????????????? adaptername, mh_reg_name, >> ?502???????????????? p2i(mh), p2i(entry_sp)); >> >> which is probably also not desired, as it is guarded by >> log_is_enabled(Info, methodhandles) earlier [1]. This could also be >> fixed in this follow-up RFE. > > Please, yes, this can be fixed with a follow-up RFE. > > Thanks, > Coleen >> >> What do you think? >> >> Best regards, >> Christian >> >> >> [1] >> http://hg.openjdk.java.net/jdk/jdk/file/748fedeb7cc1/src/hotspot/cpu/x86/methodHandles_x86.cpp#l598 >> >> >>> >>> David >>> ----- >>> >>>> Coleen >>>> >>>>> >>>>> Curiously I couldn't reproduce the crash that this bug is about. >>>>> But the hs_err file speaks for itself. >>>>> >>>>> Cheers, >>>>> David >>>>> >>>>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>>>> Hi David, hi Daniel >>>>>>> >>>>>>> Thank you for your reviews! >>>>>>> >>>>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>>>> ???? Not your bug, but the comment has two typos: >>>>>>>> >>>>>>>> ???????? s/by safer/but safer/ >>>>>>>> ???????? s/unexpensive/inexpensive/ >>>>>>> >>>>>>> Thanks, fixed it. >>>>>>> >>>>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>>>> >>>>>>>> Given all of the code is inside: >>>>>>>> >>>>>>>> ??504?? if (Verbose) { >>>>>>>> >>>>>>>> I would just move the ResourceMark to the top of that block. We >>>>>>>> are going to use it unconditionally, and we are not releasing it >>>>>>>> until the end of the method regardless. That avoids the >>>>>>>> potential concern Dan raised about extending the scope of the >>>>>>>> PRESERVE_EXCEPTION_MARK. >>>>>>> >>>>>>> This sounds reasonable. I moved it to the top of that block. >>>>>>> >>>>>>>>> There are other places like Klass::print_on(outputStream*) >>>>>>>>> where this is still the case. Should we file an RFE to clean >>>>>>>>> these up? >>>>>>>> >>>>>>>> Obviously we don't have any tests running in a mode that causes >>>>>>>> this code to be executed, so yes it is probably a good idea to >>>>>>>> take a closer look at other call-sites and file a RFE if needed. >>>>>>> >>>>>>> Yes, I filed one to take a closer look at these [1]. >>>>>>> >>>>>>> I included the changes in a new webrev: >>>>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>>>> >>>>>> Thumbs up. >>>>>> >>>>>> Dan >>>>>> >>>>>>> >>>>>>> Best regards, >>>>>>> Christian >>>>>>> >>>>>>> >>>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>>>> >>>>>> >>>> > From david.holmes at oracle.com Sat May 16 04:01:13 2020 From: david.holmes at oracle.com (David Holmes) Date: Sat, 16 May 2020 14:01:13 +1000 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: <459a6a6f-5c40-c6d2-af3b-9c76b8a4f57e@oracle.com> References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> <412316cc-fb7e-f360-5a76-78e98ddf2890@oracle.com> <459a6a6f-5c40-c6d2-af3b-9c76b8a4f57e@oracle.com> Message-ID: Hi Christian, On 16/05/2020 12:06 am, Christian Hagedorn wrote: > While working on the replacement of Verbose by Unified Logging, I've > seen that we should also fix this bug for ARM and S390 (PPC and SPARC > already have the ResourceMark set correctly and Aarch64 is not > implementing trace_method_handle_stub()). I included the necessary > changes in a new webrev: > http://cr.openjdk.java.net/~chagedorn/8244946/webrev.02/ src/hotspot/cpu/ppc/methodHandles_ppc.cpp src/hotspot/cpu/x86/methodHandles_x86.cpp You should keep the existing block so that the scope of the PRESERVE_EXCEPTION_MARK is not changed. All this is needed is to move the ResourceMark. runtime/logging/TestMethodHandlesVerbose.java Note that there is no need to declare the test in the runtime.logging package. We generally do not use packages for regression tests. > Would be good if anybody could also have a quick look at these minor > changes for ARM and S390 and verify them (I have only built and tested > it on x86): You say ARM and S390 but the webrev only has changes for ARM and PPC. Thanks, David ----- > > Thank you! > > Best regards, > Christian > > On 15.05.20 15:08, Christian Hagedorn wrote: >> On 15.05.20 14:56, coleen.phillimore at oracle.com wrote: >>> On 5/15/20 8:06 AM, Christian Hagedorn wrote: >>>> Hi Coleen >>>> >>>> On 15.05.20 12:38, coleen.phillimore at oracle.com wrote: >>>>> On 5/15/20 4:03 AM, Christian Hagedorn wrote: >>>>>> On 15.05.20 00:38, David Holmes wrote: >>>>>>> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >>>>>>>> >>>>>>>> I thought that this code under Verbose should have been changed >>>>>>>> to use logging with log_level = verbose.? We were trying to >>>>>>>> minimize/remove the Verbose flag. >>>>>>>> >>>>>>>> On 5/14/20 9:50 AM, David Holmes wrote: >>>>>>>>> +1 >>>>>>>>> >>>>>>>>> But I'd be inclined to generalize the test to run: >>>>>>>>> >>>>>>>>> -Xlog:all=trace -XX:+Verbose >>>>>>>> >>>>>>>> Don't do this, it would be tons of output and make the test >>>>>>>> unusable if it fails. >>>>>>> >>>>>>> The only way the test can fail is if it crashes. >>>>>>> >>>>>>> But we would want a level of indirection so that the test log >>>>>>> doesn't contain the full output ~11000 lines of text. >>>>>> >>>>>> Could we still generalize the test but redirect the output to a >>>>>> file with: >>>>>> >>>>>> -Xlog:all=trace:file=output.txt -XX:+Verbose ? >>>>>> >>>>>> When I just immediately return from trace_method_handle_stub() >>>>>> without printing anything, the above command will not print >>>>>> anything to the console. So, it would print the same amount of >>>>>> lines to the console as the current option in webrev.01 with >>>>>> -XX:+Verbose -Xlog:methodhandles. >>>>> >>>>> I'm confused.? Does the test crash with -Xlog:methodhandles >>>>> -XX:+Verbose?? If so, that's all that should be needed.? If it >>>>> needs -Xlog:all=trace, then don't add this test.? Even if it writes >>>>> to an output file, it would be huge and affect test execution, >>>>> particularly if more logging is added. >>>> >>>> I could already reproduce it locally on my Linux machine with >>>> -Xlog:methodhandles -XX:+Verbose. From what I understood, I thought >>>> that David is just talking about having a more general log test that >>>> covers the entire logging together with -XX:+Verbose for a >>>> HelloWorld program. But as you said, that would have some bad >>>> effects on test execution. However, such a test is not required for >>>> this bug to be triggered. >>> >>> Good! This looks good to me then. >> >> Thank you Coleen for your review! >> >>> Thank you for filing an RFE to do more logging with Unified Logging. >> >> No problem, I will take a look at it next. >> >> Best regards, >> Christian >> >>>> >>>>>> In a next step we could file an RFE to replace Verbose with >>>>>> unified logging in trace_method_handle_stub() which currently is >>>>>> only using tty. Even when using -Xlog:methodhandles alone without >>>>>> -XX:+Verbose, there is still some printing done in >>>>>> trace_method_handle_stub() to tty by >>>>>> >>>>>> ?500?? tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, >>>>>> ?501???????????????? adaptername, mh_reg_name, >>>>>> ?502???????????????? p2i(mh), p2i(entry_sp)); >>>>>> >>>>>> which is probably also not desired, as it is guarded by >>>>>> log_is_enabled(Info, methodhandles) earlier [1]. This could also >>>>>> be fixed in this follow-up RFE. >>>>> >>>>> Please, yes, this can be fixed with a follow-up RFE. >>>> >>>> Great, I filed an RFE for it [1]. >>>> >>>> Best regards, >>>> Christian >>>> >>>> >>>> [1] https://bugs.openjdk.java.net/browse/JDK-8245107 >>>> >>>> >>>>>>>>> Curiously I couldn't reproduce the crash that this bug is >>>>>>>>> about. But the hs_err file speaks for itself. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> David >>>>>>>>> >>>>>>>>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>>>>>>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>>>>>>>> Hi David, hi Daniel >>>>>>>>>>> >>>>>>>>>>> Thank you for your reviews! >>>>>>>>>>> >>>>>>>>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>>>>>>>> ???? Not your bug, but the comment has two typos: >>>>>>>>>>>> >>>>>>>>>>>> ???????? s/by safer/but safer/ >>>>>>>>>>>> ???????? s/unexpensive/inexpensive/ >>>>>>>>>>> >>>>>>>>>>> Thanks, fixed it. >>>>>>>>>>> >>>>>>>>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>>>>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>>>>>>>> >>>>>>>>>>>> Given all of the code is inside: >>>>>>>>>>>> >>>>>>>>>>>> ??504?? if (Verbose) { >>>>>>>>>>>> >>>>>>>>>>>> I would just move the ResourceMark to the top of that block. >>>>>>>>>>>> We are going to use it unconditionally, and we are not >>>>>>>>>>>> releasing it until the end of the method regardless. That >>>>>>>>>>>> avoids the potential concern Dan raised about extending the >>>>>>>>>>>> scope of the PRESERVE_EXCEPTION_MARK. >>>>>>>>>>> >>>>>>>>>>> This sounds reasonable. I moved it to the top of that block. >>>>>>>>>>> >>>>>>>>>>>>> There are other places like Klass::print_on(outputStream*) >>>>>>>>>>>>> where this is still the case. Should we file an RFE to >>>>>>>>>>>>> clean these up? >>>>>>>>>>>> >>>>>>>>>>>> Obviously we don't have any tests running in a mode that >>>>>>>>>>>> causes this code to be executed, so yes it is probably a >>>>>>>>>>>> good idea to take a closer look at other call-sites and file >>>>>>>>>>>> a RFE if needed. >>>>>>>>>>> >>>>>>>>>>> Yes, I filed one to take a closer look at these [1]. >>>>>>>>>>> >>>>>>>>>>> I included the changes in a new webrev: >>>>>>>>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>>>>>>>> >>>>>>>>>> Thumbs up. >>>>>>>>>> >>>>>>>>>> Dan >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best regards, >>>>>>>>>>> Christian >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>> >>> From david.holmes at oracle.com Sat May 16 04:27:16 2020 From: david.holmes at oracle.com (David Holmes) Date: Sat, 16 May 2020 14:27:16 +1000 Subject: RFR: 8245098: Make SafeFetch32/N available earlier In-Reply-To: References: Message-ID: <9501bdb6-c06e-5b78-0017-2de71a97c48b@oracle.com> Hi Per, On 15/05/2020 10:03 pm, Per Liden wrote: > We have a need for SafeFetch32 during initializing of ZGC (see [1]). > However, the SafeFetch32/N stubs are currently generated too late, in > the stubRoutines_init2() phase, which is after the GC has been > initialized. I propose we move the initialization of these stubs to the > stubRoutines_init1() phase, which is before the GC has been initialized, > to make them available during GC initialization. generate_safefetch includes use of a StubCodeMark. And its destructor contains: Forte::register_stub(_cdesc->name(), _cdesc->begin(), _cdesc->end()); if (JvmtiExport::should_post_dynamic_code_generated()) { JvmtiExport::post_dynamic_code_generated(_cdesc->name(), _cdesc->begin(), _cdesc->end()); } It is not at all obvious that these actions are still okay at an earlier stage of the initialization sequence. I don't know what tests we may have for this, nor in what tier they may run, but I suggest running more extensive JVM TI testing, and try to determine how the Forte code is used. Thanks, David > Please note that this patch touches platforms I can't easily test > (x86-32, arm, ppc, s390). If you have access to one of those machines, I > would appreciate if you could do some light verification. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8245098 > Webrev: http://cr.openjdk.java.net/~pliden/8245098/webrev.0 > Testing: Tier1-3 on all Oracle supported platforms. > > /Per > > [1] https://bugs.openjdk.java.net/browse/JDK-8245106 From thomas.stuefe at gmail.com Sat May 16 08:12:28 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Sat, 16 May 2020 10:12:28 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: Message-ID: Hi, On Sat, May 16, 2020 at 12:26 AM wrote: > > Hi, This looks good. The code motion makes sense so this is an improvement. > > thanks! > On 5/7/20 10:21 AM, Thomas St?fe wrote: > > Hi all, > > please take a look at the third iteration of this change: > > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ > > Changes in this version: > > - at the request of Coleen (sorry I did not see your post earlier) I > removed all platform dependent files and put the platform dependent coding > back to metaspace.cpp and compressedOops.cpp to ease the review pain. I > used plain platform defines though (#ifdef AARCH64) instead of hiding them > behind another macro to make things clearer. Note that I still intent to > put this code away into the platform corners but will do so in a follow up > RFE. > > > Oh, please don't, especially since most platforms are just a > ReservedSpace(size, align, false, NULL); The AARCH64 code isn't a big > intrusion in this code. Thank you for putting it back. It is easier to > follow how it is. > > Okay. We'll see how quickly the coding deteriorates again though. We may want to put extra stuff for ppc64 in too, and/or just AIX. See also: https://bugs.openjdk.java.net/browse/JDK-8244943 . I'll keep this off for now to a later point. > > - I reinstated, in a fashion, the special handling of reservations on > Windows. On all platforms we reserve address space to map the archive files > in with a subsequent mapping operation. However, on Windows, we cannot use > MapViewOfFile() into an existing mapping. So I remove the mapping again > before mapping the archives - see comment in code for details. > > > I missed this windows part of the change, where is it? > Follow the trail Ioi led with "MetaspaceShared::use_windows_memory_mapping()". In original coding this is a complicated dance between MetaspaceShared::reserve_address_space_for_archives and FileMapInfo::map_region, the implicit knowledge here is that these functions are called twice, first time trying to attach to the archive base address (which is the nice fast case, no relocation) and the second time trying to attach anywhere. In MetaspaceShared::reserve_address_space_for_archives, we want to reserve space for archive and ccs laying one beside the other; later we plan to map the archive file into the prereserved archive space. But that does not work on Windows, you cannot map into pre-reserved space; so the first time we do this we explicitly do *not* reserve the archive space (windows only), just the class space. Then we attempt to map the archive space into that region which is not reserved yet but hopefully unpopulated. Since that may not work, we call MetaspaceShared::reserve_address_space_for_archives a second time, this time reserving both archive space and css, and this time around instead of mapping the archive file in Ioi just reads it sequentially. This is slower than mapping the file, but his reasoning is that this does not matter since he needs to relocate anyway - different base address - so the sequential read does not hurt performance. I think this all makes sense. Its the fastest way of doing this. In my first attempt at this patch I wanted to simplify things and just always read sequentially; but that hurt startup time on WIndows when the cds was big. So I just tried to make the code a bit simpler and more faultproof: in my version of MetaspaceShared::reserve_address_space_for_archives, I always unconditionally reserve space for both archive and ccs. Then, returning from that function, in MetaspaceShared::map_archives(), I unreserve the archive again on windows, so subsequent mappings to this address can succeed. I think this makes the code a bit clearer, and an additional advantage is that since we were able to reserve at that address already once the chance is very high that mapping of the files some milliseconds later will succeed. Not guaranteed though, that is why the second time around we use the original sequential read logic. > > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/virtualSpaceList.cpp.udiff.html > > Why did you make this change? Shouldn't the caller align it? > Probably yes. I'll remove this again. > > Looks good to me. > Coleen > > Thanks! I'll post an update in the next days to incorporate yours and Iois feedback and rebase the patch. Cheers, Thomas > > All CI tests at SAP run through without problems, including on Win64 and > aarch64, but I would be happy if others were to run test too. > > Thank you, Thomas > > On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe > wrote: > >> Hi all, >> >> Could I have reviews for the following proposal of reworking cds/class >> space reservation? >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >> >> Webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >> >> (Many thanks to Ioi Lam for so patiently explaining CDS internals to me, >> and to Andrew Haley and Nick Gasson for help with aarch64!) >> >> Reservation of the compressed class space is needlessly complicated and >> has some minor issues. It can be simplified and made clearer. >> >> The complexity stems from the fact that this area lives at the >> intersection of two to three sub systems, depending on how one counts. >> Metaspace, CDS, and the platform which may or may not its own view of how >> to reserve class space. And this code has been growing organically over >> time. >> >> One small example: >> >> ReservedSpace Metaspace::reserve_preferred_space(size_t size, size_t >> alignment, >> bool large_pages, char >> *requested_addr, >> bool use_requested_addr) >> >> which I spent hours decoding, resulting in a very confused mail to >> hs-runtime and aarch64-port-dev [2]. >> >> This patch attempts to simplify cds and metaspace setup a bit; to comment >> implicit knowledge which is not immediately clear; to cleanly abstract >> platform concerns like optimized class space placement; and to disentangle >> cds from metaspace to solve issues which may bite us later with Elastic >> Metaspace [4]. >> >> --- >> >> The main change is the reworked reservation mechanism. This is based on >> Ioi's proposal [5]. >> >> When reserving class space, three things must happen: >> >> 1) reservation of the space obviously. If cds is active that space must >> be in the vicinity of cds archives to be covered by compressed class >> pointer encoding. >> 2) setting up the internal Metaspace structures atop of that space >> 3) setting up compressed class pointer encoding. >> >> In its current form, Metaspace may or may not do some or all of that in >> one function >> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base);) - if cds is active, >> it will reserve the space for Metaspace and hand it in, otherwise it will >> create it itself. >> >> When discussing this in [2], Ioi proposed to move the reservation of the >> class space completely out of Metaspace and make it a responsibility of the >> caller always. This would reduce some complexity, and this patch follows >> the proposal. >> >> I removed >> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base); and all its sub >> functions. >> >> (1) now has to always be done outside - a ReservedSpace for class space >> has to be provided by the caller. However, Metaspace now offers a utility >> function for reserving space at a "nice" location, and explicitly doing >> nothing else: >> >> ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >> >> this function can be redefined on a platform level for platform optimized >> reservation, see below for details. >> >> (2) is taken care of by a new function, >> Metaspace::initialize_class_space(ReservedSpace rs) >> >> (3) is taken care of a new function >> CompressedKlassPointers::initialize(), see below for details. >> >> >> So, class space now is set up three explicit steps: >> >> - First, reserve a suitable space by however means you want. For >> convenience you may use >> Metaspace::reserve_address_space_for_compressed_classes(), or you may roll >> your own reservation. >> - Next, tell Metaspace to use that range as backing storage for class >> space: Metaspace::initialize_class_space(ReservedSpace rs) >> - Finally, set up encoding. Encoding is independent from the concept of a >> ReservedSpace, it just gets an address range, see below for details. >> >> Separating these steps and moving them out of the responsibility of >> Metaspace makes this whole thing more flexible; it also removes unnecessary >> knowledge (e.g. Metaspace does not need to know anything about either ccp >> encoding or cds). >> >> --- >> >> How it comes together: >> >> If CDS is off, we just reserve a space using >> Metaspace::reserve_address_space_for_compressed_classes(), initialize it >> with Metaspace::initialize_class_space(ReservedSpace rs), then set up >> compressed class pointer encoding covering the range of this class space. >> >> If CDS is on (dump time), we reserve large 4G space, either at >> SharedBaseAddress or using >> Metaspace::reserve_address_space_for_compressed_classes(); we then split >> that into 3G archive space and 1G class space; we set up that space with >> Metaspace as class space; then we set up compressed class pointer encoding >> covering both archive space and cds. >> >> If CDS is on (run time), we reserve a large space, split it into archive >> space (large enough to hold both archives) and class space, then basically >> proceed as above. >> >> Note that this is almost exactly how things worked before (modulo some >> minor fixes, e.g. alignment issues), only the code is reformed and made >> more explicit. >> >> --- >> >> I moved compressed class pointer setup over to CompressedKlassPointers >> and changed the interface: >> >> -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >> metaspace_rs, address cds_base) >> +void CompressedKlassPointers::initialize(address addr, size_t len); >> >> Instead of feeding it a single ReservedSpace, which is supposed to >> represent class space, and an optional alternate base if cds is on, now we >> give it just an numeric address range. That range marks the limits to where >> Klass structures are to be expected, and is the implicit promise that >> outside that range no Klass structures will exist, so encoding has to cover >> only this range. >> >> This range may contain just the class space; or class space+cds; or >> whatever allocation scheme we come up with in the future. Encoding does not >> really care how the memory is organized as long as the input range covers >> all possible Klass locations. That way we remove knowledge about class >> space/cds from compressed class pointer encoding. >> >> Moving it away from metaspace.cpp into the CompressedKlassPointers class >> also mirrors CompressedOops::initialize(). >> >> --- >> >> I renamed _narrow_klass_range to just _range, because strictly speaking >> this is the range un-narrow Klass pointers can have. >> >> As for the implementation of CompressedKlassPointers::initialize(address >> addr, size_t len), I mimicked very closely what happened before, so there >> should be almost no differences. Since "almost no differences" sounds scary >> :) here are the differences: >> >> - When CDS is active (dump or run time) we now always, unconditionally, >> set the encoding range to 4G. This fixes a theoretical bug discussed on >> aarch64-port-dev [1]. >> >> - When CDS is not active, we set the encoding range to the minimum >> required length. Before, it was left at its default value of 4G. >> >> Both differences only affect aarch64, since they are currently the only >> one using the range field in CompressedKlassPointers. >> >> I wanted to add an assert somewhere to test encoding of the very last >> address of the CompressedKlassPointers range, again to prevent errors like >> [3]. But I did not come up with a good place for this assert which would >> cover also the encoding done by C1/C2. >> >> For the same reason I thought about introducing a mode where Klass >> structures would be allocated in reverse order, starting at the end of the >> ccs, but again left it out as too big a change. >> >> --- >> >> OS abstraction: platforms may have restrictions of what constitutes a >> valid compressed class pointer encoding base. Or if not, they may have at >> least preferences. There was logic like this in metaspace.cpp, which I >> removed and cleanly factored out into platform dependent files, giving each >> platform the option to add special logic. >> >> These are two new methods: >> >> - bool CompressedKlassPointers::is_valid_base(address p) >> >> to let the platform tell you whether it considers p to be a valid >> encoding base. The only platform having these restrictions currently is >> aarch64. >> >> - ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >> >> this hands over the process of allocating a range suitable for compressed >> class pointer encoding to the platform. Most platforms will allocate just >> anywhere, but some platforms may have a better strategy (e.g. trying low >> memory first, trying only correctly aligned addresses and so on). >> >> Beforehand, this coding existed in a similar form in metaspace.cpp for >> aarch64 and AIX. For now, I left the AIX part out - it seems only half >> done, and I want to check further if we even need it, if yes why not on >> Linux ppc, and C1 does not seem to support anything other than base+offset >> with shift either, but I may be mistaken. >> >> These two methods should give the platform enough control to implement >> their own scheme for optimized class space placement without bothering any >> shared code about it. >> >> Note about the form, I introduced two new platform dependent files, >> "metaspace_.cpp" and "compressedOops_.cpp". I am not happy about >> this but this seems to be what we generally do in hotspot, right? >> >> --- >> >> Metaspace reserve alignment vs cds alignment >> >> CDS was using Metaspace reserve alignment for CDS internal purposes. I >> guess this was just a copy paste issue. It never caused problems since >> Metaspace reserve alignment == page size, but that is not true anymore in >> the upcoming Elastic Metaspace where reserve alignment will be larger. This >> causes a number of issues. >> >> I separated those two cleanly. CDS now uses >> os::vm_allocation_granularity. Metaspace::reserve_alignment is only used in >> those two places where it is needed, when CDS creates the address space for >> class space on behalf of the Metaspace. >> >> --- >> >> Windows special handling in CDS >> >> To simplify coding I removed the windows specific handling which left out >> reservation of the archive. This was needed because windows cannot mmap >> files into reserved regions. But fallback code exists in filemap.cpp for >> this case which just reads in the region instead of mapping it. >> >> Should that turn out to be a performance problem, I will reinstate the >> feature. But a simpler way would be reserve the archive and later just >> before mmapping the archive file to release the archive space. That would >> not only be simpler but give us the best guarantee that that address space >> is actually available. But I'd be happy to leave that part out completely >> if we do not see any performance problems on windows x64. >> >> --- >> >> NMT cannot deal with spaces which are split. This problem manifests in >> that bookkeeping for class space is done under "Shared Classes", not >> "Classes" as it should. This problem exists today too at dump time and >> randomly at run time. But since I simplified the reservation, this problem >> now shows up always, whether or not we map at the SharedBaseAddress. >> While I could work around this problem, I'd prefer this problem to be >> solved at the core, and NMT to have an option to recognize reservation >> splits. So I'd rather not put a workaround for this into the patch but >> leave it for fixing as a separate issue. I opened this issue to track it >> [6]. >> >> --- >> >> Jtreg tests: >> >> I expanded the CompressedOops/CompressedClassPointers.java. I also >> extended them to Windows. The tests now optionally omit strict class space >> placement tests, since these tests heavily depend on ASLR and were the >> reason they were excluded on Windows. However I think even without checking >> for class space placement they make sense, just to see that the VM comes up >> and lives with the many different settings we can run in. >> >> --- >> >> Tests: >> >> - I ran the patch through Oracles submit repo >> - I ran tests manually for aarch64, zero, linux 32bit and windows x64 >> - The whole battery of nightly tests at SAP, including ppc, ppcle and >> aarch64, unfortunately excluding windows because of unrelated errors. >> Windows x64 tests will be redone tonight. >> >> >> Thank you, >> >> Thomas >> >> [1] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >> [2] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >> [5] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >> >> > From thomas.stuefe at gmail.com Sat May 16 08:22:21 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Sat, 16 May 2020 10:22:21 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: <1108ac1a-f693-cfc7-9f86-1a84508b1487@oracle.com> References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> <1108ac1a-f693-cfc7-9f86-1a84508b1487@oracle.com> Message-ID: On Sat, May 16, 2020 at 12:34 AM wrote: > > One more in-context comment. > > On 5/8/20 3:33 AM, Thomas St?fe wrote: > > Hi Ioi, > > thanks for taking the time to look at this! > > On Fri, May 8, 2020 at 7:37 AM Ioi Lam wrote: > >> Hi Thomas, >> >> I am running your patch in out CI pipeline now. Some comments: >> >> [1] Do we still need to use _class_space_list? It looks like we just have >> a single range. Is _class_space_list->current_virtual_space() always the >> same VS? >> >> if (_class_space_list != NULL) { >> address base = >> (address)_class_space_list->current_virtual_space()->bottom(); >> address top = base + compressed_class_space_size(); >> st->print("Compressed class space mapped at: " PTR_FORMAT "-" >> PTR_FORMAT ", size: " SIZE_FORMAT, >> p2i(base), p2i(top), top - base); >> >> > Yes it is. Has been bugging me for years too. With ccs, the > VirtualSpaceList is degenerated and only contains one node. It still makes > some sense code wise. But maybe VirtualSpace*List* is a misnomer. > > It also ties in into the question whether Metaspace reservations should be > able to grow on demand. Oracle folks I talk to tend to say yes. I > personally think the "Metaspace is infinite don't worry" meme was broken > since ccs was a thing, since CompressedClassSpaceSize is an absolute limit > already. > > I may tweak and simplify the code a bit with Elastic Metaspace. Maybe, as > you say, with ccs we can get rid of VirtualSpaceList and deal with the > range directly. Have to check. > > > Yes, this could be an improvement for a later time. The compressed class > space isn't a list. I think that was the least amount of special code > place to hook it in at the time. It makes sense for it to be a single > VirtualSpaceNode. > > CompressedClassSpace + Data (the rest of Metaspace) has a complicated > relationship for determining high water mark for starting GC, and for > reporting LowMemory thresholds. I'd have to look up what was decided there. > Data (the rest of Metaspace) does grow on demand unless someone uses > -XX:MaxMetaspaceSize and then class metaspace size is factored in (maybe > just committed space not the whole 1G). HTH. > > We stop growing when [committed class space]+[committed data] >= MaxMetaspaceSize. CompressedClassSpaceSize is actually completely uninteresting for calculating limits, it is just an additional, technical limit on the growth of [committed class space]. Even this equation: MaxMetaspaceSize = CompressedClassSpaceSize+data is actually not correct, even though the code treats it that way. We could set CompressedClassSpaceSize much higher than MaxMetaspaceSize, we only would not be able to commit the whole class space then, but it still would work. I simplified all this in Elastic Metaspace. GC threshold calculation is unaffected though. ..Thomas > Coleen > > From thomas.stuefe at gmail.com Sat May 16 11:40:27 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Sat, 16 May 2020 13:40:27 +0200 Subject: RFR (S) 8242424: Deprecate InitialBootClassLoaderMetaspaceSize In-Reply-To: <00da20b6-59a4-67ec-9023-4adcd3154a22@oracle.com> References: <00da20b6-59a4-67ec-9023-4adcd3154a22@oracle.com> Message-ID: Thank you Coleen, looks good. Cheers, Thomas On Sat, May 16, 2020 at 12:56 AM wrote: > Also: > 8243147: Deprecate UseLargePagesInMetaspace > > Summary: mark these options for deprecation > > Tested with tier1-3. > > open webrev at http://cr.openjdk.java.net/~coleenp/2020/8242424.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8242424 > bug link https://bugs.openjdk.java.net/browse/JDK-8243147 > > Thanks, > Coleen > From daniel.daugherty at oracle.com Sat May 16 13:27:00 2020 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Sat, 16 May 2020 09:27:00 -0400 Subject: RFR (S) 8242424: Deprecate InitialBootClassLoaderMetaspaceSize In-Reply-To: <00da20b6-59a4-67ec-9023-4adcd3154a22@oracle.com> References: <00da20b6-59a4-67ec-9023-4adcd3154a22@oracle.com> Message-ID: <13c0f52e-3ca9-6855-8a75-36da227dbdeb@oracle.com> On 5/15/20 6:55 PM, coleen.phillimore at oracle.com wrote: > Also: > 8243147: Deprecate UseLargePagesInMetaspace > > Summary: mark these options for deprecation > > Tested with tier1-3. > > open webrev at http://cr.openjdk.java.net/~coleenp/2020/8242424.01/webrev src/hotspot/share/runtime/arguments.cpp ??? No comments. src/hotspot/share/runtime/globals.hpp ??? No comments. test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java ??? No comments. Thumbs up on this part, but I see these options mentioned in tests: $ egrep -r 'UseLargePagesInMetaspace|InitialBootClassLoaderMetaspaceSize' open/test closed/test open/test/hotspot/jtreg/vmTestbase/metaspace/gc/MetaspaceBaseGC.java: useLargepages = PAGE_SIZE > 1_000_000 && !vmArgs.contains("-XX:-UseLargePagesInMetaspace"); open/test/hotspot/jtreg/runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java: excludeTestMaxRange("InitialBootClassLoaderMetaspaceSize"); open/test/hotspot/jtreg/runtime/cds/MaxMetaspaceSize.java: processArgs.add("-XX:InitialBootClassLoaderMetaspaceSize=1m"); open/test/hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java: "-XX:+UseLargePages", "-XX:+UseLargePagesInMetaspace"); open/test/hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java: "-XX:+UseLargePages", "-XX:+UseLargePagesInMetaspace"); open/test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeTest.java: "-XX:InitialBootClassLoaderMetaspaceSize=4195328", open/test/hotspot/jtreg/gc/TestSystemGC.java: * @run main/othervm -XX:+UseLargePages -XX:+UseLargePagesInMetaspace gc.TestSystemGC Do you know of the deprecated message will affect how these tests run? Dan > bug link https://bugs.openjdk.java.net/browse/JDK-8242424 > bug link https://bugs.openjdk.java.net/browse/JDK-8243147 > > Thanks, > Coleen From kim.barrett at oracle.com Sat May 16 15:48:03 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Sat, 16 May 2020 11:48:03 -0400 Subject: RFR: 8245098: Make SafeFetch32/N available earlier In-Reply-To: <9501bdb6-c06e-5b78-0017-2de71a97c48b@oracle.com> References: <9501bdb6-c06e-5b78-0017-2de71a97c48b@oracle.com> Message-ID: <67868905-CA21-45EA-BE8E-8AE533185211@oracle.com> > On May 16, 2020, at 12:27 AM, David Holmes wrote: > > Hi Per, > > On 15/05/2020 10:03 pm, Per Liden wrote: >> We have a need for SafeFetch32 during initializing of ZGC (see [1]). However, the SafeFetch32/N stubs are currently generated too late, in the stubRoutines_init2() phase, which is after the GC has been initialized. I propose we move the initialization of these stubs to the stubRoutines_init1() phase, which is before the GC has been initialized, to make them available during GC initialization. > > generate_safefetch includes use of a StubCodeMark. And its destructor contains: > > Forte::register_stub(_cdesc->name(), _cdesc->begin(), _cdesc->end()); > if (JvmtiExport::should_post_dynamic_code_generated()) { > JvmtiExport::post_dynamic_code_generated(_cdesc->name(), _cdesc->begin(), _cdesc->end()); > } > > It is not at all obvious that these actions are still okay at an earlier stage of the initialization sequence. I don't know what tests we may have for this, nor in what tier they may run, but I suggest running more extensive JVM TI testing, and try to determine how the Forte code is used. There are lots of other stub generators called earlier in generate_initial and which also include uses of StubCodeMark. From thomas.stuefe at gmail.com Sun May 17 07:25:18 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Sun, 17 May 2020 09:25:18 +0200 Subject: RFR (S) 8242424: Deprecate InitialBootClassLoaderMetaspaceSize In-Reply-To: <13c0f52e-3ca9-6855-8a75-36da227dbdeb@oracle.com> References: <00da20b6-59a4-67ec-9023-4adcd3154a22@oracle.com> <13c0f52e-3ca9-6855-8a75-36da227dbdeb@oracle.com> Message-ID: Hi, I ran the following tests: hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeTest.java hotspot/jtreg/runtime/cds/MaxMetaspaceSize.java hotspot/jtreg/runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java hotspot/jtreg/gc/TestSystemGC.java and all of hotspot/jtreg/vmTestbase/metaspace/... successfully with Coleens patch. Cheers, Thomas On Sat, May 16, 2020 at 3:27 PM Daniel D. Daugherty < daniel.daugherty at oracle.com> wrote: > On 5/15/20 6:55 PM, coleen.phillimore at oracle.com wrote: > > Also: > > 8243147: Deprecate UseLargePagesInMetaspace > > > > Summary: mark these options for deprecation > > > > Tested with tier1-3. > > > > open webrev at > http://cr.openjdk.java.net/~coleenp/2020/8242424.01/webrev > > src/hotspot/share/runtime/arguments.cpp > No comments. > > src/hotspot/share/runtime/globals.hpp > No comments. > > test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > No comments. > > Thumbs up on this part, but I see these options mentioned in tests: > > $ egrep -r > 'UseLargePagesInMetaspace|InitialBootClassLoaderMetaspaceSize' open/test > closed/test > open/test/hotspot/jtreg/vmTestbase/metaspace/gc/MetaspaceBaseGC.java: > useLargepages = PAGE_SIZE > 1_000_000 && > !vmArgs.contains("-XX:-UseLargePagesInMetaspace"); > open/test/hotspot/jtreg/runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java: > > excludeTestMaxRange("InitialBootClassLoaderMetaspaceSize"); > open/test/hotspot/jtreg/runtime/cds/MaxMetaspaceSize.java: > processArgs.add("-XX:InitialBootClassLoaderMetaspaceSize=1m"); > open/test/hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java: > "-XX:+UseLargePages", "-XX:+UseLargePagesInMetaspace"); > open/test/hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java: > "-XX:+UseLargePages", "-XX:+UseLargePagesInMetaspace"); > open/test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeTest.java: > "-XX:InitialBootClassLoaderMetaspaceSize=4195328", > open/test/hotspot/jtreg/gc/TestSystemGC.java: * @run main/othervm > -XX:+UseLargePages -XX:+UseLargePagesInMetaspace gc.TestSystemGC > > Do you know of the deprecated message will affect how these tests run? > > Dan > > > > > bug link https://bugs.openjdk.java.net/browse/JDK-8242424 > > bug link https://bugs.openjdk.java.net/browse/JDK-8243147 > > > > Thanks, > > Coleen > > From gnu.andrew at redhat.com Sun May 17 18:22:19 2020 From: gnu.andrew at redhat.com (Andrew Hughes) Date: Sun, 17 May 2020 19:22:19 +0100 Subject: [8u] RFR(S): 8244548: JDK 8u: sun.misc.Version.jdkUpdateVersion() returns wrong result In-Reply-To: References: <17f5dc4d7bc7d405acfd2c49642a97c1fc22689a.camel@redhat.com> <808419e3-316b-381d-f636-606aadff39b6@redhat.com> <15a2fa41f80935d8c7d3893aad7d080c1f67aa2a.camel@redhat.com> <74e4d10641c8b6acca8ada68459a767f98e23585.camel@redhat.com> Message-ID: <38d5c8bb-5be7-af41-786a-5621a118f8fd@redhat.com> On 15/05/2020 14:29, Andrew Haley wrote: > On 5/15/20 9:45 AM, Severin Gehwolf wrote: >> So, Andrew Haley, what's the verdict on this one? There is a risk, yes. >> Not fixing the bug seems riskier, though. Thoughts? > > Thanks for the detailed explanation. > > I think we have to keep sun.misc.Version working correctly but > sacrifice binary consistency in JVM_GetVersionInfo. As long as stuff > such as AOT compilation and class data sharing keep working I think > we're OK. It's not great, but it's an almost-inevitable consequence of > using a one-byte field for the minor version and bumping releases by > 10. > I see little to no risk in changing this. Anyone using 7u261 or later, or 8u262 or later, is going to get an overflowed minor JVM version and an overflowed upgrade JDK version even if we do nothing at all. What we can do is attempt to fix things in a way that doesn't change the size of the structure, just the interpretation of its contents, as proposed here. Calling code that adapts to the modified version will then be able to get the correct values. Code that doesn't will be just as broken as if we'd made no change. Arguably, even that case will be a little better, as with a non-zero micro value, the incorrect version will at least be distinct from older versions. This is for 7u & 8u, which don't have AOT compilation, so that shouldn't be an issue. I'll be approving this for 8u. Thanks, -- Andrew :) Senior Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net) Fingerprint = 5132 579D D154 0ED2 3E04 C5A0 CFDA 0F9B 3596 4222 From david.holmes at oracle.com Mon May 18 00:50:27 2020 From: david.holmes at oracle.com (David Holmes) Date: Mon, 18 May 2020 10:50:27 +1000 Subject: RFR: 8245098: Make SafeFetch32/N available earlier In-Reply-To: <67868905-CA21-45EA-BE8E-8AE533185211@oracle.com> References: <9501bdb6-c06e-5b78-0017-2de71a97c48b@oracle.com> <67868905-CA21-45EA-BE8E-8AE533185211@oracle.com> Message-ID: <9545806d-87d8-e610-ae20-a8228b44a97d@oracle.com> On 17/05/2020 1:48 am, Kim Barrett wrote: >> On May 16, 2020, at 12:27 AM, David Holmes wrote: >> >> Hi Per, >> >> On 15/05/2020 10:03 pm, Per Liden wrote: >>> We have a need for SafeFetch32 during initializing of ZGC (see [1]). However, the SafeFetch32/N stubs are currently generated too late, in the stubRoutines_init2() phase, which is after the GC has been initialized. I propose we move the initialization of these stubs to the stubRoutines_init1() phase, which is before the GC has been initialized, to make them available during GC initialization. >> >> generate_safefetch includes use of a StubCodeMark. And its destructor contains: >> >> Forte::register_stub(_cdesc->name(), _cdesc->begin(), _cdesc->end()); >> if (JvmtiExport::should_post_dynamic_code_generated()) { >> JvmtiExport::post_dynamic_code_generated(_cdesc->name(), _cdesc->begin(), _cdesc->end()); >> } >> >> It is not at all obvious that these actions are still okay at an earlier stage of the initialization sequence. I don't know what tests we may have for this, nor in what tier they may run, but I suggest running more extensive JVM TI testing, and try to determine how the Forte code is used. > > There are lots of other stub generators called earlier in generate_initial and which also > include uses of StubCodeMark. Yep - my bad. I looked at generate_throw_exception as a close-by non-trivial example and noticed it did not. I should have checked some of the others. Thanks, David From david.holmes at oracle.com Mon May 18 05:46:37 2020 From: david.holmes at oracle.com (David Holmes) Date: Mon, 18 May 2020 15:46:37 +1000 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: References: <81b1d335-d898-5bc9-8aed-b105f8d73a2e@oracle.com> Message-ID: <87a78936-d44c-6b14-49c3-ae116c5c5f5d@oracle.com> Hi Matthias, On 16/05/2020 12:59 am, Baesken, Matthias wrote: > Hello David , I get the warning always on this linux machine (I think it depends on machine setup and/or kernel version). > >> Surely it must be specified to return 0 or some such sentinel in such a case? > > The doc just says this : > > 62 /** > 63 * Returns the total amount of swap space in bytes. > 64 * > 65 * @return the total amount of swap space in bytes. > 66 */ > 67 public long getTotalSwapSpaceSize(); > > ( however it returned previously 0 before the latest changes in the container support code came in ). That sounds like a bug was introduced to me then. > Regarding the getFreeSwapSpaceSize issue , Jie provided a change to src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java > Because of this I could remove the try-construct . > > New webrev : > http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.3/ I'm quite confused by your changes. Your first version simply adjusted the test to check for the warning. But now you are also changing the main logic. I can't assess the validity of what you are doing sorry. I've cc'd Severin and Bob as I think they need to evaluate what the real problem is here and when it was introduced. David ---- > Best regards, Matthias > > > > -----Original Message----- > From: David Holmes > Sent: Donnerstag, 14. Mai 2020 07:02 > To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' > Cc: jiefu(??) > Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > On 14/05/2020 12:28 am, Baesken, Matthias wrote: >> Hello, here is a new webrev removing the comment : >> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.1/ >> >> It removes the comment , and it also handles 0 for getFreeSwapSpaceSize >> ( this can occur after JDK-8242480 : Negative value may be returned by getFreeSwapSpaceSize() in the docker was pushed recently ). > > Sorry Matthias but I still see this partly as a bug in the underlying > container support code. If you get the warning then what does > getTotalSwapSpaceSize() return? Surely it must be specified to return 0 > or some such sentinel in such a case? > > Other comments: > > ! try { > ! > out.shouldMatch("OperatingSystemMXBean\\.getFreeSwapSpaceSize: > [1-9][0-9]+"); > ! } catch(RuntimeException ex) { // allow 0, see 8242480 > ! > out.shouldContain("OperatingSystemMXBean.getFreeSwapSpaceSize: 0"); > ! } > > can't you just change the original regex to match >= 0? > > Style nit: avoid historical comments like "see 8242480" and "since > 8231111 ...". > > Thanks, > David > >> Best regards, Matthias >> >> -----Original Message----- >> From: Baesken, Matthias >> Sent: Mittwoch, 13. Mai 2020 10:11 >> To: 'David Holmes' ; 'hotspot-dev at openjdk.java.net' >> Subject: RE: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java >> >> >>> But according to the comment immediately before your changes: >>> >>> // in case of warnings like : "Your kernel does not support swap limit >>> capabilities or the cgroup is not mounted. Memory limited without swap." >>> // the getTotalSwapSpaceSize does not return the expected result, but 0 >>> >>> we should be returning zero. So this seems to be a bug in the >>> implementation not in the test. >> >> >> Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values >> that are normally expected. >> Previously we indeed got "0" . >> Probably the comment should be deleted . >> >> Best regards, Matthias >> From jiefu at tencent.com Mon May 18 06:15:38 2020 From: jiefu at tencent.com (=?utf-8?B?amllZnUo5YKF5p2wKQ==?=) Date: Mon, 18 May 2020 06:15:38 +0000 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java Message-ID: <7201E4D5-96AA-47C1-B0B7-A19E8A0F198B@tencent.com> Hi David, This bug should be introduced by JDK-8242480. I'm sorry for that. And I think the logic in src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java should be fixed. In my fix for JDK-8242480, I had ignored the case for kernels which don't support "--memory-swap 150M". In that case, containerMetrics.getMemoryAndSwapLimit()[1] will return -1. I didn't notice this bug because all our kernels support --memory-swap. Fortunately, Matthias found this bug and will fix it. Thanks a lot. Best regards, Jie [1] http://hg.openjdk.java.net/jdk/jdk/file/a95911422005/src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java#l73 ?On 2020/5/18, 1:48 PM, "David Holmes" wrote: Hi Matthias, On 16/05/2020 12:59 am, Baesken, Matthias wrote: > Hello David , I get the warning always on this linux machine (I think it depends on machine setup and/or kernel version). > >> Surely it must be specified to return 0 or some such sentinel in such a case? > > The doc just says this : > > 62 /** > 63 * Returns the total amount of swap space in bytes. > 64 * > 65 * @return the total amount of swap space in bytes. > 66 */ > 67 public long getTotalSwapSpaceSize(); > > ( however it returned previously 0 before the latest changes in the container support code came in ). That sounds like a bug was introduced to me then. > Regarding the getFreeSwapSpaceSize issue , Jie provided a change to src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java > Because of this I could remove the try-construct . > > New webrev : > http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.3/ I'm quite confused by your changes. Your first version simply adjusted the test to check for the warning. But now you are also changing the main logic. I can't assess the validity of what you are doing sorry. I've cc'd Severin and Bob as I think they need to evaluate what the real problem is here and when it was introduced. David ---- > Best regards, Matthias > > > > -----Original Message----- > From: David Holmes > Sent: Donnerstag, 14. Mai 2020 07:02 > To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' > Cc: jiefu(??) > Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > On 14/05/2020 12:28 am, Baesken, Matthias wrote: >> Hello, here is a new webrev removing the comment : >> >> http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.1/ >> >> It removes the comment , and it also handles 0 for getFreeSwapSpaceSize >> ( this can occur after JDK-8242480 : Negative value may be returned by getFreeSwapSpaceSize() in the docker was pushed recently ). > > Sorry Matthias but I still see this partly as a bug in the underlying > container support code. If you get the warning then what does > getTotalSwapSpaceSize() return? Surely it must be specified to return 0 > or some such sentinel in such a case? > > Other comments: > > ! try { > ! > out.shouldMatch("OperatingSystemMXBean\\.getFreeSwapSpaceSize: > [1-9][0-9]+"); > ! } catch(RuntimeException ex) { // allow 0, see 8242480 > ! > out.shouldContain("OperatingSystemMXBean.getFreeSwapSpaceSize: 0"); > ! } > > can't you just change the original regex to match >= 0? > > Style nit: avoid historical comments like "see 8242480" and "since > 8231111 ...". > > Thanks, > David > >> Best regards, Matthias >> >> -----Original Message----- >> From: Baesken, Matthias >> Sent: Mittwoch, 13. Mai 2020 10:11 >> To: 'David Holmes' ; 'hotspot-dev at openjdk.java.net' >> Subject: RE: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java >> >> >>> But according to the comment immediately before your changes: >>> >>> // in case of warnings like : "Your kernel does not support swap limit >>> capabilities or the cgroup is not mounted. Memory limited without swap." >>> // the getTotalSwapSpaceSize does not return the expected result, but 0 >>> >>> we should be returning zero. So this seems to be a bug in the >>> implementation not in the test. >> >> >> Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values >> that are normally expected. >> Previously we indeed got "0" . >> Probably the comment should be deleted . >> >> Best regards, Matthias >> From nick.gasson at arm.com Mon May 18 07:10:57 2020 From: nick.gasson at arm.com (Nick Gasson) Date: Mon, 18 May 2020 15:10:57 +0800 Subject: [aarch64-port-dev ] AArch64: Neoverse N1 atomic operations and -moutline-atomics In-Reply-To: References: <3a5e43e7-2039-8574-7470-5827267fb88d@redhat.com> Message-ID: <854ksd3egu.fsf@arm.com> On 05/05/20 19:40 pm, Andrew Haley wrote: > On 5/5/20 12:32 PM, Andrew Haley wrote: >> This is, as you can see, horrible, but but thankfully we don't have to >> change AArch64 HotSpot because we use GCC's atomic builtins everywhere >> in C++ code. All we'll have to do is compile HotSpot with a new GCC >> and we'll be done. > > Having said that, we should fix the AArch64 HotSpot build so that we > use the -moutline-atomics option whenever it's available. I've done some profiling of SPECjbb on N1. Building with -moutline-atomics gives around +4% on critical-jOPS and +0.2% on max-jOPS averaged over several runs. However enabling this globally will cause a slight regression on machines that don't support the LSE extension due to the overhead of the function call and branch (~1% for critical-jOPS on one machine). Question is whether it's best to enable this by default in OpenJDK or leave it to downstream distributions to decide? -- Nick From stefan.karlsson at oracle.com Mon May 18 08:17:23 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Mon, 18 May 2020 10:17:23 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> Message-ID: <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> On 2020-05-15 20:39, Kim Barrett wrote: >> On May 15, 2020, at 1:46 PM, Stefan Karlsson wrote: >> >> Hi all, >> >> Please review this patch to turn off -XX:+UseNUMAInterleaving as long as Windows GDI functions don't work with memory that crosses memory reservation boundaries. >> >> https://cr.openjdk.java.net/~stefank/8245002/webrev.00.cleanup >> https://cr.openjdk.java.net/~stefank/8245002/webrev.01.workaround >> https://cr.openjdk.java.net/~stefank/8245002/webrev.all >> >> https://bugs.openjdk.java.net/browse/JDK-8245002 > ------------------------------------------------------------------------------ > src/hotspot/os/windows/os_windows.cpp > 2753 void* verify_mem = ::malloc(4 * 1024); > > There's no corresponding free. Fixed. > > ------------------------------------------------------------------------------ > src/hotspot/os/windows/os_windows.cpp > 2776 assert(false, "Reservation failed"); > 2791 assert(false, "Failed to commit memory"); > 2802 assert(false, "Failed to commit memory"); > > These look like leftover debugging code? Not really. I left them in place so that it we would fail early in our testing, if we couldn't even create a few pages. However, I realize that this might be more problematic when we start to use this function for large pages, so I'll remove them. Thanks, StefanK > > ------------------------------------------------------------------------------ > From stefan.karlsson at oracle.com Mon May 18 08:17:31 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Mon, 18 May 2020 10:17:31 +0200 Subject: RFR: 8245000: Windows GDI functions don't support large pages In-Reply-To: References: Message-ID: <2d6b8bdf-493a-5c6f-9372-46537ca77451@oracle.com> On 2020-05-15 20:26, Kim Barrett wrote: >> On May 15, 2020, at 1:47 PM, Stefan Karlsson wrote: >> >> Hi all, >> >> Please review this patch to turn off -XX:+UseLargePages as long as Windows GDI functions don't work with large pages. >> >> https://cr.openjdk.java.net/~stefank/8245000/webrev.00.cleanup >> https://cr.openjdk.java.net/~stefank/8245000/webrev.01.workaround >> https://cr.openjdk.java.net/~stefank/8245000/webrev.all/ > A couple of things that may or may not need to be addressed. > Otherwise, looks good to me. > > ------------------------------------------------------------------------------ > src/hotspot/os/windows/os_windows.cpp > 3118 if (_large_page_size > default_page_size) { > ... > 3124 UseLargePages = _large_page_size != 0; > > Should UseLargePages be true if _large_page_size > 0 but < default_page_size? > (Assuming that's actually possible, but I see nothing in this code > that makes that impossible. The OS might...) > > The old code did the same thing, so if it's a bug, it's been there all along. The code that sets up _lage_page_size uses GetLargePageMinimum, which sets the size to the system's large page size or 0. It's not explicitly stated that it's not < default_page_size (4 * 1024), but it would really surprise me if it was: https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getlargepageminimum It also uses LargePageSizeInBytes, but only if it is larger than, and aligned to, the given GetLargePageMinimum value. So, I think we are OK here. > > ------------------------------------------------------------------------------ > src/hotspot/os/windows/os_windows.cpp > 2920 if (!gdi_can_use_split_reservation_memory(UseLargePages, min_interleave_granularity)) { > 2921 WARN("Windows GDI cannot handle split reservations."); > 2922 WARN("...Ignoring UseNUMAInterleaving flag."); > > If the failure is due to UseLargePages, it might be worth mentioning > that in the warning. We've already done the large pages checks when this code runs. If we can't use large pages, the UseLargePages flag will have been turned off at this point. Thanks, StefanK > > ------------------------------------------------------------------------------ > From christian.hagedorn at oracle.com Mon May 18 08:56:22 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Mon, 18 May 2020 10:56:22 +0200 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> <412316cc-fb7e-f360-5a76-78e98ddf2890@oracle.com> <459a6a6f-5c40-c6d2-af3b-9c76b8a4f57e@oracle.com> Message-ID: Hi David On 16.05.20 06:01, David Holmes wrote: > On 16/05/2020 12:06 am, Christian Hagedorn wrote: >> While working on the replacement of Verbose by Unified Logging, I've >> seen that we should also fix this bug for ARM and S390 (PPC and SPARC >> already have the ResourceMark set correctly and Aarch64 is not >> implementing trace_method_handle_stub()). I included the necessary >> changes in a new webrev: >> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.02/ > > src/hotspot/cpu/ppc/methodHandles_ppc.cpp > src/hotspot/cpu/x86/methodHandles_x86.cpp > > You should keep the existing block so that the scope of the > PRESERVE_EXCEPTION_MARK is not changed. All this is needed is to move > the ResourceMark. I see, I fixed that. > runtime/logging/TestMethodHandlesVerbose.java > > Note that there is no need to declare the test in the runtime.logging > package. We generally do not use packages for regression tests. I adjusted the test to not use a package. >> Would be good if anybody could also have a quick look at these minor >> changes for ARM and S390 and verify them (I have only built and tested >> it on x86): > > You say ARM and S390 but the webrev only has changes for ARM and PPC. Oh, sorry. Was meant to say ARM and PPC! I included the changes in a new webrev. I also moved the ResourceMark on ARM to the start of the block while keeping the scope for PRESERVE_EXCEPTION_MARK as done for x86 and PPC now: http://cr.openjdk.java.net/~chagedorn/8244946/webrev.03/ Best regards, Christian > Thanks, > David > ----- > >> >> Thank you! >> >> Best regards, >> Christian >> >> On 15.05.20 15:08, Christian Hagedorn wrote: >>> On 15.05.20 14:56, coleen.phillimore at oracle.com wrote: >>>> On 5/15/20 8:06 AM, Christian Hagedorn wrote: >>>>> Hi Coleen >>>>> >>>>> On 15.05.20 12:38, coleen.phillimore at oracle.com wrote: >>>>>> On 5/15/20 4:03 AM, Christian Hagedorn wrote: >>>>>>> On 15.05.20 00:38, David Holmes wrote: >>>>>>>> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >>>>>>>>> >>>>>>>>> I thought that this code under Verbose should have been changed >>>>>>>>> to use logging with log_level = verbose.? We were trying to >>>>>>>>> minimize/remove the Verbose flag. >>>>>>>>> >>>>>>>>> On 5/14/20 9:50 AM, David Holmes wrote: >>>>>>>>>> +1 >>>>>>>>>> >>>>>>>>>> But I'd be inclined to generalize the test to run: >>>>>>>>>> >>>>>>>>>> -Xlog:all=trace -XX:+Verbose >>>>>>>>> >>>>>>>>> Don't do this, it would be tons of output and make the test >>>>>>>>> unusable if it fails. >>>>>>>> >>>>>>>> The only way the test can fail is if it crashes. >>>>>>>> >>>>>>>> But we would want a level of indirection so that the test log >>>>>>>> doesn't contain the full output ~11000 lines of text. >>>>>>> >>>>>>> Could we still generalize the test but redirect the output to a >>>>>>> file with: >>>>>>> >>>>>>> -Xlog:all=trace:file=output.txt -XX:+Verbose ? >>>>>>> >>>>>>> When I just immediately return from trace_method_handle_stub() >>>>>>> without printing anything, the above command will not print >>>>>>> anything to the console. So, it would print the same amount of >>>>>>> lines to the console as the current option in webrev.01 with >>>>>>> -XX:+Verbose -Xlog:methodhandles. >>>>>> >>>>>> I'm confused.? Does the test crash with -Xlog:methodhandles >>>>>> -XX:+Verbose?? If so, that's all that should be needed.? If it >>>>>> needs -Xlog:all=trace, then don't add this test.? Even if it >>>>>> writes to an output file, it would be huge and affect test >>>>>> execution, particularly if more logging is added. >>>>> >>>>> I could already reproduce it locally on my Linux machine with >>>>> -Xlog:methodhandles -XX:+Verbose. From what I understood, I thought >>>>> that David is just talking about having a more general log test >>>>> that covers the entire logging together with -XX:+Verbose for a >>>>> HelloWorld program. But as you said, that would have some bad >>>>> effects on test execution. However, such a test is not required for >>>>> this bug to be triggered. >>>> >>>> Good! This looks good to me then. >>> >>> Thank you Coleen for your review! >>> >>>> Thank you for filing an RFE to do more logging with Unified Logging. >>> >>> No problem, I will take a look at it next. >>> >>> Best regards, >>> Christian >>> >>>>> >>>>>>> In a next step we could file an RFE to replace Verbose with >>>>>>> unified logging in trace_method_handle_stub() which currently is >>>>>>> only using tty. Even when using -Xlog:methodhandles alone without >>>>>>> -XX:+Verbose, there is still some printing done in >>>>>>> trace_method_handle_stub() to tty by >>>>>>> >>>>>>> ?500?? tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, >>>>>>> ?501???????????????? adaptername, mh_reg_name, >>>>>>> ?502???????????????? p2i(mh), p2i(entry_sp)); >>>>>>> >>>>>>> which is probably also not desired, as it is guarded by >>>>>>> log_is_enabled(Info, methodhandles) earlier [1]. This could also >>>>>>> be fixed in this follow-up RFE. >>>>>> >>>>>> Please, yes, this can be fixed with a follow-up RFE. >>>>> >>>>> Great, I filed an RFE for it [1]. >>>>> >>>>> Best regards, >>>>> Christian >>>>> >>>>> >>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8245107 >>>>> >>>>> >>>>>>>>>> Curiously I couldn't reproduce the crash that this bug is >>>>>>>>>> about. But the hs_err file speaks for itself. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> David >>>>>>>>>> >>>>>>>>>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>>>>>>>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>>>>>>>>> Hi David, hi Daniel >>>>>>>>>>>> >>>>>>>>>>>> Thank you for your reviews! >>>>>>>>>>>> >>>>>>>>>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>>>>>>>>> ???? Not your bug, but the comment has two typos: >>>>>>>>>>>>> >>>>>>>>>>>>> ???????? s/by safer/but safer/ >>>>>>>>>>>>> ???????? s/unexpensive/inexpensive/ >>>>>>>>>>>> >>>>>>>>>>>> Thanks, fixed it. >>>>>>>>>>>> >>>>>>>>>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>>>>>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>>>>>>>>> >>>>>>>>>>>>> Given all of the code is inside: >>>>>>>>>>>>> >>>>>>>>>>>>> ??504?? if (Verbose) { >>>>>>>>>>>>> >>>>>>>>>>>>> I would just move the ResourceMark to the top of that >>>>>>>>>>>>> block. We are going to use it unconditionally, and we are >>>>>>>>>>>>> not releasing it until the end of the method regardless. >>>>>>>>>>>>> That avoids the potential concern Dan raised about >>>>>>>>>>>>> extending the scope of the PRESERVE_EXCEPTION_MARK. >>>>>>>>>>>> >>>>>>>>>>>> This sounds reasonable. I moved it to the top of that block. >>>>>>>>>>>> >>>>>>>>>>>>>> There are other places like Klass::print_on(outputStream*) >>>>>>>>>>>>>> where this is still the case. Should we file an RFE to >>>>>>>>>>>>>> clean these up? >>>>>>>>>>>>> >>>>>>>>>>>>> Obviously we don't have any tests running in a mode that >>>>>>>>>>>>> causes this code to be executed, so yes it is probably a >>>>>>>>>>>>> good idea to take a closer look at other call-sites and >>>>>>>>>>>>> file a RFE if needed. >>>>>>>>>>>> >>>>>>>>>>>> Yes, I filed one to take a closer look at these [1]. >>>>>>>>>>>> >>>>>>>>>>>> I included the changes in a new webrev: >>>>>>>>>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>>>>>>>>> >>>>>>>>>>> Thumbs up. >>>>>>>>>>> >>>>>>>>>>> Dan >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Best regards, >>>>>>>>>>>> Christian >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>> >>>> From aph at redhat.com Mon May 18 09:53:25 2020 From: aph at redhat.com (Andrew Haley) Date: Mon, 18 May 2020 10:53:25 +0100 Subject: [aarch64-port-dev ] AArch64: Neoverse N1 atomic operations and -moutline-atomics In-Reply-To: <854ksd3egu.fsf@arm.com> References: <3a5e43e7-2039-8574-7470-5827267fb88d@redhat.com> <854ksd3egu.fsf@arm.com> Message-ID: <280d2657-7997-1dac-f370-8ff41a0b3035@redhat.com> On 5/18/20 8:10 AM, Nick Gasson wrote: > I've done some profiling of SPECjbb on N1. Building with > -moutline-atomics gives around +4% on critical-jOPS and +0.2% on > max-jOPS averaged over several runs. However enabling this globally will > cause a slight regression on machines that don't support the LSE > extension due to the overhead of the function call and branch (~1% for > critical-jOPS on one machine). Mmm. That's not a nice decision to have to make. > Question is whether it's best to enable this by default in OpenJDK or > leave it to downstream distributions to decide? I don't really understand why there is a conditional branch there. They could just do the test at startup and patch the code. Maybe it'd hurt DSO startup time or something? Maybe they don't like the idea of temporarily enabling exec privilege on a page of writable data. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From sgehwolf at redhat.com Mon May 18 10:08:38 2020 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Mon, 18 May 2020 12:08:38 +0200 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: <7201E4D5-96AA-47C1-B0B7-A19E8A0F198B@tencent.com> References: <7201E4D5-96AA-47C1-B0B7-A19E8A0F198B@tencent.com> Message-ID: <51519874123ae26a4d3ebf4863d80e682c13b989.camel@redhat.com> Hi, On Mon, 2020-05-18 at 06:15 +0000, jiefu(??) wrote: > Hi David, > > This bug should be introduced by JDK-8242480. > I'm sorry for that. > > And I think the logic in src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java should be fixed. > > In my fix for JDK-8242480, I had ignored the case for kernels which don't support "--memory-swap 150M". > In that case, containerMetrics.getMemoryAndSwapLimit()[1] will return -1. It appears so that containerMetrics.getMemoryAndSwapLimit() returned 0 prior JDK-8231111 and -1 after. I'm not convinced JDK-8242480 introduced this. > I didn't notice this bug because all our kernels support --memory-swap. > Fortunately, Matthias found this bug and will fix it. I'm still not sure what's happening in Matthias' case on that system. It would be best to first try to understand the actual problem better. The info I have is: - It's a cgroup v1 system - The system now returns large values for OperatingSystemMXBean.getTotalSwapSpaceSize() and OperatingSystemMXBean.getFreeSwapSpaceSize() It would be good to know what the actual return values are for the container metrics on this system with swap limit disabled at kernel level. And also whether or not relevant cgroup files exist on such a system (within docker). FWIW, parts of this work-around got added by JDK-8236617. Looking at the bug it was observed that: containerMetrics.getMemoryAndSwapLimit() == 0 and containerMetrics.getMemoryLimit() returned a large value. So then with JDK-8231111 I suspect containerMetrics.getMemoryAndSwapLimit() now returns -1 and, thus, getTotalSwapSpaceSize0() is being used instead of the container values. So far the theory, I'd like to gather more data before we come up with another patch, though. In JDK-8236617 it wasn't investigated why containerMetrics.getMemoryAndSwapLimit() returned 0 on such a system. It would be good to know why. Could it be that file 'memory.memsw.limit_in_bytes' doesn't exist on such a system? Unfortunately, I don't know what config one needs in order to reproduce so we'll have to rely on Matthias to gather the info. Thanks, Severin > Thanks a lot. > Best regards, > Jie > > [1] http://hg.openjdk.java.net/jdk/jdk/file/a95911422005/src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java#l73 > > ?On 2020/5/18, 1:48 PM, "David Holmes" wrote: > > Hi Matthias, > > On 16/05/2020 12:59 am, Baesken, Matthias wrote: > > Hello David , I get the warning always on this linux machine (I think it depends on machine setup and/or kernel version). > > > >> Surely it must be specified to return 0 or some such sentinel in such a case? > > > > The doc just says this : > > > > 62 /** > > 63 * Returns the total amount of swap space in bytes. > > 64 * > > 65 * @return the total amount of swap space in bytes. > > 66 */ > > 67 public long getTotalSwapSpaceSize(); > > > > ( however it returned previously 0 before the latest changes in the container support code came in ). > > That sounds like a bug was introduced to me then. > > > Regarding the getFreeSwapSpaceSize issue , Jie provided a change to src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java > > Because of this I could remove the try-construct . > > > > New webrev : > > http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.3/ > > I'm quite confused by your changes. Your first version simply adjusted > the test to check for the warning. But now you are also changing the > main logic. I can't assess the validity of what you are doing sorry. > > I've cc'd Severin and Bob as I think they need to evaluate what the real > problem is here and when it was introduced. > > David > ---- > > > Best regards, Matthias > > > > > > > > -----Original Message----- > > From: David Holmes > > Sent: Donnerstag, 14. Mai 2020 07:02 > > To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' > > Cc: jiefu(??) > > Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > > > On 14/05/2020 12:28 am, Baesken, Matthias wrote: > >> Hello, here is a new webrev removing the comment : > >> > >> http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.1/ > >> > >> It removes the comment , and it also handles 0 for getFreeSwapSpaceSize > >> ( this can occur after JDK-8242480 : Negative value may be returned by getFreeSwapSpaceSize() in the docker was pushed recently ). > > > > Sorry Matthias but I still see this partly as a bug in the underlying > > container support code. If you get the warning then what does > > getTotalSwapSpaceSize() return? Surely it must be specified to return 0 > > or some such sentinel in such a case? > > > > Other comments: > > > > ! try { > > ! > > out.shouldMatch("OperatingSystemMXBean\\.getFreeSwapSpaceSize: > > [1-9][0-9]+"); > > ! } catch(RuntimeException ex) { // allow 0, see 8242480 > > ! > > out.shouldContain("OperatingSystemMXBean.getFreeSwapSpaceSize: 0"); > > ! } > > > > can't you just change the original regex to match >= 0? > > > > Style nit: avoid historical comments like "see 8242480" and "since > > 8231111 ...". > > > > Thanks, > > David > > > >> Best regards, Matthias > >> > >> -----Original Message----- > >> From: Baesken, Matthias > >> Sent: Mittwoch, 13. Mai 2020 10:11 > >> To: 'David Holmes' ; 'hotspot-dev at openjdk.java.net' > >> Subject: RE: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > >> > >> > >>> But according to the comment immediately before your changes: > >>> > >>> // in case of warnings like : "Your kernel does not support swap limit > >>> capabilities or the cgroup is not mounted. Memory limited without swap." > >>> // the getTotalSwapSpaceSize does not return the expected result, but 0 > >>> > >>> we should be returning zero. So this seems to be a bug in the > >>> implementation not in the test. > >> > >> > >> Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values > >> that are normally expected. > >> Previously we indeed got "0" . > >> Probably the comment should be deleted . > >> > >> Best regards, Matthias > >> > > > From per.liden at oracle.com Mon May 18 10:22:29 2020 From: per.liden at oracle.com (Per Liden) Date: Mon, 18 May 2020 12:22:29 +0200 Subject: RFR: 8245098: Make SafeFetch32/N available earlier In-Reply-To: <1F383A58-BDF8-4197-BD58-1E7530E4411D@oracle.com> References: <1F383A58-BDF8-4197-BD58-1E7530E4411D@oracle.com> Message-ID: <252be529-2ad1-c870-667c-df376c14cdd7@oracle.com> On 5/15/20 6:07 PM, Kim Barrett wrote: >> On May 15, 2020, at 8:03 AM, Per Liden wrote: >> >> We have a need for SafeFetch32 during initializing of ZGC (see [1]). However, the SafeFetch32/N stubs are currently generated too late, in the stubRoutines_init2() phase, which is after the GC has been initialized. I propose we move the initialization of these stubs to the stubRoutines_init1() phase, which is before the GC has been initialized, to make them available during GC initialization. >> >> Please note that this patch touches platforms I can't easily test (x86-32, arm, ppc, s390). If you have access to one of those machines, I would appreciate if you could do some light verification. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8245098 >> Webrev: http://cr.openjdk.java.net/~pliden/8245098/webrev.0 >> Testing: Tier1-3 on all Oracle supported platforms. >> >> /Per >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8245106 > > Looks good. > Thanks Kim! /Per From per.liden at oracle.com Mon May 18 10:23:25 2020 From: per.liden at oracle.com (Per Liden) Date: Mon, 18 May 2020 12:23:25 +0200 Subject: RFR: 8245098: Make SafeFetch32/N available earlier In-Reply-To: <9545806d-87d8-e610-ae20-a8228b44a97d@oracle.com> References: <9501bdb6-c06e-5b78-0017-2de71a97c48b@oracle.com> <67868905-CA21-45EA-BE8E-8AE533185211@oracle.com> <9545806d-87d8-e610-ae20-a8228b44a97d@oracle.com> Message-ID: <445cf7ec-92fa-4068-a6a3-660e46c4a1a1@oracle.com> On 5/18/20 2:50 AM, David Holmes wrote: > On 17/05/2020 1:48 am, Kim Barrett wrote: >>> On May 16, 2020, at 12:27 AM, David Holmes >>> wrote: >>> >>> Hi Per, >>> >>> On 15/05/2020 10:03 pm, Per Liden wrote: >>>> We have a need for SafeFetch32 during initializing of ZGC (see [1]). >>>> However, the SafeFetch32/N stubs are currently generated too late, >>>> in the stubRoutines_init2() phase, which is after the GC has been >>>> initialized. I propose we move the initialization of these stubs to >>>> the stubRoutines_init1() phase, which is before the GC has been >>>> initialized, to make them available during GC initialization. >>> >>> generate_safefetch includes use of a StubCodeMark. And its destructor >>> contains: >>> >>> ? Forte::register_stub(_cdesc->name(), _cdesc->begin(), _cdesc->end()); >>> ? if (JvmtiExport::should_post_dynamic_code_generated()) { >>> ??? JvmtiExport::post_dynamic_code_generated(_cdesc->name(), >>> _cdesc->begin(), _cdesc->end()); >>> ? } >>> >>> It is not at all obvious that these actions are still okay at an >>> earlier stage of the initialization sequence. I don't know what tests >>> we may have for this, nor in what tier they may run, but I suggest >>> running more extensive JVM TI testing, and try to determine how the >>> Forte code is used. >> >> There are lots of other stub generators called earlier in >> generate_initial and which also >> include uses of StubCodeMark. > > Yep - my bad. I looked at generate_throw_exception as a close-by > non-trivial example and noticed it did not. I should have checked some > of the others. Thanks for reviewing, David! /Per From kim.barrett at oracle.com Mon May 18 11:56:17 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Mon, 18 May 2020 07:56:17 -0400 Subject: RFR: 8245000: Windows GDI functions don't support large pages In-Reply-To: <2d6b8bdf-493a-5c6f-9372-46537ca77451@oracle.com> References: <2d6b8bdf-493a-5c6f-9372-46537ca77451@oracle.com> Message-ID: > On May 18, 2020, at 4:17 AM, Stefan Karlsson wrote: > > On 2020-05-15 20:26, Kim Barrett wrote: >>> On May 15, 2020, at 1:47 PM, Stefan Karlsson wrote: >>> >>> Hi all, >>> >>> Please review this patch to turn off -XX:+UseLargePages as long as Windows GDI functions don't work with large pages. >>> >>> https://cr.openjdk.java.net/~stefank/8245000/webrev.00.cleanup >>> https://cr.openjdk.java.net/~stefank/8245000/webrev.01.workaround >>> https://cr.openjdk.java.net/~stefank/8245000/webrev.all/ >> A couple of things that may or may not need to be addressed. >> Otherwise, looks good to me. >> >> ------------------------------------------------------------------------------ >> src/hotspot/os/windows/os_windows.cpp >> 3118 if (_large_page_size > default_page_size) { >> ... >> 3124 UseLargePages = _large_page_size != 0; >> >> Should UseLargePages be true if _large_page_size > 0 but < default_page_size? >> (Assuming that's actually possible, but I see nothing in this code >> that makes that impossible. The OS might...) >> >> The old code did the same thing, so if it's a bug, it's been there all along. > > The code that sets up _lage_page_size uses GetLargePageMinimum, which sets the size to the system's large page size or 0. It's not explicitly stated that it's not < default_page_size (4 * 1024), but it would really surprise me if it was: > > https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getlargepageminimum > > It also uses LargePageSizeInBytes, but only if it is larger than, and aligned to, the given GetLargePageMinimum value. > > So, I think we are OK here. Thanks for the explanation and the doc pointer. Agreed this is okay. >> src/hotspot/os/windows/os_windows.cpp >> 2920 if (!gdi_can_use_split_reservation_memory(UseLargePages, min_interleave_granularity)) { >> 2921 WARN("Windows GDI cannot handle split reservations."); >> 2922 WARN("...Ignoring UseNUMAInterleaving flag."); >> >> If the failure is due to UseLargePages, it might be worth mentioning >> that in the warning. > > We've already done the large pages checks when this code runs. If we can't use large pages, the UseLargePages flag will have been turned off at this point. Oh, of course. Looks good. From stefan.karlsson at oracle.com Mon May 18 12:08:31 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Mon, 18 May 2020 14:08:31 +0200 Subject: RFR: 8245000: Windows GDI functions don't support large pages In-Reply-To: References: <2d6b8bdf-493a-5c6f-9372-46537ca77451@oracle.com> Message-ID: Thanks for reviewing! StefanK On 2020-05-18 13:56, Kim Barrett wrote: >> On May 18, 2020, at 4:17 AM, Stefan Karlsson wrote: >> >> On 2020-05-15 20:26, Kim Barrett wrote: >>>> On May 15, 2020, at 1:47 PM, Stefan Karlsson wrote: >>>> >>>> Hi all, >>>> >>>> Please review this patch to turn off -XX:+UseLargePages as long as Windows GDI functions don't work with large pages. >>>> >>>> https://cr.openjdk.java.net/~stefank/8245000/webrev.00.cleanup >>>> https://cr.openjdk.java.net/~stefank/8245000/webrev.01.workaround >>>> https://cr.openjdk.java.net/~stefank/8245000/webrev.all/ >>> A couple of things that may or may not need to be addressed. >>> Otherwise, looks good to me. >>> >>> ------------------------------------------------------------------------------ >>> src/hotspot/os/windows/os_windows.cpp >>> 3118 if (_large_page_size > default_page_size) { >>> ... >>> 3124 UseLargePages = _large_page_size != 0; >>> >>> Should UseLargePages be true if _large_page_size > 0 but < default_page_size? >>> (Assuming that's actually possible, but I see nothing in this code >>> that makes that impossible. The OS might...) >>> >>> The old code did the same thing, so if it's a bug, it's been there all along. >> The code that sets up _lage_page_size uses GetLargePageMinimum, which sets the size to the system's large page size or 0. It's not explicitly stated that it's not < default_page_size (4 * 1024), but it would really surprise me if it was: >> >> https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getlargepageminimum >> >> It also uses LargePageSizeInBytes, but only if it is larger than, and aligned to, the given GetLargePageMinimum value. >> >> So, I think we are OK here. > Thanks for the explanation and the doc pointer. Agreed this is okay. > >>> src/hotspot/os/windows/os_windows.cpp >>> 2920 if (!gdi_can_use_split_reservation_memory(UseLargePages, min_interleave_granularity)) { >>> 2921 WARN("Windows GDI cannot handle split reservations."); >>> 2922 WARN("...Ignoring UseNUMAInterleaving flag."); >>> >>> If the failure is due to UseLargePages, it might be worth mentioning >>> that in the warning. >> We've already done the large pages checks when this code runs. If we can't use large pages, the UseLargePages flag will have been turned off at this point. > Oh, of course. > > Looks good. > From coleen.phillimore at oracle.com Mon May 18 12:46:06 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 18 May 2020 08:46:06 -0400 Subject: RFR (S) 8242424: Deprecate InitialBootClassLoaderMetaspaceSize In-Reply-To: References: <00da20b6-59a4-67ec-9023-4adcd3154a22@oracle.com> <13c0f52e-3ca9-6855-8a75-36da227dbdeb@oracle.com> Message-ID: Hi, Dan and Thomas, thank you for looking at the patch. I ran tier1-3 which runs these tests with no failures.? I think we should continue testing with these options and examine the tests for rewriting or removal once we deprecate the options.? Thank you for pointing this out though.? I'll add this to the RFE (to be filed) for making these options obsolete. Thanks, Coleen On 5/17/20 3:25 AM, Thomas St?fe wrote: > Hi, > > I ran the following tests: > > hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeTest.java > hotspot/jtreg/runtime/cds/MaxMetaspaceSize.java > hotspot/jtreg/runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java > hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java > hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > hotspot/jtreg/gc/TestSystemGC.java > > and all of hotspot/jtreg/vmTestbase/metaspace/... > > successfully with Coleens patch. > > Cheers, Thomas > > > > On Sat, May 16, 2020 at 3:27 PM Daniel D. Daugherty > > wrote: > > On 5/15/20 6:55 PM, coleen.phillimore at oracle.com > wrote: > > Also: > > 8243147: Deprecate UseLargePagesInMetaspace > > > > Summary: mark these options for deprecation > > > > Tested with tier1-3. > > > > open webrev at > http://cr.openjdk.java.net/~coleenp/2020/8242424.01/webrev > > src/hotspot/share/runtime/arguments.cpp > ???? No comments. > > src/hotspot/share/runtime/globals.hpp > ???? No comments. > > test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java > ???? No comments. > > Thumbs up on this part, but I see these options mentioned in tests: > > $ egrep -r > 'UseLargePagesInMetaspace|InitialBootClassLoaderMetaspaceSize' > open/test > closed/test > open/test/hotspot/jtreg/vmTestbase/metaspace/gc/MetaspaceBaseGC.java: > useLargepages = PAGE_SIZE > 1_000_000 && > !vmArgs.contains("-XX:-UseLargePagesInMetaspace"); > open/test/hotspot/jtreg/runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java: > > excludeTestMaxRange("InitialBootClassLoaderMetaspaceSize"); > open/test/hotspot/jtreg/runtime/cds/MaxMetaspaceSize.java: > processArgs.add("-XX:InitialBootClassLoaderMetaspaceSize=1m"); > open/test/hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java: > > "-XX:+UseLargePages", "-XX:+UseLargePagesInMetaspace"); > open/test/hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java: > > "-XX:+UseLargePages", "-XX:+UseLargePagesInMetaspace"); > open/test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeTest.java: > "-XX:InitialBootClassLoaderMetaspaceSize=4195328", > open/test/hotspot/jtreg/gc/TestSystemGC.java: * @run main/othervm > -XX:+UseLargePages -XX:+UseLargePagesInMetaspace gc.TestSystemGC > > Do you know of the deprecated message will affect how these tests run? > > Dan > > > > > bug link https://bugs.openjdk.java.net/browse/JDK-8242424 > > bug link https://bugs.openjdk.java.net/browse/JDK-8243147 > > > > Thanks, > > Coleen > From coleen.phillimore at oracle.com Mon May 18 12:51:52 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Mon, 18 May 2020 08:51:52 -0400 Subject: RFR (S) 8242424: Deprecate InitialBootClassLoaderMetaspaceSize In-Reply-To: References: <00da20b6-59a4-67ec-9023-4adcd3154a22@oracle.com> <13c0f52e-3ca9-6855-8a75-36da227dbdeb@oracle.com> Message-ID: On 5/18/20 8:46 AM, coleen.phillimore at oracle.com wrote: > > Hi, Dan and Thomas, thank you for looking at the patch. > > I ran tier1-3 which runs these tests with no failures.? I think we > should continue testing with these options and examine the tests for > rewriting or removal once we deprecate the options.? Thank you for > pointing this out though.? I'll add this to the RFE (to be filed) for > making these options obsolete. Sorry I meant "obsolete these options" above. Coleen > > Thanks, > Coleen > > On 5/17/20 3:25 AM, Thomas St?fe wrote: >> Hi, >> >> I ran the following tests: >> >> hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeTest.java >> hotspot/jtreg/runtime/cds/MaxMetaspaceSize.java >> hotspot/jtreg/runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java >> >> hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java >> hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java >> hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java >> hotspot/jtreg/gc/TestSystemGC.java >> >> and all of hotspot/jtreg/vmTestbase/metaspace/... >> >> successfully with Coleens patch. >> >> Cheers, Thomas >> >> >> >> On Sat, May 16, 2020 at 3:27 PM Daniel D. Daugherty >> > >> wrote: >> >> ??? On 5/15/20 6:55 PM, coleen.phillimore at oracle.com >> ??? wrote: >> ??? > Also: >> ??? > 8243147: Deprecate UseLargePagesInMetaspace >> ??? > >> ??? > Summary: mark these options for deprecation >> ??? > >> ??? > Tested with tier1-3. >> ??? > >> ??? > open webrev at >> ??? http://cr.openjdk.java.net/~coleenp/2020/8242424.01/webrev >> >> ??? src/hotspot/share/runtime/arguments.cpp >> ??? ???? No comments. >> >> ??? src/hotspot/share/runtime/globals.hpp >> ??? ???? No comments. >> >> test/hotspot/jtreg/runtime/CommandLine/VMDeprecatedOptions.java >> ??? ???? No comments. >> >> ??? Thumbs up on this part, but I see these options mentioned in tests: >> >> ??? $ egrep -r >> 'UseLargePagesInMetaspace|InitialBootClassLoaderMetaspaceSize' >> ??? open/test >> ??? closed/test >> open/test/hotspot/jtreg/vmTestbase/metaspace/gc/MetaspaceBaseGC.java: >> ??? useLargepages = PAGE_SIZE > 1_000_000 && >> ??? !vmArgs.contains("-XX:-UseLargePagesInMetaspace"); >> open/test/hotspot/jtreg/runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java: >> >> ??? excludeTestMaxRange("InitialBootClassLoaderMetaspaceSize"); >> ??? open/test/hotspot/jtreg/runtime/cds/MaxMetaspaceSize.java: >> processArgs.add("-XX:InitialBootClassLoaderMetaspaceSize=1m"); >> open/test/hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java: >> >> ??? "-XX:+UseLargePages", "-XX:+UseLargePagesInMetaspace"); >> open/test/hotspot/jtreg/runtime/cds/appcds/sharedStrings/LargePages.java: >> >> ??? "-XX:+UseLargePages", "-XX:+UseLargePagesInMetaspace"); >> open/test/hotspot/jtreg/runtime/Metaspace/MaxMetaspaceSizeTest.java: >> ??? "-XX:InitialBootClassLoaderMetaspaceSize=4195328", >> ??? open/test/hotspot/jtreg/gc/TestSystemGC.java: * @run main/othervm >> ??? -XX:+UseLargePages -XX:+UseLargePagesInMetaspace gc.TestSystemGC >> >> ??? Do you know of the deprecated message will affect how these tests >> run? >> >> ??? Dan >> >> >> >> ??? > bug link https://bugs.openjdk.java.net/browse/JDK-8242424 >> ??? > bug link https://bugs.openjdk.java.net/browse/JDK-8243147 >> ??? > >> ??? > Thanks, >> ??? > Coleen >> > From thomas.stuefe at gmail.com Mon May 18 14:37:30 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Mon, 18 May 2020 16:37:30 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> Message-ID: Hi all, fourth webrev: http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ I rebased atop of changeset: 59325:1ba9a9a3f948 summary: 8244433: Remove saving of RSP in Assembler::pusha_uncached() so, atop of "8241825: Make compressed oops and compressed class pointers independent (x86_64, PPC, S390)". The changes are very small this time, see notes below: @Ioi: >Hi Thomas, >I am running your patch in out CI pipeline now. Some comments: >[2] Does JDK-8243535 exist with the current jdk/jdk repo? I assigned JDK-8243535 to me and I'm working on a patch. But I would like to keep that separate from this work, since this patch is already large. And I would like to fix JDK-8243535 after this remodel patch has been pushed, which makes some minor things easier. That would leave a small time window where NMT does mis-track class space as shared class space, but that is not a big issue, it does not even disturb any test. Which in itself is something we may want to fix, come to think about it. > [3] I think this can be put in header file. > > bool Metaspace::class_space_is_initialized() { > return _class_space_list != NULL; > } Done. > [7] argument.cpp checks this: > > if (UseSharedSpaces || DumpSharedSpaces) { > if (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { > log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for this platform, option will be ignored.", > p2i((address)SharedBaseAddress)); > FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); > } > } > > but initialize_dumptime_shared_and_meta_spaces aligns up the SharedBaseAddress before doing the assert. > > void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { > assert(DumpSharedSpaces, "should be called for dump time only"); > > const size_t reserve_alignment = MetaspaceShared::reserved_space_alignment(); > char* shared_base = (char*)align_up((char*)SharedBaseAddress, reserve_alignment); > > #ifdef _LP64 > assert(CompressedKlassPointers::is_valid_base((address)shared_base), "Sanity"); > > So theoretically shared_base may no longer be is_valid_base after the align_up. It probably won't happen, but I think the code will be > much clearer tofirst to align_up, then check for is_valid_base and reset to default, and finally assert. The problem with that is that at the time this coding runs os::init() did not yet run and we do not know yet allocation granularity. So I changed verification of SharedBaseAddress: I added the alignment as you suggested and moved it to a later time in VM initialization, to CDS initialization. >[8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the following block can be simplified: > > if (!_shared_rs.is_reserved()) { > // Get a reserved space anywhere if attaching at the SharedBaseAddress fails: > if (UseCompressedClassPointers) { > // If we need a compressed class space too, let the platform handle the reservation. > _shared_rs = Metaspace::reserve_address_space_for_compressed_classes(cds_total); > } else { > // anywhere is fine. > _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* large */, (char*)NULL); > } > } > > ... if you change the declaration to: > > static ReservedSpace reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return ReservedSpace();}); I'd like to avoid keep the method LP64 only, but I rewrote the section somewhat to be more clearer. > [9] I think the #ifdef _LP64 is not necessary: > > #ifdef _LP64 > // Some sanity checks after reserving address spaces for archives and class space. > assert(archive_space_rs.is_reserved(), "Sanity"); > if (Metaspace::using_class_space()) { > // Class space must closely follow the archive space. Both spaces must be aligned correctly. > assert(class_space_rs.is_reserved(), "A class space should have been reserved"); > assert(class_space_rs.base() >= archive_space_rs.end(), "class space should follow the cds archive space"); > assert(is_aligned(archive_space_rs.base(), MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); > assert(is_aligned(class_space_rs.base(), Metaspace::reserve_alignment()), "class space misaligned"); > } > #endif I removed the #ifdef, but replaced it with an #ifdef ASSERT. Which is also not needed, technically, but makes it clear that this section is debugging checks only. ---- @Coleen: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/ >virtualSpaceList.cpp.udiff.html > >Why did you make this change? Shouldn't the caller align it? You are right, I removed the assert. --- Thanks all for your review work, ..Thomas On Fri, May 8, 2020 at 7:37 AM Ioi Lam wrote: > Hi Thomas, > > I am running your patch in out CI pipeline now. Some comments: > > [1] Do we still need to use _class_space_list? It looks like we just have > a single range. Is _class_space_list->current_virtual_space() always the > same VS? > > if (_class_space_list != NULL) { > address base = > (address)_class_space_list->current_virtual_space()->bottom(); > address top = base + compressed_class_space_size(); > st->print("Compressed class space mapped at: " PTR_FORMAT "-" > PTR_FORMAT ", size: " SIZE_FORMAT, > p2i(base), p2i(top), top - base); > > [2] Does JDK-8243535 exist with the current jdk/jdk repo? > > // This does currently not work because rs may be the result of a split > operation > // and NMT seems not to be able to handle splits. > // See JDK-8243535. > // MemTracker::record_virtual_memory_type((address)rs.base(), mtClass); > > > [3] I think this can be put in header file. > > bool Metaspace::class_space_is_initialized() { > return _class_space_list != NULL; > } > > [4] Why does the CCS need to follow UseLargePagesInMetaspace? This is the > reason for the > gap in the following chart: > > // We do this by reserving space for the ccs behind the archives. Note > however that > // ccs follows a different alignment (Metaspace::reserve_alignment), > so there may > // be a gap between ccs and cds. > // We use a similar layout at runtime, see > reserve_address_space_for_archives(). > // > // +-- SharedBaseAddress (default = > 0x800000000) > // v > // +-..---------+---------+ ... > +----+----+----+--------+--------------------+ > // | Heap | Archive | | MC | RW | RO | [gap] | class > space | > // +-..---------+---------+ ... > +----+----+----+--------+--------------------+ > // |<-- MaxHeapSize -->| |<-- UnscaledClassSpaceMax = 4GB -->| > > > _reserve_alignment is determined here: > > void Metaspace::ergo_initialize() { > if (DumpSharedSpaces) { > // Using large pages when dumping the shared archive is currently not > implemented. > FLAG_SET_ERGO(UseLargePagesInMetaspace, false); > } > > size_t page_size = os::vm_page_size(); > if (UseLargePages && UseLargePagesInMetaspace) { > page_size = os::large_page_size(); > } > > _commit_alignment = page_size; > _reserve_alignment = MAX2(page_size, > (size_t)os::vm_allocation_granularity()); > > But when CDS is enabled, the RS is reserved without large pages, so it > should be possible to incrementally commit the CCS using just > os::vm_allocation_granularity(). > > [5] I see the aarch64 code in > Metaspace::reserve_address_space_for_compressed_classes has been changed. > Is this necessary for this RFE, or is it a separate improvement that can be > done in a different RFE? > > [6] For AARCH64, should we skip the part that reserves it right above the > heap? Or does AARCH64 always allocate the heap such that it's a preferable > address? > > // case (b) > ReservedSpace rs; > > // Try right above the Java heap... > address base = align_up(CompressedOops::end(), > Metaspace::reserve_alignment()); > assert(base != NULL, "Sanity"); > > const size_t size = align_up(CompressedClassSpaceSize, > Metaspace::reserve_alignment()); > if (CompressedKlassPointers::is_valid_base(base)) { > rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* > large */, (char*)base); > } > > // ...failing that, reserve anywhere, but let platform do optimized > placement: > if (!rs.is_reserved()) { > rs = Metaspace::reserve_address_space_for_compressed_classes(size); > } > > [7] argument.cpp checks this: > > if (UseSharedSpaces || DumpSharedSpaces) { > if > (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { > log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for > this platform, option will be ignored.", > p2i((address)SharedBaseAddress)); > FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); > } > } > > but initialize_dumptime_shared_and_meta_spaces aligns up the > SharedBaseAddress before doing the assert. > > void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { > assert(DumpSharedSpaces, "should be called for dump time only"); > > const size_t reserve_alignment = > MetaspaceShared::reserved_space_alignment(); > char* shared_base = (char*)align_up((char*)SharedBaseAddress, > reserve_alignment); > > #ifdef _LP64 > assert(CompressedKlassPointers::is_valid_base((address)shared_base), > "Sanity"); > > So theoretically shared_base may no longer be is_valid_base after the > align_up. It probably won't happen, but I think the code will be much > clearer tofirst to align_up, then check for is_valid_base and reset to > default, and finally assert. > > [8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the > following block can be simplified: > > if (!_shared_rs.is_reserved()) { > // Get a reserved space anywhere if attaching at the SharedBaseAddress > fails: > if (UseCompressedClassPointers) { > // If we need a compressed class space too, let the platform handle > the reservation. > _shared_rs = > Metaspace::reserve_address_space_for_compressed_classes(cds_total); > } else { > // anywhere is fine. > _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* > large */, (char*)NULL); > } > } > > ... if you change the declaration to: > > static ReservedSpace reserve_address_space_for_compressed_classes(size_t > size) NOT_LP64({ return ReservedSpace();}); > > [9] I think the #ifdef _LP64 is not necessary: > > #ifdef _LP64 > // Some sanity checks after reserving address spaces for archives and > class space. > assert(archive_space_rs.is_reserved(), "Sanity"); > if (Metaspace::using_class_space()) { > // Class space must closely follow the archive space. Both spaces > must be aligned correctly. > assert(class_space_rs.is_reserved(), "A class space should have been > reserved"); > assert(class_space_rs.base() >= archive_space_rs.end(), "class space > should follow the cds archive space"); > assert(is_aligned(archive_space_rs.base(), > MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); > assert(is_aligned(class_space_rs.base(), > Metaspace::reserve_alignment()), "class space misaligned"); > } > #endif > > The rest of the code looks OK to me, but I may take a look at it again > after getting more sleep :-) > > Thanks > - Ioi > > > > > On 5/7/20 7:21 AM, Thomas St?fe wrote: > > Hi all, > > please take a look at the third iteration of this change: > > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ > > Changes in this version: > > - at the request of Coleen (sorry I did not see your post earlier) I > removed all platform dependent files and put the platform dependent coding > back to metaspace.cpp and compressedOops.cpp to ease the review pain. I > used plain platform defines though (#ifdef AARCH64) instead of hiding them > behind another macro to make things clearer. Note that I still intent to > put this code away into the platform corners but will do so in a follow up > RFE. > > - I reinstated, in a fashion, the special handling of reservations on > Windows. On all platforms we reserve address space to map the archive files > in with a subsequent mapping operation. However, on Windows, we cannot use > MapViewOfFile() into an existing mapping. So I remove the mapping again > before mapping the archives - see comment in code for details. > > All CI tests at SAP run through without problems, including on Win64 and > aarch64, but I would be happy if others were to run test too. > > Thank you, Thomas > > On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe > wrote: > >> Hi all, >> >> Could I have reviews for the following proposal of reworking cds/class >> space reservation? >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >> >> Webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >> >> (Many thanks to Ioi Lam for so patiently explaining CDS internals to me, >> and to Andrew Haley and Nick Gasson for help with aarch64!) >> >> Reservation of the compressed class space is needlessly complicated and >> has some minor issues. It can be simplified and made clearer. >> >> The complexity stems from the fact that this area lives at the >> intersection of two to three sub systems, depending on how one counts. >> Metaspace, CDS, and the platform which may or may not its own view of how >> to reserve class space. And this code has been growing organically over >> time. >> >> One small example: >> >> ReservedSpace Metaspace::reserve_preferred_space(size_t size, size_t >> alignment, >> bool large_pages, char >> *requested_addr, >> bool use_requested_addr) >> >> which I spent hours decoding, resulting in a very confused mail to >> hs-runtime and aarch64-port-dev [2]. >> >> This patch attempts to simplify cds and metaspace setup a bit; to comment >> implicit knowledge which is not immediately clear; to cleanly abstract >> platform concerns like optimized class space placement; and to disentangle >> cds from metaspace to solve issues which may bite us later with Elastic >> Metaspace [4]. >> >> --- >> >> The main change is the reworked reservation mechanism. This is based on >> Ioi's proposal [5]. >> >> When reserving class space, three things must happen: >> >> 1) reservation of the space obviously. If cds is active that space must >> be in the vicinity of cds archives to be covered by compressed class >> pointer encoding. >> 2) setting up the internal Metaspace structures atop of that space >> 3) setting up compressed class pointer encoding. >> >> In its current form, Metaspace may or may not do some or all of that in >> one function >> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base);) - if cds is active, >> it will reserve the space for Metaspace and hand it in, otherwise it will >> create it itself. >> >> When discussing this in [2], Ioi proposed to move the reservation of the >> class space completely out of Metaspace and make it a responsibility of the >> caller always. This would reduce some complexity, and this patch follows >> the proposal. >> >> I removed >> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base); and all its sub >> functions. >> >> (1) now has to always be done outside - a ReservedSpace for class space >> has to be provided by the caller. However, Metaspace now offers a utility >> function for reserving space at a "nice" location, and explicitly doing >> nothing else: >> >> ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >> >> this function can be redefined on a platform level for platform optimized >> reservation, see below for details. >> >> (2) is taken care of by a new function, >> Metaspace::initialize_class_space(ReservedSpace rs) >> >> (3) is taken care of a new function >> CompressedKlassPointers::initialize(), see below for details. >> >> >> So, class space now is set up three explicit steps: >> >> - First, reserve a suitable space by however means you want. For >> convenience you may use >> Metaspace::reserve_address_space_for_compressed_classes(), or you may roll >> your own reservation. >> - Next, tell Metaspace to use that range as backing storage for class >> space: Metaspace::initialize_class_space(ReservedSpace rs) >> - Finally, set up encoding. Encoding is independent from the concept of a >> ReservedSpace, it just gets an address range, see below for details. >> >> Separating these steps and moving them out of the responsibility of >> Metaspace makes this whole thing more flexible; it also removes unnecessary >> knowledge (e.g. Metaspace does not need to know anything about either ccp >> encoding or cds). >> >> --- >> >> How it comes together: >> >> If CDS is off, we just reserve a space using >> Metaspace::reserve_address_space_for_compressed_classes(), initialize it >> with Metaspace::initialize_class_space(ReservedSpace rs), then set up >> compressed class pointer encoding covering the range of this class space. >> >> If CDS is on (dump time), we reserve large 4G space, either at >> SharedBaseAddress or using >> Metaspace::reserve_address_space_for_compressed_classes(); we then split >> that into 3G archive space and 1G class space; we set up that space with >> Metaspace as class space; then we set up compressed class pointer encoding >> covering both archive space and cds. >> >> If CDS is on (run time), we reserve a large space, split it into archive >> space (large enough to hold both archives) and class space, then basically >> proceed as above. >> >> Note that this is almost exactly how things worked before (modulo some >> minor fixes, e.g. alignment issues), only the code is reformed and made >> more explicit. >> >> --- >> >> I moved compressed class pointer setup over to CompressedKlassPointers >> and changed the interface: >> >> -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >> metaspace_rs, address cds_base) >> +void CompressedKlassPointers::initialize(address addr, size_t len); >> >> Instead of feeding it a single ReservedSpace, which is supposed to >> represent class space, and an optional alternate base if cds is on, now we >> give it just an numeric address range. That range marks the limits to where >> Klass structures are to be expected, and is the implicit promise that >> outside that range no Klass structures will exist, so encoding has to cover >> only this range. >> >> This range may contain just the class space; or class space+cds; or >> whatever allocation scheme we come up with in the future. Encoding does not >> really care how the memory is organized as long as the input range covers >> all possible Klass locations. That way we remove knowledge about class >> space/cds from compressed class pointer encoding. >> >> Moving it away from metaspace.cpp into the CompressedKlassPointers class >> also mirrors CompressedOops::initialize(). >> >> --- >> >> I renamed _narrow_klass_range to just _range, because strictly speaking >> this is the range un-narrow Klass pointers can have. >> >> As for the implementation of CompressedKlassPointers::initialize(address >> addr, size_t len), I mimicked very closely what happened before, so there >> should be almost no differences. Since "almost no differences" sounds scary >> :) here are the differences: >> >> - When CDS is active (dump or run time) we now always, unconditionally, >> set the encoding range to 4G. This fixes a theoretical bug discussed on >> aarch64-port-dev [1]. >> >> - When CDS is not active, we set the encoding range to the minimum >> required length. Before, it was left at its default value of 4G. >> >> Both differences only affect aarch64, since they are currently the only >> one using the range field in CompressedKlassPointers. >> >> I wanted to add an assert somewhere to test encoding of the very last >> address of the CompressedKlassPointers range, again to prevent errors like >> [3]. But I did not come up with a good place for this assert which would >> cover also the encoding done by C1/C2. >> >> For the same reason I thought about introducing a mode where Klass >> structures would be allocated in reverse order, starting at the end of the >> ccs, but again left it out as too big a change. >> >> --- >> >> OS abstraction: platforms may have restrictions of what constitutes a >> valid compressed class pointer encoding base. Or if not, they may have at >> least preferences. There was logic like this in metaspace.cpp, which I >> removed and cleanly factored out into platform dependent files, giving each >> platform the option to add special logic. >> >> These are two new methods: >> >> - bool CompressedKlassPointers::is_valid_base(address p) >> >> to let the platform tell you whether it considers p to be a valid >> encoding base. The only platform having these restrictions currently is >> aarch64. >> >> - ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >> >> this hands over the process of allocating a range suitable for compressed >> class pointer encoding to the platform. Most platforms will allocate just >> anywhere, but some platforms may have a better strategy (e.g. trying low >> memory first, trying only correctly aligned addresses and so on). >> >> Beforehand, this coding existed in a similar form in metaspace.cpp for >> aarch64 and AIX. For now, I left the AIX part out - it seems only half >> done, and I want to check further if we even need it, if yes why not on >> Linux ppc, and C1 does not seem to support anything other than base+offset >> with shift either, but I may be mistaken. >> >> These two methods should give the platform enough control to implement >> their own scheme for optimized class space placement without bothering any >> shared code about it. >> >> Note about the form, I introduced two new platform dependent files, >> "metaspace_.cpp" and "compressedOops_.cpp". I am not happy about >> this but this seems to be what we generally do in hotspot, right? >> >> --- >> >> Metaspace reserve alignment vs cds alignment >> >> CDS was using Metaspace reserve alignment for CDS internal purposes. I >> guess this was just a copy paste issue. It never caused problems since >> Metaspace reserve alignment == page size, but that is not true anymore in >> the upcoming Elastic Metaspace where reserve alignment will be larger. This >> causes a number of issues. >> >> I separated those two cleanly. CDS now uses >> os::vm_allocation_granularity. Metaspace::reserve_alignment is only used in >> those two places where it is needed, when CDS creates the address space for >> class space on behalf of the Metaspace. >> >> --- >> >> Windows special handling in CDS >> >> To simplify coding I removed the windows specific handling which left out >> reservation of the archive. This was needed because windows cannot mmap >> files into reserved regions. But fallback code exists in filemap.cpp for >> this case which just reads in the region instead of mapping it. >> >> Should that turn out to be a performance problem, I will reinstate the >> feature. But a simpler way would be reserve the archive and later just >> before mmapping the archive file to release the archive space. That would >> not only be simpler but give us the best guarantee that that address space >> is actually available. But I'd be happy to leave that part out completely >> if we do not see any performance problems on windows x64. >> >> --- >> >> NMT cannot deal with spaces which are split. This problem manifests in >> that bookkeeping for class space is done under "Shared Classes", not >> "Classes" as it should. This problem exists today too at dump time and >> randomly at run time. But since I simplified the reservation, this problem >> now shows up always, whether or not we map at the SharedBaseAddress. >> While I could work around this problem, I'd prefer this problem to be >> solved at the core, and NMT to have an option to recognize reservation >> splits. So I'd rather not put a workaround for this into the patch but >> leave it for fixing as a separate issue. I opened this issue to track it >> [6]. >> >> --- >> >> Jtreg tests: >> >> I expanded the CompressedOops/CompressedClassPointers.java. I also >> extended them to Windows. The tests now optionally omit strict class space >> placement tests, since these tests heavily depend on ASLR and were the >> reason they were excluded on Windows. However I think even without checking >> for class space placement they make sense, just to see that the VM comes up >> and lives with the many different settings we can run in. >> >> --- >> >> Tests: >> >> - I ran the patch through Oracles submit repo >> - I ran tests manually for aarch64, zero, linux 32bit and windows x64 >> - The whole battery of nightly tests at SAP, including ppc, ppcle and >> aarch64, unfortunately excluding windows because of unrelated errors. >> Windows x64 tests will be redone tonight. >> >> >> Thank you, >> >> Thomas >> >> [1] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >> [2] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >> [5] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >> >> > From evgeny.nikitin at oracle.com Mon May 18 14:46:58 2020 From: evgeny.nikitin at oracle.com (Evgeny Nikitin) Date: Mon, 18 May 2020 16:46:58 +0200 Subject: RFR(L): 8229186: Improve error messages for TestStringIntrinsics failures Message-ID: <2fa31b05-c37a-1367-a7dc-5ae2b13133be@oracle.com> Hi, Bug: https://bugs.openjdk.java.net/browse/JDK-8229186 Webrev: http://cr.openjdk.java.net/~enikitin/8229186/webrev.00/ Error reporting was improved by writing a C-style escaped string representations for the variables passed to the methods being tested. For array comparisons, a dedicated diff-formatter was implemented. Sample output for comparing byte arrays (with artificial failure): ----------System.err:(21/1553)---------- Result: (false) of 'arrayEqualsB' is not equal to expected (true) Arrays differ starting from [index: 7]: ... 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, ... ... 5, 6, 125, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, ... ^^^^ java.lang.RuntimeException: Result: (false) of 'arrayEqualsB' is not equal to expected (true) at compiler.intrinsics.string.TestStringIntrinsics.invokeAndCheckArrays(TestStringIntrinsics.java:273) at ... stack trace continues - E.N. Sample output for comparing char arrays: ----------System.err:(21/1579)*---------- Result: (false) of 'arrayEqualsC' is not equal to expected (true) Arrays differ starting from [index: 7]: ... \\u0005, \\u0006, \\u0007, \\u0008, \\u0009, \\n, \\u000B, \\u000C, \\r, \\u000E, ... ... \\u0005, \\u0006, }, \\u0008, \\u0009, \\n, \\u000B, \\u000C, \\r, \\u000E, ... ^^^^^^^ java.lang.RuntimeException: Result: (false) of 'arrayEqualsC' is not equal to expected (true) at compiler.intrinsics.string.TestStringIntrinsics.invokeAndCheckArrays(TestStringIntrinsics.java:280) at ... and so on - E.N. Please review. Thanks in advance, /Evgeny Nikitin. From paul.sandoz at oracle.com Mon May 18 19:13:59 2020 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 18 May 2020 12:13:59 -0700 Subject: Update Re: RFR JDK-8223347 Integration of Vector API (Incubator): Java API, implementation, and tests In-Reply-To: <49F81016-55DF-4E3C-940A-A8DF8C712259@oracle.com> References: <49F81016-55DF-4E3C-940A-A8DF8C712259@oracle.com> Message-ID: <9D074341-7234-4D9F-8E99-0E73F5BB5407@oracle.com> HI, Here?s an update of the API and implementation webrevs based on (mostly) API feedback. We have removed the performance tests to make the review easier (those will be dealt with separately to integration as a follow on task). I think over the past year and recently via the CSR the API has had significant review. Reviews focusing on the Java implementation and tests would be greatly appreciated. It?s worth reiterating that the implementation and tests are quite formulaic, there is a lot of code that is generated from templates, so it's possible to focus on the template + subset (e.g. byte, long, float, and sizes of say 256, max). Notable changes from the prior webrev are: - Removal of the fromValues methods to construct a vector from a varargs array. Feedback indicated this was a misleading way to obtain a vector. - Unification of byte array and ByteBuffer load/store method signatures and unification of the implementations with more tests (including out-of-bounds tests for all the kinds of loads/stores). Paul. -- Latest javadoc http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/docs-2020-05-15-88a83f7238d8/api/jdk.incubator.vector/jdk/incubator/vector/package-summary.html Latest specdiff http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/specdiff-jdk-2020-05-15-88a83f7238d8/overview-summary.html Incremental specdiff http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/specdiff-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/overview-summary.html Latest implementation webrev http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev-2020-05-16-default-2020-05-15-88a83f7238d8/webrev/ Incremental Implementation webrev http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/webrev/ Latest test webrev http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev-2020-05-16-default-2020-05-15-88a83f7238d8/webrev/ Incremental test webrev http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/webrev/ > On Apr 1, 2020, at 3:46 PM, Paul Sandoz wrote: > > Hi, > > A prior email sent out a request for review of the Vector API in preparation for JEP 338: Vector API (Incubator) [1] to be proposed for target: > > https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html > > This email expands the review of the API to the Java implementation and Java tests: > > http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev/webrev/ > > http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev/webrev/ > > (Further emails will sent for review of general HotSpot changes CPU architecture specific HotSpot changes, x64 and aarch64, and performance tests. For an early peek see the links in the description of JDK-8223347.) > > ? > > The Vector API provides > > - the public Vector class with vector operations common to all supported primitive types > > - public primitive specializations, such as IntVector, with common operations associated with the primitive type > > - internal concrete specializations for the vector size, such as Int256Vector, for a vector holding 8 ints. > > Operations generally defer to one of approximately 20 vector intrinsics. Some operations may be composed of other operations. > > Explicit casts are performed by vector operations to ensure vectors arguments are of the required shape (bit size) and element type. > > A vector intrinsic is an internal low-level vector operation. The last argument to the intrinsic is fall back behavior in Java, implementing the scalar operation over the number of elements held by the vector. Thus, If the intrinsic is not supported in C2 for the other arguments then the Java implementation is executed (the Java implementation is always executed when running in the interpreter or for C1). > > The public primitive specializations and the internal concrete implementations are generated from SSP template files, X-Vector.java.template and X-VectorBits.java.template respectively. > > Overall the implementation approach is quite formulaic and rather repetitive. Once you grok the pattern It should be easier to review if a little laborious. > > Unit tests are auto-generated by composing templates for each operation into an SSP template file which is then used to generate unit test files for each concrete vector class. The tests are quite extensive and have found many a HotSpot issue in development across a wide range of platforms. > > Paul. > > [1] https://bugs.openjdk.java.net/browse/JDK-8201271 > From ioi.lam at oracle.com Tue May 19 04:37:48 2020 From: ioi.lam at oracle.com (Ioi Lam) Date: Mon, 18 May 2020 21:37:48 -0700 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> Message-ID: <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> On 5/18/20 7:37 AM, Thomas St?fe wrote: > Hi all, > > fourth webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ > > I rebased atop of > > changeset: ? 59325:1ba9a9a3f948 > summary: ? ? 8244433: Remove saving of RSP in Assembler::pusha_uncached() > > so, atop of "8241825: Make compressed oops and compressed class > pointers independent (x86_64, PPC, S390)". > > The changes are very small this time, see notes below: > > @Ioi: > > >Hi Thomas, > > >I am running your patch in out CI pipeline now. Some comments: > > >[2] Does JDK-8243535 exist with the current jdk/jdk repo? > > I assigned JDK-8243535 to me and I'm working on a patch. But I would > like to keep that separate from this work, since this patch is already > large. And I would like to fix JDK-8243535 after this remodel patch > has been pushed, which makes some minor things easier. That would > leave a small time window where NMT does mis-track class space as > shared class space, but that is not a big issue, it does not even > disturb any test. Which in itself is something we may want to fix, > come to think about it. > > > > [3] I think this can be put in header file. > > > > bool Metaspace::class_space_is_initialized() { > > ? return _class_space_list != NULL; > > } > > Done. > > > [7] argument.cpp checks this: > > > > ? if (UseSharedSpaces || DumpSharedSpaces) { > > ? ? if > (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { > > ? ? ? log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid > for this platform, option will be ignored.", > > ? ? ? ? ? ? ? ? ? ? ? ?p2i((address)SharedBaseAddress)); > > ? ? ? FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); > > ? ? } > > ? } > > > > but initialize_dumptime_shared_and_meta_spaces aligns up the > SharedBaseAddress before doing the assert. > > > > ? void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { > > ? ? assert(DumpSharedSpaces, "should be called for dump time only"); > > > > ? ? const size_t reserve_alignment = > MetaspaceShared::reserved_space_alignment(); > > ? ? char* shared_base = (char*)align_up((char*)SharedBaseAddress, > reserve_alignment); > > > > ? #ifdef _LP64 > > assert(CompressedKlassPointers::is_valid_base((address)shared_base), > "Sanity"); > > > > So theoretically shared_base may no longer be is_valid_base after > the align_up. It probably won't happen, but I think the code will be > > much clearer tofirst to align_up, then check for is_valid_base and > reset to default, and finally assert. > > The problem with that is that at the time this coding runs os::init() > did not yet run and we do not know yet allocation granularity. > > So I changed verification of SharedBaseAddress: I added the alignment > as you suggested and moved it to a later time in VM initialization, to > CDS initialization. > Is the call to? check_SharedBaseAddress() necessary inside MetaspaceShared::initialize_runtime_shared_and_meta_spaces()? SharedBaseAddress is currently ignored at run time. Does your patch change that behavior? > >[8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, > the following block can be simplified: > > > > ?if (!_shared_rs.is_reserved()) { > > ? ?// Get a reserved space anywhere if attaching at the > SharedBaseAddress fails: > > ? ?if (UseCompressedClassPointers) { > > ? ? ?// If we need a compressed class space too, let the platform > handle the reservation. > > ? ? ?_shared_rs = > Metaspace::reserve_address_space_for_compressed_classes(cds_total); > > ? ?} else { > > ? ? ?// anywhere is fine. > > ? ? ?_shared_rs = ReservedSpace(cds_total, reserve_alignment, false > /* large */, (char*)NULL); > > ? ?} > > ?} > > > > ? ... if you change the declaration to: > > > > ?static ReservedSpace > reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ > return ReservedSpace();}); > > I'd like to avoid keep the method LP64 only, but I rewrote the section > somewhat to be more clearer. > The new code looks good. > > [9] I think the #ifdef _LP64 is not necessary: > > > > #ifdef _LP64 > > ? ? // Some sanity checks after reserving address spaces for > archives and class space. > > ? ? assert(archive_space_rs.is_reserved(), "Sanity"); > > ? ? if (Metaspace::using_class_space()) { > > ? ? ? // Class space must closely follow the archive space. Both > spaces must be aligned correctly. > > ? ? ? assert(class_space_rs.is_reserved(), "A class space should > have been reserved"); > > ? ? ? assert(class_space_rs.base() >= archive_space_rs.end(), "class > space should follow the cds archive space"); > > ? ? ? assert(is_aligned(archive_space_rs.base(), > MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); > > ? ? ? assert(is_aligned(class_space_rs.base(), > Metaspace::reserve_alignment()), "class space misaligned"); > > ? ? } > > #endif > > I removed the #ifdef, but replaced it with an #ifdef ASSERT. Which is > also not needed, technically, but makes it clear that this section is > debugging checks only. > OK. The rest of the code looks good to me. I am going to run your latest patch in our CI and will report the results. Thanks - Ioi > ---- > > @Coleen: > > >http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/>virtualSpaceList.cpp.udiff.html > > > >Why did you make this change?? Shouldn't the caller align it? > > > You are right, I removed the assert. > > --- > > Thanks all for your review work, > > ..Thomas > > > > > > > > > > > > > On Fri, May 8, 2020 at 7:37 AM Ioi Lam > wrote: > > Hi Thomas, > > I am running your patch in out CI pipeline now. Some comments: > > [1] Do we still need to use _class_space_list? It looks like we > just have a single range. Is > _class_space_list->current_virtual_space() always the same VS? > > ? if (_class_space_list != NULL) { > ??? address base = > (address)_class_space_list->current_virtual_space()->bottom(); > ??? address top = base + compressed_class_space_size(); > ??? st->print("Compressed class space mapped at: " PTR_FORMAT "-" > PTR_FORMAT ", size: " SIZE_FORMAT, > ?????????????? p2i(base), p2i(top), top - base); > > [2] Does JDK-8243535 exist with the current jdk/jdk repo? > > ? // This does currently not work because rs may be the result of > a split operation > ? // and NMT seems not to be able to handle splits. > ? // See JDK-8243535. > ? // MemTracker::record_virtual_memory_type((address)rs.base(), > mtClass); > > > [3] I think this can be put in header file. > > bool Metaspace::class_space_is_initialized() { > ? return _class_space_list != NULL; > } > > [4] Why does the CCS need to follow UseLargePagesInMetaspace? This > is the reason for the > ??? gap in the following chart: > > ??? // We do this by reserving space for the ccs behind the > archives. Note however that > ??? //? ccs follows a different alignment > (Metaspace::reserve_alignment), so there may > ??? //? be a gap between ccs and cds. > ??? // We use a similar layout at runtime, see > reserve_address_space_for_archives(). > ??? // > ??? //????????????????????????????? +-- SharedBaseAddress (default > = 0x800000000) > ??? //????????????????????????????? v > ??? // +-..---------+---------+ ... > +----+----+----+--------+--------------------+ > ??? // |??? Heap??? | Archive |???? | MC | RW | RO | [gap]? |??? > class space???? | > ??? // +-..---------+---------+ ... > +----+----+----+--------+--------------------+ > ??? // |<--?? MaxHeapSize? -->|???? |<-- UnscaledClassSpaceMax = > 4GB -->| > > > _reserve_alignment is determined here: > > void Metaspace::ergo_initialize() { > ? if (DumpSharedSpaces) { > ??? // Using large pages when dumping the shared archive is > currently not implemented. > ??? FLAG_SET_ERGO(UseLargePagesInMetaspace, false); > ? } > > ? size_t page_size = os::vm_page_size(); > ? if (UseLargePages && UseLargePagesInMetaspace) { > ??? page_size = os::large_page_size(); > ? } > > ? _commit_alignment? = page_size; > ? _reserve_alignment = MAX2(page_size, > (size_t)os::vm_allocation_granularity()); > > But when CDS is enabled, the RS is reserved without large pages, > so it should be possible to incrementally commit the CCS using > just os::vm_allocation_granularity(). > > [5] I see the aarch64 code in > Metaspace::reserve_address_space_for_compressed_classes has been > changed. Is this necessary for this RFE, or is it a separate > improvement that can be done in a different RFE? > > [6] For AARCH64, should we skip the part that reserves it right > above the heap? Or does AARCH64 always allocate the heap such that > it's a preferable address? > > ??? // case (b) > ??? ReservedSpace rs; > > ??? // Try right above the Java heap... > ??? address base = align_up(CompressedOops::end(), > Metaspace::reserve_alignment()); > ??? assert(base != NULL, "Sanity"); > > ??? const size_t size = align_up(CompressedClassSpaceSize, > Metaspace::reserve_alignment()); > ??? if (CompressedKlassPointers::is_valid_base(base)) { > ????? rs = ReservedSpace(size, Metaspace::reserve_alignment(), > false /* large */, (char*)base); > ??? } > > ??? // ...failing that, reserve anywhere, but let platform do > optimized placement: > ??? if (!rs.is_reserved()) { > ????? rs = > Metaspace::reserve_address_space_for_compressed_classes(size); > ??? } > > [7] argument.cpp checks this: > > ? if (UseSharedSpaces || DumpSharedSpaces) { > ??? if > (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) > { > ????? log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is > invalid for this platform, option will be ignored.", > ?????????????????????? p2i((address)SharedBaseAddress)); > ????? FLAG_SET_DEFAULT(SharedBaseAddress, > default_SharedBaseAddress()); > ??? } > ? } > > but initialize_dumptime_shared_and_meta_spaces aligns up the > SharedBaseAddress before doing the assert. > > ? void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { > ??? assert(DumpSharedSpaces, "should be called for dump time only"); > > ??? const size_t reserve_alignment = > MetaspaceShared::reserved_space_alignment(); > ??? char* shared_base = (char*)align_up((char*)SharedBaseAddress, > reserve_alignment); > > ? #ifdef _LP64 > assert(CompressedKlassPointers::is_valid_base((address)shared_base), > "Sanity"); > > So theoretically shared_base may no longer be is_valid_base after > the align_up. It probably won't happen, but I think the code will > be much clearer tofirst to align_up, then check for is_valid_base > and reset to default, and finally assert. > > [8] In > MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the > following block can be simplified: > > ? if (!_shared_rs.is_reserved()) { > ??? // Get a reserved space anywhere if attaching at the > SharedBaseAddress fails: > ??? if (UseCompressedClassPointers) { > ????? // If we need a compressed class space too, let the platform > handle the reservation. > ????? _shared_rs = > Metaspace::reserve_address_space_for_compressed_classes(cds_total); > ??? } else { > ????? // anywhere is fine. > ????? _shared_rs = ReservedSpace(cds_total, reserve_alignment, > false /* large */, (char*)NULL); > ??? } > ? } > > ?? ... if you change the declaration to: > > ? static ReservedSpace > reserve_address_space_for_compressed_classes(size_t size) > NOT_LP64({ return ReservedSpace();}); > > [9] I think the #ifdef _LP64 is not necessary: > > #ifdef _LP64 > ??? // Some sanity checks after reserving address spaces for > archives and class space. > ??? assert(archive_space_rs.is_reserved(), "Sanity"); > ??? if (Metaspace::using_class_space()) { > ????? // Class space must closely follow the archive space. Both > spaces must be aligned correctly. > ????? assert(class_space_rs.is_reserved(), "A class space should > have been reserved"); > ????? assert(class_space_rs.base() >= archive_space_rs.end(), > "class space should follow the cds archive space"); > ????? assert(is_aligned(archive_space_rs.base(), > MetaspaceShared::reserved_space_alignment()), "Archive space > misaligned"); > ????? assert(is_aligned(class_space_rs.base(), > Metaspace::reserve_alignment()), "class space misaligned"); > ??? } > #endif > > The rest of the code looks OK to me, but I may take a look at it > again after getting more sleep :-) > > Thanks > - Ioi > > > > > On 5/7/20 7:21 AM, Thomas St?fe wrote: >> Hi all, >> >> please take a look at the third iteration of this change: >> >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >> >> Changes in this version: >> >> - at the request of Coleen (sorry I did not see your post >> earlier) I removed all platform dependent?files and put the >> platform dependent coding back to metaspace.cpp and >> compressedOops.cpp to ease the review pain. I used plain platform >> defines though (#ifdef AARCH64) instead of hiding them behind >> another macro to make things clearer. Note that I still intent to >> put this code away into the platform corners but will do so in a >> follow up RFE. >> >> - I reinstated, in a fashion, the special handling of >> reservations on Windows. On all platforms we reserve address >> space to map the archive files in with a subsequent mapping >> operation. However, on Windows, we cannot use MapViewOfFile() >> into an existing mapping. So I remove the mapping again before >> mapping the archives - see comment in code for details. >> >> All CI tests at SAP run through without problems, including on >> Win64 and aarch64, but I would be happy if others were to run >> test too. >> >> Thank you, Thomas >> >> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >> > wrote: >> >> Hi all, >> >> Could I have reviews for the following proposal of reworking >> cds/class space reservation? >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >> >> Webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >> >> (Many thanks to Ioi Lam for so patiently explaining CDS >> internals to me, and to Andrew Haley and Nick Gasson for help >> with aarch64!) >> >> Reservation of the compressed class space is needlessly >> complicated and has some minor issues. It can be simplified >> and made clearer. >> >> The complexity stems from the fact that this area lives at >> the intersection of two to three sub systems, depending on >> how one counts. Metaspace, CDS, and the platform which may or >> may not its own view of how to reserve class space. And this >> code has been growing organically over time. >> >> One small example: >> >> ReservedSpace Metaspace::reserve_preferred_space(size_t size, >> size_t alignment, >> ?bool large_pages, char *requested_addr, >> ?bool use_requested_addr) >> >> which I spent hours decoding, resulting in a very confused >> mail to hs-runtime and aarch64-port-dev [2]. >> >> This patch attempts to simplify cds and metaspace setup a >> bit; to comment implicit knowledge which is not immediately >> clear; to cleanly abstract platform concerns like optimized >> class space placement; and to disentangle cds from metaspace >> to solve issues which may bite us later with Elastic >> Metaspace [4]. >> >> --- >> >> The main change is the reworked reservation mechanism. This >> is based on Ioi's proposal [5]. >> >> When reserving class space, three things must happen: >> >> 1) reservation of the space obviously. If cds is active that >> space must be in the vicinity of cds archives to be covered >> by compressed class pointer encoding. >> 2) setting up the internal Metaspace structures atop of that >> space >> 3) setting up compressed class pointer encoding. >> >> In its current form, Metaspace may or may not do some or all >> of that in one function >> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base);) - if >> cds is active, it will reserve the space for Metaspace and >> hand it in, otherwise it will create it itself. >> >> When discussing this in [2], Ioi proposed to move the >> reservation of the class space completely out of Metaspace >> and make it a responsibility of the caller always. This would >> reduce some complexity, and this patch follows the proposal. >> >> I removed >> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >> metaspace_rs, char* requested_addr, address cds_base); and >> all its sub functions. >> >> (1) now has to always be done outside - a ReservedSpace for >> class space has to be provided by the caller. However, >> Metaspace now offers a utility function for reserving space >> at a "nice" location, and explicitly doing nothing else: >> >> ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t >> size); >> >> this function can be redefined on a platform level for >> platform optimized reservation, see below for details. >> >> (2) is taken care of by a new function, >> Metaspace::initialize_class_space(ReservedSpace rs) >> >> (3) is taken care of a new function >> CompressedKlassPointers::initialize(), see below for details. >> >> >> So, class space now is set up three explicit steps: >> >> - First, reserve a suitable space by however means you want. >> For convenience you may use >> Metaspace::reserve_address_space_for_compressed_classes(), or >> you may roll your own reservation. >> - Next, tell Metaspace to use that range as backing storage >> for class space: >> Metaspace::initialize_class_space(ReservedSpace rs) >> - Finally, set up encoding. Encoding is independent from the >> concept of a ReservedSpace, it just gets an address range, >> see below for details. >> >> Separating these steps and moving them out of the >> responsibility of Metaspace makes this whole thing more >> flexible; it also removes unnecessary knowledge (e.g. >> Metaspace does not need to know anything about either ccp >> encoding or cds). >> >> --- >> >> How it comes together: >> >> If CDS is off, we just reserve a space using >> Metaspace::reserve_address_space_for_compressed_classes(), >> initialize it with >> Metaspace::initialize_class_space(ReservedSpace rs), then set >> up compressed class pointer encoding covering the range of >> this class space. >> >> If CDS is on (dump time), we reserve large 4G space, either >> at SharedBaseAddress or using >> Metaspace::reserve_address_space_for_compressed_classes(); we >> then split that into 3G archive space and 1G class space; we >> set up that space with Metaspace as class space; then we set >> up?compressed class pointer encoding covering both archive >> space and cds. >> >> If CDS is on (run time), we reserve a large space, split it >> into archive space (large enough to hold both archives) and >> class space, then basically proceed as above. >> >> Note that this is almost exactly how things worked before >> (modulo some minor fixes, e.g. alignment issues), only the >> code is reformed and made more explicit. >> >> --- >> >> I moved compressed class pointer setup over to >> CompressedKlassPointers and changed the interface: >> >> -void >> Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >> metaspace_rs, address cds_base) >> +void CompressedKlassPointers::initialize(address addr, >> size_t len); >> >> Instead of feeding it a single ReservedSpace, which is >> supposed to represent class space, and an optional alternate >> base if cds is on, now we give it just an numeric address >> range. That range marks the limits to where Klass structures >> are to be expected, and is the implicit promise that outside >> that range no Klass structures will exist, so encoding has to >> cover only this range. >> >> This range may contain just the class space; or class >> space+cds; or whatever allocation scheme we come up with in >> the future. Encoding does not really care how the memory is >> organized as long as the input range covers all possible >> Klass locations. That way we remove knowledge about class >> space/cds from compressed class pointer encoding. >> >> Moving it away from metaspace.cpp into the >> CompressedKlassPointers class also mirrors >> CompressedOops::initialize(). >> >> --- >> >> I renamed _narrow_klass_range to just _range, because >> strictly speaking this is the range un-narrow Klass pointers >> can have. >> >> As for the implementation of >> CompressedKlassPointers::initialize(address addr, size_t >> len), I mimicked very closely what happened before, so there >> should be almost no differences. Since "almost no >> differences" sounds scary :) here are the differences: >> >> - When CDS is active (dump or run time) we now always, >> unconditionally, set the encoding range to 4G. This fixes a >> theoretical bug discussed on aarch64-port-dev [1]. >> >> - When CDS is not active, we set the encoding range to the >> minimum required length. Before, it was left at its default >> value of 4G. >> >> Both differences only affect aarch64, since they are >> currently the only one using the range field in >> CompressedKlassPointers. >> >> I wanted to add an assert somewhere to test encoding of the >> very last address of the CompressedKlassPointers range, again >> to prevent errors like [3]. But I did not come up with a good >> place for this assert which would cover also the encoding >> done by C1/C2. >> >> For the same reason I thought about introducing a mode where >> Klass structures would be allocated in reverse order, >> starting at the end of the ccs, but again left it out as too >> big a change. >> >> --- >> >> OS abstraction: platforms may have restrictions of what >> constitutes a valid compressed class pointer encoding base. >> Or if not, they may have at least preferences. There was >> logic like this in metaspace.cpp, which I removed and cleanly >> factored out into platform dependent files, giving each >> platform the option to add special logic. >> >> These are two new methods: >> >> - bool CompressedKlassPointers::is_valid_base(address p) >> >> to let the platform tell you whether it considers p to be a >> valid encoding base. The only platform having these >> restrictions currently is aarch64. >> >> - ReservedSpace >> Metaspace::reserve_address_space_for_compressed_classes(size_t >> size); >> >> this hands over the process of allocating a range suitable >> for compressed class pointer encoding to the platform. Most >> platforms will allocate just anywhere, but some platforms may >> have a better strategy (e.g. trying low memory first, trying >> only correctly aligned addresses and so on). >> >> Beforehand, this coding existed in a similar form in >> metaspace.cpp for aarch64 and AIX. For now, I left the AIX >> part out - it seems only half done, and I want to check >> further if we even need it, if yes why not on Linux ppc, and >> C1 does not seem to support anything other than base+offset >> with shift either, but I may be mistaken. >> >> These two methods should give the platform enough control to >> implement their own scheme for optimized class space >> placement without bothering any shared code about it. >> >> Note about the form, I introduced two new platform dependent >> files, "metaspace_.cpp" and "compressedOops_.cpp". >> I am not happy about this but this seems to be what we >> generally do in hotspot, right? >> >> --- >> >> Metaspace reserve alignment vs cds alignment >> >> CDS was using Metaspace reserve alignment for CDS internal >> purposes. I guess this was just a copy paste issue. It never >> caused problems since Metaspace reserve alignment == page >> size, but that is not true anymore in the upcoming Elastic >> Metaspace where reserve alignment will be larger. This causes >> a number of issues. >> >> I separated those two cleanly. CDS now uses >> os::vm_allocation_granularity. Metaspace::reserve_alignment >> is only used in those two places where it is needed, when CDS >> creates the address space for class space on behalf of the >> Metaspace. >> >> --- >> >> Windows special handling in CDS >> >> To simplify coding I removed the windows specific handling >> which left out reservation of the archive. This was needed >> because windows cannot mmap files into reserved regions. But >> fallback code exists in filemap.cpp for this case which just >> reads in the region instead of mapping?it. >> >> Should that turn out to be a performance problem, I will >> reinstate the feature. But a simpler way would be reserve the >> archive and later just before mmapping?the archive file to >> release the archive space. That would not only be simpler but >> give us the best guarantee that that address space is >> actually available. But I'd be happy to leave that part out >> completely if we do not see any performance problems on >> windows x64. >> >> --- >> >> NMT cannot deal with spaces which are split. This problem >> manifests in that bookkeeping for class space is done under >> "Shared Classes", not "Classes" as it should. This problem >> exists today too at dump?time and randomly at run time. But >> since I simplified the reservation, this problem now shows up >> always, whether or not we map at the SharedBaseAddress. >> While I could work around this problem, I'd prefer this >> problem to be solved at the core, and NMT to have an option >> to recognize reservation splits. So I'd rather not put a >> workaround for this into the patch but leave it for fixing as >> a separate issue. I opened this issue to track it [6]. >> >> --- >> >> Jtreg tests: >> >> I expanded the CompressedOops/CompressedClassPointers.java. I >> also extended them to Windows. The tests now optionally omit >> strict class space placement tests, since these tests heavily >> depend on ASLR and were the reason they were excluded on >> Windows. However I think even without checking for class >> space placement they make sense, just to see that the VM >> comes up and lives with the many different settings we can >> run in. >> >> --- >> >> Tests: >> >> - I ran the patch through Oracles submit repo >> - I ran tests manually for aarch64, zero, linux 32bit and >> windows x64 >> - The whole battery of nightly tests at SAP, including ppc, >> ppcle and aarch64, unfortunately excluding windows because of >> unrelated errors. Windows x64 tests will be redone tonight. >> >> >> Thank you, >> >> Thomas >> >> [1] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >> [2] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >> [5] >> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >> > From thomas.stuefe at gmail.com Tue May 19 04:42:51 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 19 May 2020 06:42:51 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> Message-ID: Hi Ioi, On Tue, May 19, 2020 at 6:38 AM Ioi Lam wrote: > > > On 5/18/20 7:37 AM, Thomas St?fe wrote: > > Hi all, > > fourth webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ > > I rebased atop of > > changeset: 59325:1ba9a9a3f948 > summary: 8244433: Remove saving of RSP in Assembler::pusha_uncached() > > so, atop of "8241825: Make compressed oops and compressed class pointers > independent (x86_64, PPC, S390)". > > The changes are very small this time, see notes below: > > @Ioi: > > >Hi Thomas, > > >I am running your patch in out CI pipeline now. Some comments: > > >[2] Does JDK-8243535 exist with the current jdk/jdk repo? > > I assigned JDK-8243535 to me and I'm working on a patch. But I would like > to keep that separate from this work, since this patch is already large. > And I would like to fix JDK-8243535 after this remodel patch has been > pushed, which makes some minor things easier. That would leave a small time > window where NMT does mis-track class space as shared class space, but that > is not a big issue, it does not even disturb any test. Which in itself is > something we may want to fix, come to think about it. > > > > [3] I think this can be put in header file. > > > > bool Metaspace::class_space_is_initialized() { > > return _class_space_list != NULL; > > } > > Done. > > > [7] argument.cpp checks this: > > > > if (UseSharedSpaces || DumpSharedSpaces) { > > if > (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { > > log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for > this platform, option will be ignored.", > > p2i((address)SharedBaseAddress)); > > FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); > > } > > } > > > > but initialize_dumptime_shared_and_meta_spaces aligns up the > SharedBaseAddress before doing the assert. > > > > void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { > > assert(DumpSharedSpaces, "should be called for dump time only"); > > > > const size_t reserve_alignment = > MetaspaceShared::reserved_space_alignment(); > > char* shared_base = (char*)align_up((char*)SharedBaseAddress, > reserve_alignment); > > > > #ifdef _LP64 > > assert(CompressedKlassPointers::is_valid_base((address)shared_base), > "Sanity"); > > > > So theoretically shared_base may no longer be is_valid_base after the > align_up. It probably won't happen, but I think the code will be > much > clearer tofirst to align_up, then check for is_valid_base and reset to > default, and finally assert. > > The problem with that is that at the time this coding runs os::init() did > not yet run and we do not know yet allocation granularity. > > So I changed verification of SharedBaseAddress: I added the alignment as > you suggested and moved it to a later time in VM initialization, to CDS > initialization. > > > Is the call to check_SharedBaseAddress() necessary inside > MetaspaceShared::initialize_runtime_shared_and_meta_spaces()? > SharedBaseAddress is currently ignored at run time. Does your patch change > that behavior? > > No, you are right. I'll remove the call. > >[8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the > following block can be simplified: > > > > if (!_shared_rs.is_reserved()) { > > // Get a reserved space anywhere if attaching at the > SharedBaseAddress fails: > > if (UseCompressedClassPointers) { > > // If we need a compressed class space too, let the platform handle > the reservation. > > _shared_rs = > Metaspace::reserve_address_space_for_compressed_classes(cds_total); > > } else { > > // anywhere is fine. > > _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* > large */, (char*)NULL); > > } > > } > > > > ... if you change the declaration to: > > > > static ReservedSpace > reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return > ReservedSpace();}); > > I'd like to avoid keep the method LP64 only, but I rewrote the section > somewhat to be more clearer. > > The new code looks good. > > > [9] I think the #ifdef _LP64 is not necessary: > > > > #ifdef _LP64 > > // Some sanity checks after reserving address spaces for archives > and class space. > > assert(archive_space_rs.is_reserved(), "Sanity"); > > if (Metaspace::using_class_space()) { > > // Class space must closely follow the archive space. Both spaces > must be aligned correctly. > > assert(class_space_rs.is_reserved(), "A class space should have > been reserved"); > > assert(class_space_rs.base() >= archive_space_rs.end(), "class > space should follow the cds archive space"); > > assert(is_aligned(archive_space_rs.base(), > MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); > > assert(is_aligned(class_space_rs.base(), > Metaspace::reserve_alignment()), "class space misaligned"); > > } > > #endif > > I removed the #ifdef, but replaced it with an #ifdef ASSERT. Which is also > not needed, technically, but makes it clear that this section is debugging > checks only. > > > OK. > > The rest of the code looks good to me. I am going to run your latest patch > in our CI and will report the results. > > Thanks > - Ioi > > Thanks a lot! ..Thomas > ---- > > @Coleen: > > > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/ > >virtualSpaceList.cpp.udiff.html > > > >Why did you make this change? Shouldn't the caller align it? > > > You are right, I removed the assert. > > --- > > Thanks all for your review work, > > ..Thomas > > > > > > > > > > > > > On Fri, May 8, 2020 at 7:37 AM Ioi Lam wrote: > >> Hi Thomas, >> >> I am running your patch in out CI pipeline now. Some comments: >> >> [1] Do we still need to use _class_space_list? It looks like we just have >> a single range. Is _class_space_list->current_virtual_space() always the >> same VS? >> >> if (_class_space_list != NULL) { >> address base = >> (address)_class_space_list->current_virtual_space()->bottom(); >> address top = base + compressed_class_space_size(); >> st->print("Compressed class space mapped at: " PTR_FORMAT "-" >> PTR_FORMAT ", size: " SIZE_FORMAT, >> p2i(base), p2i(top), top - base); >> >> [2] Does JDK-8243535 exist with the current jdk/jdk repo? >> >> // This does currently not work because rs may be the result of a split >> operation >> // and NMT seems not to be able to handle splits. >> // See JDK-8243535. >> // MemTracker::record_virtual_memory_type((address)rs.base(), mtClass); >> >> >> [3] I think this can be put in header file. >> >> bool Metaspace::class_space_is_initialized() { >> return _class_space_list != NULL; >> } >> >> [4] Why does the CCS need to follow UseLargePagesInMetaspace? This is the >> reason for the >> gap in the following chart: >> >> // We do this by reserving space for the ccs behind the archives. >> Note however that >> // ccs follows a different alignment (Metaspace::reserve_alignment), >> so there may >> // be a gap between ccs and cds. >> // We use a similar layout at runtime, see >> reserve_address_space_for_archives(). >> // >> // +-- SharedBaseAddress (default = >> 0x800000000) >> // v >> // +-..---------+---------+ ... >> +----+----+----+--------+--------------------+ >> // | Heap | Archive | | MC | RW | RO | [gap] | class >> space | >> // +-..---------+---------+ ... >> +----+----+----+--------+--------------------+ >> // |<-- MaxHeapSize -->| |<-- UnscaledClassSpaceMax = 4GB -->| >> >> >> _reserve_alignment is determined here: >> >> void Metaspace::ergo_initialize() { >> if (DumpSharedSpaces) { >> // Using large pages when dumping the shared archive is currently not >> implemented. >> FLAG_SET_ERGO(UseLargePagesInMetaspace, false); >> } >> >> size_t page_size = os::vm_page_size(); >> if (UseLargePages && UseLargePagesInMetaspace) { >> page_size = os::large_page_size(); >> } >> >> _commit_alignment = page_size; >> _reserve_alignment = MAX2(page_size, >> (size_t)os::vm_allocation_granularity()); >> >> But when CDS is enabled, the RS is reserved without large pages, so it >> should be possible to incrementally commit the CCS using just >> os::vm_allocation_granularity(). >> >> [5] I see the aarch64 code in >> Metaspace::reserve_address_space_for_compressed_classes has been changed. >> Is this necessary for this RFE, or is it a separate improvement that can be >> done in a different RFE? >> >> [6] For AARCH64, should we skip the part that reserves it right above the >> heap? Or does AARCH64 always allocate the heap such that it's a preferable >> address? >> >> // case (b) >> ReservedSpace rs; >> >> // Try right above the Java heap... >> address base = align_up(CompressedOops::end(), >> Metaspace::reserve_alignment()); >> assert(base != NULL, "Sanity"); >> >> const size_t size = align_up(CompressedClassSpaceSize, >> Metaspace::reserve_alignment()); >> if (CompressedKlassPointers::is_valid_base(base)) { >> rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* >> large */, (char*)base); >> } >> >> // ...failing that, reserve anywhere, but let platform do optimized >> placement: >> if (!rs.is_reserved()) { >> rs = Metaspace::reserve_address_space_for_compressed_classes(size); >> } >> >> [7] argument.cpp checks this: >> >> if (UseSharedSpaces || DumpSharedSpaces) { >> if >> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >> log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for >> this platform, option will be ignored.", >> p2i((address)SharedBaseAddress)); >> FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >> } >> } >> >> but initialize_dumptime_shared_and_meta_spaces aligns up the >> SharedBaseAddress before doing the assert. >> >> void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >> assert(DumpSharedSpaces, "should be called for dump time only"); >> >> const size_t reserve_alignment = >> MetaspaceShared::reserved_space_alignment(); >> char* shared_base = (char*)align_up((char*)SharedBaseAddress, >> reserve_alignment); >> >> #ifdef _LP64 >> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >> "Sanity"); >> >> So theoretically shared_base may no longer be is_valid_base after the >> align_up. It probably won't happen, but I think the code will be much >> clearer tofirst to align_up, then check for is_valid_base and reset to >> default, and finally assert. >> >> [8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >> following block can be simplified: >> >> if (!_shared_rs.is_reserved()) { >> // Get a reserved space anywhere if attaching at the >> SharedBaseAddress fails: >> if (UseCompressedClassPointers) { >> // If we need a compressed class space too, let the platform handle >> the reservation. >> _shared_rs = >> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >> } else { >> // anywhere is fine. >> _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >> large */, (char*)NULL); >> } >> } >> >> ... if you change the declaration to: >> >> static ReservedSpace >> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >> ReservedSpace();}); >> >> [9] I think the #ifdef _LP64 is not necessary: >> >> #ifdef _LP64 >> // Some sanity checks after reserving address spaces for archives and >> class space. >> assert(archive_space_rs.is_reserved(), "Sanity"); >> if (Metaspace::using_class_space()) { >> // Class space must closely follow the archive space. Both spaces >> must be aligned correctly. >> assert(class_space_rs.is_reserved(), "A class space should have >> been reserved"); >> assert(class_space_rs.base() >= archive_space_rs.end(), "class >> space should follow the cds archive space"); >> assert(is_aligned(archive_space_rs.base(), >> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >> assert(is_aligned(class_space_rs.base(), >> Metaspace::reserve_alignment()), "class space misaligned"); >> } >> #endif >> >> The rest of the code looks OK to me, but I may take a look at it again >> after getting more sleep :-) >> >> Thanks >> - Ioi >> >> >> >> >> On 5/7/20 7:21 AM, Thomas St?fe wrote: >> >> Hi all, >> >> please take a look at the third iteration of this change: >> >> >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >> >> Changes in this version: >> >> - at the request of Coleen (sorry I did not see your post earlier) I >> removed all platform dependent files and put the platform dependent coding >> back to metaspace.cpp and compressedOops.cpp to ease the review pain. I >> used plain platform defines though (#ifdef AARCH64) instead of hiding them >> behind another macro to make things clearer. Note that I still intent to >> put this code away into the platform corners but will do so in a follow up >> RFE. >> >> - I reinstated, in a fashion, the special handling of reservations on >> Windows. On all platforms we reserve address space to map the archive files >> in with a subsequent mapping operation. However, on Windows, we cannot use >> MapViewOfFile() into an existing mapping. So I remove the mapping again >> before mapping the archives - see comment in code for details. >> >> All CI tests at SAP run through without problems, including on Win64 and >> aarch64, but I would be happy if others were to run test too. >> >> Thank you, Thomas >> >> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >> wrote: >> >>> Hi all, >>> >>> Could I have reviews for the following proposal of reworking cds/class >>> space reservation? >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >>> >>> (Many thanks to Ioi Lam for so patiently explaining CDS internals to me, >>> and to Andrew Haley and Nick Gasson for help with aarch64!) >>> >>> Reservation of the compressed class space is needlessly complicated and >>> has some minor issues. It can be simplified and made clearer. >>> >>> The complexity stems from the fact that this area lives at the >>> intersection of two to three sub systems, depending on how one counts. >>> Metaspace, CDS, and the platform which may or may not its own view of how >>> to reserve class space. And this code has been growing organically over >>> time. >>> >>> One small example: >>> >>> ReservedSpace Metaspace::reserve_preferred_space(size_t size, size_t >>> alignment, >>> bool large_pages, char >>> *requested_addr, >>> bool >>> use_requested_addr) >>> >>> which I spent hours decoding, resulting in a very confused mail to >>> hs-runtime and aarch64-port-dev [2]. >>> >>> This patch attempts to simplify cds and metaspace setup a bit; to >>> comment implicit knowledge which is not immediately clear; to cleanly >>> abstract platform concerns like optimized class space placement; and to >>> disentangle cds from metaspace to solve issues which may bite us later with >>> Elastic Metaspace [4]. >>> >>> --- >>> >>> The main change is the reworked reservation mechanism. This is based on >>> Ioi's proposal [5]. >>> >>> When reserving class space, three things must happen: >>> >>> 1) reservation of the space obviously. If cds is active that space must >>> be in the vicinity of cds archives to be covered by compressed class >>> pointer encoding. >>> 2) setting up the internal Metaspace structures atop of that space >>> 3) setting up compressed class pointer encoding. >>> >>> In its current form, Metaspace may or may not do some or all of that in >>> one function >>> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>> metaspace_rs, char* requested_addr, address cds_base);) - if cds is active, >>> it will reserve the space for Metaspace and hand it in, otherwise it will >>> create it itself. >>> >>> When discussing this in [2], Ioi proposed to move the reservation of the >>> class space completely out of Metaspace and make it a responsibility of the >>> caller always. This would reduce some complexity, and this patch follows >>> the proposal. >>> >>> I removed >>> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>> metaspace_rs, char* requested_addr, address cds_base); and all its sub >>> functions. >>> >>> (1) now has to always be done outside - a ReservedSpace for class space >>> has to be provided by the caller. However, Metaspace now offers a utility >>> function for reserving space at a "nice" location, and explicitly doing >>> nothing else: >>> >>> ReservedSpace >>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>> >>> this function can be redefined on a platform level for platform >>> optimized reservation, see below for details. >>> >>> (2) is taken care of by a new function, >>> Metaspace::initialize_class_space(ReservedSpace rs) >>> >>> (3) is taken care of a new function >>> CompressedKlassPointers::initialize(), see below for details. >>> >>> >>> So, class space now is set up three explicit steps: >>> >>> - First, reserve a suitable space by however means you want. For >>> convenience you may use >>> Metaspace::reserve_address_space_for_compressed_classes(), or you may roll >>> your own reservation. >>> - Next, tell Metaspace to use that range as backing storage for class >>> space: Metaspace::initialize_class_space(ReservedSpace rs) >>> - Finally, set up encoding. Encoding is independent from the concept of >>> a ReservedSpace, it just gets an address range, see below for details. >>> >>> Separating these steps and moving them out of the responsibility of >>> Metaspace makes this whole thing more flexible; it also removes unnecessary >>> knowledge (e.g. Metaspace does not need to know anything about either ccp >>> encoding or cds). >>> >>> --- >>> >>> How it comes together: >>> >>> If CDS is off, we just reserve a space using >>> Metaspace::reserve_address_space_for_compressed_classes(), initialize it >>> with Metaspace::initialize_class_space(ReservedSpace rs), then set up >>> compressed class pointer encoding covering the range of this class space. >>> >>> If CDS is on (dump time), we reserve large 4G space, either at >>> SharedBaseAddress or using >>> Metaspace::reserve_address_space_for_compressed_classes(); we then split >>> that into 3G archive space and 1G class space; we set up that space with >>> Metaspace as class space; then we set up compressed class pointer encoding >>> covering both archive space and cds. >>> >>> If CDS is on (run time), we reserve a large space, split it into archive >>> space (large enough to hold both archives) and class space, then basically >>> proceed as above. >>> >>> Note that this is almost exactly how things worked before (modulo some >>> minor fixes, e.g. alignment issues), only the code is reformed and made >>> more explicit. >>> >>> --- >>> >>> I moved compressed class pointer setup over to CompressedKlassPointers >>> and changed the interface: >>> >>> -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >>> metaspace_rs, address cds_base) >>> +void CompressedKlassPointers::initialize(address addr, size_t len); >>> >>> Instead of feeding it a single ReservedSpace, which is supposed to >>> represent class space, and an optional alternate base if cds is on, now we >>> give it just an numeric address range. That range marks the limits to where >>> Klass structures are to be expected, and is the implicit promise that >>> outside that range no Klass structures will exist, so encoding has to cover >>> only this range. >>> >>> This range may contain just the class space; or class space+cds; or >>> whatever allocation scheme we come up with in the future. Encoding does not >>> really care how the memory is organized as long as the input range covers >>> all possible Klass locations. That way we remove knowledge about class >>> space/cds from compressed class pointer encoding. >>> >>> Moving it away from metaspace.cpp into the CompressedKlassPointers class >>> also mirrors CompressedOops::initialize(). >>> >>> --- >>> >>> I renamed _narrow_klass_range to just _range, because strictly speaking >>> this is the range un-narrow Klass pointers can have. >>> >>> As for the implementation of CompressedKlassPointers::initialize(address >>> addr, size_t len), I mimicked very closely what happened before, so there >>> should be almost no differences. Since "almost no differences" sounds scary >>> :) here are the differences: >>> >>> - When CDS is active (dump or run time) we now always, unconditionally, >>> set the encoding range to 4G. This fixes a theoretical bug discussed on >>> aarch64-port-dev [1]. >>> >>> - When CDS is not active, we set the encoding range to the minimum >>> required length. Before, it was left at its default value of 4G. >>> >>> Both differences only affect aarch64, since they are currently the only >>> one using the range field in CompressedKlassPointers. >>> >>> I wanted to add an assert somewhere to test encoding of the very last >>> address of the CompressedKlassPointers range, again to prevent errors like >>> [3]. But I did not come up with a good place for this assert which would >>> cover also the encoding done by C1/C2. >>> >>> For the same reason I thought about introducing a mode where Klass >>> structures would be allocated in reverse order, starting at the end of the >>> ccs, but again left it out as too big a change. >>> >>> --- >>> >>> OS abstraction: platforms may have restrictions of what constitutes a >>> valid compressed class pointer encoding base. Or if not, they may have at >>> least preferences. There was logic like this in metaspace.cpp, which I >>> removed and cleanly factored out into platform dependent files, giving each >>> platform the option to add special logic. >>> >>> These are two new methods: >>> >>> - bool CompressedKlassPointers::is_valid_base(address p) >>> >>> to let the platform tell you whether it considers p to be a valid >>> encoding base. The only platform having these restrictions currently is >>> aarch64. >>> >>> - ReservedSpace >>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>> >>> this hands over the process of allocating a range suitable for >>> compressed class pointer encoding to the platform. Most platforms will >>> allocate just anywhere, but some platforms may have a better strategy (e.g. >>> trying low memory first, trying only correctly aligned addresses and so >>> on). >>> >>> Beforehand, this coding existed in a similar form in metaspace.cpp for >>> aarch64 and AIX. For now, I left the AIX part out - it seems only half >>> done, and I want to check further if we even need it, if yes why not on >>> Linux ppc, and C1 does not seem to support anything other than base+offset >>> with shift either, but I may be mistaken. >>> >>> These two methods should give the platform enough control to implement >>> their own scheme for optimized class space placement without bothering any >>> shared code about it. >>> >>> Note about the form, I introduced two new platform dependent files, >>> "metaspace_.cpp" and "compressedOops_.cpp". I am not happy about >>> this but this seems to be what we generally do in hotspot, right? >>> >>> --- >>> >>> Metaspace reserve alignment vs cds alignment >>> >>> CDS was using Metaspace reserve alignment for CDS internal purposes. I >>> guess this was just a copy paste issue. It never caused problems since >>> Metaspace reserve alignment == page size, but that is not true anymore in >>> the upcoming Elastic Metaspace where reserve alignment will be larger. This >>> causes a number of issues. >>> >>> I separated those two cleanly. CDS now uses >>> os::vm_allocation_granularity. Metaspace::reserve_alignment is only used in >>> those two places where it is needed, when CDS creates the address space for >>> class space on behalf of the Metaspace. >>> >>> --- >>> >>> Windows special handling in CDS >>> >>> To simplify coding I removed the windows specific handling which left >>> out reservation of the archive. This was needed because windows cannot mmap >>> files into reserved regions. But fallback code exists in filemap.cpp for >>> this case which just reads in the region instead of mapping it. >>> >>> Should that turn out to be a performance problem, I will reinstate the >>> feature. But a simpler way would be reserve the archive and later just >>> before mmapping the archive file to release the archive space. That would >>> not only be simpler but give us the best guarantee that that address space >>> is actually available. But I'd be happy to leave that part out completely >>> if we do not see any performance problems on windows x64. >>> >>> --- >>> >>> NMT cannot deal with spaces which are split. This problem manifests in >>> that bookkeeping for class space is done under "Shared Classes", not >>> "Classes" as it should. This problem exists today too at dump time and >>> randomly at run time. But since I simplified the reservation, this problem >>> now shows up always, whether or not we map at the SharedBaseAddress. >>> While I could work around this problem, I'd prefer this problem to be >>> solved at the core, and NMT to have an option to recognize reservation >>> splits. So I'd rather not put a workaround for this into the patch but >>> leave it for fixing as a separate issue. I opened this issue to track it >>> [6]. >>> >>> --- >>> >>> Jtreg tests: >>> >>> I expanded the CompressedOops/CompressedClassPointers.java. I also >>> extended them to Windows. The tests now optionally omit strict class space >>> placement tests, since these tests heavily depend on ASLR and were the >>> reason they were excluded on Windows. However I think even without checking >>> for class space placement they make sense, just to see that the VM comes up >>> and lives with the many different settings we can run in. >>> >>> --- >>> >>> Tests: >>> >>> - I ran the patch through Oracles submit repo >>> - I ran tests manually for aarch64, zero, linux 32bit and windows x64 >>> - The whole battery of nightly tests at SAP, including ppc, ppcle and >>> aarch64, unfortunately excluding windows because of unrelated errors. >>> Windows x64 tests will be redone tonight. >>> >>> >>> Thank you, >>> >>> Thomas >>> >>> [1] >>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >>> [2] >>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >>> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >>> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >>> [5] >>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >>> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >>> >>> >> > From kim.barrett at oracle.com Tue May 19 05:24:36 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Tue, 19 May 2020 01:24:36 -0400 Subject: RFR: 8245236: Remove MO_VOLATILE Access decorator Message-ID: Please review this change with replaces uses of the MO_VOLATILE memory order Access decorator with uses of the MO_RELAXED decorator. Uses of MO_VOLATILE generally assume non-word-tearing semantics, so a relaxed atomic access is more appropriate. As a result, we can eliminate the MO_VOLATILE decorator. CR: https://bugs.openjdk.java.net/browse/JDK-8245236 Webrev: https://cr.openjdk.java.net/~kbarrett/8245236/open.00/ Testing: mach5 tier1. From david.holmes at oracle.com Tue May 19 05:50:47 2020 From: david.holmes at oracle.com (David Holmes) Date: Tue, 19 May 2020 15:50:47 +1000 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> <412316cc-fb7e-f360-5a76-78e98ddf2890@oracle.com> <459a6a6f-5c40-c6d2-af3b-9c76b8a4f57e@oracle.com> Message-ID: <6a9f8fd0-1beb-4ad8-2a24-76c19ddbe349@oracle.com> Ship it! :) Thanks for your patience on this. David On 18/05/2020 6:56 pm, Christian Hagedorn wrote: > Hi David > > On 16.05.20 06:01, David Holmes wrote: >> On 16/05/2020 12:06 am, Christian Hagedorn wrote: >>> While working on the replacement of Verbose by Unified Logging, I've >>> seen that we should also fix this bug for ARM and S390 (PPC and SPARC >>> already have the ResourceMark set correctly and Aarch64 is not >>> implementing trace_method_handle_stub()). I included the necessary >>> changes in a new webrev: >>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.02/ >> >> src/hotspot/cpu/ppc/methodHandles_ppc.cpp >> src/hotspot/cpu/x86/methodHandles_x86.cpp >> >> You should keep the existing block so that the scope of the >> PRESERVE_EXCEPTION_MARK is not changed. All this is needed is to move >> the ResourceMark. > > I see, I fixed that. > >> runtime/logging/TestMethodHandlesVerbose.java >> >> Note that there is no need to declare the test in the runtime.logging >> package. We generally do not use packages for regression tests. > > I adjusted the test to not use a package. > >>> Would be good if anybody could also have a quick look at these minor >>> changes for ARM and S390 and verify them (I have only built and >>> tested it on x86): >> >> You say ARM and S390 but the webrev only has changes for ARM and PPC. > > Oh, sorry. Was meant to say ARM and PPC! > > I included the changes in a new webrev. I also moved the ResourceMark on > ARM to the start of the block while keeping the scope for > PRESERVE_EXCEPTION_MARK as done for x86 and PPC now: > http://cr.openjdk.java.net/~chagedorn/8244946/webrev.03/ > > Best regards, > Christian > >> Thanks, >> David >> ----- >> >>> >>> Thank you! >>> >>> Best regards, >>> Christian >>> >>> On 15.05.20 15:08, Christian Hagedorn wrote: >>>> On 15.05.20 14:56, coleen.phillimore at oracle.com wrote: >>>>> On 5/15/20 8:06 AM, Christian Hagedorn wrote: >>>>>> Hi Coleen >>>>>> >>>>>> On 15.05.20 12:38, coleen.phillimore at oracle.com wrote: >>>>>>> On 5/15/20 4:03 AM, Christian Hagedorn wrote: >>>>>>>> On 15.05.20 00:38, David Holmes wrote: >>>>>>>>> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >>>>>>>>>> >>>>>>>>>> I thought that this code under Verbose should have been >>>>>>>>>> changed to use logging with log_level = verbose.? We were >>>>>>>>>> trying to minimize/remove the Verbose flag. >>>>>>>>>> >>>>>>>>>> On 5/14/20 9:50 AM, David Holmes wrote: >>>>>>>>>>> +1 >>>>>>>>>>> >>>>>>>>>>> But I'd be inclined to generalize the test to run: >>>>>>>>>>> >>>>>>>>>>> -Xlog:all=trace -XX:+Verbose >>>>>>>>>> >>>>>>>>>> Don't do this, it would be tons of output and make the test >>>>>>>>>> unusable if it fails. >>>>>>>>> >>>>>>>>> The only way the test can fail is if it crashes. >>>>>>>>> >>>>>>>>> But we would want a level of indirection so that the test log >>>>>>>>> doesn't contain the full output ~11000 lines of text. >>>>>>>> >>>>>>>> Could we still generalize the test but redirect the output to a >>>>>>>> file with: >>>>>>>> >>>>>>>> -Xlog:all=trace:file=output.txt -XX:+Verbose ? >>>>>>>> >>>>>>>> When I just immediately return from trace_method_handle_stub() >>>>>>>> without printing anything, the above command will not print >>>>>>>> anything to the console. So, it would print the same amount of >>>>>>>> lines to the console as the current option in webrev.01 with >>>>>>>> -XX:+Verbose -Xlog:methodhandles. >>>>>>> >>>>>>> I'm confused.? Does the test crash with -Xlog:methodhandles >>>>>>> -XX:+Verbose?? If so, that's all that should be needed.? If it >>>>>>> needs -Xlog:all=trace, then don't add this test.? Even if it >>>>>>> writes to an output file, it would be huge and affect test >>>>>>> execution, particularly if more logging is added. >>>>>> >>>>>> I could already reproduce it locally on my Linux machine with >>>>>> -Xlog:methodhandles -XX:+Verbose. From what I understood, I >>>>>> thought that David is just talking about having a more general log >>>>>> test that covers the entire logging together with -XX:+Verbose for >>>>>> a HelloWorld program. But as you said, that would have some bad >>>>>> effects on test execution. However, such a test is not required >>>>>> for this bug to be triggered. >>>>> >>>>> Good! This looks good to me then. >>>> >>>> Thank you Coleen for your review! >>>> >>>>> Thank you for filing an RFE to do more logging with Unified Logging. >>>> >>>> No problem, I will take a look at it next. >>>> >>>> Best regards, >>>> Christian >>>> >>>>>> >>>>>>>> In a next step we could file an RFE to replace Verbose with >>>>>>>> unified logging in trace_method_handle_stub() which currently is >>>>>>>> only using tty. Even when using -Xlog:methodhandles alone >>>>>>>> without -XX:+Verbose, there is still some printing done in >>>>>>>> trace_method_handle_stub() to tty by >>>>>>>> >>>>>>>> ?500?? tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, >>>>>>>> ?501???????????????? adaptername, mh_reg_name, >>>>>>>> ?502???????????????? p2i(mh), p2i(entry_sp)); >>>>>>>> >>>>>>>> which is probably also not desired, as it is guarded by >>>>>>>> log_is_enabled(Info, methodhandles) earlier [1]. This could also >>>>>>>> be fixed in this follow-up RFE. >>>>>>> >>>>>>> Please, yes, this can be fixed with a follow-up RFE. >>>>>> >>>>>> Great, I filed an RFE for it [1]. >>>>>> >>>>>> Best regards, >>>>>> Christian >>>>>> >>>>>> >>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8245107 >>>>>> >>>>>> >>>>>>>>>>> Curiously I couldn't reproduce the crash that this bug is >>>>>>>>>>> about. But the hs_err file speaks for itself. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> David >>>>>>>>>>> >>>>>>>>>>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>>>>>>>>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>>>>>>>>>> Hi David, hi Daniel >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you for your reviews! >>>>>>>>>>>>> >>>>>>>>>>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>>>>>>>>>> ???? Not your bug, but the comment has two typos: >>>>>>>>>>>>>> >>>>>>>>>>>>>> ???????? s/by safer/but safer/ >>>>>>>>>>>>>> ???????? s/unexpensive/inexpensive/ >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, fixed it. >>>>>>>>>>>>> >>>>>>>>>>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>>>>>>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>>>>>>>>>> >>>>>>>>>>>>>> Given all of the code is inside: >>>>>>>>>>>>>> >>>>>>>>>>>>>> ??504?? if (Verbose) { >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would just move the ResourceMark to the top of that >>>>>>>>>>>>>> block. We are going to use it unconditionally, and we are >>>>>>>>>>>>>> not releasing it until the end of the method regardless. >>>>>>>>>>>>>> That avoids the potential concern Dan raised about >>>>>>>>>>>>>> extending the scope of the PRESERVE_EXCEPTION_MARK. >>>>>>>>>>>>> >>>>>>>>>>>>> This sounds reasonable. I moved it to the top of that block. >>>>>>>>>>>>> >>>>>>>>>>>>>>> There are other places like >>>>>>>>>>>>>>> Klass::print_on(outputStream*) where this is still the >>>>>>>>>>>>>>> case. Should we file an RFE to clean these up? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Obviously we don't have any tests running in a mode that >>>>>>>>>>>>>> causes this code to be executed, so yes it is probably a >>>>>>>>>>>>>> good idea to take a closer look at other call-sites and >>>>>>>>>>>>>> file a RFE if needed. >>>>>>>>>>>>> >>>>>>>>>>>>> Yes, I filed one to take a closer look at these [1]. >>>>>>>>>>>>> >>>>>>>>>>>>> I included the changes in a new webrev: >>>>>>>>>>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>>>>>>>>>> >>>>>>>>>>>> Thumbs up. >>>>>>>>>>>> >>>>>>>>>>>> Dan >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Best regards, >>>>>>>>>>>>> Christian >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> From stefan.karlsson at oracle.com Tue May 19 06:25:26 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Tue, 19 May 2020 08:25:26 +0200 Subject: RFR: 8245236: Remove MO_VOLATILE Access decorator In-Reply-To: References: Message-ID: <6806a61b-f46a-022c-9a55-c7530fbfe084@oracle.com> On 2020-05-19 07:24, Kim Barrett wrote: > Please review this change with replaces uses of the MO_VOLATILE memory > order Access decorator with uses of the MO_RELAXED decorator. Uses of > MO_VOLATILE generally assume non-word-tearing semantics, so a relaxed > atomic access is more appropriate. As a result, we can eliminate the > MO_VOLATILE decorator. > > CR: > https://bugs.openjdk.java.net/browse/JDK-8245236 > > Webrev: > https://cr.openjdk.java.net/~kbarrett/8245236/open.00/ Looks good. One reflection after reading this. markWord oopDesc::mark() const { - uintptr_t v = HeapAccess::load_at(as_oop(), mark_offset_in_bytes()); + uintptr_t v = HeapAccess::load_at(as_oop(), mark_offset_in_bytes()); return markWord(v); } markWord oopDesc::mark_raw() const { return Atomic::load(&_mark); and void oopDesc::set_mark(markWord m) { - HeapAccess::store_at(as_oop(), mark_offset_in_bytes(), m.value()); + HeapAccess::store_at(as_oop(), mark_offset_in_bytes(), m.value()); } void oopDesc::set_mark_raw(markWord m) { Atomic::store(&_mark, m); } I noticed that the normal versions use HeapAccess and the _raw versions use Atomic::. The _raw versions where added to support Shenandoah's load barriers, but they have since changed their load barriers and don't need these functions anymore (AFAIK). Maybe its time to clean this up and get rid of all the _raw versions added for Shenandoah. StefanK > > Testing: > mach5 tier1. > From christian.hagedorn at oracle.com Tue May 19 07:35:50 2020 From: christian.hagedorn at oracle.com (Christian Hagedorn) Date: Tue, 19 May 2020 09:35:50 +0200 Subject: [15] RFR(XS): 8244946: fatal error: memory leak: allocating without ResourceMark with -XX:+Verbose -Xlog:methodhandles In-Reply-To: <6a9f8fd0-1beb-4ad8-2a24-76c19ddbe349@oracle.com> References: <4fe3803c-f3cb-96db-7718-c661142d9ffe@oracle.com> <89ecab87-0739-9453-2bca-61f37eeefe38@oracle.com> <7f55ee28-f3a8-c477-ec6f-3ed48a723e66@oracle.com> <412316cc-fb7e-f360-5a76-78e98ddf2890@oracle.com> <459a6a6f-5c40-c6d2-af3b-9c76b8a4f57e@oracle.com> <6a9f8fd0-1beb-4ad8-2a24-76c19ddbe349@oracle.com> Message-ID: Thanks a lot for your review David! :) Best regards, Christian On 19.05.20 07:50, David Holmes wrote: > Ship it! :) > > Thanks for your patience on this. > > David > > On 18/05/2020 6:56 pm, Christian Hagedorn wrote: >> Hi David >> >> On 16.05.20 06:01, David Holmes wrote: >>> On 16/05/2020 12:06 am, Christian Hagedorn wrote: >>>> While working on the replacement of Verbose by Unified Logging, I've >>>> seen that we should also fix this bug for ARM and S390 (PPC and >>>> SPARC already have the ResourceMark set correctly and Aarch64 is not >>>> implementing trace_method_handle_stub()). I included the necessary >>>> changes in a new webrev: >>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.02/ >>> >>> src/hotspot/cpu/ppc/methodHandles_ppc.cpp >>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>> >>> You should keep the existing block so that the scope of the >>> PRESERVE_EXCEPTION_MARK is not changed. All this is needed is to move >>> the ResourceMark. >> >> I see, I fixed that. >> >>> runtime/logging/TestMethodHandlesVerbose.java >>> >>> Note that there is no need to declare the test in the runtime.logging >>> package. We generally do not use packages for regression tests. >> >> I adjusted the test to not use a package. >> >>>> Would be good if anybody could also have a quick look at these minor >>>> changes for ARM and S390 and verify them (I have only built and >>>> tested it on x86): >>> >>> You say ARM and S390 but the webrev only has changes for ARM and PPC. >> >> Oh, sorry. Was meant to say ARM and PPC! >> >> I included the changes in a new webrev. I also moved the ResourceMark >> on ARM to the start of the block while keeping the scope for >> PRESERVE_EXCEPTION_MARK as done for x86 and PPC now: >> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.03/ >> >> Best regards, >> Christian >> >>> Thanks, >>> David >>> ----- >>> >>>> >>>> Thank you! >>>> >>>> Best regards, >>>> Christian >>>> >>>> On 15.05.20 15:08, Christian Hagedorn wrote: >>>>> On 15.05.20 14:56, coleen.phillimore at oracle.com wrote: >>>>>> On 5/15/20 8:06 AM, Christian Hagedorn wrote: >>>>>>> Hi Coleen >>>>>>> >>>>>>> On 15.05.20 12:38, coleen.phillimore at oracle.com wrote: >>>>>>>> On 5/15/20 4:03 AM, Christian Hagedorn wrote: >>>>>>>>> On 15.05.20 00:38, David Holmes wrote: >>>>>>>>>> On 15/05/2020 12:02 am, coleen.phillimore at oracle.com wrote: >>>>>>>>>>> >>>>>>>>>>> I thought that this code under Verbose should have been >>>>>>>>>>> changed to use logging with log_level = verbose.? We were >>>>>>>>>>> trying to minimize/remove the Verbose flag. >>>>>>>>>>> >>>>>>>>>>> On 5/14/20 9:50 AM, David Holmes wrote: >>>>>>>>>>>> +1 >>>>>>>>>>>> >>>>>>>>>>>> But I'd be inclined to generalize the test to run: >>>>>>>>>>>> >>>>>>>>>>>> -Xlog:all=trace -XX:+Verbose >>>>>>>>>>> >>>>>>>>>>> Don't do this, it would be tons of output and make the test >>>>>>>>>>> unusable if it fails. >>>>>>>>>> >>>>>>>>>> The only way the test can fail is if it crashes. >>>>>>>>>> >>>>>>>>>> But we would want a level of indirection so that the test log >>>>>>>>>> doesn't contain the full output ~11000 lines of text. >>>>>>>>> >>>>>>>>> Could we still generalize the test but redirect the output to a >>>>>>>>> file with: >>>>>>>>> >>>>>>>>> -Xlog:all=trace:file=output.txt -XX:+Verbose ? >>>>>>>>> >>>>>>>>> When I just immediately return from trace_method_handle_stub() >>>>>>>>> without printing anything, the above command will not print >>>>>>>>> anything to the console. So, it would print the same amount of >>>>>>>>> lines to the console as the current option in webrev.01 with >>>>>>>>> -XX:+Verbose -Xlog:methodhandles. >>>>>>>> >>>>>>>> I'm confused.? Does the test crash with -Xlog:methodhandles >>>>>>>> -XX:+Verbose?? If so, that's all that should be needed.? If it >>>>>>>> needs -Xlog:all=trace, then don't add this test.? Even if it >>>>>>>> writes to an output file, it would be huge and affect test >>>>>>>> execution, particularly if more logging is added. >>>>>>> >>>>>>> I could already reproduce it locally on my Linux machine with >>>>>>> -Xlog:methodhandles -XX:+Verbose. From what I understood, I >>>>>>> thought that David is just talking about having a more general >>>>>>> log test that covers the entire logging together with >>>>>>> -XX:+Verbose for a HelloWorld program. But as you said, that >>>>>>> would have some bad effects on test execution. However, such a >>>>>>> test is not required for this bug to be triggered. >>>>>> >>>>>> Good! This looks good to me then. >>>>> >>>>> Thank you Coleen for your review! >>>>> >>>>>> Thank you for filing an RFE to do more logging with Unified Logging. >>>>> >>>>> No problem, I will take a look at it next. >>>>> >>>>> Best regards, >>>>> Christian >>>>> >>>>>>> >>>>>>>>> In a next step we could file an RFE to replace Verbose with >>>>>>>>> unified logging in trace_method_handle_stub() which currently >>>>>>>>> is only using tty. Even when using -Xlog:methodhandles alone >>>>>>>>> without -XX:+Verbose, there is still some printing done in >>>>>>>>> trace_method_handle_stub() to tty by >>>>>>>>> >>>>>>>>> ?500?? tty->print_cr("MH %s %s=" PTR_FORMAT " sp=" PTR_FORMAT, >>>>>>>>> ?501???????????????? adaptername, mh_reg_name, >>>>>>>>> ?502???????????????? p2i(mh), p2i(entry_sp)); >>>>>>>>> >>>>>>>>> which is probably also not desired, as it is guarded by >>>>>>>>> log_is_enabled(Info, methodhandles) earlier [1]. This could >>>>>>>>> also be fixed in this follow-up RFE. >>>>>>>> >>>>>>>> Please, yes, this can be fixed with a follow-up RFE. >>>>>>> >>>>>>> Great, I filed an RFE for it [1]. >>>>>>> >>>>>>> Best regards, >>>>>>> Christian >>>>>>> >>>>>>> >>>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8245107 >>>>>>> >>>>>>> >>>>>>>>>>>> Curiously I couldn't reproduce the crash that this bug is >>>>>>>>>>>> about. But the hs_err file speaks for itself. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> David >>>>>>>>>>>> >>>>>>>>>>>> On 14/05/2020 11:28 pm, Daniel D. Daugherty wrote: >>>>>>>>>>>>> On 5/14/20 4:00 AM, Christian Hagedorn wrote: >>>>>>>>>>>>>> Hi David, hi Daniel >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you for your reviews! >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 13.05.20 17:54, Daniel D. Daugherty wrote: >>>>>>>>>>>>>>> ???? Not your bug, but the comment has two typos: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ???????? s/by safer/but safer/ >>>>>>>>>>>>>>> ???????? s/unexpensive/inexpensive/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, fixed it. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 14.05.20 01:51, David Holmes wrote: >>>>>>>>>>>>>>> src/hotspot/cpu/x86/methodHandles_x86.cpp >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Given all of the code is inside: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ??504?? if (Verbose) { >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I would just move the ResourceMark to the top of that >>>>>>>>>>>>>>> block. We are going to use it unconditionally, and we are >>>>>>>>>>>>>>> not releasing it until the end of the method regardless. >>>>>>>>>>>>>>> That avoids the potential concern Dan raised about >>>>>>>>>>>>>>> extending the scope of the PRESERVE_EXCEPTION_MARK. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This sounds reasonable. I moved it to the top of that block. >>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There are other places like >>>>>>>>>>>>>>>> Klass::print_on(outputStream*) where this is still the >>>>>>>>>>>>>>>> case. Should we file an RFE to clean these up? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Obviously we don't have any tests running in a mode that >>>>>>>>>>>>>>> causes this code to be executed, so yes it is probably a >>>>>>>>>>>>>>> good idea to take a closer look at other call-sites and >>>>>>>>>>>>>>> file a RFE if needed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, I filed one to take a closer look at these [1]. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I included the changes in a new webrev: >>>>>>>>>>>>>> http://cr.openjdk.java.net/~chagedorn/8244946/webrev.01/ >>>>>>>>>>>>> >>>>>>>>>>>>> Thumbs up. >>>>>>>>>>>>> >>>>>>>>>>>>> Dan >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>> Christian >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8244999 >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>> From thomas.schatzl at oracle.com Tue May 19 08:06:59 2020 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 19 May 2020 10:06:59 +0200 Subject: RFR: 8245236: Remove MO_VOLATILE Access decorator In-Reply-To: References: Message-ID: <86c4e8a6-abd2-f15c-19fc-36832b4ca19e@oracle.com> Hi, On 19.05.20 07:24, Kim Barrett wrote: > Please review this change with replaces uses of the MO_VOLATILE memory > order Access decorator with uses of the MO_RELAXED decorator. Uses of > MO_VOLATILE generally assume non-word-tearing semantics, so a relaxed > atomic access is more appropriate. As a result, we can eliminate the > MO_VOLATILE decorator. > > CR: > https://bugs.openjdk.java.net/browse/JDK-8245236 > > Webrev: > https://cr.openjdk.java.net/~kbarrett/8245236/open.00/ > > Testing: > mach5 tier1. > looks good. Thomas From aph at redhat.com Tue May 19 09:26:36 2020 From: aph at redhat.com (Andrew Haley) Date: Tue, 19 May 2020 10:26:36 +0100 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: Message-ID: <7362f008-ed11-eca3-4b2e-d0c38a531e1d@redhat.com> Hi, In general, please wrap comment lines at 80 characters. Some of these lines are too long to read. On 5/7/20 3:21 PM, Thomas St?fe wrote: + // Note: on AARCH64, if the code above does not find any good placement, we + // have no recurse. We return an empty space and the VM will exit. return ReservedSpace(); Spelling: "we have no recourse" + // We do this by reserving space for the ccs behind the archives. Note however that Should be "below the archives", surely? Or is that "above"? Which is it? + bool use_requested_base = true; // Whether to use SharedBaseAddress as attach point. What exactly is an "attach point" ? The phrase is used few times in comments but never defined. +// class space according to CompressedClassSpaceSize. Its start address will be aligned to etaspace::reserve_alignment "Metaspace" -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From weijun.wang at oracle.com Tue May 19 09:32:20 2020 From: weijun.wang at oracle.com (Weijun Wang) Date: Tue, 19 May 2020 17:32:20 +0800 Subject: RFR 8245280: Add runtime/cds/appcds/SignedJar.java to problem list Message-ID: <60C599B9-414F-486B-A938-A3EB25FED221@oracle.com> This test always fails now. https://bugs.openjdk.java.net/browse/JDK-8245264 is filed. At the meantime, add it to problem list first. Here is the patch, please take a review: diff --git a/test/hotspot/jtreg/ProblemList.txt b/test/hotspot/jtreg/ProblemList.txt --- a/test/hotspot/jtreg/ProblemList.txt +++ b/test/hotspot/jtreg/ProblemList.txt @@ -96,6 +96,8 @@ runtime/jni/terminatedThread/TestTerminatedThread.java 8219652 aix-ppc64 runtime/ReservedStack/ReservedStackTest.java 8231031 generic-all +runtime/cds/appcds/SignedJar.java 8245264 generic-all + ############################################################################# # :hotspot_serviceability Thanks, Max From thomas.stuefe at gmail.com Tue May 19 10:06:25 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 19 May 2020 12:06:25 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: <7362f008-ed11-eca3-4b2e-d0c38a531e1d@redhat.com> References: <7362f008-ed11-eca3-4b2e-d0c38a531e1d@redhat.com> Message-ID: On Tue, May 19, 2020 at 11:26 AM Andrew Haley wrote: > Hi, > > In general, please wrap comment lines at 80 characters. Some of these lines > are too long to read. > okay. > > On 5/7/20 3:21 PM, Thomas St?fe wrote: > > > + // Note: on AARCH64, if the code above does not find any good > placement, we > + // have no recurse. We return an empty space and the VM will exit. > return ReservedSpace(); > > Spelling: "we have no recourse" > > Will fix. > + // We do this by reserving space for the ccs behind the archives. > Note however that > > Should be "below the archives", surely? Or is that "above"? Which is it? > Above. Will fix. > > + bool use_requested_base = true; // Whether to use SharedBaseAddress as > attach point. > > What exactly is an "attach point" ? The phrase is used few times in > comments but never defined. > > Attach address. I'll clarify the comments. > +// class space according to CompressedClassSpaceSize. Its start > address will be aligned to etaspace::reserve_alignment > > "Metaspace" > > Thanks for looking this over Andrew. I'll wait for Coleens feedback and will post an update. ..Thomas > -- > Andrew Haley (he/him) > Java Platform Lead Engineer > Red Hat UK Ltd. > https://keybase.io/andrewhaley > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 > > From nick.gasson at arm.com Tue May 19 10:45:40 2020 From: nick.gasson at arm.com (Nick Gasson) Date: Tue, 19 May 2020 18:45:40 +0800 Subject: [aarch64-port-dev ] AArch64: Neoverse N1 atomic operations and -moutline-atomics In-Reply-To: <280d2657-7997-1dac-f370-8ff41a0b3035@redhat.com> References: <3a5e43e7-2039-8574-7470-5827267fb88d@redhat.com> <854ksd3egu.fsf@arm.com> <280d2657-7997-1dac-f370-8ff41a0b3035@redhat.com> Message-ID: On 05/18/20 17:53 PM, Andrew Haley wrote: > >> Question is whether it's best to enable this by default in OpenJDK or >> leave it to downstream distributions to decide? > > I don't really understand why there is a conditional branch there. > They could just do the test at startup and patch the code. Maybe it'd > hurt DSO startup time or something? Maybe they don't like the idea of > temporarily enabling exec privilege on a page of writable data. I asked internally and it's basically the reasons you mention above, plus patching could cause excessive copy-on-writes of the code pages. The other way to do it is to build one libjvm.so without LSE support and then a separate atomics/libjvm.so with -march set to enable the LSE instructions. Then ld.so will load the appropriate library based on the hwcaps. This should give the best performance but the build time and package size balloons. -- Nick From coleen.phillimore at oracle.com Tue May 19 10:50:21 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 19 May 2020 06:50:21 -0400 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <7362f008-ed11-eca3-4b2e-d0c38a531e1d@redhat.com> Message-ID: Hi, I have no more comments.? It all looks good to me.? I think we're waiting for Ioi to run through CI and he might have a comment on the check_SharedBaseAddress change which seems reasonable to me. I like that it's not in arguments.cpp.? Too much dispersion of knowledge to arguments.cpp. http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/src/hotspot/share/runtime/arguments.cpp.udiff.html You may not need this change anymore. Thanks, Coleen On 5/19/20 6:06 AM, Thomas St?fe wrote: > > > On Tue, May 19, 2020 at 11:26 AM Andrew Haley > wrote: > > Hi, > > In general, please wrap comment lines at 80 characters. Some of > these lines > are too long to read. > > > okay. > > > On 5/7/20 3:21 PM, Thomas St?fe wrote: > > > +? // Note: on AARCH64, if the code above does not find any good > placement, we > +? // have no recurse. We return an empty space and the VM will exit. > ? ?return ReservedSpace(); > > Spelling: "we have no recourse" > > > Will fix. > > +? ? // We do this by reserving space for the ccs behind the > archives. Note however that > > Should be "below the archives", surely? Or is that "above"? Which > is it? > > > Above. Will fix. > > > +? bool use_requested_base = true; // Whether to use > SharedBaseAddress as attach point. > > What exactly is an "attach point" ? The phrase is used few times in > comments but never defined. > > > Attach address. I'll clarify the comments. > > +//? ? ? class space according to CompressedClassSpaceSize. Its > start address will be aligned to etaspace::reserve_alignment > > "Metaspace" > > > Thanks for looking this over Andrew. I'll wait for Coleens feedback > and will post an update. > > ..Thomas > > -- > Andrew Haley? (he/him) > Java Platform Lead Engineer > Red Hat UK Ltd. > https://keybase.io/andrewhaley > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 > From Yang.Zhang at arm.com Tue May 19 10:55:09 2020 From: Yang.Zhang at arm.com (Yang Zhang) Date: Tue, 19 May 2020 10:55:09 +0000 Subject: RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes Message-ID: Hi, Following up on review requests of API [0], Java implementation and test [1], General Hotspot changes[2] for Vector API and x86 backend changes [3]. Here's a request for review of AArch64 backend changes required for supporting the Vector API: JEP: https://openjdk.java.net/jeps/338 JBS: https://bugs.openjdk.java.net/browse/JDK-8223347 Webrev: http://cr.openjdk.java.net/~yzhang/vectorapi/vectorapi.rfr/aarch64_webrev/webrev.01/ Complete implementation resides in vector-unstable branch of panama/dev repository [4]. Looking forward to your feedback. Best Regards, Yang [0] https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html [1] https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-April/065587.html [2] https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037798.html [3] https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-April/037801.html [4] https://openjdk.java.net/projects/panama/ $ hg clone http://hg.openjdk.java.net/panama/dev/ -b vector-unstable From coleen.phillimore at oracle.com Tue May 19 11:00:46 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 19 May 2020 07:00:46 -0400 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <7362f008-ed11-eca3-4b2e-d0c38a531e1d@redhat.com> Message-ID: <51605673-b784-e4aa-0ac8-08b99f19e6f4@oracle.com> Hi, I have no more comments.? It all looks good to me.? I think we're waiting for Ioi to run through CI and he might have a comment on the check_SharedBaseAddress change which seems reasonable to me. I like that it's not in arguments.cpp.? Too much dispersion of knowledge to arguments.cpp. http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/src/hotspot/share/runtime/arguments.cpp.udiff.html You may not need this change anymore. Thanks, Coleen On 5/19/20 6:06 AM, Thomas St?fe wrote: > > > On Tue, May 19, 2020 at 11:26 AM Andrew Haley > wrote: > > Hi, > > In general, please wrap comment lines at 80 characters. Some of > these lines > are too long to read. > > > okay. > > > On 5/7/20 3:21 PM, Thomas St?fe wrote: > > > +? // Note: on AARCH64, if the code above does not find any good > placement, we > +? // have no recurse. We return an empty space and the VM will exit. > ? ?return ReservedSpace(); > > Spelling: "we have no recourse" > > > Will fix. > > +? ? // We do this by reserving space for the ccs behind the > archives. Note however that > > Should be "below the archives", surely? Or is that "above"? Which > is it? > > > Above. Will fix. > > > +? bool use_requested_base = true; // Whether to use > SharedBaseAddress as attach point. > > What exactly is an "attach point" ? The phrase is used few times in > comments but never defined. > > > Attach address. I'll clarify the comments. > > +//? ? ? class space according to CompressedClassSpaceSize. Its > start address will be aligned to etaspace::reserve_alignment > > "Metaspace" > > > Thanks for looking this over Andrew. I'll wait for Coleens feedback > and will post an update. > > ..Thomas > > -- > Andrew Haley? (he/him) > Java Platform Lead Engineer > Red Hat UK Ltd. > https://keybase.io/andrewhaley > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 > From stefan.karlsson at oracle.com Tue May 19 11:53:17 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Tue, 19 May 2020 13:53:17 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> Message-ID: Updated webrevs: ?https://cr.openjdk.java.net/~stefank/8245002/webrev.02.workaround.kims-review ?https://cr.openjdk.java.net/~stefank/8245002/webrev.all.kims-review StefanK On 2020-05-18 10:17, Stefan Karlsson wrote: > On 2020-05-15 20:39, Kim Barrett wrote: >>> On May 15, 2020, at 1:46 PM, Stefan Karlsson >>> wrote: >>> >>> Hi all, >>> >>> Please review this patch to turn off -XX:+UseNUMAInterleaving as >>> long as Windows GDI functions don't work with memory that crosses >>> memory reservation boundaries. >>> >>> https://cr.openjdk.java.net/~stefank/8245002/webrev.00.cleanup >>> https://cr.openjdk.java.net/~stefank/8245002/webrev.01.workaround >>> https://cr.openjdk.java.net/~stefank/8245002/webrev.all >>> >>> https://bugs.openjdk.java.net/browse/JDK-8245002 >> ------------------------------------------------------------------------------ >> >> src/hotspot/os/windows/os_windows.cpp >> 2753?? void* verify_mem = ::malloc(4 * 1024); >> >> There's no corresponding free. > > Fixed. > >> >> ------------------------------------------------------------------------------ >> >> src/hotspot/os/windows/os_windows.cpp >> 2776???? assert(false, "Reservation failed"); >> 2791???? assert(false, "Failed to commit memory"); >> 2802???? assert(false, "Failed to commit memory"); >> >> These look like leftover debugging code? > Not really. I left them in place so that it we would fail early in our > testing, if we couldn't even create a few pages. However, I realize > that this might be more problematic when we start to use this function > for large pages, so I'll remove them. > > Thanks, > StefanK > >> >> ------------------------------------------------------------------------------ >> >> > From stefan.johansson at oracle.com Tue May 19 12:10:14 2020 From: stefan.johansson at oracle.com (stefan.johansson at oracle.com) Date: Tue, 19 May 2020 14:10:14 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> Message-ID: <2aa1bb15-444e-4967-2850-c098fb5be0f9@oracle.com> Hi Stefan, On 2020-05-19 13:53, Stefan Karlsson wrote: > Updated webrevs: > ?https://cr.openjdk.java.net/~stefank/8245002/webrev.02.workaround.kims-review > ?https://cr.openjdk.java.net/~stefank/8245002/webrev.all.kims-review > Looks good, StefanJ > StefanK > > On 2020-05-18 10:17, Stefan Karlsson wrote: >> On 2020-05-15 20:39, Kim Barrett wrote: >>>> On May 15, 2020, at 1:46 PM, Stefan Karlsson >>>> wrote: >>>> >>>> Hi all, >>>> >>>> Please review this patch to turn off -XX:+UseNUMAInterleaving as >>>> long as Windows GDI functions don't work with memory that crosses >>>> memory reservation boundaries. >>>> >>>> https://cr.openjdk.java.net/~stefank/8245002/webrev.00.cleanup >>>> https://cr.openjdk.java.net/~stefank/8245002/webrev.01.workaround >>>> https://cr.openjdk.java.net/~stefank/8245002/webrev.all >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8245002 >>> ------------------------------------------------------------------------------ >>> >>> src/hotspot/os/windows/os_windows.cpp >>> 2753?? void* verify_mem = ::malloc(4 * 1024); >>> >>> There's no corresponding free. >> >> Fixed. >> >>> >>> ------------------------------------------------------------------------------ >>> >>> src/hotspot/os/windows/os_windows.cpp >>> 2776???? assert(false, "Reservation failed"); >>> 2791???? assert(false, "Failed to commit memory"); >>> 2802???? assert(false, "Failed to commit memory"); >>> >>> These look like leftover debugging code? >> Not really. I left them in place so that it we would fail early in our >> testing, if we couldn't even create a few pages. However, I realize >> that this might be more problematic when we start to use this function >> for large pages, so I'll remove them. >> >> Thanks, >> StefanK >> >>> >>> ------------------------------------------------------------------------------ >>> >>> >> > From thomas.schatzl at oracle.com Tue May 19 12:23:53 2020 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 19 May 2020 14:23:53 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> Message-ID: <4cf0265c-f901-e1fb-edf6-dc7bdb0a0bc7@oracle.com> Hi, On 19.05.20 13:53, Stefan Karlsson wrote: > Updated webrevs: > ?https://cr.openjdk.java.net/~stefank/8245002/webrev.02.workaround.kims-review > ?https://cr.openjdk.java.net/~stefank/8245002/webrev.all.kims-review - s/adjecent/adjacent - From CreateBitmap documentation (https://docs.microsoft.com/en-us/windows/win32/api/wingdi/nf-wingdi-createbitmap): "A pointer to an array of color data used to set the colors in a rectangle of pixels. Each scan line in the rectangle must be word aligned (scan lines that are not word aligned must be padded with zeros). If this parameter is NULL, the contents of the new bitmap is undefined." I am not sure if that alignment requirement refers to the start of the scanline or the length of the scanline (or both), it seems that this refers to the length only though. In your example the start of the first (and only) scanline is guaranteed to be not word aligned (2808 void* bitmapBits = (char*)mem1 - 1;) so this test may fail (on different Windows versions?) for different reasons. Since the code seems to work, I am okay with it though. If you only intend to fix the typo, I do not need a re-review. Looks good. Thanks, Thomas From stefan.johansson at oracle.com Tue May 19 13:16:32 2020 From: stefan.johansson at oracle.com (stefan.johansson at oracle.com) Date: Tue, 19 May 2020 15:16:32 +0200 Subject: RFR: 8245000: Windows GDI functions don't support large pages In-Reply-To: References: Message-ID: Hi Stefan, On 2020-05-15 19:47, Stefan Karlsson wrote: > Hi all, > > Please review this patch to turn off -XX:+UseLargePages as long as > Windows GDI functions don't work with large pages. > > https://cr.openjdk.java.net/~stefank/8245000/webrev.00.cleanup > https://cr.openjdk.java.net/~stefank/8245000/webrev.01.workaround > https://cr.openjdk.java.net/~stefank/8245000/webrev.all/ Thanks for splitting those changes up into cleanup and fix, really makes it easy to review. Looks good, StefanJ > > The patches build upon the changes in: > 8245002: Windows GDI functions don't support NUMA interleaving > https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-May/041798.html > > During the investigation of JDK-8240654 > it was found that > some of the Windows GDI functions don't work with large pages > (-XX:+UseLargePages). This causes severe repaint issues. > > The problem can easily be seen with the Java2D demos, where the drawn UI > is grayed out instead of containing all relevant components. > > This can be reproduced by running building J2DBench and running: > java -Xmx1g -Xms1g -XX:+UseLargePages -Dsun.java2d.opengl=False > -Dsun.java2d.d3d=False -jar dist/J2DBench.jar > > The -Dsun.java2d.d3d property is used to ensure that that java2d doesn't > enable the alternative, D3D implementation instead of GDI. > -Dsun.java2d.opengl is turned off for the same reason. > > The cleanup patch simplifies request_lock_memory_privilege and > large_page_init, to make it easier to insert the necessary checks. > > The code first checks if memory allocated with large pages can be used > by GDI functions. If this check fails it forcefully turns off > -XX:+UseLargePages. > > Just like with JDK-8245002, the intention is that as long as the GDI > functions can't use large pages, the -XX:+UseLargePages flag will be > turned off. If Microsoft changes the GDI code, so that it works with > large pages, then the proposed patch will automatically allow the flag > to be enabled. > > There's a second case that we need to consider for large pages. If the > GDI functions start to allow large pages, we have a flag > (-XX:+UseLargePagesIndividualAllocation) that reserves and commits large > pages in page-sized granules. This could cause the same > multi-reservation problem described in JDK-8245002. I've also added a > check for that, and turn off -XX:+UseLargePagesIndividualAllocation if > it fails. > > I've run this through tier1-3 and also tested this manually. I've > manually turned off the different checks to verify that it turns on the > two flags. > > Thanks, > StefanK From daniel.daugherty at oracle.com Tue May 19 13:35:07 2020 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 19 May 2020 09:35:07 -0400 Subject: RFR 8245280: Add runtime/cds/appcds/SignedJar.java to problem list In-Reply-To: <60C599B9-414F-486B-A938-A3EB25FED221@oracle.com> References: <60C599B9-414F-486B-A938-A3EB25FED221@oracle.com> Message-ID: <723c1141-a577-7c3a-8ea0-a307ece45ef9@oracle.com> Thumbs up! This is a trivial fix, only needs a single reviewer and does not have to wait for 24 hours. Dan On 5/19/20 5:32 AM, Weijun Wang wrote: > This test always fails now. https://bugs.openjdk.java.net/browse/JDK-8245264 is filed. At the meantime, add it to problem list first. > > Here is the patch, please take a review: > > diff --git a/test/hotspot/jtreg/ProblemList.txt b/test/hotspot/jtreg/ProblemList.txt > --- a/test/hotspot/jtreg/ProblemList.txt > +++ b/test/hotspot/jtreg/ProblemList.txt > @@ -96,6 +96,8 @@ > runtime/jni/terminatedThread/TestTerminatedThread.java 8219652 aix-ppc64 > runtime/ReservedStack/ReservedStackTest.java 8231031 generic-all > > +runtime/cds/appcds/SignedJar.java 8245264 generic-all > + > ############################################################################# > > # :hotspot_serviceability > > Thanks, > Max > From stefan.karlsson at oracle.com Tue May 19 14:04:43 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Tue, 19 May 2020 16:04:43 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: <4cf0265c-f901-e1fb-edf6-dc7bdb0a0bc7@oracle.com> References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> <4cf0265c-f901-e1fb-edf6-dc7bdb0a0bc7@oracle.com> Message-ID: <8e4aeff8-df10-538b-d65f-b8ed05217244@oracle.com> On 2020-05-19 14:23, Thomas Schatzl wrote: > Hi, > > On 19.05.20 13:53, Stefan Karlsson wrote: >> Updated webrevs: >> ??https://cr.openjdk.java.net/~stefank/8245002/webrev.02.workaround.kims-review >> >> ??https://cr.openjdk.java.net/~stefank/8245002/webrev.all.kims-review > > - s/adjecent/adjacent Fixed. > > - From CreateBitmap documentation > (https://docs.microsoft.com/en-us/windows/win32/api/wingdi/nf-wingdi-createbitmap): > > > "A pointer to an array of color data used to set the colors in a > rectangle of pixels. Each scan line in the rectangle must be word > aligned (scan lines that are not word aligned must be padded with > zeros). If this parameter is NULL, the contents of the new bitmap is > undefined." > > I am not sure if that alignment requirement refers to the start of the > scanline or the length of the scanline (or both), it seems that this > refers to the length only though. > > In your example the start of the first (and only) scanline is guaranteed > to be not word aligned (2808?? void* bitmapBits = (char*)mem1 - 1;) so > this test may fail (on different Windows versions?) for different reasons. > > Since the code seems to work, I am okay with it though. > > If you only intend to fix the typo, I do not need a re-review. I've tested by misaligning the memory, and it seems to work. However, I've updated the code to ensure that both memory and and width bytes are aligned, just to be sure. I've also added some more comments to describe the chosen value: https://cr.openjdk.java.net/~stefank/8245002/webrev.03.workaround.thomas-review/ https://cr.openjdk.java.net/~stefank/8245002/webrev.all.thomas-review/ Thanks, StefanK > > Looks good. > > Thanks, > ? Thomas From stefan.karlsson at oracle.com Tue May 19 14:11:59 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Tue, 19 May 2020 16:11:59 +0200 Subject: RFR: 8245000: Windows GDI functions don't support large pages In-Reply-To: References: Message-ID: <192ae913-f58d-2b5b-1fb8-32c68b0aaba5@oracle.com> Thanks for reviewing! StefanK On 2020-05-19 15:16, stefan.johansson at oracle.com wrote: > Hi Stefan, > > On 2020-05-15 19:47, Stefan Karlsson wrote: >> Hi all, >> >> Please review this patch to turn off -XX:+UseLargePages as long as >> Windows GDI functions don't work with large pages. >> >> https://cr.openjdk.java.net/~stefank/8245000/webrev.00.cleanup >> https://cr.openjdk.java.net/~stefank/8245000/webrev.01.workaround >> https://cr.openjdk.java.net/~stefank/8245000/webrev.all/ > > Thanks for splitting those changes up into cleanup and fix, really > makes it easy to review. > > Looks good, > StefanJ >> >> The patches build upon the changes in: >> 8245002: Windows GDI functions don't support NUMA interleaving >> https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-May/041798.html >> >> During the investigation of JDK-8240654 >> it was found that >> some of the Windows GDI functions don't work with large pages >> (-XX:+UseLargePages). This causes severe repaint issues. >> >> The problem can easily be seen with the Java2D demos, where the drawn >> UI is grayed out instead of containing all relevant components. >> >> This can be reproduced by running building J2DBench and running: >> java -Xmx1g -Xms1g -XX:+UseLargePages -Dsun.java2d.opengl=False >> -Dsun.java2d.d3d=False -jar dist/J2DBench.jar >> >> The -Dsun.java2d.d3d property is used to ensure that that java2d >> doesn't enable the alternative, D3D implementation instead of GDI. >> -Dsun.java2d.opengl is turned off for the same reason. >> >> The cleanup patch simplifies request_lock_memory_privilege and >> large_page_init, to make it easier to insert the necessary checks. >> >> The code first checks if memory allocated with large pages can be >> used by GDI functions. If this check fails it forcefully turns off >> -XX:+UseLargePages. >> >> Just like with JDK-8245002, the intention is that as long as the GDI >> functions can't use large pages, the -XX:+UseLargePages flag will be >> turned off. If Microsoft changes the GDI code, so that it works with >> large pages, then the proposed patch will automatically allow the >> flag to be enabled. >> >> There's a second case that we need to consider for large pages. If >> the GDI functions start to allow large pages, we have a flag >> (-XX:+UseLargePagesIndividualAllocation) that reserves and commits >> large pages in page-sized granules. This could cause the same >> multi-reservation problem described in JDK-8245002. I've also added a >> check for that, and turn off -XX:+UseLargePagesIndividualAllocation >> if it fails. >> >> I've run this through tier1-3 and also tested this manually. I've >> manually turned off the different checks to verify that it turns on >> the two flags. >> >> Thanks, >> StefanK From aph at redhat.com Tue May 19 14:27:48 2020 From: aph at redhat.com (Andrew Haley) Date: Tue, 19 May 2020 15:27:48 +0100 Subject: [aarch64-port-dev ] AArch64: Neoverse N1 atomic operations and -moutline-atomics In-Reply-To: References: <3a5e43e7-2039-8574-7470-5827267fb88d@redhat.com> <854ksd3egu.fsf@arm.com> <280d2657-7997-1dac-f370-8ff41a0b3035@redhat.com> Message-ID: <760f4899-1235-2699-ca45-58a68e2dcc0a@redhat.com> On 5/19/20 11:45 AM, Nick Gasson wrote: > On 05/18/20 17:53 PM, Andrew Haley wrote: >> >>> Question is whether it's best to enable this by default in OpenJDK or >>> leave it to downstream distributions to decide? >> >> I don't really understand why there is a conditional branch there. >> They could just do the test at startup and patch the code. Maybe it'd >> hurt DSO startup time or something? Maybe they don't like the idea of >> temporarily enabling exec privilege on a page of writable data. > > I asked internally and it's basically the reasons you mention above, > plus patching could cause excessive copy-on-writes of the code pages. > > The other way to do it is to build one libjvm.so without LSE support and > then a separate atomics/libjvm.so with -march set to enable the LSE > instructions. Then ld.so will load the appropriate library based on the > hwcaps. This should give the best performance but the build time and > package size balloons. Hmm, no. :-) I think we'd better leave it as it is. I've done a few experiments with the Neoverse N1 and it's all rather murky: it's not clear to me that LSE atomics are an overall win in the Java context. We don't make heavy use of pthread mutexes except when we've decided to block, in which case the execution time is completely dominated by the cost of a system call. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From thomas.stuefe at gmail.com Tue May 19 14:40:06 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 19 May 2020 16:40:06 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> Message-ID: Hi guys, new webrev: full: http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ delta: http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ Mostly format changes, breaking up at column 80 as Andrew requested. The only changes which matter is the removal of the unnecessary include in arguments.cpp as Coleen suggested; and to not validate SharedBaseAddress anymore at runtime, as Ioi requested. Also rebased to head. Thanks, Thomas On Tue, May 19, 2020 at 6:38 AM Ioi Lam wrote: > > > On 5/18/20 7:37 AM, Thomas St?fe wrote: > > Hi all, > > fourth webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ > > I rebased atop of > > changeset: 59325:1ba9a9a3f948 > summary: 8244433: Remove saving of RSP in Assembler::pusha_uncached() > > so, atop of "8241825: Make compressed oops and compressed class pointers > independent (x86_64, PPC, S390)". > > The changes are very small this time, see notes below: > > @Ioi: > > >Hi Thomas, > > >I am running your patch in out CI pipeline now. Some comments: > > >[2] Does JDK-8243535 exist with the current jdk/jdk repo? > > I assigned JDK-8243535 to me and I'm working on a patch. But I would like > to keep that separate from this work, since this patch is already large. > And I would like to fix JDK-8243535 after this remodel patch has been > pushed, which makes some minor things easier. That would leave a small time > window where NMT does mis-track class space as shared class space, but that > is not a big issue, it does not even disturb any test. Which in itself is > something we may want to fix, come to think about it. > > > > [3] I think this can be put in header file. > > > > bool Metaspace::class_space_is_initialized() { > > return _class_space_list != NULL; > > } > > Done. > > > [7] argument.cpp checks this: > > > > if (UseSharedSpaces || DumpSharedSpaces) { > > if > (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { > > log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for > this platform, option will be ignored.", > > p2i((address)SharedBaseAddress)); > > FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); > > } > > } > > > > but initialize_dumptime_shared_and_meta_spaces aligns up the > SharedBaseAddress before doing the assert. > > > > void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { > > assert(DumpSharedSpaces, "should be called for dump time only"); > > > > const size_t reserve_alignment = > MetaspaceShared::reserved_space_alignment(); > > char* shared_base = (char*)align_up((char*)SharedBaseAddress, > reserve_alignment); > > > > #ifdef _LP64 > > assert(CompressedKlassPointers::is_valid_base((address)shared_base), > "Sanity"); > > > > So theoretically shared_base may no longer be is_valid_base after the > align_up. It probably won't happen, but I think the code will be > much > clearer tofirst to align_up, then check for is_valid_base and reset to > default, and finally assert. > > The problem with that is that at the time this coding runs os::init() did > not yet run and we do not know yet allocation granularity. > > So I changed verification of SharedBaseAddress: I added the alignment as > you suggested and moved it to a later time in VM initialization, to CDS > initialization. > > > Is the call to check_SharedBaseAddress() necessary inside > MetaspaceShared::initialize_runtime_shared_and_meta_spaces()? > SharedBaseAddress is currently ignored at run time. Does your patch change > that behavior? > > >[8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the > following block can be simplified: > > > > if (!_shared_rs.is_reserved()) { > > // Get a reserved space anywhere if attaching at the > SharedBaseAddress fails: > > if (UseCompressedClassPointers) { > > // If we need a compressed class space too, let the platform handle > the reservation. > > _shared_rs = > Metaspace::reserve_address_space_for_compressed_classes(cds_total); > > } else { > > // anywhere is fine. > > _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* > large */, (char*)NULL); > > } > > } > > > > ... if you change the declaration to: > > > > static ReservedSpace > reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return > ReservedSpace();}); > > I'd like to avoid keep the method LP64 only, but I rewrote the section > somewhat to be more clearer. > > The new code looks good. > > > [9] I think the #ifdef _LP64 is not necessary: > > > > #ifdef _LP64 > > // Some sanity checks after reserving address spaces for archives > and class space. > > assert(archive_space_rs.is_reserved(), "Sanity"); > > if (Metaspace::using_class_space()) { > > // Class space must closely follow the archive space. Both spaces > must be aligned correctly. > > assert(class_space_rs.is_reserved(), "A class space should have > been reserved"); > > assert(class_space_rs.base() >= archive_space_rs.end(), "class > space should follow the cds archive space"); > > assert(is_aligned(archive_space_rs.base(), > MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); > > assert(is_aligned(class_space_rs.base(), > Metaspace::reserve_alignment()), "class space misaligned"); > > } > > #endif > > I removed the #ifdef, but replaced it with an #ifdef ASSERT. Which is also > not needed, technically, but makes it clear that this section is debugging > checks only. > > > OK. > > The rest of the code looks good to me. I am going to run your latest patch > in our CI and will report the results. > > Thanks > - Ioi > > ---- > > @Coleen: > > > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/ > >virtualSpaceList.cpp.udiff.html > > > >Why did you make this change? Shouldn't the caller align it? > > > You are right, I removed the assert. > > --- > > Thanks all for your review work, > > ..Thomas > > > > > > > > > > > > > On Fri, May 8, 2020 at 7:37 AM Ioi Lam wrote: > >> Hi Thomas, >> >> I am running your patch in out CI pipeline now. Some comments: >> >> [1] Do we still need to use _class_space_list? It looks like we just have >> a single range. Is _class_space_list->current_virtual_space() always the >> same VS? >> >> if (_class_space_list != NULL) { >> address base = >> (address)_class_space_list->current_virtual_space()->bottom(); >> address top = base + compressed_class_space_size(); >> st->print("Compressed class space mapped at: " PTR_FORMAT "-" >> PTR_FORMAT ", size: " SIZE_FORMAT, >> p2i(base), p2i(top), top - base); >> >> [2] Does JDK-8243535 exist with the current jdk/jdk repo? >> >> // This does currently not work because rs may be the result of a split >> operation >> // and NMT seems not to be able to handle splits. >> // See JDK-8243535. >> // MemTracker::record_virtual_memory_type((address)rs.base(), mtClass); >> >> >> [3] I think this can be put in header file. >> >> bool Metaspace::class_space_is_initialized() { >> return _class_space_list != NULL; >> } >> >> [4] Why does the CCS need to follow UseLargePagesInMetaspace? This is the >> reason for the >> gap in the following chart: >> >> // We do this by reserving space for the ccs behind the archives. >> Note however that >> // ccs follows a different alignment (Metaspace::reserve_alignment), >> so there may >> // be a gap between ccs and cds. >> // We use a similar layout at runtime, see >> reserve_address_space_for_archives(). >> // >> // +-- SharedBaseAddress (default = >> 0x800000000) >> // v >> // +-..---------+---------+ ... >> +----+----+----+--------+--------------------+ >> // | Heap | Archive | | MC | RW | RO | [gap] | class >> space | >> // +-..---------+---------+ ... >> +----+----+----+--------+--------------------+ >> // |<-- MaxHeapSize -->| |<-- UnscaledClassSpaceMax = 4GB -->| >> >> >> _reserve_alignment is determined here: >> >> void Metaspace::ergo_initialize() { >> if (DumpSharedSpaces) { >> // Using large pages when dumping the shared archive is currently not >> implemented. >> FLAG_SET_ERGO(UseLargePagesInMetaspace, false); >> } >> >> size_t page_size = os::vm_page_size(); >> if (UseLargePages && UseLargePagesInMetaspace) { >> page_size = os::large_page_size(); >> } >> >> _commit_alignment = page_size; >> _reserve_alignment = MAX2(page_size, >> (size_t)os::vm_allocation_granularity()); >> >> But when CDS is enabled, the RS is reserved without large pages, so it >> should be possible to incrementally commit the CCS using just >> os::vm_allocation_granularity(). >> >> [5] I see the aarch64 code in >> Metaspace::reserve_address_space_for_compressed_classes has been changed. >> Is this necessary for this RFE, or is it a separate improvement that can be >> done in a different RFE? >> >> [6] For AARCH64, should we skip the part that reserves it right above the >> heap? Or does AARCH64 always allocate the heap such that it's a preferable >> address? >> >> // case (b) >> ReservedSpace rs; >> >> // Try right above the Java heap... >> address base = align_up(CompressedOops::end(), >> Metaspace::reserve_alignment()); >> assert(base != NULL, "Sanity"); >> >> const size_t size = align_up(CompressedClassSpaceSize, >> Metaspace::reserve_alignment()); >> if (CompressedKlassPointers::is_valid_base(base)) { >> rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* >> large */, (char*)base); >> } >> >> // ...failing that, reserve anywhere, but let platform do optimized >> placement: >> if (!rs.is_reserved()) { >> rs = Metaspace::reserve_address_space_for_compressed_classes(size); >> } >> >> [7] argument.cpp checks this: >> >> if (UseSharedSpaces || DumpSharedSpaces) { >> if >> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >> log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for >> this platform, option will be ignored.", >> p2i((address)SharedBaseAddress)); >> FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >> } >> } >> >> but initialize_dumptime_shared_and_meta_spaces aligns up the >> SharedBaseAddress before doing the assert. >> >> void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >> assert(DumpSharedSpaces, "should be called for dump time only"); >> >> const size_t reserve_alignment = >> MetaspaceShared::reserved_space_alignment(); >> char* shared_base = (char*)align_up((char*)SharedBaseAddress, >> reserve_alignment); >> >> #ifdef _LP64 >> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >> "Sanity"); >> >> So theoretically shared_base may no longer be is_valid_base after the >> align_up. It probably won't happen, but I think the code will be much >> clearer tofirst to align_up, then check for is_valid_base and reset to >> default, and finally assert. >> >> [8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >> following block can be simplified: >> >> if (!_shared_rs.is_reserved()) { >> // Get a reserved space anywhere if attaching at the >> SharedBaseAddress fails: >> if (UseCompressedClassPointers) { >> // If we need a compressed class space too, let the platform handle >> the reservation. >> _shared_rs = >> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >> } else { >> // anywhere is fine. >> _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >> large */, (char*)NULL); >> } >> } >> >> ... if you change the declaration to: >> >> static ReservedSpace >> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >> ReservedSpace();}); >> >> [9] I think the #ifdef _LP64 is not necessary: >> >> #ifdef _LP64 >> // Some sanity checks after reserving address spaces for archives and >> class space. >> assert(archive_space_rs.is_reserved(), "Sanity"); >> if (Metaspace::using_class_space()) { >> // Class space must closely follow the archive space. Both spaces >> must be aligned correctly. >> assert(class_space_rs.is_reserved(), "A class space should have >> been reserved"); >> assert(class_space_rs.base() >= archive_space_rs.end(), "class >> space should follow the cds archive space"); >> assert(is_aligned(archive_space_rs.base(), >> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >> assert(is_aligned(class_space_rs.base(), >> Metaspace::reserve_alignment()), "class space misaligned"); >> } >> #endif >> >> The rest of the code looks OK to me, but I may take a look at it again >> after getting more sleep :-) >> >> Thanks >> - Ioi >> >> >> >> >> On 5/7/20 7:21 AM, Thomas St?fe wrote: >> >> Hi all, >> >> please take a look at the third iteration of this change: >> >> >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >> >> Changes in this version: >> >> - at the request of Coleen (sorry I did not see your post earlier) I >> removed all platform dependent files and put the platform dependent coding >> back to metaspace.cpp and compressedOops.cpp to ease the review pain. I >> used plain platform defines though (#ifdef AARCH64) instead of hiding them >> behind another macro to make things clearer. Note that I still intent to >> put this code away into the platform corners but will do so in a follow up >> RFE. >> >> - I reinstated, in a fashion, the special handling of reservations on >> Windows. On all platforms we reserve address space to map the archive files >> in with a subsequent mapping operation. However, on Windows, we cannot use >> MapViewOfFile() into an existing mapping. So I remove the mapping again >> before mapping the archives - see comment in code for details. >> >> All CI tests at SAP run through without problems, including on Win64 and >> aarch64, but I would be happy if others were to run test too. >> >> Thank you, Thomas >> >> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >> wrote: >> >>> Hi all, >>> >>> Could I have reviews for the following proposal of reworking cds/class >>> space reservation? >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >>> >>> (Many thanks to Ioi Lam for so patiently explaining CDS internals to me, >>> and to Andrew Haley and Nick Gasson for help with aarch64!) >>> >>> Reservation of the compressed class space is needlessly complicated and >>> has some minor issues. It can be simplified and made clearer. >>> >>> The complexity stems from the fact that this area lives at the >>> intersection of two to three sub systems, depending on how one counts. >>> Metaspace, CDS, and the platform which may or may not its own view of how >>> to reserve class space. And this code has been growing organically over >>> time. >>> >>> One small example: >>> >>> ReservedSpace Metaspace::reserve_preferred_space(size_t size, size_t >>> alignment, >>> bool large_pages, char >>> *requested_addr, >>> bool >>> use_requested_addr) >>> >>> which I spent hours decoding, resulting in a very confused mail to >>> hs-runtime and aarch64-port-dev [2]. >>> >>> This patch attempts to simplify cds and metaspace setup a bit; to >>> comment implicit knowledge which is not immediately clear; to cleanly >>> abstract platform concerns like optimized class space placement; and to >>> disentangle cds from metaspace to solve issues which may bite us later with >>> Elastic Metaspace [4]. >>> >>> --- >>> >>> The main change is the reworked reservation mechanism. This is based on >>> Ioi's proposal [5]. >>> >>> When reserving class space, three things must happen: >>> >>> 1) reservation of the space obviously. If cds is active that space must >>> be in the vicinity of cds archives to be covered by compressed class >>> pointer encoding. >>> 2) setting up the internal Metaspace structures atop of that space >>> 3) setting up compressed class pointer encoding. >>> >>> In its current form, Metaspace may or may not do some or all of that in >>> one function >>> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>> metaspace_rs, char* requested_addr, address cds_base);) - if cds is active, >>> it will reserve the space for Metaspace and hand it in, otherwise it will >>> create it itself. >>> >>> When discussing this in [2], Ioi proposed to move the reservation of the >>> class space completely out of Metaspace and make it a responsibility of the >>> caller always. This would reduce some complexity, and this patch follows >>> the proposal. >>> >>> I removed >>> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>> metaspace_rs, char* requested_addr, address cds_base); and all its sub >>> functions. >>> >>> (1) now has to always be done outside - a ReservedSpace for class space >>> has to be provided by the caller. However, Metaspace now offers a utility >>> function for reserving space at a "nice" location, and explicitly doing >>> nothing else: >>> >>> ReservedSpace >>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>> >>> this function can be redefined on a platform level for platform >>> optimized reservation, see below for details. >>> >>> (2) is taken care of by a new function, >>> Metaspace::initialize_class_space(ReservedSpace rs) >>> >>> (3) is taken care of a new function >>> CompressedKlassPointers::initialize(), see below for details. >>> >>> >>> So, class space now is set up three explicit steps: >>> >>> - First, reserve a suitable space by however means you want. For >>> convenience you may use >>> Metaspace::reserve_address_space_for_compressed_classes(), or you may roll >>> your own reservation. >>> - Next, tell Metaspace to use that range as backing storage for class >>> space: Metaspace::initialize_class_space(ReservedSpace rs) >>> - Finally, set up encoding. Encoding is independent from the concept of >>> a ReservedSpace, it just gets an address range, see below for details. >>> >>> Separating these steps and moving them out of the responsibility of >>> Metaspace makes this whole thing more flexible; it also removes unnecessary >>> knowledge (e.g. Metaspace does not need to know anything about either ccp >>> encoding or cds). >>> >>> --- >>> >>> How it comes together: >>> >>> If CDS is off, we just reserve a space using >>> Metaspace::reserve_address_space_for_compressed_classes(), initialize it >>> with Metaspace::initialize_class_space(ReservedSpace rs), then set up >>> compressed class pointer encoding covering the range of this class space. >>> >>> If CDS is on (dump time), we reserve large 4G space, either at >>> SharedBaseAddress or using >>> Metaspace::reserve_address_space_for_compressed_classes(); we then split >>> that into 3G archive space and 1G class space; we set up that space with >>> Metaspace as class space; then we set up compressed class pointer encoding >>> covering both archive space and cds. >>> >>> If CDS is on (run time), we reserve a large space, split it into archive >>> space (large enough to hold both archives) and class space, then basically >>> proceed as above. >>> >>> Note that this is almost exactly how things worked before (modulo some >>> minor fixes, e.g. alignment issues), only the code is reformed and made >>> more explicit. >>> >>> --- >>> >>> I moved compressed class pointer setup over to CompressedKlassPointers >>> and changed the interface: >>> >>> -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >>> metaspace_rs, address cds_base) >>> +void CompressedKlassPointers::initialize(address addr, size_t len); >>> >>> Instead of feeding it a single ReservedSpace, which is supposed to >>> represent class space, and an optional alternate base if cds is on, now we >>> give it just an numeric address range. That range marks the limits to where >>> Klass structures are to be expected, and is the implicit promise that >>> outside that range no Klass structures will exist, so encoding has to cover >>> only this range. >>> >>> This range may contain just the class space; or class space+cds; or >>> whatever allocation scheme we come up with in the future. Encoding does not >>> really care how the memory is organized as long as the input range covers >>> all possible Klass locations. That way we remove knowledge about class >>> space/cds from compressed class pointer encoding. >>> >>> Moving it away from metaspace.cpp into the CompressedKlassPointers class >>> also mirrors CompressedOops::initialize(). >>> >>> --- >>> >>> I renamed _narrow_klass_range to just _range, because strictly speaking >>> this is the range un-narrow Klass pointers can have. >>> >>> As for the implementation of CompressedKlassPointers::initialize(address >>> addr, size_t len), I mimicked very closely what happened before, so there >>> should be almost no differences. Since "almost no differences" sounds scary >>> :) here are the differences: >>> >>> - When CDS is active (dump or run time) we now always, unconditionally, >>> set the encoding range to 4G. This fixes a theoretical bug discussed on >>> aarch64-port-dev [1]. >>> >>> - When CDS is not active, we set the encoding range to the minimum >>> required length. Before, it was left at its default value of 4G. >>> >>> Both differences only affect aarch64, since they are currently the only >>> one using the range field in CompressedKlassPointers. >>> >>> I wanted to add an assert somewhere to test encoding of the very last >>> address of the CompressedKlassPointers range, again to prevent errors like >>> [3]. But I did not come up with a good place for this assert which would >>> cover also the encoding done by C1/C2. >>> >>> For the same reason I thought about introducing a mode where Klass >>> structures would be allocated in reverse order, starting at the end of the >>> ccs, but again left it out as too big a change. >>> >>> --- >>> >>> OS abstraction: platforms may have restrictions of what constitutes a >>> valid compressed class pointer encoding base. Or if not, they may have at >>> least preferences. There was logic like this in metaspace.cpp, which I >>> removed and cleanly factored out into platform dependent files, giving each >>> platform the option to add special logic. >>> >>> These are two new methods: >>> >>> - bool CompressedKlassPointers::is_valid_base(address p) >>> >>> to let the platform tell you whether it considers p to be a valid >>> encoding base. The only platform having these restrictions currently is >>> aarch64. >>> >>> - ReservedSpace >>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>> >>> this hands over the process of allocating a range suitable for >>> compressed class pointer encoding to the platform. Most platforms will >>> allocate just anywhere, but some platforms may have a better strategy (e.g. >>> trying low memory first, trying only correctly aligned addresses and so >>> on). >>> >>> Beforehand, this coding existed in a similar form in metaspace.cpp for >>> aarch64 and AIX. For now, I left the AIX part out - it seems only half >>> done, and I want to check further if we even need it, if yes why not on >>> Linux ppc, and C1 does not seem to support anything other than base+offset >>> with shift either, but I may be mistaken. >>> >>> These two methods should give the platform enough control to implement >>> their own scheme for optimized class space placement without bothering any >>> shared code about it. >>> >>> Note about the form, I introduced two new platform dependent files, >>> "metaspace_.cpp" and "compressedOops_.cpp". I am not happy about >>> this but this seems to be what we generally do in hotspot, right? >>> >>> --- >>> >>> Metaspace reserve alignment vs cds alignment >>> >>> CDS was using Metaspace reserve alignment for CDS internal purposes. I >>> guess this was just a copy paste issue. It never caused problems since >>> Metaspace reserve alignment == page size, but that is not true anymore in >>> the upcoming Elastic Metaspace where reserve alignment will be larger. This >>> causes a number of issues. >>> >>> I separated those two cleanly. CDS now uses >>> os::vm_allocation_granularity. Metaspace::reserve_alignment is only used in >>> those two places where it is needed, when CDS creates the address space for >>> class space on behalf of the Metaspace. >>> >>> --- >>> >>> Windows special handling in CDS >>> >>> To simplify coding I removed the windows specific handling which left >>> out reservation of the archive. This was needed because windows cannot mmap >>> files into reserved regions. But fallback code exists in filemap.cpp for >>> this case which just reads in the region instead of mapping it. >>> >>> Should that turn out to be a performance problem, I will reinstate the >>> feature. But a simpler way would be reserve the archive and later just >>> before mmapping the archive file to release the archive space. That would >>> not only be simpler but give us the best guarantee that that address space >>> is actually available. But I'd be happy to leave that part out completely >>> if we do not see any performance problems on windows x64. >>> >>> --- >>> >>> NMT cannot deal with spaces which are split. This problem manifests in >>> that bookkeeping for class space is done under "Shared Classes", not >>> "Classes" as it should. This problem exists today too at dump time and >>> randomly at run time. But since I simplified the reservation, this problem >>> now shows up always, whether or not we map at the SharedBaseAddress. >>> While I could work around this problem, I'd prefer this problem to be >>> solved at the core, and NMT to have an option to recognize reservation >>> splits. So I'd rather not put a workaround for this into the patch but >>> leave it for fixing as a separate issue. I opened this issue to track it >>> [6]. >>> >>> --- >>> >>> Jtreg tests: >>> >>> I expanded the CompressedOops/CompressedClassPointers.java. I also >>> extended them to Windows. The tests now optionally omit strict class space >>> placement tests, since these tests heavily depend on ASLR and were the >>> reason they were excluded on Windows. However I think even without checking >>> for class space placement they make sense, just to see that the VM comes up >>> and lives with the many different settings we can run in. >>> >>> --- >>> >>> Tests: >>> >>> - I ran the patch through Oracles submit repo >>> - I ran tests manually for aarch64, zero, linux 32bit and windows x64 >>> - The whole battery of nightly tests at SAP, including ppc, ppcle and >>> aarch64, unfortunately excluding windows because of unrelated errors. >>> Windows x64 tests will be redone tonight. >>> >>> >>> Thank you, >>> >>> Thomas >>> >>> [1] >>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >>> [2] >>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >>> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >>> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >>> [5] >>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >>> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >>> >>> >> > From stefan.johansson at oracle.com Tue May 19 15:21:32 2020 From: stefan.johansson at oracle.com (stefan.johansson at oracle.com) Date: Tue, 19 May 2020 17:21:32 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: <8e4aeff8-df10-538b-d65f-b8ed05217244@oracle.com> References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> <4cf0265c-f901-e1fb-edf6-dc7bdb0a0bc7@oracle.com> <8e4aeff8-df10-538b-d65f-b8ed05217244@oracle.com> Message-ID: <622ef9c9-3790-94d0-c8c0-a3f070ca2dc4@oracle.com> Hi Stefan, On 2020-05-19 16:04, Stefan Karlsson wrote: > > However, I've updated the code to ensure that both memory and and width > bytes are aligned, just to be sure. I've also added some more comments > to describe the chosen value: > > https://cr.openjdk.java.net/~stefank/8245002/webrev.03.workaround.thomas-review/ > > https://cr.openjdk.java.net/~stefank/8245002/webrev.all.thomas-review/ Still good, but if you like you can add parentheses around the division in the nBitCount calculation to make it easier to read. Thanks, Stefan > > Thanks, > StefanK > >> >> Looks good. >> >> Thanks, >> ?? Thomas From thomas.schatzl at oracle.com Tue May 19 15:28:00 2020 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 19 May 2020 17:28:00 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: <8e4aeff8-df10-538b-d65f-b8ed05217244@oracle.com> References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> <4cf0265c-f901-e1fb-edf6-dc7bdb0a0bc7@oracle.com> <8e4aeff8-df10-538b-d65f-b8ed05217244@oracle.com> Message-ID: Hi, On 19.05.20 16:04, Stefan Karlsson wrote: > > > On 2020-05-19 14:23, Thomas Schatzl wrote: >> Hi, >> >> On 19.05.20 13:53, Stefan Karlsson wrote: >>> Updated webrevs: >>> ??https://cr.openjdk.java.net/~stefank/8245002/webrev.02.workaround.kims-review >>> >>> ??https://cr.openjdk.java.net/~stefank/8245002/webrev.all.kims-review >> >> - s/adjecent/adjacent > > Fixed. > >> [...]>> In your example the start of the first (and only) scanline is >> guaranteed to be not word aligned (2808?? void* bitmapBits = >> (char*)mem1 - 1;) so this test may fail (on different Windows >> versions?) for different reasons. >> >> Since the code seems to work, I am okay with it though. >> >> If you only intend to fix the typo, I do not need a re-review. > > I've tested by misaligning the memory, and it seems to work. > > However, I've updated the code to ensure that both memory and and width > bytes are aligned, just to be sure. I've also added some more comments > to describe the chosen value: > > https://cr.openjdk.java.net/~stefank/8245002/webrev.03.workaround.thomas-review/ > > https://cr.openjdk.java.net/~stefank/8245002/webrev.all.thomas-review/ > lgtm. Thomas From stefan.karlsson at oracle.com Tue May 19 16:42:40 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Tue, 19 May 2020 18:42:40 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> <4cf0265c-f901-e1fb-edf6-dc7bdb0a0bc7@oracle.com> <8e4aeff8-df10-538b-d65f-b8ed05217244@oracle.com> Message-ID: <93500c5a-aa11-05c3-6508-d50ebb2dfe82@oracle.com> On 2020-05-19 17:28, Thomas Schatzl wrote: > Hi, > > On 19.05.20 16:04, Stefan Karlsson wrote: >> >> >> On 2020-05-19 14:23, Thomas Schatzl wrote: >>> Hi, >>> >>> On 19.05.20 13:53, Stefan Karlsson wrote: >>>> Updated webrevs: >>>> ??https://cr.openjdk.java.net/~stefank/8245002/webrev.02.workaround.kims-review >>>> >>>> ??https://cr.openjdk.java.net/~stefank/8245002/webrev.all.kims-review >>> >>> - s/adjecent/adjacent >> >> Fixed. >> >>> > [...]>> In your example the start of the first (and only) scanline is >>> guaranteed to be not word aligned (2808?? void* bitmapBits = >>> (char*)mem1 - 1;) so this test may fail (on different Windows >>> versions?) for different reasons. >>> >>> Since the code seems to work, I am okay with it though. >>> >>> If you only intend to fix the typo, I do not need a re-review. >> >> I've tested by misaligning the memory, and it seems to work. >> >> However, I've updated the code to ensure that both memory and and >> width bytes are aligned, just to be sure. I've also added some more >> comments to describe the chosen value: >> >> https://cr.openjdk.java.net/~stefank/8245002/webrev.03.workaround.thomas-review/ >> >> https://cr.openjdk.java.net/~stefank/8245002/webrev.all.thomas-review/ >> > > ? lgtm. Thanks, Thomas. StefanK > > Thomas From stefan.karlsson at oracle.com Tue May 19 16:43:06 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Tue, 19 May 2020 18:43:06 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: <622ef9c9-3790-94d0-c8c0-a3f070ca2dc4@oracle.com> References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> <4cf0265c-f901-e1fb-edf6-dc7bdb0a0bc7@oracle.com> <8e4aeff8-df10-538b-d65f-b8ed05217244@oracle.com> <622ef9c9-3790-94d0-c8c0-a3f070ca2dc4@oracle.com> Message-ID: <5bb80650-890b-b694-91a9-15c2a758c862@oracle.com> On 2020-05-19 17:21, stefan.johansson at oracle.com wrote: > Hi Stefan, > > On 2020-05-19 16:04, Stefan Karlsson wrote: >> >> However, I've updated the code to ensure that both memory and and >> width bytes are aligned, just to be sure. I've also added some more >> comments to describe the chosen value: >> >> https://cr.openjdk.java.net/~stefank/8245002/webrev.03.workaround.thomas-review/ >> >> https://cr.openjdk.java.net/~stefank/8245002/webrev.all.thomas-review/ > > Still good, but if you like you can add parentheses around the > division in the nBitCount calculation to make it easier to read. OK. Will add parentheses. Thanks, StefanK > > Thanks, > Stefan > >> >> Thanks, >> StefanK >> >>> >>> Looks good. >>> >>> Thanks, >>> ?? Thomas From kim.barrett at oracle.com Tue May 19 19:53:14 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Tue, 19 May 2020 15:53:14 -0400 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: <8e4aeff8-df10-538b-d65f-b8ed05217244@oracle.com> References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> <4cf0265c-f901-e1fb-edf6-dc7bdb0a0bc7@oracle.com> <8e4aeff8-df10-538b-d65f-b8ed05217244@oracle.com> Message-ID: <3F0B7551-32DA-4169-81FA-728872C750AA@oracle.com> > On May 19, 2020, at 10:04 AM, Stefan Karlsson wrote: > https://cr.openjdk.java.net/~stefank/8245002/webrev.03.workaround.thomas-review/ > https://cr.openjdk.java.net/~stefank/8245002/webrev.all.thomas-review/ Looks good. A few trivial typos: 2770 // If this fail, the testing method above isn't reliable. s/fail/fails/ 2825 // Set the bitmaps bits to point one "width" bytes before, so that s/bitmaps/bitmap's/ 2826 // the bitmap extends a cross the reservation boundary. s/a cross/across/ From kim.barrett at oracle.com Tue May 19 22:18:08 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Tue, 19 May 2020 18:18:08 -0400 Subject: RFR: 8245236: Remove MO_VOLATILE Access decorator In-Reply-To: <6806a61b-f46a-022c-9a55-c7530fbfe084@oracle.com> References: <6806a61b-f46a-022c-9a55-c7530fbfe084@oracle.com> Message-ID: <744F2FA9-6AFF-408A-8B3D-EB3B9C15E633@oracle.com> > On May 19, 2020, at 2:25 AM, Stefan Karlsson wrote: > > On 2020-05-19 07:24, Kim Barrett wrote: >> Please review this change with replaces uses of the MO_VOLATILE memory >> order Access decorator with uses of the MO_RELAXED decorator. Uses of >> MO_VOLATILE generally assume non-word-tearing semantics, so a relaxed >> atomic access is more appropriate. As a result, we can eliminate the >> MO_VOLATILE decorator. >> >> CR: >> https://bugs.openjdk.java.net/browse/JDK-8245236 >> >> Webrev: >> https://cr.openjdk.java.net/~kbarrett/8245236/open.00/ > Looks good. Thanks. > One reflection after reading this. > > markWord oopDesc::mark() const { > - uintptr_t v = HeapAccess::load_at(as_oop(), mark_offset_in_bytes()); > + uintptr_t v = HeapAccess::load_at(as_oop(), mark_offset_in_bytes()); > return markWord(v); > } > markWord oopDesc::mark_raw() const { > return Atomic::load(&_mark); > > and > > void oopDesc::set_mark(markWord m) { > - HeapAccess::store_at(as_oop(), mark_offset_in_bytes(), m.value()); > + HeapAccess::store_at(as_oop(), mark_offset_in_bytes(), m.value()); > } > void oopDesc::set_mark_raw(markWord m) { > Atomic::store(&_mark, m); > } > > I noticed that the normal versions use HeapAccess and the _raw versions use Atomic::. The _raw versions where added to support Shenandoah's load barriers, but they have since changed their load barriers and don't need these functions anymore (AFAIK). Maybe its time to clean this up and get rid of all the _raw versions added for Shenandoah. > > StefanK I haven't been following the details of Shenandoah's barrier changes. Assuming we don't need those functions anymore, getting rid of them would be a nice additional cleanup for a new RFE. From kim.barrett at oracle.com Tue May 19 22:18:19 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Tue, 19 May 2020 18:18:19 -0400 Subject: RFR: 8245236: Remove MO_VOLATILE Access decorator In-Reply-To: <86c4e8a6-abd2-f15c-19fc-36832b4ca19e@oracle.com> References: <86c4e8a6-abd2-f15c-19fc-36832b4ca19e@oracle.com> Message-ID: <997B59DF-1139-456B-9BAE-5A228272197A@oracle.com> > On May 19, 2020, at 4:06 AM, Thomas Schatzl wrote: > > Hi, > > On 19.05.20 07:24, Kim Barrett wrote: >> Please review this change with replaces uses of the MO_VOLATILE memory >> order Access decorator with uses of the MO_RELAXED decorator. Uses of >> MO_VOLATILE generally assume non-word-tearing semantics, so a relaxed >> atomic access is more appropriate. As a result, we can eliminate the >> MO_VOLATILE decorator. >> CR: >> https://bugs.openjdk.java.net/browse/JDK-8245236 >> Webrev: >> https://cr.openjdk.java.net/~kbarrett/8245236/open.00/ >> Testing: >> mach5 tier1. > > looks good. > > Thomas Thanks. From paul.sandoz at oracle.com Tue May 19 22:25:27 2020 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 19 May 2020 15:25:27 -0700 Subject: Update Re: RFR JDK-8223347 Integration of Vector API (Incubator): Java API, implementation, and tests In-Reply-To: <9D074341-7234-4D9F-8E99-0E73F5BB5407@oracle.com> References: <49F81016-55DF-4E3C-940A-A8DF8C712259@oracle.com> <9D074341-7234-4D9F-8E99-0E73F5BB5407@oracle.com> Message-ID: I just realized that the vector tests have not been included in any JDK test category e.g. tier1. Ordinarily I would expect the tests to be added to tier1 since the tests exercise intrinsics. However, those intrinsics are only enabled with the Vector API module so we could place in another tier to potentially reduce the cost of testing. Advice very much appreciate on which tier the tests should belong to. Paul. > On May 18, 2020, at 12:13 PM, Paul Sandoz wrote: > > HI, > > Here?s an update of the API and implementation webrevs based on (mostly) API feedback. We have removed the performance tests to make the review easier (those will be dealt with separately to integration as a follow on task). > > I think over the past year and recently via the CSR the API has had significant review. Reviews focusing on the Java implementation and tests would be greatly appreciated. > > It?s worth reiterating that the implementation and tests are quite formulaic, there is a lot of code that is generated from templates, so it's possible to focus on the template + subset (e.g. byte, long, float, and sizes of say 256, max). > > Notable changes from the prior webrev are: > > - Removal of the fromValues methods to construct a vector from a varargs array. Feedback indicated this was a misleading way to obtain a vector. > > - Unification of byte array and ByteBuffer load/store method signatures and unification of the implementations with more tests (including out-of-bounds tests for all the kinds of loads/stores). > > Paul. > > -- > > Latest javadoc > http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/docs-2020-05-15-88a83f7238d8/api/jdk.incubator.vector/jdk/incubator/vector/package-summary.html > > Latest specdiff > http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/specdiff-jdk-2020-05-15-88a83f7238d8/overview-summary.html > > Incremental specdiff > http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/specdiff-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/overview-summary.html > > > Latest implementation webrev > http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev-2020-05-16-default-2020-05-15-88a83f7238d8/webrev/ > > Incremental Implementation webrev > http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/webrev/ > > Latest test webrev > http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev-2020-05-16-default-2020-05-15-88a83f7238d8/webrev/ > > Incremental test webrev > http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/webrev/ > > >> On Apr 1, 2020, at 3:46 PM, Paul Sandoz wrote: >> >> Hi, >> >> A prior email sent out a request for review of the Vector API in preparation for JEP 338: Vector API (Incubator) [1] to be proposed for target: >> >> https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html >> >> This email expands the review of the API to the Java implementation and Java tests: >> >> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev/webrev/ >> >> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev/webrev/ >> >> (Further emails will sent for review of general HotSpot changes CPU architecture specific HotSpot changes, x64 and aarch64, and performance tests. For an early peek see the links in the description of JDK-8223347.) >> >> ? >> >> The Vector API provides >> >> - the public Vector class with vector operations common to all supported primitive types >> >> - public primitive specializations, such as IntVector, with common operations associated with the primitive type >> >> - internal concrete specializations for the vector size, such as Int256Vector, for a vector holding 8 ints. >> >> Operations generally defer to one of approximately 20 vector intrinsics. Some operations may be composed of other operations. >> >> Explicit casts are performed by vector operations to ensure vectors arguments are of the required shape (bit size) and element type. >> >> A vector intrinsic is an internal low-level vector operation. The last argument to the intrinsic is fall back behavior in Java, implementing the scalar operation over the number of elements held by the vector. Thus, If the intrinsic is not supported in C2 for the other arguments then the Java implementation is executed (the Java implementation is always executed when running in the interpreter or for C1). >> >> The public primitive specializations and the internal concrete implementations are generated from SSP template files, X-Vector.java.template and X-VectorBits.java.template respectively. >> >> Overall the implementation approach is quite formulaic and rather repetitive. Once you grok the pattern It should be easier to review if a little laborious. >> >> Unit tests are auto-generated by composing templates for each operation into an SSP template file which is then used to generate unit test files for each concrete vector class. The tests are quite extensive and have found many a HotSpot issue in development across a wide range of platforms. >> >> Paul. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8201271 >> > From ioi.lam at oracle.com Tue May 19 23:41:03 2020 From: ioi.lam at oracle.com (Ioi Lam) Date: Tue, 19 May 2020 16:41:03 -0700 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> Message-ID: <8bf736b3-9ab1-86b7-fd22-230e449b1b9f@oracle.com> Hi Thomas, The testing took a little longer than I expected. I've run your previous version: http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ In our CI tiers 1/4. No failures. I didn't run your latest version. I would suggesting testing it with your own CI as well as in jdk-submit before pushing (after you get OKs from other reviewers). Thumbs up. - Ioi On 5/19/20 7:40 AM, Thomas St?fe wrote: > Hi guys, > > new webrev: > > full: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ > delta: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ > > Mostly format changes, breaking up at column 80 as Andrew requested. > > The only changes which matter is the removal of the unnecessary > include in arguments.cpp as Coleen suggested; and to not validate > SharedBaseAddress anymore at runtime, as Ioi requested. > > Also rebased to head. > > Thanks, Thomas > > > > > On Tue, May 19, 2020 at 6:38 AM Ioi Lam > wrote: > > > > On 5/18/20 7:37 AM, Thomas St?fe wrote: >> Hi all, >> >> fourth webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ >> >> I rebased atop of >> >> changeset: ? 59325:1ba9a9a3f948 >> summary: ? ? 8244433: Remove saving of RSP in >> Assembler::pusha_uncached() >> >> so, atop of "8241825: Make compressed oops and compressed class >> pointers independent (x86_64, PPC, S390)". >> >> The changes are very small this time, see notes below: >> >> @Ioi: >> >> >Hi Thomas, >> >> >I am running your patch in out CI pipeline now. Some comments: >> >> >[2] Does JDK-8243535 exist with the current jdk/jdk repo? >> >> I assigned JDK-8243535 to me and I'm working on a patch. But I >> would like to keep that separate from this work, since this patch >> is already large. And I would like to fix JDK-8243535 after this >> remodel patch has been pushed, which makes some minor things >> easier. That would leave a small time window where NMT does >> mis-track class space as shared class space, but that is not a >> big issue, it does not even disturb any test. Which in itself is >> something we may want to fix, come to think about it. >> >> >> > [3] I think this can be put in header file. >> > >> > bool Metaspace::class_space_is_initialized() { >> > ? return _class_space_list != NULL; >> > } >> >> Done. >> >> > [7] argument.cpp checks this: >> > >> > ? if (UseSharedSpaces || DumpSharedSpaces) { >> > ? ? if >> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) >> { >> > ? ? ? log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is >> invalid for this platform, option will be ignored.", >> > ?p2i((address)SharedBaseAddress)); >> > ? ? ? FLAG_SET_DEFAULT(SharedBaseAddress, >> default_SharedBaseAddress()); >> > ? ? } >> > ? } >> > >> > but initialize_dumptime_shared_and_meta_spaces aligns up the >> SharedBaseAddress before doing the assert. >> > >> > ? void >> MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >> > ? ? assert(DumpSharedSpaces, "should be called for dump time >> only"); >> > >> > ? ? const size_t reserve_alignment = >> MetaspaceShared::reserved_space_alignment(); >> > ? ? char* shared_base = >> (char*)align_up((char*)SharedBaseAddress, reserve_alignment); >> > >> > ? #ifdef _LP64 >> > >> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >> "Sanity"); >> > >> > So theoretically shared_base may no longer be is_valid_base >> after the align_up. It probably won't happen, but I think the >> code will be > much clearer tofirst to align_up, then check for >> is_valid_base and reset to default, and finally assert. >> >> The problem with that is that at the time this coding runs >> os::init() did not yet run and we do not know yet allocation >> granularity. >> >> So I changed verification of SharedBaseAddress: I added the >> alignment as you suggested and moved it to a later time in VM >> initialization, to CDS initialization. >> > > Is the call to? check_SharedBaseAddress() necessary inside > MetaspaceShared::initialize_runtime_shared_and_meta_spaces()? > SharedBaseAddress is currently ignored at run time. Does your > patch change that behavior? > >> >[8] In >> MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >> following block can be simplified: >> > >> > ?if (!_shared_rs.is_reserved()) { >> > ? ?// Get a reserved space anywhere if attaching at the >> SharedBaseAddress fails: >> > ? ?if (UseCompressedClassPointers) { >> > ? ? ?// If we need a compressed class space too, let the >> platform handle the reservation. >> > ? ? ?_shared_rs = >> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >> > ? ?} else { >> > ? ? ?// anywhere is fine. >> > ? ? ?_shared_rs = ReservedSpace(cds_total, reserve_alignment, >> false /* large */, (char*)NULL); >> > ? ?} >> > ?} >> > >> > ? ... if you change the declaration to: >> > >> > ?static ReservedSpace >> reserve_address_space_for_compressed_classes(size_t size) >> NOT_LP64({ return ReservedSpace();}); >> >> I'd like to avoid keep the method LP64 only, but I rewrote the >> section somewhat to be more clearer. >> > The new code looks good. > >> > [9] I think the #ifdef _LP64 is not necessary: >> > >> > #ifdef _LP64 >> > ? ? // Some sanity checks after reserving address spaces for >> archives and class space. >> > ? ? assert(archive_space_rs.is_reserved(), "Sanity"); >> > ? ? if (Metaspace::using_class_space()) { >> > ? ? ? // Class space must closely follow the archive space. >> Both spaces must be aligned correctly. >> > ? ? ? assert(class_space_rs.is_reserved(), "A class space >> should have been reserved"); >> > ? ? ? assert(class_space_rs.base() >= archive_space_rs.end(), >> "class space should follow the cds archive space"); >> > assert(is_aligned(archive_space_rs.base(), >> MetaspaceShared::reserved_space_alignment()), "Archive space >> misaligned"); >> > assert(is_aligned(class_space_rs.base(), >> Metaspace::reserve_alignment()), "class space misaligned"); >> > ? ? } >> > #endif >> >> I removed the #ifdef, but replaced it with an #ifdef ASSERT. >> Which is also not needed, technically, but makes it clear that >> this section is debugging checks only. >> > > OK. > > The rest of the code looks good to me. I am going to run your > latest patch in our CI and will report the results. > > Thanks > - Ioi >> ---- >> >> @Coleen: >> >> >http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/>virtualSpaceList.cpp.udiff.html >> > >> >Why did you make this change?? Shouldn't the caller align it? >> >> >> You are right, I removed the assert. >> >> --- >> >> Thanks all for your review work, >> >> ..Thomas >> >> >> >> >> >> >> >> >> >> >> >> >> On Fri, May 8, 2020 at 7:37 AM Ioi Lam > > wrote: >> >> Hi Thomas, >> >> I am running your patch in out CI pipeline now. Some comments: >> >> [1] Do we still need to use _class_space_list? It looks like >> we just have a single range. Is >> _class_space_list->current_virtual_space() always the same VS? >> >> ? if (_class_space_list != NULL) { >> ??? address base = >> (address)_class_space_list->current_virtual_space()->bottom(); >> ??? address top = base + compressed_class_space_size(); >> ??? st->print("Compressed class space mapped at: " PTR_FORMAT >> "-" PTR_FORMAT ", size: " SIZE_FORMAT, >> ?????????????? p2i(base), p2i(top), top - base); >> >> [2] Does JDK-8243535 exist with the current jdk/jdk repo? >> >> ? // This does currently not work because rs may be the >> result of a split operation >> ? // and NMT seems not to be able to handle splits. >> ? // See JDK-8243535. >> ? // >> MemTracker::record_virtual_memory_type((address)rs.base(), >> mtClass); >> >> >> [3] I think this can be put in header file. >> >> bool Metaspace::class_space_is_initialized() { >> ? return _class_space_list != NULL; >> } >> >> [4] Why does the CCS need to follow UseLargePagesInMetaspace? >> This is the reason for the >> ??? gap in the following chart: >> >> ??? // We do this by reserving space for the ccs behind the >> archives. Note however that >> ??? //? ccs follows a different alignment >> (Metaspace::reserve_alignment), so there may >> ??? //? be a gap between ccs and cds. >> ??? // We use a similar layout at runtime, see >> reserve_address_space_for_archives(). >> ??? // >> ??? //????????????????????????????? +-- SharedBaseAddress >> (default = 0x800000000) >> ??? //????????????????????????????? v >> ??? // +-..---------+---------+ ... >> +----+----+----+--------+--------------------+ >> ??? // |??? Heap??? | Archive |???? | MC | RW | RO | [gap]? >> |??? class space???? | >> ??? // +-..---------+---------+ ... >> +----+----+----+--------+--------------------+ >> ??? // |<--?? MaxHeapSize? -->|???? |<-- >> UnscaledClassSpaceMax = 4GB -->| >> >> >> _reserve_alignment is determined here: >> >> void Metaspace::ergo_initialize() { >> ? if (DumpSharedSpaces) { >> ??? // Using large pages when dumping the shared archive is >> currently not implemented. >> ??? FLAG_SET_ERGO(UseLargePagesInMetaspace, false); >> ? } >> >> ? size_t page_size = os::vm_page_size(); >> ? if (UseLargePages && UseLargePagesInMetaspace) { >> ??? page_size = os::large_page_size(); >> ? } >> >> ? _commit_alignment? = page_size; >> ? _reserve_alignment = MAX2(page_size, >> (size_t)os::vm_allocation_granularity()); >> >> But when CDS is enabled, the RS is reserved without large >> pages, so it should be possible to incrementally commit the >> CCS using just os::vm_allocation_granularity(). >> >> [5] I see the aarch64 code in >> Metaspace::reserve_address_space_for_compressed_classes has >> been changed. Is this necessary for this RFE, or is it a >> separate improvement that can be done in a different RFE? >> >> [6] For AARCH64, should we skip the part that reserves it >> right above the heap? Or does AARCH64 always allocate the >> heap such that it's a preferable address? >> >> ??? // case (b) >> ??? ReservedSpace rs; >> >> ??? // Try right above the Java heap... >> ??? address base = align_up(CompressedOops::end(), >> Metaspace::reserve_alignment()); >> ??? assert(base != NULL, "Sanity"); >> >> ??? const size_t size = align_up(CompressedClassSpaceSize, >> Metaspace::reserve_alignment()); >> ??? if (CompressedKlassPointers::is_valid_base(base)) { >> ????? rs = ReservedSpace(size, >> Metaspace::reserve_alignment(), false /* large */, (char*)base); >> ??? } >> >> ??? // ...failing that, reserve anywhere, but let platform do >> optimized placement: >> ??? if (!rs.is_reserved()) { >> ????? rs = >> Metaspace::reserve_address_space_for_compressed_classes(size); >> ??? } >> >> [7] argument.cpp checks this: >> >> ? if (UseSharedSpaces || DumpSharedSpaces) { >> ??? if >> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) >> { >> ????? log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is >> invalid for this platform, option will be ignored.", >> p2i((address)SharedBaseAddress)); >> ????? FLAG_SET_DEFAULT(SharedBaseAddress, >> default_SharedBaseAddress()); >> ??? } >> ? } >> >> but initialize_dumptime_shared_and_meta_spaces aligns up the >> SharedBaseAddress before doing the assert. >> >> ? void >> MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >> ??? assert(DumpSharedSpaces, "should be called for dump time >> only"); >> >> ??? const size_t reserve_alignment = >> MetaspaceShared::reserved_space_alignment(); >> ??? char* shared_base = >> (char*)align_up((char*)SharedBaseAddress, reserve_alignment); >> >> ? #ifdef _LP64 >> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >> "Sanity"); >> >> So theoretically shared_base may no longer be is_valid_base >> after the align_up. It probably won't happen, but I think the >> code will be much clearer tofirst to align_up, then check for >> is_valid_base and reset to default, and finally assert. >> >> [8] In >> MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, >> the following block can be simplified: >> >> ? if (!_shared_rs.is_reserved()) { >> ??? // Get a reserved space anywhere if attaching at the >> SharedBaseAddress fails: >> ??? if (UseCompressedClassPointers) { >> ????? // If we need a compressed class space too, let the >> platform handle the reservation. >> ????? _shared_rs = >> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >> ??? } else { >> ????? // anywhere is fine. >> ????? _shared_rs = ReservedSpace(cds_total, >> reserve_alignment, false /* large */, (char*)NULL); >> ??? } >> ? } >> >> ?? ... if you change the declaration to: >> >> ? static ReservedSpace >> reserve_address_space_for_compressed_classes(size_t size) >> NOT_LP64({ return ReservedSpace();}); >> >> [9] I think the #ifdef _LP64 is not necessary: >> >> #ifdef _LP64 >> ??? // Some sanity checks after reserving address spaces for >> archives and class space. >> ??? assert(archive_space_rs.is_reserved(), "Sanity"); >> ??? if (Metaspace::using_class_space()) { >> ????? // Class space must closely follow the archive space. >> Both spaces must be aligned correctly. >> ????? assert(class_space_rs.is_reserved(), "A class space >> should have been reserved"); >> ????? assert(class_space_rs.base() >= archive_space_rs.end(), >> "class space should follow the cds archive space"); >> ????? assert(is_aligned(archive_space_rs.base(), >> MetaspaceShared::reserved_space_alignment()), "Archive space >> misaligned"); >> ????? assert(is_aligned(class_space_rs.base(), >> Metaspace::reserve_alignment()), "class space misaligned"); >> ??? } >> #endif >> >> The rest of the code looks OK to me, but I may take a look at >> it again after getting more sleep :-) >> >> Thanks >> - Ioi >> >> >> >> >> On 5/7/20 7:21 AM, Thomas St?fe wrote: >>> Hi all, >>> >>> please take a look at the third iteration of this change: >>> >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >>> >>> Changes in this version: >>> >>> - at the request of Coleen (sorry I did not see your post >>> earlier) I removed all platform dependent?files and put the >>> platform dependent coding back to metaspace.cpp and >>> compressedOops.cpp to ease the review pain. I used plain >>> platform defines though (#ifdef AARCH64) instead of hiding >>> them behind another macro to make things clearer. Note that >>> I still intent to put this code away into the platform >>> corners but will do so in a follow up RFE. >>> >>> - I reinstated, in a fashion, the special handling of >>> reservations on Windows. On all platforms we reserve address >>> space to map the archive files in with a subsequent mapping >>> operation. However, on Windows, we cannot use >>> MapViewOfFile() into an existing mapping. So I remove the >>> mapping again before mapping the archives - see comment in >>> code for details. >>> >>> All CI tests at SAP run through without problems, including >>> on Win64 and aarch64, but I would be happy if others were to >>> run test too. >>> >>> Thank you, Thomas >>> >>> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >>> > >>> wrote: >>> >>> Hi all, >>> >>> Could I have reviews for the following proposal of >>> reworking cds/class space reservation? >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >>> >>> (Many thanks to Ioi Lam for so patiently explaining CDS >>> internals to me, and to Andrew Haley and Nick Gasson for >>> help with aarch64!) >>> >>> Reservation of the compressed class space is needlessly >>> complicated and has some minor issues. It can be >>> simplified and made clearer. >>> >>> The complexity stems from the fact that this area lives >>> at the intersection of two to three sub systems, >>> depending on how one counts. Metaspace, CDS, and the >>> platform which may or may not its own view of how to >>> reserve class space. And this code has been growing >>> organically over time. >>> >>> One small example: >>> >>> ReservedSpace Metaspace::reserve_preferred_space(size_t >>> size, size_t alignment, >>> ? ? ? ?bool large_pages, char *requested_addr, >>> ? ? ? ?bool use_requested_addr) >>> >>> which I spent hours decoding, resulting in a very >>> confused mail to hs-runtime and aarch64-port-dev [2]. >>> >>> This patch attempts to simplify cds and metaspace setup >>> a bit; to comment implicit knowledge which is not >>> immediately clear; to cleanly abstract platform concerns >>> like optimized class space placement; and to disentangle >>> cds from metaspace to solve issues which may bite us >>> later with Elastic Metaspace [4]. >>> >>> --- >>> >>> The main change is the reworked reservation mechanism. >>> This is based on Ioi's proposal [5]. >>> >>> When reserving class space, three things must happen: >>> >>> 1) reservation of the space obviously. If cds is active >>> that space must be in the vicinity of cds archives to be >>> covered by compressed class pointer encoding. >>> 2) setting up the internal Metaspace structures atop of >>> that space >>> 3) setting up compressed class pointer encoding. >>> >>> In its current form, Metaspace may or may not do some or >>> all of that in one function >>> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>> metaspace_rs, char* requested_addr, address cds_base);) >>> - if cds is active, it will reserve the space for >>> Metaspace and hand it in, otherwise it will create it >>> itself. >>> >>> When discussing this in [2], Ioi proposed to move the >>> reservation of the class space completely out of >>> Metaspace and make it a responsibility of the caller >>> always. This would reduce some complexity, and this >>> patch follows the proposal. >>> >>> I removed >>> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>> metaspace_rs, char* requested_addr, address cds_base); >>> and all its sub functions. >>> >>> (1) now has to always be done outside - a ReservedSpace >>> for class space has to be provided by the caller. >>> However, Metaspace now offers a utility function for >>> reserving space at a "nice" location, and explicitly >>> doing nothing else: >>> >>> ReservedSpace >>> Metaspace::reserve_address_space_for_compressed_classes(size_t >>> size); >>> >>> this function can be redefined on a platform level for >>> platform optimized reservation, see below for details. >>> >>> (2) is taken care of by a new function, >>> Metaspace::initialize_class_space(ReservedSpace rs) >>> >>> (3) is taken care of a new function >>> CompressedKlassPointers::initialize(), see below for >>> details. >>> >>> >>> So, class space now is set up three explicit steps: >>> >>> - First, reserve a suitable space by however means you >>> want. For convenience you may use >>> Metaspace::reserve_address_space_for_compressed_classes(), >>> or you may roll your own reservation. >>> - Next, tell Metaspace to use that range as backing >>> storage for class space: >>> Metaspace::initialize_class_space(ReservedSpace rs) >>> - Finally, set up encoding. Encoding is independent from >>> the concept of a ReservedSpace, it just gets an address >>> range, see below for details. >>> >>> Separating these steps and moving them out of the >>> responsibility of Metaspace makes this whole thing more >>> flexible; it also removes unnecessary knowledge (e.g. >>> Metaspace does not need to know anything about either >>> ccp encoding or cds). >>> >>> --- >>> >>> How it comes together: >>> >>> If CDS is off, we just reserve a space using >>> Metaspace::reserve_address_space_for_compressed_classes(), >>> initialize it with >>> Metaspace::initialize_class_space(ReservedSpace rs), >>> then set up compressed class pointer encoding covering >>> the range of this class space. >>> >>> If CDS is on (dump time), we reserve large 4G space, >>> either at SharedBaseAddress or using >>> Metaspace::reserve_address_space_for_compressed_classes(); >>> we then split that into 3G archive space and 1G class >>> space; we set up that space with Metaspace as class >>> space; then we set up?compressed class pointer encoding >>> covering both archive space and cds. >>> >>> If CDS is on (run time), we reserve a large space, split >>> it into archive space (large enough to hold both >>> archives) and class space, then basically proceed as above. >>> >>> Note that this is almost exactly how things worked >>> before (modulo some minor fixes, e.g. alignment issues), >>> only the code is reformed and made more explicit. >>> >>> --- >>> >>> I moved compressed class pointer setup over to >>> CompressedKlassPointers and changed the interface: >>> >>> -void >>> Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >>> metaspace_rs, address cds_base) >>> +void CompressedKlassPointers::initialize(address addr, >>> size_t len); >>> >>> Instead of feeding it a single ReservedSpace, which is >>> supposed to represent class space, and an optional >>> alternate base if cds is on, now we give it just an >>> numeric address range. That range marks the limits to >>> where Klass structures are to be expected, and is the >>> implicit promise that outside that range no Klass >>> structures will exist, so encoding has to cover only >>> this range. >>> >>> This range may contain just the class space; or class >>> space+cds; or whatever allocation scheme we come up with >>> in the future. Encoding does not really care how the >>> memory is organized as long as the input range covers >>> all possible Klass locations. That way we remove >>> knowledge about class space/cds from compressed class >>> pointer encoding. >>> >>> Moving it away from metaspace.cpp into the >>> CompressedKlassPointers class also mirrors >>> CompressedOops::initialize(). >>> >>> --- >>> >>> I renamed _narrow_klass_range to just _range, because >>> strictly speaking this is the range un-narrow Klass >>> pointers can have. >>> >>> As for the implementation of >>> CompressedKlassPointers::initialize(address addr, size_t >>> len), I mimicked very closely what happened before, so >>> there should be almost no differences. Since "almost no >>> differences" sounds scary :) here are the differences: >>> >>> - When CDS is active (dump or run time) we now always, >>> unconditionally, set the encoding range to 4G. This >>> fixes a theoretical bug discussed on aarch64-port-dev [1]. >>> >>> - When CDS is not active, we set the encoding range to >>> the minimum required length. Before, it was left at its >>> default value of 4G. >>> >>> Both differences only affect aarch64, since they are >>> currently the only one using the range field in >>> CompressedKlassPointers. >>> >>> I wanted to add an assert somewhere to test encoding of >>> the very last address of the CompressedKlassPointers >>> range, again to prevent errors like [3]. But I did not >>> come up with a good place for this assert which would >>> cover also the encoding done by C1/C2. >>> >>> For the same reason I thought about introducing a mode >>> where Klass structures would be allocated in reverse >>> order, starting at the end of the ccs, but again left it >>> out as too big a change. >>> >>> --- >>> >>> OS abstraction: platforms may have restrictions of what >>> constitutes a valid compressed class pointer encoding >>> base. Or if not, they may have at least preferences. >>> There was logic like this in metaspace.cpp, which I >>> removed and cleanly factored out into platform dependent >>> files, giving each platform the option to add special logic. >>> >>> These are two new methods: >>> >>> - bool CompressedKlassPointers::is_valid_base(address p) >>> >>> to let the platform tell you whether it considers p to >>> be a valid encoding base. The only platform having these >>> restrictions currently is aarch64. >>> >>> - ReservedSpace >>> Metaspace::reserve_address_space_for_compressed_classes(size_t >>> size); >>> >>> this hands over the process of allocating a range >>> suitable for compressed class pointer encoding to the >>> platform. Most platforms will allocate just anywhere, >>> but some platforms may have a better strategy (e.g. >>> trying low memory first, trying only correctly aligned >>> addresses and so on). >>> >>> Beforehand, this coding existed in a similar form in >>> metaspace.cpp for aarch64 and AIX. For now, I left the >>> AIX part out - it seems only half done, and I want to >>> check further if we even need it, if yes why not on >>> Linux ppc, and C1 does not seem to support anything >>> other than base+offset with shift either, but I may be >>> mistaken. >>> >>> These two methods should give the platform enough >>> control to implement their own scheme for optimized >>> class space placement without bothering any shared code >>> about it. >>> >>> Note about the form, I introduced two new platform >>> dependent files, "metaspace_.cpp" and >>> "compressedOops_.cpp". I am not happy about this >>> but this seems to be what we generally do in hotspot, right? >>> >>> --- >>> >>> Metaspace reserve alignment vs cds alignment >>> >>> CDS was using Metaspace reserve alignment for CDS >>> internal purposes. I guess this was just a copy paste >>> issue. It never caused problems since Metaspace reserve >>> alignment == page size, but that is not true anymore in >>> the upcoming Elastic Metaspace where reserve alignment >>> will be larger. This causes a number of issues. >>> >>> I separated those two cleanly. CDS now uses >>> os::vm_allocation_granularity. >>> Metaspace::reserve_alignment is only used in those two >>> places where it is needed, when CDS creates the address >>> space for class space on behalf of the Metaspace. >>> >>> --- >>> >>> Windows special handling in CDS >>> >>> To simplify coding I removed the windows specific >>> handling which left out reservation of the archive. This >>> was needed because windows cannot mmap files into >>> reserved regions. But fallback code exists in >>> filemap.cpp for this case which just reads in the region >>> instead of mapping?it. >>> >>> Should that turn out to be a performance problem, I will >>> reinstate the feature. But a simpler way would be >>> reserve the archive and later just before mmapping?the >>> archive file to release the archive space. That would >>> not only be simpler but give us the best guarantee that >>> that address space is actually available. But I'd be >>> happy to leave that part out completely if we do not see >>> any performance problems on windows x64. >>> >>> --- >>> >>> NMT cannot deal with spaces which are split. This >>> problem manifests in that bookkeeping for class space is >>> done under "Shared Classes", not "Classes" as it should. >>> This problem exists today too at dump?time and randomly >>> at run time. But since I simplified the reservation, >>> this problem now shows up always, whether or not we map >>> at the SharedBaseAddress. >>> While I could work around this problem, I'd prefer this >>> problem to be solved at the core, and NMT to have an >>> option to recognize reservation splits. So I'd rather >>> not put a workaround for this into the patch but leave >>> it for fixing as a separate issue. I opened this issue >>> to track it [6]. >>> >>> --- >>> >>> Jtreg tests: >>> >>> I expanded the >>> CompressedOops/CompressedClassPointers.java. I also >>> extended them to Windows. The tests now optionally omit >>> strict class space placement tests, since these tests >>> heavily depend on ASLR and were the reason they were >>> excluded on Windows. However I think even without >>> checking for class space placement they make sense, just >>> to see that the VM comes up and lives with the many >>> different settings we can run in. >>> >>> --- >>> >>> Tests: >>> >>> - I ran the patch through Oracles submit repo >>> - I ran tests manually for aarch64, zero, linux 32bit >>> and windows x64 >>> - The whole battery of nightly tests at SAP, including >>> ppc, ppcle and aarch64, unfortunately excluding windows >>> because of unrelated errors. Windows x64 tests will be >>> redone tonight. >>> >>> >>> Thank you, >>> >>> Thomas >>> >>> [1] >>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >>> [2] >>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >>> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >>> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >>> [5] >>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >>> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >>> >> > From ekaterina.pavlova at oracle.com Wed May 20 00:06:28 2020 From: ekaterina.pavlova at oracle.com (Ekaterina Pavlova) Date: Tue, 19 May 2020 17:06:28 -0700 Subject: Update Re: RFR JDK-8223347 Integration of Vector API (Incubator): Java API, implementation, and tests In-Reply-To: References: <49F81016-55DF-4E3C-940A-A8DF8C712259@oracle.com> <9D074341-7234-4D9F-8E99-0E73F5BB5407@oracle.com> Message-ID: <5bf2b471-36c0-f28b-b7c9-06ff5fded10c@oracle.com> As I wrote to openjdk alias tier3 seems to be more reasonable tier for incubating feature tests. Once the tests will be integrated we will also need to add these tests into hs-comp tiers to be tested with additional compiler flags (like -Xcomp). -katya On 5/19/20 3:25 PM, Paul Sandoz wrote: > I just realized that the vector tests have not been included in any JDK test category e.g. tier1. > > Ordinarily I would expect the tests to be added to tier1 since the tests exercise intrinsics. However, those intrinsics are only enabled with the Vector API module so we could place in another tier to potentially reduce the cost of testing. > > Advice very much appreciate on which tier the tests should belong to. > > Paul. > >> On May 18, 2020, at 12:13 PM, Paul Sandoz wrote: >> >> HI, >> >> Here?s an update of the API and implementation webrevs based on (mostly) API feedback. We have removed the performance tests to make the review easier (those will be dealt with separately to integration as a follow on task). >> >> I think over the past year and recently via the CSR the API has had significant review. Reviews focusing on the Java implementation and tests would be greatly appreciated. >> >> It?s worth reiterating that the implementation and tests are quite formulaic, there is a lot of code that is generated from templates, so it's possible to focus on the template + subset (e.g. byte, long, float, and sizes of say 256, max). >> >> Notable changes from the prior webrev are: >> >> - Removal of the fromValues methods to construct a vector from a varargs array. Feedback indicated this was a misleading way to obtain a vector. >> >> - Unification of byte array and ByteBuffer load/store method signatures and unification of the implementations with more tests (including out-of-bounds tests for all the kinds of loads/stores). >> >> Paul. >> >> -- >> >> Latest javadoc >> http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/docs-2020-05-15-88a83f7238d8/api/jdk.incubator.vector/jdk/incubator/vector/package-summary.html >> >> Latest specdiff >> http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/specdiff-jdk-2020-05-15-88a83f7238d8/overview-summary.html >> >> Incremental specdiff >> http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/specdiff-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/overview-summary.html >> >> >> Latest implementation webrev >> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev-2020-05-16-default-2020-05-15-88a83f7238d8/webrev/ >> >> Incremental Implementation webrev >> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/webrev/ >> >> Latest test webrev >> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev-2020-05-16-default-2020-05-15-88a83f7238d8/webrev/ >> >> Incremental test webrev >> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/webrev/ >> >> >>> On Apr 1, 2020, at 3:46 PM, Paul Sandoz wrote: >>> >>> Hi, >>> >>> A prior email sent out a request for review of the Vector API in preparation for JEP 338: Vector API (Incubator) [1] to be proposed for target: >>> >>> https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html >>> >>> This email expands the review of the API to the Java implementation and Java tests: >>> >>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev/webrev/ >>> >>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev/webrev/ >>> >>> (Further emails will sent for review of general HotSpot changes CPU architecture specific HotSpot changes, x64 and aarch64, and performance tests. For an early peek see the links in the description of JDK-8223347.) >>> >>> ? >>> >>> The Vector API provides >>> >>> - the public Vector class with vector operations common to all supported primitive types >>> >>> - public primitive specializations, such as IntVector, with common operations associated with the primitive type >>> >>> - internal concrete specializations for the vector size, such as Int256Vector, for a vector holding 8 ints. >>> >>> Operations generally defer to one of approximately 20 vector intrinsics. Some operations may be composed of other operations. >>> >>> Explicit casts are performed by vector operations to ensure vectors arguments are of the required shape (bit size) and element type. >>> >>> A vector intrinsic is an internal low-level vector operation. The last argument to the intrinsic is fall back behavior in Java, implementing the scalar operation over the number of elements held by the vector. Thus, If the intrinsic is not supported in C2 for the other arguments then the Java implementation is executed (the Java implementation is always executed when running in the interpreter or for C1). >>> >>> The public primitive specializations and the internal concrete implementations are generated from SSP template files, X-Vector.java.template and X-VectorBits.java.template respectively. >>> >>> Overall the implementation approach is quite formulaic and rather repetitive. Once you grok the pattern It should be easier to review if a little laborious. >>> >>> Unit tests are auto-generated by composing templates for each operation into an SSP template file which is then used to generate unit test files for each concrete vector class. The tests are quite extensive and have found many a HotSpot issue in development across a wide range of platforms. >>> >>> Paul. >>> >>> [1] https://bugs.openjdk.java.net/browse/JDK-8201271 >>> >> > From thomas.stuefe at gmail.com Wed May 20 03:54:53 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 20 May 2020 05:54:53 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: <8bf736b3-9ab1-86b7-fd22-230e449b1b9f@oracle.com> References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> <8bf736b3-9ab1-86b7-fd22-230e449b1b9f@oracle.com> Message-ID: Great, Ioi, thanks a lot for your help! On Wed, May 20, 2020 at 1:43 AM Ioi Lam wrote: > Hi Thomas, > > The testing took a little longer than I expected. I've run your previous > version: > > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ > > In our CI tiers 1/4. No failures. > > I didn't run your latest version. I would suggesting testing it with your > own CI as well as in jdk-submit before pushing (after you get OKs from > other reviewers). > > Thumbs up. > - Ioi > > > On 5/19/20 7:40 AM, Thomas St?fe wrote: > > Hi guys, > > new webrev: > > full: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ > delta: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ > > Mostly format changes, breaking up at column 80 as Andrew requested. > > The only changes which matter is the removal of the unnecessary include in > arguments.cpp as Coleen suggested; and to not validate SharedBaseAddress > anymore at runtime, as Ioi requested. > > Also rebased to head. > > Thanks, Thomas > > > > > On Tue, May 19, 2020 at 6:38 AM Ioi Lam wrote: > >> >> >> On 5/18/20 7:37 AM, Thomas St?fe wrote: >> >> Hi all, >> >> fourth webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ >> >> I rebased atop of >> >> changeset: 59325:1ba9a9a3f948 >> summary: 8244433: Remove saving of RSP in Assembler::pusha_uncached() >> >> so, atop of "8241825: Make compressed oops and compressed class pointers >> independent (x86_64, PPC, S390)". >> >> The changes are very small this time, see notes below: >> >> @Ioi: >> >> >Hi Thomas, >> >> >I am running your patch in out CI pipeline now. Some comments: >> >> >[2] Does JDK-8243535 exist with the current jdk/jdk repo? >> >> I assigned JDK-8243535 to me and I'm working on a patch. But I would like >> to keep that separate from this work, since this patch is already large. >> And I would like to fix JDK-8243535 after this remodel patch has been >> pushed, which makes some minor things easier. That would leave a small time >> window where NMT does mis-track class space as shared class space, but that >> is not a big issue, it does not even disturb any test. Which in itself is >> something we may want to fix, come to think about it. >> >> >> > [3] I think this can be put in header file. >> > >> > bool Metaspace::class_space_is_initialized() { >> > return _class_space_list != NULL; >> > } >> >> Done. >> >> > [7] argument.cpp checks this: >> > >> > if (UseSharedSpaces || DumpSharedSpaces) { >> > if >> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >> > log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for >> this platform, option will be ignored.", >> > p2i((address)SharedBaseAddress)); >> > FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >> > } >> > } >> > >> > but initialize_dumptime_shared_and_meta_spaces aligns up the >> SharedBaseAddress before doing the assert. >> > >> > void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >> > assert(DumpSharedSpaces, "should be called for dump time only"); >> > >> > const size_t reserve_alignment = >> MetaspaceShared::reserved_space_alignment(); >> > char* shared_base = (char*)align_up((char*)SharedBaseAddress, >> reserve_alignment); >> > >> > #ifdef _LP64 >> > >> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >> "Sanity"); >> > >> > So theoretically shared_base may no longer be is_valid_base after the >> align_up. It probably won't happen, but I think the code will be > much >> clearer tofirst to align_up, then check for is_valid_base and reset to >> default, and finally assert. >> >> The problem with that is that at the time this coding runs os::init() did >> not yet run and we do not know yet allocation granularity. >> >> So I changed verification of SharedBaseAddress: I added the alignment as >> you suggested and moved it to a later time in VM initialization, to CDS >> initialization. >> >> >> Is the call to check_SharedBaseAddress() necessary inside >> MetaspaceShared::initialize_runtime_shared_and_meta_spaces()? >> SharedBaseAddress is currently ignored at run time. Does your patch change >> that behavior? >> >> >[8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >> following block can be simplified: >> > >> > if (!_shared_rs.is_reserved()) { >> > // Get a reserved space anywhere if attaching at the >> SharedBaseAddress fails: >> > if (UseCompressedClassPointers) { >> > // If we need a compressed class space too, let the platform >> handle the reservation. >> > _shared_rs = >> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >> > } else { >> > // anywhere is fine. >> > _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >> large */, (char*)NULL); >> > } >> > } >> > >> > ... if you change the declaration to: >> > >> > static ReservedSpace >> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >> ReservedSpace();}); >> >> I'd like to avoid keep the method LP64 only, but I rewrote the section >> somewhat to be more clearer. >> >> The new code looks good. >> >> > [9] I think the #ifdef _LP64 is not necessary: >> > >> > #ifdef _LP64 >> > // Some sanity checks after reserving address spaces for archives >> and class space. >> > assert(archive_space_rs.is_reserved(), "Sanity"); >> > if (Metaspace::using_class_space()) { >> > // Class space must closely follow the archive space. Both spaces >> must be aligned correctly. >> > assert(class_space_rs.is_reserved(), "A class space should have >> been reserved"); >> > assert(class_space_rs.base() >= archive_space_rs.end(), "class >> space should follow the cds archive space"); >> > assert(is_aligned(archive_space_rs.base(), >> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >> > assert(is_aligned(class_space_rs.base(), >> Metaspace::reserve_alignment()), "class space misaligned"); >> > } >> > #endif >> >> I removed the #ifdef, but replaced it with an #ifdef ASSERT. Which is >> also not needed, technically, but makes it clear that this section is >> debugging checks only. >> >> >> OK. >> >> The rest of the code looks good to me. I am going to run your latest >> patch in our CI and will report the results. >> >> Thanks >> - Ioi >> >> ---- >> >> @Coleen: >> >> > >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/ >> >virtualSpaceList.cpp.udiff.html >> > >> >Why did you make this change? Shouldn't the caller align it? >> >> >> You are right, I removed the assert. >> >> --- >> >> Thanks all for your review work, >> >> ..Thomas >> >> >> >> >> >> >> >> >> >> >> >> >> On Fri, May 8, 2020 at 7:37 AM Ioi Lam wrote: >> >>> Hi Thomas, >>> >>> I am running your patch in out CI pipeline now. Some comments: >>> >>> [1] Do we still need to use _class_space_list? It looks like we just >>> have a single range. Is _class_space_list->current_virtual_space() always >>> the same VS? >>> >>> if (_class_space_list != NULL) { >>> address base = >>> (address)_class_space_list->current_virtual_space()->bottom(); >>> address top = base + compressed_class_space_size(); >>> st->print("Compressed class space mapped at: " PTR_FORMAT "-" >>> PTR_FORMAT ", size: " SIZE_FORMAT, >>> p2i(base), p2i(top), top - base); >>> >>> [2] Does JDK-8243535 exist with the current jdk/jdk repo? >>> >>> // This does currently not work because rs may be the result of a >>> split operation >>> // and NMT seems not to be able to handle splits. >>> // See JDK-8243535. >>> // MemTracker::record_virtual_memory_type((address)rs.base(), mtClass); >>> >>> >>> [3] I think this can be put in header file. >>> >>> bool Metaspace::class_space_is_initialized() { >>> return _class_space_list != NULL; >>> } >>> >>> [4] Why does the CCS need to follow UseLargePagesInMetaspace? This is >>> the reason for the >>> gap in the following chart: >>> >>> // We do this by reserving space for the ccs behind the archives. >>> Note however that >>> // ccs follows a different alignment >>> (Metaspace::reserve_alignment), so there may >>> // be a gap between ccs and cds. >>> // We use a similar layout at runtime, see >>> reserve_address_space_for_archives(). >>> // >>> // +-- SharedBaseAddress (default = >>> 0x800000000) >>> // v >>> // +-..---------+---------+ ... >>> +----+----+----+--------+--------------------+ >>> // | Heap | Archive | | MC | RW | RO | [gap] | class >>> space | >>> // +-..---------+---------+ ... >>> +----+----+----+--------+--------------------+ >>> // |<-- MaxHeapSize -->| |<-- UnscaledClassSpaceMax = 4GB -->| >>> >>> >>> _reserve_alignment is determined here: >>> >>> void Metaspace::ergo_initialize() { >>> if (DumpSharedSpaces) { >>> // Using large pages when dumping the shared archive is currently >>> not implemented. >>> FLAG_SET_ERGO(UseLargePagesInMetaspace, false); >>> } >>> >>> size_t page_size = os::vm_page_size(); >>> if (UseLargePages && UseLargePagesInMetaspace) { >>> page_size = os::large_page_size(); >>> } >>> >>> _commit_alignment = page_size; >>> _reserve_alignment = MAX2(page_size, >>> (size_t)os::vm_allocation_granularity()); >>> >>> But when CDS is enabled, the RS is reserved without large pages, so it >>> should be possible to incrementally commit the CCS using just >>> os::vm_allocation_granularity(). >>> >>> [5] I see the aarch64 code in >>> Metaspace::reserve_address_space_for_compressed_classes has been changed. >>> Is this necessary for this RFE, or is it a separate improvement that can be >>> done in a different RFE? >>> >>> [6] For AARCH64, should we skip the part that reserves it right above >>> the heap? Or does AARCH64 always allocate the heap such that it's a >>> preferable address? >>> >>> // case (b) >>> ReservedSpace rs; >>> >>> // Try right above the Java heap... >>> address base = align_up(CompressedOops::end(), >>> Metaspace::reserve_alignment()); >>> assert(base != NULL, "Sanity"); >>> >>> const size_t size = align_up(CompressedClassSpaceSize, >>> Metaspace::reserve_alignment()); >>> if (CompressedKlassPointers::is_valid_base(base)) { >>> rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* >>> large */, (char*)base); >>> } >>> >>> // ...failing that, reserve anywhere, but let platform do optimized >>> placement: >>> if (!rs.is_reserved()) { >>> rs = Metaspace::reserve_address_space_for_compressed_classes(size); >>> } >>> >>> [7] argument.cpp checks this: >>> >>> if (UseSharedSpaces || DumpSharedSpaces) { >>> if >>> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >>> log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for >>> this platform, option will be ignored.", >>> p2i((address)SharedBaseAddress)); >>> FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >>> } >>> } >>> >>> but initialize_dumptime_shared_and_meta_spaces aligns up the >>> SharedBaseAddress before doing the assert. >>> >>> void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >>> assert(DumpSharedSpaces, "should be called for dump time only"); >>> >>> const size_t reserve_alignment = >>> MetaspaceShared::reserved_space_alignment(); >>> char* shared_base = (char*)align_up((char*)SharedBaseAddress, >>> reserve_alignment); >>> >>> #ifdef _LP64 >>> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >>> "Sanity"); >>> >>> So theoretically shared_base may no longer be is_valid_base after the >>> align_up. It probably won't happen, but I think the code will be much >>> clearer tofirst to align_up, then check for is_valid_base and reset to >>> default, and finally assert. >>> >>> [8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >>> following block can be simplified: >>> >>> if (!_shared_rs.is_reserved()) { >>> // Get a reserved space anywhere if attaching at the >>> SharedBaseAddress fails: >>> if (UseCompressedClassPointers) { >>> // If we need a compressed class space too, let the platform >>> handle the reservation. >>> _shared_rs = >>> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >>> } else { >>> // anywhere is fine. >>> _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >>> large */, (char*)NULL); >>> } >>> } >>> >>> ... if you change the declaration to: >>> >>> static ReservedSpace >>> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >>> ReservedSpace();}); >>> >>> [9] I think the #ifdef _LP64 is not necessary: >>> >>> #ifdef _LP64 >>> // Some sanity checks after reserving address spaces for archives >>> and class space. >>> assert(archive_space_rs.is_reserved(), "Sanity"); >>> if (Metaspace::using_class_space()) { >>> // Class space must closely follow the archive space. Both spaces >>> must be aligned correctly. >>> assert(class_space_rs.is_reserved(), "A class space should have >>> been reserved"); >>> assert(class_space_rs.base() >= archive_space_rs.end(), "class >>> space should follow the cds archive space"); >>> assert(is_aligned(archive_space_rs.base(), >>> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >>> assert(is_aligned(class_space_rs.base(), >>> Metaspace::reserve_alignment()), "class space misaligned"); >>> } >>> #endif >>> >>> The rest of the code looks OK to me, but I may take a look at it again >>> after getting more sleep :-) >>> >>> Thanks >>> - Ioi >>> >>> >>> >>> >>> On 5/7/20 7:21 AM, Thomas St?fe wrote: >>> >>> Hi all, >>> >>> please take a look at the third iteration of this change: >>> >>> >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >>> >>> Changes in this version: >>> >>> - at the request of Coleen (sorry I did not see your post earlier) I >>> removed all platform dependent files and put the platform dependent coding >>> back to metaspace.cpp and compressedOops.cpp to ease the review pain. I >>> used plain platform defines though (#ifdef AARCH64) instead of hiding them >>> behind another macro to make things clearer. Note that I still intent to >>> put this code away into the platform corners but will do so in a follow up >>> RFE. >>> >>> - I reinstated, in a fashion, the special handling of reservations on >>> Windows. On all platforms we reserve address space to map the archive files >>> in with a subsequent mapping operation. However, on Windows, we cannot use >>> MapViewOfFile() into an existing mapping. So I remove the mapping again >>> before mapping the archives - see comment in code for details. >>> >>> All CI tests at SAP run through without problems, including on Win64 and >>> aarch64, but I would be happy if others were to run test too. >>> >>> Thank you, Thomas >>> >>> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >>> wrote: >>> >>>> Hi all, >>>> >>>> Could I have reviews for the following proposal of reworking cds/class >>>> space reservation? >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >>>> >>>> (Many thanks to Ioi Lam for so patiently explaining CDS internals to >>>> me, and to Andrew Haley and Nick Gasson for help with aarch64!) >>>> >>>> Reservation of the compressed class space is needlessly complicated and >>>> has some minor issues. It can be simplified and made clearer. >>>> >>>> The complexity stems from the fact that this area lives at the >>>> intersection of two to three sub systems, depending on how one counts. >>>> Metaspace, CDS, and the platform which may or may not its own view of how >>>> to reserve class space. And this code has been growing organically over >>>> time. >>>> >>>> One small example: >>>> >>>> ReservedSpace Metaspace::reserve_preferred_space(size_t size, size_t >>>> alignment, >>>> bool large_pages, char >>>> *requested_addr, >>>> bool >>>> use_requested_addr) >>>> >>>> which I spent hours decoding, resulting in a very confused mail to >>>> hs-runtime and aarch64-port-dev [2]. >>>> >>>> This patch attempts to simplify cds and metaspace setup a bit; to >>>> comment implicit knowledge which is not immediately clear; to cleanly >>>> abstract platform concerns like optimized class space placement; and to >>>> disentangle cds from metaspace to solve issues which may bite us later with >>>> Elastic Metaspace [4]. >>>> >>>> --- >>>> >>>> The main change is the reworked reservation mechanism. This is based on >>>> Ioi's proposal [5]. >>>> >>>> When reserving class space, three things must happen: >>>> >>>> 1) reservation of the space obviously. If cds is active that space must >>>> be in the vicinity of cds archives to be covered by compressed class >>>> pointer encoding. >>>> 2) setting up the internal Metaspace structures atop of that space >>>> 3) setting up compressed class pointer encoding. >>>> >>>> In its current form, Metaspace may or may not do some or all of that in >>>> one function >>>> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>>> metaspace_rs, char* requested_addr, address cds_base);) - if cds is active, >>>> it will reserve the space for Metaspace and hand it in, otherwise it will >>>> create it itself. >>>> >>>> When discussing this in [2], Ioi proposed to move the reservation of >>>> the class space completely out of Metaspace and make it a responsibility of >>>> the caller always. This would reduce some complexity, and this patch >>>> follows the proposal. >>>> >>>> I removed >>>> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>>> metaspace_rs, char* requested_addr, address cds_base); and all its sub >>>> functions. >>>> >>>> (1) now has to always be done outside - a ReservedSpace for class space >>>> has to be provided by the caller. However, Metaspace now offers a utility >>>> function for reserving space at a "nice" location, and explicitly doing >>>> nothing else: >>>> >>>> ReservedSpace >>>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>>> >>>> this function can be redefined on a platform level for platform >>>> optimized reservation, see below for details. >>>> >>>> (2) is taken care of by a new function, >>>> Metaspace::initialize_class_space(ReservedSpace rs) >>>> >>>> (3) is taken care of a new function >>>> CompressedKlassPointers::initialize(), see below for details. >>>> >>>> >>>> So, class space now is set up three explicit steps: >>>> >>>> - First, reserve a suitable space by however means you want. For >>>> convenience you may use >>>> Metaspace::reserve_address_space_for_compressed_classes(), or you may roll >>>> your own reservation. >>>> - Next, tell Metaspace to use that range as backing storage for class >>>> space: Metaspace::initialize_class_space(ReservedSpace rs) >>>> - Finally, set up encoding. Encoding is independent from the concept of >>>> a ReservedSpace, it just gets an address range, see below for details. >>>> >>>> Separating these steps and moving them out of the responsibility of >>>> Metaspace makes this whole thing more flexible; it also removes unnecessary >>>> knowledge (e.g. Metaspace does not need to know anything about either ccp >>>> encoding or cds). >>>> >>>> --- >>>> >>>> How it comes together: >>>> >>>> If CDS is off, we just reserve a space using >>>> Metaspace::reserve_address_space_for_compressed_classes(), initialize it >>>> with Metaspace::initialize_class_space(ReservedSpace rs), then set up >>>> compressed class pointer encoding covering the range of this class space. >>>> >>>> If CDS is on (dump time), we reserve large 4G space, either at >>>> SharedBaseAddress or using >>>> Metaspace::reserve_address_space_for_compressed_classes(); we then split >>>> that into 3G archive space and 1G class space; we set up that space with >>>> Metaspace as class space; then we set up compressed class pointer encoding >>>> covering both archive space and cds. >>>> >>>> If CDS is on (run time), we reserve a large space, split it into >>>> archive space (large enough to hold both archives) and class space, then >>>> basically proceed as above. >>>> >>>> Note that this is almost exactly how things worked before (modulo some >>>> minor fixes, e.g. alignment issues), only the code is reformed and made >>>> more explicit. >>>> >>>> --- >>>> >>>> I moved compressed class pointer setup over to CompressedKlassPointers >>>> and changed the interface: >>>> >>>> -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >>>> metaspace_rs, address cds_base) >>>> +void CompressedKlassPointers::initialize(address addr, size_t len); >>>> >>>> Instead of feeding it a single ReservedSpace, which is supposed to >>>> represent class space, and an optional alternate base if cds is on, now we >>>> give it just an numeric address range. That range marks the limits to where >>>> Klass structures are to be expected, and is the implicit promise that >>>> outside that range no Klass structures will exist, so encoding has to cover >>>> only this range. >>>> >>>> This range may contain just the class space; or class space+cds; or >>>> whatever allocation scheme we come up with in the future. Encoding does not >>>> really care how the memory is organized as long as the input range covers >>>> all possible Klass locations. That way we remove knowledge about class >>>> space/cds from compressed class pointer encoding. >>>> >>>> Moving it away from metaspace.cpp into the CompressedKlassPointers >>>> class also mirrors CompressedOops::initialize(). >>>> >>>> --- >>>> >>>> I renamed _narrow_klass_range to just _range, because strictly speaking >>>> this is the range un-narrow Klass pointers can have. >>>> >>>> As for the implementation of >>>> CompressedKlassPointers::initialize(address addr, size_t len), I mimicked >>>> very closely what happened before, so there should be almost no >>>> differences. Since "almost no differences" sounds scary :) here are the >>>> differences: >>>> >>>> - When CDS is active (dump or run time) we now always, unconditionally, >>>> set the encoding range to 4G. This fixes a theoretical bug discussed on >>>> aarch64-port-dev [1]. >>>> >>>> - When CDS is not active, we set the encoding range to the minimum >>>> required length. Before, it was left at its default value of 4G. >>>> >>>> Both differences only affect aarch64, since they are currently the only >>>> one using the range field in CompressedKlassPointers. >>>> >>>> I wanted to add an assert somewhere to test encoding of the very last >>>> address of the CompressedKlassPointers range, again to prevent errors like >>>> [3]. But I did not come up with a good place for this assert which would >>>> cover also the encoding done by C1/C2. >>>> >>>> For the same reason I thought about introducing a mode where Klass >>>> structures would be allocated in reverse order, starting at the end of the >>>> ccs, but again left it out as too big a change. >>>> >>>> --- >>>> >>>> OS abstraction: platforms may have restrictions of what constitutes a >>>> valid compressed class pointer encoding base. Or if not, they may have at >>>> least preferences. There was logic like this in metaspace.cpp, which I >>>> removed and cleanly factored out into platform dependent files, giving each >>>> platform the option to add special logic. >>>> >>>> These are two new methods: >>>> >>>> - bool CompressedKlassPointers::is_valid_base(address p) >>>> >>>> to let the platform tell you whether it considers p to be a valid >>>> encoding base. The only platform having these restrictions currently is >>>> aarch64. >>>> >>>> - ReservedSpace >>>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>>> >>>> this hands over the process of allocating a range suitable for >>>> compressed class pointer encoding to the platform. Most platforms will >>>> allocate just anywhere, but some platforms may have a better strategy (e.g. >>>> trying low memory first, trying only correctly aligned addresses and so >>>> on). >>>> >>>> Beforehand, this coding existed in a similar form in metaspace.cpp for >>>> aarch64 and AIX. For now, I left the AIX part out - it seems only half >>>> done, and I want to check further if we even need it, if yes why not on >>>> Linux ppc, and C1 does not seem to support anything other than base+offset >>>> with shift either, but I may be mistaken. >>>> >>>> These two methods should give the platform enough control to implement >>>> their own scheme for optimized class space placement without bothering any >>>> shared code about it. >>>> >>>> Note about the form, I introduced two new platform dependent files, >>>> "metaspace_.cpp" and "compressedOops_.cpp". I am not happy about >>>> this but this seems to be what we generally do in hotspot, right? >>>> >>>> --- >>>> >>>> Metaspace reserve alignment vs cds alignment >>>> >>>> CDS was using Metaspace reserve alignment for CDS internal purposes. I >>>> guess this was just a copy paste issue. It never caused problems since >>>> Metaspace reserve alignment == page size, but that is not true anymore in >>>> the upcoming Elastic Metaspace where reserve alignment will be larger. This >>>> causes a number of issues. >>>> >>>> I separated those two cleanly. CDS now uses >>>> os::vm_allocation_granularity. Metaspace::reserve_alignment is only used in >>>> those two places where it is needed, when CDS creates the address space for >>>> class space on behalf of the Metaspace. >>>> >>>> --- >>>> >>>> Windows special handling in CDS >>>> >>>> To simplify coding I removed the windows specific handling which left >>>> out reservation of the archive. This was needed because windows cannot mmap >>>> files into reserved regions. But fallback code exists in filemap.cpp for >>>> this case which just reads in the region instead of mapping it. >>>> >>>> Should that turn out to be a performance problem, I will reinstate the >>>> feature. But a simpler way would be reserve the archive and later just >>>> before mmapping the archive file to release the archive space. That would >>>> not only be simpler but give us the best guarantee that that address space >>>> is actually available. But I'd be happy to leave that part out completely >>>> if we do not see any performance problems on windows x64. >>>> >>>> --- >>>> >>>> NMT cannot deal with spaces which are split. This problem manifests in >>>> that bookkeeping for class space is done under "Shared Classes", not >>>> "Classes" as it should. This problem exists today too at dump time and >>>> randomly at run time. But since I simplified the reservation, this problem >>>> now shows up always, whether or not we map at the SharedBaseAddress. >>>> While I could work around this problem, I'd prefer this problem to be >>>> solved at the core, and NMT to have an option to recognize reservation >>>> splits. So I'd rather not put a workaround for this into the patch but >>>> leave it for fixing as a separate issue. I opened this issue to track it >>>> [6]. >>>> >>>> --- >>>> >>>> Jtreg tests: >>>> >>>> I expanded the CompressedOops/CompressedClassPointers.java. I also >>>> extended them to Windows. The tests now optionally omit strict class space >>>> placement tests, since these tests heavily depend on ASLR and were the >>>> reason they were excluded on Windows. However I think even without checking >>>> for class space placement they make sense, just to see that the VM comes up >>>> and lives with the many different settings we can run in. >>>> >>>> --- >>>> >>>> Tests: >>>> >>>> - I ran the patch through Oracles submit repo >>>> - I ran tests manually for aarch64, zero, linux 32bit and windows x64 >>>> - The whole battery of nightly tests at SAP, including ppc, ppcle and >>>> aarch64, unfortunately excluding windows because of unrelated errors. >>>> Windows x64 tests will be redone tonight. >>>> >>>> >>>> Thank you, >>>> >>>> Thomas >>>> >>>> [1] >>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >>>> [2] >>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >>>> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >>>> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >>>> [5] >>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >>>> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >>>> >>>> >>> >> > From stefan.karlsson at oracle.com Wed May 20 06:45:15 2020 From: stefan.karlsson at oracle.com (Stefan Karlsson) Date: Wed, 20 May 2020 08:45:15 +0200 Subject: RFR: 8245002: Windows GDI functions don't support NUMA interleaving In-Reply-To: <3F0B7551-32DA-4169-81FA-728872C750AA@oracle.com> References: <2D7DEE5B-8D23-4E6A-914B-F3DA949F99B1@oracle.com> <841690cf-193e-df2e-9643-eaaceae935a5@oracle.com> <4cf0265c-f901-e1fb-edf6-dc7bdb0a0bc7@oracle.com> <8e4aeff8-df10-538b-d65f-b8ed05217244@oracle.com> <3F0B7551-32DA-4169-81FA-728872C750AA@oracle.com> Message-ID: <20f5c17a-2291-ea91-5f6b-878ddfddb8f1@oracle.com> On 2020-05-19 21:53, Kim Barrett wrote: >> On May 19, 2020, at 10:04 AM, Stefan Karlsson wrote: >> https://cr.openjdk.java.net/~stefank/8245002/webrev.03.workaround.thomas-review/ >> https://cr.openjdk.java.net/~stefank/8245002/webrev.all.thomas-review/ > Looks good. > > A few trivial typos: > > 2770 // If this fail, the testing method above isn't reliable. > s/fail/fails/ > > 2825 // Set the bitmaps bits to point one "width" bytes before, so that > s/bitmaps/bitmap's/ > > 2826 // the bitmap extends a cross the reservation boundary. > s/a cross/across/ > All fixed. Thanks! StefanK From per.liden at oracle.com Wed May 20 11:04:02 2020 From: per.liden at oracle.com (Per Liden) Date: Wed, 20 May 2020 13:04:02 +0200 Subject: RFR: 8244416: Remove incorrect assert during inline cache cleaning In-Reply-To: <5a4574d6-0d7d-9d52-19b0-cbacebb1fba5@oracle.com> References: <5a4574d6-0d7d-9d52-19b0-cbacebb1fba5@oracle.com> Message-ID: Looks good! /Per On 5/14/20 3:07 PM, Erik ?sterlund wrote: > Hi, > > Months of stress testing triggered the following assert in inline cache > cleaning: > > assert(!from->is_zombie(), "should not clean inline caches on zombies"); > > It used to be assert(!from->is_alive()), but after finding a race that > resulted in > false positives for unloaded nmethods, the assert was relaxed to check > only that > we shouldn't have zombies here, with a fat comment explaining why. > > Unfortunately, the relaxed assert also gets false positives, so I have > chosen to > remove it, and further increase the size of that fat comment, explaning > why we can't > assert that it isn't a zombie either. > > What it boils down to is a race where inline cache cleaning is performed > from > a concurrent GC thread, concurrent to the sweeper making an nmethod zombie. > > The order of the race ingredients making the nmethod zombie from the > sweeper is > the following: > > 1. Transition state to zombie > 2. Unregister nmethod form GC (importantly triggers a wait for > concurrent code cache unloading to finish) > 3. clear_ic_callsites() importantly releasing all IC stubs > > The race happens when the GC finds an is_alive && !is_unloading nmethod and > triggers inline cache cleaning on it before 1, but then when racingly > checking > again in the assert, we are past 1. > > Since the assert will spuriously fail even though there is no actual > problem, and > we are at a point where there isn't much else it could assert, I chose > to remove it. > > Other than triggering the assert, it is worth noting that this racingly > zombiefied > nmethod will have its inline caches cleaned *with* IC stubs from the > concurrent > GC thread. Subsequent IC finalization in safepoint cleanup requires live > IC stubs > to belong to is_alive() nmethods, so the IC stubs must be released > before the next > safepoint for this race to not be dangerous. That is ensured, because > the racing > sweeper will get stuck waiting for concurrent code cache unloading to > finish, > and when it wakes up, it will clear_ic_callsites(), releasing the > spurious ICStubs, > before finishing the make_zombie() function. Then it will eventually hit > a safepoint > poll, when the nmethod has been purged from IC stubs. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8244416 > > Webrev: > http://cr.openjdk.java.net/~eosterlund/8244416/webrev.00/ > > Thanks, > /Erik From matthias.baesken at sap.com Wed May 20 11:07:15 2020 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Wed, 20 May 2020 11:07:15 +0000 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: <51519874123ae26a4d3ebf4863d80e682c13b989.camel@redhat.com> References: <7201E4D5-96AA-47C1-B0B7-A19E8A0F198B@tencent.com> <51519874123ae26a4d3ebf4863d80e682c13b989.camel@redhat.com> Message-ID: Hi Severin , > FWIW, parts of this work-around got added by JDK-8236617. Looking at > the bug it was observed that: > > containerMetrics.getMemoryAndSwapLimit() == 0 and > containerMetrics.getMemoryLimit() returned a large value. > > So then with JDK-8231111 I suspect > containerMetrics.getMemoryAndSwapLimit() now returns -1 and, thus, > getTotalSwapSpaceSize0() is being used instead of the container values. I added some tracing output to OperatingSystemImpl.getTotalSwapSpaceSize() and get these values : containerMetrics.getMemoryAndSwapLimit():-1 containerMetrics.getMemoryLimit():-1 ****** getTotalSwapSpaceSize0() returned 16106061824 >So far the theory, I'd like to gather more data before we come up with >another patch, though. In JDK-8236617 it wasn't investigated why >containerMetrics.getMemoryAndSwapLimit() returned 0 on such a system. >It would be good to know why. Could it be that file >'memory.memsw.limit_in_bytes' doesn't exist on such a system? True, it does not exist : ls -alL /sys/fs/cgroup/memory/ total 0 drwxr-xr-x 3 root root 0 Mar 16 11:00 . drwxr-xr-x 11 root root 260 Mar 16 11:00 .. -rw-r--r-- 1 root root 0 Mar 16 11:00 cgroup.clone_children --w--w--w- 1 root root 0 Mar 16 11:00 cgroup.event_control -rw-r--r-- 1 root root 0 May 20 13:00 cgroup.procs -r--r--r-- 1 root root 0 Mar 16 11:00 cgroup.sane_behavior drwxr-xr-x 2 root root 0 May 20 12:52 docker -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.failcnt --w------- 1 root root 0 Mar 16 11:00 memory.force_empty -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.limit_in_bytes -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.low_limit_in_bytes -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.max_usage_in_bytes -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.move_charge_at_immigrate -r--r--r-- 1 root root 0 Mar 16 11:00 memory.numa_stat -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.oom_control ---------- 1 root root 0 Mar 16 11:00 memory.pressure_level -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.soft_limit_in_bytes -r--r--r-- 1 root root 0 Mar 16 11:00 memory.stat -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.swappiness -r--r--r-- 1 root root 0 Mar 16 11:00 memory.usage_in_bytes -rw-r--r-- 1 root root 0 May 20 13:00 memory.use_hierarchy -rw-r--r-- 1 root root 0 Mar 16 11:00 notify_on_release -rw-r--r-- 1 root root 0 Mar 16 11:00 release_agent -rw-r--r-- 1 root root 0 Mar 16 11:00 tasks Best regards, Matthias -----Original Message----- From: Severin Gehwolf Sent: Montag, 18. Mai 2020 12:09 To: jiefu(??) ; David Holmes ; Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' Cc: Bob Vandette Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java Hi, On Mon, 2020-05-18 at 06:15 +0000, jiefu(??) wrote: > Hi David, > > This bug should be introduced by JDK-8242480. > I'm sorry for that. > > And I think the logic in src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java should be fixed. > > In my fix for JDK-8242480, I had ignored the case for kernels which don't support "--memory-swap 150M". > In that case, containerMetrics.getMemoryAndSwapLimit()[1] will return -1. It appears so that containerMetrics.getMemoryAndSwapLimit() returned 0 prior JDK-8231111 and -1 after. I'm not convinced JDK-8242480 introduced this. > I didn't notice this bug because all our kernels support --memory-swap. > Fortunately, Matthias found this bug and will fix it. I'm still not sure what's happening in Matthias' case on that system. It would be best to first try to understand the actual problem better. The info I have is: - It's a cgroup v1 system - The system now returns large values for OperatingSystemMXBean.getTotalSwapSpaceSize() and OperatingSystemMXBean.getFreeSwapSpaceSize() It would be good to know what the actual return values are for the container metrics on this system with swap limit disabled at kernel level. And also whether or not relevant cgroup files exist on such a system (within docker). FWIW, parts of this work-around got added by JDK-8236617. Looking at the bug it was observed that: containerMetrics.getMemoryAndSwapLimit() == 0 and containerMetrics.getMemoryLimit() returned a large value. So then with JDK-8231111 I suspect containerMetrics.getMemoryAndSwapLimit() now returns -1 and, thus, getTotalSwapSpaceSize0() is being used instead of the container values. So far the theory, I'd like to gather more data before we come up with another patch, though. In JDK-8236617 it wasn't investigated why containerMetrics.getMemoryAndSwapLimit() returned 0 on such a system. It would be good to know why. Could it be that file 'memory.memsw.limit_in_bytes' doesn't exist on such a system? Unfortunately, I don't know what config one needs in order to reproduce so we'll have to rely on Matthias to gather the info. Thanks, Severin > Thanks a lot. > Best regards, > Jie > > [1] http://hg.openjdk.java.net/jdk/jdk/file/a95911422005/src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java#l73 > > ?On 2020/5/18, 1:48 PM, "David Holmes" wrote: > > Hi Matthias, > > On 16/05/2020 12:59 am, Baesken, Matthias wrote: > > Hello David , I get the warning always on this linux machine (I think it depends on machine setup and/or kernel version). > > > >> Surely it must be specified to return 0 or some such sentinel in such a case? > > > > The doc just says this : > > > > 62 /** > > 63 * Returns the total amount of swap space in bytes. > > 64 * > > 65 * @return the total amount of swap space in bytes. > > 66 */ > > 67 public long getTotalSwapSpaceSize(); > > > > ( however it returned previously 0 before the latest changes in the container support code came in ). > > That sounds like a bug was introduced to me then. > > > Regarding the getFreeSwapSpaceSize issue , Jie provided a change to src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java > > Because of this I could remove the try-construct . > > > > New webrev : > > http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.3/ > > I'm quite confused by your changes. Your first version simply adjusted > the test to check for the warning. But now you are also changing the > main logic. I can't assess the validity of what you are doing sorry. > > I've cc'd Severin and Bob as I think they need to evaluate what the real > problem is here and when it was introduced. > > David > ---- > > > Best regards, Matthias > > > > > > > > -----Original Message----- > > From: David Holmes > > Sent: Donnerstag, 14. Mai 2020 07:02 > > To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' > > Cc: jiefu(??) > > Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > > > On 14/05/2020 12:28 am, Baesken, Matthias wrote: > >> Hello, here is a new webrev removing the comment : > >> > >> http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.1/ > >> > >> It removes the comment , and it also handles 0 for getFreeSwapSpaceSize > >> ( this can occur after JDK-8242480 : Negative value may be returned by getFreeSwapSpaceSize() in the docker was pushed recently ). > > > > Sorry Matthias but I still see this partly as a bug in the underlying > > container support code. If you get the warning then what does > > getTotalSwapSpaceSize() return? Surely it must be specified to return 0 > > or some such sentinel in such a case? > > > > Other comments: > > > > ! try { > > ! > > out.shouldMatch("OperatingSystemMXBean\\.getFreeSwapSpaceSize: > > [1-9][0-9]+"); > > ! } catch(RuntimeException ex) { // allow 0, see 8242480 > > ! > > out.shouldContain("OperatingSystemMXBean.getFreeSwapSpaceSize: 0"); > > ! } > > > > can't you just change the original regex to match >= 0? > > > > Style nit: avoid historical comments like "see 8242480" and "since > > 8231111 ...". > > > > Thanks, > > David > > > >> Best regards, Matthias > >> > >> -----Original Message----- > >> From: Baesken, Matthias > >> Sent: Mittwoch, 13. Mai 2020 10:11 > >> To: 'David Holmes' ; 'hotspot-dev at openjdk.java.net' > >> Subject: RE: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > >> > >> > >>> But according to the comment immediately before your changes: > >>> > >>> // in case of warnings like : "Your kernel does not support swap limit > >>> capabilities or the cgroup is not mounted. Memory limited without swap." > >>> // the getTotalSwapSpaceSize does not return the expected result, but 0 > >>> > >>> we should be returning zero. So this seems to be a bug in the > >>> implementation not in the test. > >> > >> > >> Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values > >> that are normally expected. > >> Previously we indeed got "0" . > >> Probably the comment should be deleted . > >> > >> Best regards, Matthias > >> > > > From erik.osterlund at oracle.com Wed May 20 11:28:44 2020 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Wed, 20 May 2020 13:28:44 +0200 Subject: RFR: 8244416: Remove incorrect assert during inline cache cleaning In-Reply-To: References: <5a4574d6-0d7d-9d52-19b0-cbacebb1fba5@oracle.com> Message-ID: <9a9cbcfe-77eb-5c79-2f67-052042002ac5@oracle.com> Hi Per, Thanks for the review! /Erik On 2020-05-20 13:04, Per Liden wrote: > Looks good! > > /Per > > On 5/14/20 3:07 PM, Erik ?sterlund wrote: >> Hi, >> >> Months of stress testing triggered the following assert in inline >> cache cleaning: >> >> assert(!from->is_zombie(), "should not clean inline caches on zombies"); >> >> It used to be assert(!from->is_alive()), but after finding a race >> that resulted in >> false positives for unloaded nmethods, the assert was relaxed to >> check only that >> we shouldn't have zombies here, with a fat comment explaining why. >> >> Unfortunately, the relaxed assert also gets false positives, so I >> have chosen to >> remove it, and further increase the size of that fat comment, >> explaning why we can't >> assert that it isn't a zombie either. >> >> What it boils down to is a race where inline cache cleaning is >> performed from >> a concurrent GC thread, concurrent to the sweeper making an nmethod >> zombie. >> >> The order of the race ingredients making the nmethod zombie from the >> sweeper is >> the following: >> >> 1. Transition state to zombie >> 2. Unregister nmethod form GC (importantly triggers a wait for >> concurrent code cache unloading to finish) >> 3. clear_ic_callsites() importantly releasing all IC stubs >> >> The race happens when the GC finds an is_alive && !is_unloading >> nmethod and >> triggers inline cache cleaning on it before 1, but then when racingly >> checking >> again in the assert, we are past 1. >> >> Since the assert will spuriously fail even though there is no actual >> problem, and >> we are at a point where there isn't much else it could assert, I >> chose to remove it. >> >> Other than triggering the assert, it is worth noting that this >> racingly zombiefied >> nmethod will have its inline caches cleaned *with* IC stubs from the >> concurrent >> GC thread. Subsequent IC finalization in safepoint cleanup requires >> live IC stubs >> to belong to is_alive() nmethods, so the IC stubs must be released >> before the next >> safepoint for this race to not be dangerous. That is ensured, because >> the racing >> sweeper will get stuck waiting for concurrent code cache unloading to >> finish, >> and when it wakes up, it will clear_ic_callsites(), releasing the >> spurious ICStubs, >> before finishing the make_zombie() function. Then it will eventually >> hit a safepoint >> poll, when the nmethod has been purged from IC stubs. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8244416 >> >> Webrev: >> http://cr.openjdk.java.net/~eosterlund/8244416/webrev.00/ >> >> Thanks, >> /Erik From paul.sandoz at oracle.com Wed May 20 17:29:50 2020 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 20 May 2020 10:29:50 -0700 Subject: Update Re: RFR JDK-8223347 Integration of Vector API (Incubator): Java API, implementation, and tests In-Reply-To: <5bf2b471-36c0-f28b-b7c9-06ff5fded10c@oracle.com> References: <49F81016-55DF-4E3C-940A-A8DF8C712259@oracle.com> <9D074341-7234-4D9F-8E99-0E73F5BB5407@oracle.com> <5bf2b471-36c0-f28b-b7c9-06ff5fded10c@oracle.com> Message-ID: <6EB0C8F6-EB7F-4CBA-BDE1-4CE4F978D624@oracle.com> Thanks, tier3 it is! I am curious about how those tests can be co-opted for HS tiers. Can you share details on the panama-dev thread? Paul. diff -r a606409980d6 test/jdk/TEST.groups --- a/test/jdk/TEST.groups Fri May 15 17:23:27 2020 -0700 +++ b/test/jdk/TEST.groups Wed May 20 10:28:34 2020 -0700 @@ -72,6 +72,7 @@ tier3 = \ :build \ + :jdk_vector :jdk_rmi \ :jdk_beans \ :jdk_imageio \ @@ -338,6 +339,9 @@ jdk_foreign = \ java/foreign +jdk_vector = \ + jdk/incubator/vector + ############################# # > On May 19, 2020, at 5:06 PM, Ekaterina Pavlova wrote: > > As I wrote to openjdk alias tier3 seems to be more reasonable tier for incubating feature tests. > > Once the tests will be integrated we will also need to add these tests into hs-comp tiers to be tested with > additional compiler flags (like -Xcomp). > > -katya > > > On 5/19/20 3:25 PM, Paul Sandoz wrote: >> I just realized that the vector tests have not been included in any JDK test category e.g. tier1. >> Ordinarily I would expect the tests to be added to tier1 since the tests exercise intrinsics. However, those intrinsics are only enabled with the Vector API module so we could place in another tier to potentially reduce the cost of testing. >> Advice very much appreciate on which tier the tests should belong to. >> Paul. >>> On May 18, 2020, at 12:13 PM, Paul Sandoz wrote: >>> >>> HI, >>> >>> Here?s an update of the API and implementation webrevs based on (mostly) API feedback. We have removed the performance tests to make the review easier (those will be dealt with separately to integration as a follow on task). >>> >>> I think over the past year and recently via the CSR the API has had significant review. Reviews focusing on the Java implementation and tests would be greatly appreciated. >>> >>> It?s worth reiterating that the implementation and tests are quite formulaic, there is a lot of code that is generated from templates, so it's possible to focus on the template + subset (e.g. byte, long, float, and sizes of say 256, max). >>> >>> Notable changes from the prior webrev are: >>> >>> - Removal of the fromValues methods to construct a vector from a varargs array. Feedback indicated this was a misleading way to obtain a vector. >>> >>> - Unification of byte array and ByteBuffer load/store method signatures and unification of the implementations with more tests (including out-of-bounds tests for all the kinds of loads/stores). >>> >>> Paul. >>> >>> -- >>> >>> Latest javadoc >>> http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/docs-2020-05-15-88a83f7238d8/api/jdk.incubator.vector/jdk/incubator/vector/package-summary.html >>> >>> Latest specdiff >>> http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/specdiff-jdk-2020-05-15-88a83f7238d8/overview-summary.html >>> >>> Incremental specdiff >>> http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/specdiff-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/overview-summary.html >>> >>> >>> Latest implementation webrev >>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev-2020-05-16-default-2020-05-15-88a83f7238d8/webrev/ >>> >>> Incremental Implementation webrev >>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/webrev/ >>> >>> Latest test webrev >>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev-2020-05-16-default-2020-05-15-88a83f7238d8/webrev/ >>> >>> Incremental test webrev >>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/webrev/ >>> >>> >>>> On Apr 1, 2020, at 3:46 PM, Paul Sandoz wrote: >>>> >>>> Hi, >>>> >>>> A prior email sent out a request for review of the Vector API in preparation for JEP 338: Vector API (Incubator) [1] to be proposed for target: >>>> >>>> https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html >>>> >>>> This email expands the review of the API to the Java implementation and Java tests: >>>> >>>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev/webrev/ >>>> >>>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev/webrev/ >>>> >>>> (Further emails will sent for review of general HotSpot changes CPU architecture specific HotSpot changes, x64 and aarch64, and performance tests. For an early peek see the links in the description of JDK-8223347.) >>>> >>>> ? >>>> >>>> The Vector API provides >>>> >>>> - the public Vector class with vector operations common to all supported primitive types >>>> >>>> - public primitive specializations, such as IntVector, with common operations associated with the primitive type >>>> >>>> - internal concrete specializations for the vector size, such as Int256Vector, for a vector holding 8 ints. >>>> >>>> Operations generally defer to one of approximately 20 vector intrinsics. Some operations may be composed of other operations. >>>> >>>> Explicit casts are performed by vector operations to ensure vectors arguments are of the required shape (bit size) and element type. >>>> >>>> A vector intrinsic is an internal low-level vector operation. The last argument to the intrinsic is fall back behavior in Java, implementing the scalar operation over the number of elements held by the vector. Thus, If the intrinsic is not supported in C2 for the other arguments then the Java implementation is executed (the Java implementation is always executed when running in the interpreter or for C1). >>>> >>>> The public primitive specializations and the internal concrete implementations are generated from SSP template files, X-Vector.java.template and X-VectorBits.java.template respectively. >>>> >>>> Overall the implementation approach is quite formulaic and rather repetitive. Once you grok the pattern It should be easier to review if a little laborious. >>>> >>>> Unit tests are auto-generated by composing templates for each operation into an SSP template file which is then used to generate unit test files for each concrete vector class. The tests are quite extensive and have found many a HotSpot issue in development across a wide range of platforms. >>>> >>>> Paul. >>>> >>>> [1] https://bugs.openjdk.java.net/browse/JDK-8201271 >>>> >>> > From sgehwolf at redhat.com Wed May 20 19:29:49 2020 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Wed, 20 May 2020 21:29:49 +0200 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: References: <7201E4D5-96AA-47C1-B0B7-A19E8A0F198B@tencent.com> <51519874123ae26a4d3ebf4863d80e682c13b989.camel@redhat.com> Message-ID: Hi Matthias, On Wed, 2020-05-20 at 11:07 +0000, Baesken, Matthias wrote: > Hi Severin , > > > FWIW, parts of this work-around got added by JDK-8236617. Looking at > > the bug it was observed that: > > > > containerMetrics.getMemoryAndSwapLimit() == 0 and > > containerMetrics.getMemoryLimit() returned a large value. > > > > So then with JDK-8231111 I suspect > > containerMetrics.getMemoryAndSwapLimit() now returns -1 and, thus, > > getTotalSwapSpaceSize0() is being used instead of the container values. > > I added some tracing output to OperatingSystemImpl.getTotalSwapSpaceSize() and get these values : > > containerMetrics.getMemoryAndSwapLimit():-1 containerMetrics.getMemoryLimit():-1 > ****** getTotalSwapSpaceSize0() returned 16106061824 Thanks for this. This seems to confirm my suspicion. So my working theory here is that the system where you are observing the issue now is similar to the one when JDK-8236617 was discovered. For some reason containerMetrics.getMemoryAndSwapLimit() returned 0 for the case when the necessary files were missing (which was wrong IMO). After JDK-8231111 there is a clear distinction for this case: -1 is being returned. That makes more sense to me, as there wouldn't be any container limit to take into account. Your trace output seems to confirm that: containerMetrics.getMemoryAndSwapLimit():-1 containerMetrics.getMemoryLimit():-1 > > > So far the theory, I'd like to gather more data before we come up with > > another patch, though. In JDK-8236617 it wasn't investigated why > > containerMetrics.getMemoryAndSwapLimit() returned 0 on such a system. > > It would be good to know why. Could it be that file > > 'memory.memsw.limit_in_bytes' doesn't exist on such a system? > > True, it does not exist : > > ls -alL /sys/fs/cgroup/memory/ > total 0 > drwxr-xr-x 3 root root 0 Mar 16 11:00 . > drwxr-xr-x 11 root root 260 Mar 16 11:00 .. > -rw-r--r-- 1 root root 0 Mar 16 11:00 cgroup.clone_children > --w--w--w- 1 root root 0 Mar 16 11:00 cgroup.event_control > -rw-r--r-- 1 root root 0 May 20 13:00 cgroup.procs > -r--r--r-- 1 root root 0 Mar 16 11:00 cgroup.sane_behavior > drwxr-xr-x 2 root root 0 May 20 12:52 docker > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.failcnt > --w------- 1 root root 0 Mar 16 11:00 memory.force_empty > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.limit_in_bytes > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.low_limit_in_bytes > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.max_usage_in_bytes > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.move_charge_at_immigrate > -r--r--r-- 1 root root 0 Mar 16 11:00 memory.numa_stat > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.oom_control > ---------- 1 root root 0 Mar 16 11:00 memory.pressure_level > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.soft_limit_in_bytes > -r--r--r-- 1 root root 0 Mar 16 11:00 memory.stat > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.swappiness > -r--r--r-- 1 root root 0 Mar 16 11:00 memory.usage_in_bytes > -rw-r--r-- 1 root root 0 May 20 13:00 memory.use_hierarchy > -rw-r--r-- 1 root root 0 Mar 16 11:00 notify_on_release > -rw-r--r-- 1 root root 0 Mar 16 11:00 release_agent > -rw-r--r-- 1 root root 0 Mar 16 11:00 tasks Hmm, is this inside a container on this system with --memory and -- memory-swap being specified on the docker command line? Anyway, I think we should change the test code similar to this: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8244500/01/webrev/ If this works for you, then that's what I'd suggest as a fix. Note that I haven't tested this patch. I need to recreate my set-up for proper docker testing here :-/ Thanks, Severin > Best regards, Matthias > > > > -----Original Message----- > From: Severin Gehwolf > Sent: Montag, 18. Mai 2020 12:09 > To: jiefu(??) ; David Holmes ; Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' > Cc: Bob Vandette > Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > Hi, > > On Mon, 2020-05-18 at 06:15 +0000, jiefu(??) wrote: > > Hi David, > > > > This bug should be introduced by JDK-8242480. > > I'm sorry for that. > > > > And I think the logic in src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java should be fixed. > > > > In my fix for JDK-8242480, I had ignored the case for kernels which don't support "--memory-swap 150M". > > In that case, containerMetrics.getMemoryAndSwapLimit()[1] will return -1. > > It appears so that containerMetrics.getMemoryAndSwapLimit() returned 0 > prior JDK-8231111 and -1 after. I'm not convinced JDK-8242480 > introduced this. > > > I didn't notice this bug because all our kernels support --memory-swap. > > Fortunately, Matthias found this bug and will fix it. > > I'm still not sure what's happening in Matthias' case on that system. > It would be best to first try to understand the actual problem better. > The info I have is: > > - It's a cgroup v1 system > - The system now returns large values for > OperatingSystemMXBean.getTotalSwapSpaceSize() and > OperatingSystemMXBean.getFreeSwapSpaceSize() > > It would be good to know what the actual return values are for the > container metrics on this system with swap limit disabled at kernel > level. And also whether or not relevant cgroup files exist on such a > system (within docker). > > FWIW, parts of this work-around got added by JDK-8236617. Looking at > the bug it was observed that: > > containerMetrics.getMemoryAndSwapLimit() == 0 and > containerMetrics.getMemoryLimit() returned a large value. > > So then with JDK-8231111 I suspect > containerMetrics.getMemoryAndSwapLimit() now returns -1 and, thus, > getTotalSwapSpaceSize0() is being used instead of the container values. > > So far the theory, I'd like to gather more data before we come up with > another patch, though. In JDK-8236617 it wasn't investigated why > containerMetrics.getMemoryAndSwapLimit() returned 0 on such a system. > It would be good to know why. Could it be that file > 'memory.memsw.limit_in_bytes' doesn't exist on such a system? > > Unfortunately, I don't know what config one needs in order to reproduce > so we'll have to rely on Matthias to gather the info. > > Thanks, > Severin > > > Thanks a lot. > > Best regards, > > Jie > > > > [1] http://hg.openjdk.java.net/jdk/jdk/file/a95911422005/src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java#l73 > > > > ?On 2020/5/18, 1:48 PM, "David Holmes" wrote: > > > > Hi Matthias, > > > > On 16/05/2020 12:59 am, Baesken, Matthias wrote: > > > Hello David , I get the warning always on this linux machine (I think it depends on machine setup and/or kernel version). > > > > > >> Surely it must be specified to return 0 or some such sentinel in such a case? > > > > > > The doc just says this : > > > > > > 62 /** > > > 63 * Returns the total amount of swap space in bytes. > > > 64 * > > > 65 * @return the total amount of swap space in bytes. > > > 66 */ > > > 67 public long getTotalSwapSpaceSize(); > > > > > > ( however it returned previously 0 before the latest changes in the container support code came in ). > > > > That sounds like a bug was introduced to me then. > > > > > Regarding the getFreeSwapSpaceSize issue , Jie provided a change to src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java > > > Because of this I could remove the try-construct . > > > > > > New webrev : > > > http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.3/ > > > > I'm quite confused by your changes. Your first version simply adjusted > > the test to check for the warning. But now you are also changing the > > main logic. I can't assess the validity of what you are doing sorry. > > > > I've cc'd Severin and Bob as I think they need to evaluate what the real > > problem is here and when it was introduced. > > > > David > > ---- > > > > > Best regards, Matthias > > > > > > > > > > > > -----Original Message----- > > > From: David Holmes > > > Sent: Donnerstag, 14. Mai 2020 07:02 > > > To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' > > > Cc: jiefu(??) > > > Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > > > > > On 14/05/2020 12:28 am, Baesken, Matthias wrote: > > >> Hello, here is a new webrev removing the comment : > > >> > > >> http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.1/ > > >> > > >> It removes the comment , and it also handles 0 for getFreeSwapSpaceSize > > >> ( this can occur after JDK-8242480 : Negative value may be returned by getFreeSwapSpaceSize() in the docker was pushed recently ). > > > > > > Sorry Matthias but I still see this partly as a bug in the underlying > > > container support code. If you get the warning then what does > > > getTotalSwapSpaceSize() return? Surely it must be specified to return 0 > > > or some such sentinel in such a case? > > > > > > Other comments: > > > > > > ! try { > > > ! > > > out.shouldMatch("OperatingSystemMXBean\\.getFreeSwapSpaceSize: > > > [1-9][0-9]+"); > > > ! } catch(RuntimeException ex) { // allow 0, see 8242480 > > > ! > > > out.shouldContain("OperatingSystemMXBean.getFreeSwapSpaceSize: 0"); > > > ! } > > > > > > can't you just change the original regex to match >= 0? > > > > > > Style nit: avoid historical comments like "see 8242480" and "since > > > 8231111 ...". > > > > > > Thanks, > > > David > > > > > >> Best regards, Matthias > > >> > > >> -----Original Message----- > > >> From: Baesken, Matthias > > >> Sent: Mittwoch, 13. Mai 2020 10:11 > > >> To: 'David Holmes' ; 'hotspot-dev at openjdk.java.net' > > >> Subject: RE: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > >> > > >> > > >>> But according to the comment immediately before your changes: > > >>> > > >>> // in case of warnings like : "Your kernel does not support swap limit > > >>> capabilities or the cgroup is not mounted. Memory limited without swap." > > >>> // the getTotalSwapSpaceSize does not return the expected result, but 0 > > >>> > > >>> we should be returning zero. So this seems to be a bug in the > > >>> implementation not in the test. > > >> > > >> > > >> Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values > > >> that are normally expected. > > >> Previously we indeed got "0" . > > >> Probably the comment should be deleted . > > >> > > >> Best regards, Matthias > > >> > > > > > > From bob.vandette at oracle.com Wed May 20 20:16:32 2020 From: bob.vandette at oracle.com (Bob Vandette) Date: Wed, 20 May 2020 16:16:32 -0400 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: References: <7201E4D5-96AA-47C1-B0B7-A19E8A0F198B@tencent.com> <51519874123ae26a4d3ebf4863d80e682c13b989.camel@redhat.com> Message-ID: What do you think about fixing getTotalSwapSpaceSze and getFreeSwapSpaceSize to return 0L if either of the underlying Metrics used to calculate the result return -1. Of course we only do this if containerMetrics != null. Failure to read these values, means we have no swap. If we have no swap we also have no free swap. 0 seems like an appropriate value to represent this state. It seems as though some kernels report 0 when kernel swap limit is not enabled while others don?t populate the memory status files. Bob. > On May 20, 2020, at 3:29 PM, Severin Gehwolf wrote: > > Hi Matthias, > > On Wed, 2020-05-20 at 11:07 +0000, Baesken, Matthias wrote: >> Hi Severin , >> >>> FWIW, parts of this work-around got added by JDK-8236617. Looking at >>> the bug it was observed that: >>> >>> containerMetrics.getMemoryAndSwapLimit() == 0 and >>> containerMetrics.getMemoryLimit() returned a large value. >>> >>> So then with JDK-8231111 I suspect >>> containerMetrics.getMemoryAndSwapLimit() now returns -1 and, thus, >>> getTotalSwapSpaceSize0() is being used instead of the container values. >> >> I added some tracing output to OperatingSystemImpl.getTotalSwapSpaceSize() and get these values : >> >> containerMetrics.getMemoryAndSwapLimit():-1 containerMetrics.getMemoryLimit():-1 >> ****** getTotalSwapSpaceSize0() returned 16106061824 > > Thanks for this. This seems to confirm my suspicion. > > So my working theory here is that the system where you are observing > the issue now is similar to the one when JDK-8236617 was discovered. > For some reason containerMetrics.getMemoryAndSwapLimit() returned 0 for > the case when the necessary files were missing (which was wrong IMO). > After JDK-8231111 there is a clear distinction for this case: -1 is > being returned. That makes more sense to me, as there wouldn't be any > container limit to take into account. > > Your trace output seems to confirm that: > > containerMetrics.getMemoryAndSwapLimit():-1 > containerMetrics.getMemoryLimit():-1 > >> >>> So far the theory, I'd like to gather more data before we come up with >>> another patch, though. In JDK-8236617 it wasn't investigated why >>> containerMetrics.getMemoryAndSwapLimit() returned 0 on such a system. >>> It would be good to know why. Could it be that file >>> 'memory.memsw.limit_in_bytes' doesn't exist on such a system? >> >> True, it does not exist : >> >> ls -alL /sys/fs/cgroup/memory/ >> total 0 >> drwxr-xr-x 3 root root 0 Mar 16 11:00 . >> drwxr-xr-x 11 root root 260 Mar 16 11:00 .. >> -rw-r--r-- 1 root root 0 Mar 16 11:00 cgroup.clone_children >> --w--w--w- 1 root root 0 Mar 16 11:00 cgroup.event_control >> -rw-r--r-- 1 root root 0 May 20 13:00 cgroup.procs >> -r--r--r-- 1 root root 0 Mar 16 11:00 cgroup.sane_behavior >> drwxr-xr-x 2 root root 0 May 20 12:52 docker >> -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.failcnt >> --w------- 1 root root 0 Mar 16 11:00 memory.force_empty >> -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.limit_in_bytes >> -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.low_limit_in_bytes >> -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.max_usage_in_bytes >> -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.move_charge_at_immigrate >> -r--r--r-- 1 root root 0 Mar 16 11:00 memory.numa_stat >> -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.oom_control >> ---------- 1 root root 0 Mar 16 11:00 memory.pressure_level >> -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.soft_limit_in_bytes >> -r--r--r-- 1 root root 0 Mar 16 11:00 memory.stat >> -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.swappiness >> -r--r--r-- 1 root root 0 Mar 16 11:00 memory.usage_in_bytes >> -rw-r--r-- 1 root root 0 May 20 13:00 memory.use_hierarchy >> -rw-r--r-- 1 root root 0 Mar 16 11:00 notify_on_release >> -rw-r--r-- 1 root root 0 Mar 16 11:00 release_agent >> -rw-r--r-- 1 root root 0 Mar 16 11:00 tasks > > Hmm, is this inside a container on this system with --memory and -- > memory-swap being specified on the docker command line? > > Anyway, I think we should change the test code similar to this: > http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8244500/01/webrev/ > > If this works for you, then that's what I'd suggest as a fix. > > Note that I haven't tested this patch. I need to recreate my set-up for > proper docker testing here :-/ > > Thanks, > Severin > >> Best regards, Matthias >> >> >> >> -----Original Message----- >> From: Severin Gehwolf >> Sent: Montag, 18. Mai 2020 12:09 >> To: jiefu(??) ; David Holmes ; Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' >> Cc: Bob Vandette >> Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java >> >> Hi, >> >> On Mon, 2020-05-18 at 06:15 +0000, jiefu(??) wrote: >>> Hi David, >>> >>> This bug should be introduced by JDK-8242480. >>> I'm sorry for that. >>> >>> And I think the logic in src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java should be fixed. >>> >>> In my fix for JDK-8242480, I had ignored the case for kernels which don't support "--memory-swap 150M". >>> In that case, containerMetrics.getMemoryAndSwapLimit()[1] will return -1. >> >> It appears so that containerMetrics.getMemoryAndSwapLimit() returned 0 >> prior JDK-8231111 and -1 after. I'm not convinced JDK-8242480 >> introduced this. >> >>> I didn't notice this bug because all our kernels support --memory-swap. >>> Fortunately, Matthias found this bug and will fix it. >> >> I'm still not sure what's happening in Matthias' case on that system. >> It would be best to first try to understand the actual problem better. >> The info I have is: >> >> - It's a cgroup v1 system >> - The system now returns large values for >> OperatingSystemMXBean.getTotalSwapSpaceSize() and >> OperatingSystemMXBean.getFreeSwapSpaceSize() >> >> It would be good to know what the actual return values are for the >> container metrics on this system with swap limit disabled at kernel >> level. And also whether or not relevant cgroup files exist on such a >> system (within docker). >> >> FWIW, parts of this work-around got added by JDK-8236617. Looking at >> the bug it was observed that: >> >> containerMetrics.getMemoryAndSwapLimit() == 0 and >> containerMetrics.getMemoryLimit() returned a large value. >> >> So then with JDK-8231111 I suspect >> containerMetrics.getMemoryAndSwapLimit() now returns -1 and, thus, >> getTotalSwapSpaceSize0() is being used instead of the container values. >> >> So far the theory, I'd like to gather more data before we come up with >> another patch, though. In JDK-8236617 it wasn't investigated why >> containerMetrics.getMemoryAndSwapLimit() returned 0 on such a system. >> It would be good to know why. Could it be that file >> 'memory.memsw.limit_in_bytes' doesn't exist on such a system? >> >> Unfortunately, I don't know what config one needs in order to reproduce >> so we'll have to rely on Matthias to gather the info. >> >> Thanks, >> Severin >> >>> Thanks a lot. >>> Best regards, >>> Jie >>> >>> [1] http://hg.openjdk.java.net/jdk/jdk/file/a95911422005/src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java#l73 >>> >>> ?On 2020/5/18, 1:48 PM, "David Holmes" wrote: >>> >>> Hi Matthias, >>> >>> On 16/05/2020 12:59 am, Baesken, Matthias wrote: >>>> Hello David , I get the warning always on this linux machine (I think it depends on machine setup and/or kernel version). >>>> >>>>> Surely it must be specified to return 0 or some such sentinel in such a case? >>>> >>>> The doc just says this : >>>> >>>> 62 /** >>>> 63 * Returns the total amount of swap space in bytes. >>>> 64 * >>>> 65 * @return the total amount of swap space in bytes. >>>> 66 */ >>>> 67 public long getTotalSwapSpaceSize(); >>>> >>>> ( however it returned previously 0 before the latest changes in the container support code came in ). >>> >>> That sounds like a bug was introduced to me then. >>> >>>> Regarding the getFreeSwapSpaceSize issue , Jie provided a change to src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java >>>> Because of this I could remove the try-construct . >>>> >>>> New webrev : >>>> http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.3/ >>> >>> I'm quite confused by your changes. Your first version simply adjusted >>> the test to check for the warning. But now you are also changing the >>> main logic. I can't assess the validity of what you are doing sorry. >>> >>> I've cc'd Severin and Bob as I think they need to evaluate what the real >>> problem is here and when it was introduced. >>> >>> David >>> ---- >>> >>>> Best regards, Matthias >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: David Holmes >>>> Sent: Donnerstag, 14. Mai 2020 07:02 >>>> To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' >>>> Cc: jiefu(??) >>>> Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java >>>> >>>> On 14/05/2020 12:28 am, Baesken, Matthias wrote: >>>>> Hello, here is a new webrev removing the comment : >>>>> >>>>> http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.1/ >>>>> >>>>> It removes the comment , and it also handles 0 for getFreeSwapSpaceSize >>>>> ( this can occur after JDK-8242480 : Negative value may be returned by getFreeSwapSpaceSize() in the docker was pushed recently ). >>>> >>>> Sorry Matthias but I still see this partly as a bug in the underlying >>>> container support code. If you get the warning then what does >>>> getTotalSwapSpaceSize() return? Surely it must be specified to return 0 >>>> or some such sentinel in such a case? >>>> >>>> Other comments: >>>> >>>> ! try { >>>> ! >>>> out.shouldMatch("OperatingSystemMXBean\\.getFreeSwapSpaceSize: >>>> [1-9][0-9]+"); >>>> ! } catch(RuntimeException ex) { // allow 0, see 8242480 >>>> ! >>>> out.shouldContain("OperatingSystemMXBean.getFreeSwapSpaceSize: 0"); >>>> ! } >>>> >>>> can't you just change the original regex to match >= 0? >>>> >>>> Style nit: avoid historical comments like "see 8242480" and "since >>>> 8231111 ...". >>>> >>>> Thanks, >>>> David >>>> >>>>> Best regards, Matthias >>>>> >>>>> -----Original Message----- >>>>> From: Baesken, Matthias >>>>> Sent: Mittwoch, 13. Mai 2020 10:11 >>>>> To: 'David Holmes' ; 'hotspot-dev at openjdk.java.net' >>>>> Subject: RE: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java >>>>> >>>>> >>>>>> But according to the comment immediately before your changes: >>>>>> >>>>>> // in case of warnings like : "Your kernel does not support swap limit >>>>>> capabilities or the cgroup is not mounted. Memory limited without swap." >>>>>> // the getTotalSwapSpaceSize does not return the expected result, but 0 >>>>>> >>>>>> we should be returning zero. So this seems to be a bug in the >>>>>> implementation not in the test. >>>>> >>>>> >>>>> Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values >>>>> that are normally expected. >>>>> Previously we indeed got "0" . >>>>> Probably the comment should be deleted . >>>>> >>>>> Best regards, Matthias >>>>> >>> >>> >>> > From ekaterina.pavlova at oracle.com Wed May 20 22:55:13 2020 From: ekaterina.pavlova at oracle.com (Ekaterina Pavlova) Date: Wed, 20 May 2020 15:55:13 -0700 Subject: Update Re: RFR JDK-8223347 Integration of Vector API (Incubator): Java API, implementation, and tests In-Reply-To: <6EB0C8F6-EB7F-4CBA-BDE1-4CE4F978D624@oracle.com> References: <49F81016-55DF-4E3C-940A-A8DF8C712259@oracle.com> <9D074341-7234-4D9F-8E99-0E73F5BB5407@oracle.com> <5bf2b471-36c0-f28b-b7c9-06ff5fded10c@oracle.com> <6EB0C8F6-EB7F-4CBA-BDE1-4CE4F978D624@oracle.com> Message-ID: On 5/20/20 10:29 AM, Paul Sandoz wrote: > Thanks, tier3 it is! Great, now Vector API tests could be automatically run with default HS flags as part of tier3 testing in mach5. > I am curious about how those tests can be co-opted for HS tiers. Can you share details on the panama-dev thread? I would suggest just to add open/test/jdk/jdk/incubator/vector tests into corresponding HS compiler testing tiers (mach5 tasks) which do run tests with -Xcomp and other compiler flags. I can take care of this once Vector API is integrated. -katya > Paul. > > diff -r a606409980d6 test/jdk/TEST.groups > --- a/test/jdk/TEST.groups Fri May 15 17:23:27 2020 -0700 > +++ b/test/jdk/TEST.groups Wed May 20 10:28:34 2020 -0700 > @@ -72,6 +72,7 @@ > > tier3 = \ > :build \ > + :jdk_vector > :jdk_rmi \ > :jdk_beans \ > :jdk_imageio \ > @@ -338,6 +339,9 @@ > jdk_foreign = \ > java/foreign > > +jdk_vector = \ > + jdk/incubator/vector > + > ############################# > > # > > >> On May 19, 2020, at 5:06 PM, Ekaterina Pavlova wrote: >> >> As I wrote to openjdk alias tier3 seems to be more reasonable tier for incubating feature tests. >> >> Once the tests will be integrated we will also need to add these tests into hs-comp tiers to be tested with >> additional compiler flags (like -Xcomp). >> >> -katya >> >> >> On 5/19/20 3:25 PM, Paul Sandoz wrote: >>> I just realized that the vector tests have not been included in any JDK test category e.g. tier1. >>> Ordinarily I would expect the tests to be added to tier1 since the tests exercise intrinsics. However, those intrinsics are only enabled with the Vector API module so we could place in another tier to potentially reduce the cost of testing. >>> Advice very much appreciate on which tier the tests should belong to. >>> Paul. >>>> On May 18, 2020, at 12:13 PM, Paul Sandoz wrote: >>>> >>>> HI, >>>> >>>> Here?s an update of the API and implementation webrevs based on (mostly) API feedback. We have removed the performance tests to make the review easier (those will be dealt with separately to integration as a follow on task). >>>> >>>> I think over the past year and recently via the CSR the API has had significant review. Reviews focusing on the Java implementation and tests would be greatly appreciated. >>>> >>>> It?s worth reiterating that the implementation and tests are quite formulaic, there is a lot of code that is generated from templates, so it's possible to focus on the template + subset (e.g. byte, long, float, and sizes of say 256, max). >>>> >>>> Notable changes from the prior webrev are: >>>> >>>> - Removal of the fromValues methods to construct a vector from a varargs array. Feedback indicated this was a misleading way to obtain a vector. >>>> >>>> - Unification of byte array and ByteBuffer load/store method signatures and unification of the implementations with more tests (including out-of-bounds tests for all the kinds of loads/stores). >>>> >>>> Paul. >>>> >>>> -- >>>> >>>> Latest javadoc >>>> http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/docs-2020-05-15-88a83f7238d8/api/jdk.incubator.vector/jdk/incubator/vector/package-summary.html >>>> >>>> Latest specdiff >>>> http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/specdiff-jdk-2020-05-15-88a83f7238d8/overview-summary.html >>>> >>>> Incremental specdiff >>>> http://cr.openjdk.java.net/~psandoz/panama/vector-api-review/specdiff-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/overview-summary.html >>>> >>>> >>>> Latest implementation webrev >>>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev-2020-05-16-default-2020-05-15-88a83f7238d8/webrev/ >>>> >>>> Incremental Implementation webrev >>>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/webrev/ >>>> >>>> Latest test webrev >>>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev-2020-05-16-default-2020-05-15-88a83f7238d8/webrev/ >>>> >>>> Incremental test webrev >>>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev-2020-05-11-38dd763d023e-2020-05-15-88a83f7238d8/webrev/ >>>> >>>> >>>>> On Apr 1, 2020, at 3:46 PM, Paul Sandoz wrote: >>>>> >>>>> Hi, >>>>> >>>>> A prior email sent out a request for review of the Vector API in preparation for JEP 338: Vector API (Incubator) [1] to be proposed for target: >>>>> >>>>> https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-March/065345.html >>>>> >>>>> This email expands the review of the API to the Java implementation and Java tests: >>>>> >>>>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_src_webrev/webrev/ >>>>> >>>>> http://cr.openjdk.java.net/~psandoz/panama/JDK-8223347-vector-api-integration-java/jdk_test_webrev/webrev/ >>>>> >>>>> (Further emails will sent for review of general HotSpot changes CPU architecture specific HotSpot changes, x64 and aarch64, and performance tests. For an early peek see the links in the description of JDK-8223347.) >>>>> >>>>> ? >>>>> >>>>> The Vector API provides >>>>> >>>>> - the public Vector class with vector operations common to all supported primitive types >>>>> >>>>> - public primitive specializations, such as IntVector, with common operations associated with the primitive type >>>>> >>>>> - internal concrete specializations for the vector size, such as Int256Vector, for a vector holding 8 ints. >>>>> >>>>> Operations generally defer to one of approximately 20 vector intrinsics. Some operations may be composed of other operations. >>>>> >>>>> Explicit casts are performed by vector operations to ensure vectors arguments are of the required shape (bit size) and element type. >>>>> >>>>> A vector intrinsic is an internal low-level vector operation. The last argument to the intrinsic is fall back behavior in Java, implementing the scalar operation over the number of elements held by the vector. Thus, If the intrinsic is not supported in C2 for the other arguments then the Java implementation is executed (the Java implementation is always executed when running in the interpreter or for C1). >>>>> >>>>> The public primitive specializations and the internal concrete implementations are generated from SSP template files, X-Vector.java.template and X-VectorBits.java.template respectively. >>>>> >>>>> Overall the implementation approach is quite formulaic and rather repetitive. Once you grok the pattern It should be easier to review if a little laborious. >>>>> >>>>> Unit tests are auto-generated by composing templates for each operation into an SSP template file which is then used to generate unit test files for each concrete vector class. The tests are quite extensive and have found many a HotSpot issue in development across a wide range of platforms. >>>>> >>>>> Paul. >>>>> >>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8201271 >>>>> >>>> >> > From thomas.stuefe at gmail.com Thu May 21 06:26:31 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 21 May 2020 08:26:31 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: <8bf736b3-9ab1-86b7-fd22-230e449b1b9f@oracle.com> References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> <8bf736b3-9ab1-86b7-fd22-230e449b1b9f@oracle.com> Message-ID: Hi all, tests went through successfully at SAP two nights in a row. Ioi gave his okay, and I would like to push today or tomorrow. Any remaining objections`? The current, final version is: full: http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ delta: http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ Thank you, Thomas On Wed, May 20, 2020 at 1:43 AM Ioi Lam wrote: > Hi Thomas, > > The testing took a little longer than I expected. I've run your previous > version: > > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ > > In our CI tiers 1/4. No failures. > > I didn't run your latest version. I would suggesting testing it with your > own CI as well as in jdk-submit before pushing (after you get OKs from > other reviewers). > > Thumbs up. > - Ioi > > > On 5/19/20 7:40 AM, Thomas St?fe wrote: > > Hi guys, > > new webrev: > > full: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ > delta: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ > > Mostly format changes, breaking up at column 80 as Andrew requested. > > The only changes which matter is the removal of the unnecessary include in > arguments.cpp as Coleen suggested; and to not validate SharedBaseAddress > anymore at runtime, as Ioi requested. > > Also rebased to head. > > Thanks, Thomas > > > > > On Tue, May 19, 2020 at 6:38 AM Ioi Lam wrote: > >> >> >> On 5/18/20 7:37 AM, Thomas St?fe wrote: >> >> Hi all, >> >> fourth webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ >> >> I rebased atop of >> >> changeset: 59325:1ba9a9a3f948 >> summary: 8244433: Remove saving of RSP in Assembler::pusha_uncached() >> >> so, atop of "8241825: Make compressed oops and compressed class pointers >> independent (x86_64, PPC, S390)". >> >> The changes are very small this time, see notes below: >> >> @Ioi: >> >> >Hi Thomas, >> >> >I am running your patch in out CI pipeline now. Some comments: >> >> >[2] Does JDK-8243535 exist with the current jdk/jdk repo? >> >> I assigned JDK-8243535 to me and I'm working on a patch. But I would like >> to keep that separate from this work, since this patch is already large. >> And I would like to fix JDK-8243535 after this remodel patch has been >> pushed, which makes some minor things easier. That would leave a small time >> window where NMT does mis-track class space as shared class space, but that >> is not a big issue, it does not even disturb any test. Which in itself is >> something we may want to fix, come to think about it. >> >> >> > [3] I think this can be put in header file. >> > >> > bool Metaspace::class_space_is_initialized() { >> > return _class_space_list != NULL; >> > } >> >> Done. >> >> > [7] argument.cpp checks this: >> > >> > if (UseSharedSpaces || DumpSharedSpaces) { >> > if >> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >> > log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for >> this platform, option will be ignored.", >> > p2i((address)SharedBaseAddress)); >> > FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >> > } >> > } >> > >> > but initialize_dumptime_shared_and_meta_spaces aligns up the >> SharedBaseAddress before doing the assert. >> > >> > void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >> > assert(DumpSharedSpaces, "should be called for dump time only"); >> > >> > const size_t reserve_alignment = >> MetaspaceShared::reserved_space_alignment(); >> > char* shared_base = (char*)align_up((char*)SharedBaseAddress, >> reserve_alignment); >> > >> > #ifdef _LP64 >> > >> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >> "Sanity"); >> > >> > So theoretically shared_base may no longer be is_valid_base after the >> align_up. It probably won't happen, but I think the code will be > much >> clearer tofirst to align_up, then check for is_valid_base and reset to >> default, and finally assert. >> >> The problem with that is that at the time this coding runs os::init() did >> not yet run and we do not know yet allocation granularity. >> >> So I changed verification of SharedBaseAddress: I added the alignment as >> you suggested and moved it to a later time in VM initialization, to CDS >> initialization. >> >> >> Is the call to check_SharedBaseAddress() necessary inside >> MetaspaceShared::initialize_runtime_shared_and_meta_spaces()? >> SharedBaseAddress is currently ignored at run time. Does your patch change >> that behavior? >> >> >[8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >> following block can be simplified: >> > >> > if (!_shared_rs.is_reserved()) { >> > // Get a reserved space anywhere if attaching at the >> SharedBaseAddress fails: >> > if (UseCompressedClassPointers) { >> > // If we need a compressed class space too, let the platform >> handle the reservation. >> > _shared_rs = >> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >> > } else { >> > // anywhere is fine. >> > _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >> large */, (char*)NULL); >> > } >> > } >> > >> > ... if you change the declaration to: >> > >> > static ReservedSpace >> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >> ReservedSpace();}); >> >> I'd like to avoid keep the method LP64 only, but I rewrote the section >> somewhat to be more clearer. >> >> The new code looks good. >> >> > [9] I think the #ifdef _LP64 is not necessary: >> > >> > #ifdef _LP64 >> > // Some sanity checks after reserving address spaces for archives >> and class space. >> > assert(archive_space_rs.is_reserved(), "Sanity"); >> > if (Metaspace::using_class_space()) { >> > // Class space must closely follow the archive space. Both spaces >> must be aligned correctly. >> > assert(class_space_rs.is_reserved(), "A class space should have >> been reserved"); >> > assert(class_space_rs.base() >= archive_space_rs.end(), "class >> space should follow the cds archive space"); >> > assert(is_aligned(archive_space_rs.base(), >> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >> > assert(is_aligned(class_space_rs.base(), >> Metaspace::reserve_alignment()), "class space misaligned"); >> > } >> > #endif >> >> I removed the #ifdef, but replaced it with an #ifdef ASSERT. Which is >> also not needed, technically, but makes it clear that this section is >> debugging checks only. >> >> >> OK. >> >> The rest of the code looks good to me. I am going to run your latest >> patch in our CI and will report the results. >> >> Thanks >> - Ioi >> >> ---- >> >> @Coleen: >> >> > >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/ >> >virtualSpaceList.cpp.udiff.html >> > >> >Why did you make this change? Shouldn't the caller align it? >> >> >> You are right, I removed the assert. >> >> --- >> >> Thanks all for your review work, >> >> ..Thomas >> >> >> >> >> >> >> >> >> >> >> >> >> On Fri, May 8, 2020 at 7:37 AM Ioi Lam wrote: >> >>> Hi Thomas, >>> >>> I am running your patch in out CI pipeline now. Some comments: >>> >>> [1] Do we still need to use _class_space_list? It looks like we just >>> have a single range. Is _class_space_list->current_virtual_space() always >>> the same VS? >>> >>> if (_class_space_list != NULL) { >>> address base = >>> (address)_class_space_list->current_virtual_space()->bottom(); >>> address top = base + compressed_class_space_size(); >>> st->print("Compressed class space mapped at: " PTR_FORMAT "-" >>> PTR_FORMAT ", size: " SIZE_FORMAT, >>> p2i(base), p2i(top), top - base); >>> >>> [2] Does JDK-8243535 exist with the current jdk/jdk repo? >>> >>> // This does currently not work because rs may be the result of a >>> split operation >>> // and NMT seems not to be able to handle splits. >>> // See JDK-8243535. >>> // MemTracker::record_virtual_memory_type((address)rs.base(), mtClass); >>> >>> >>> [3] I think this can be put in header file. >>> >>> bool Metaspace::class_space_is_initialized() { >>> return _class_space_list != NULL; >>> } >>> >>> [4] Why does the CCS need to follow UseLargePagesInMetaspace? This is >>> the reason for the >>> gap in the following chart: >>> >>> // We do this by reserving space for the ccs behind the archives. >>> Note however that >>> // ccs follows a different alignment >>> (Metaspace::reserve_alignment), so there may >>> // be a gap between ccs and cds. >>> // We use a similar layout at runtime, see >>> reserve_address_space_for_archives(). >>> // >>> // +-- SharedBaseAddress (default = >>> 0x800000000) >>> // v >>> // +-..---------+---------+ ... >>> +----+----+----+--------+--------------------+ >>> // | Heap | Archive | | MC | RW | RO | [gap] | class >>> space | >>> // +-..---------+---------+ ... >>> +----+----+----+--------+--------------------+ >>> // |<-- MaxHeapSize -->| |<-- UnscaledClassSpaceMax = 4GB -->| >>> >>> >>> _reserve_alignment is determined here: >>> >>> void Metaspace::ergo_initialize() { >>> if (DumpSharedSpaces) { >>> // Using large pages when dumping the shared archive is currently >>> not implemented. >>> FLAG_SET_ERGO(UseLargePagesInMetaspace, false); >>> } >>> >>> size_t page_size = os::vm_page_size(); >>> if (UseLargePages && UseLargePagesInMetaspace) { >>> page_size = os::large_page_size(); >>> } >>> >>> _commit_alignment = page_size; >>> _reserve_alignment = MAX2(page_size, >>> (size_t)os::vm_allocation_granularity()); >>> >>> But when CDS is enabled, the RS is reserved without large pages, so it >>> should be possible to incrementally commit the CCS using just >>> os::vm_allocation_granularity(). >>> >>> [5] I see the aarch64 code in >>> Metaspace::reserve_address_space_for_compressed_classes has been changed. >>> Is this necessary for this RFE, or is it a separate improvement that can be >>> done in a different RFE? >>> >>> [6] For AARCH64, should we skip the part that reserves it right above >>> the heap? Or does AARCH64 always allocate the heap such that it's a >>> preferable address? >>> >>> // case (b) >>> ReservedSpace rs; >>> >>> // Try right above the Java heap... >>> address base = align_up(CompressedOops::end(), >>> Metaspace::reserve_alignment()); >>> assert(base != NULL, "Sanity"); >>> >>> const size_t size = align_up(CompressedClassSpaceSize, >>> Metaspace::reserve_alignment()); >>> if (CompressedKlassPointers::is_valid_base(base)) { >>> rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* >>> large */, (char*)base); >>> } >>> >>> // ...failing that, reserve anywhere, but let platform do optimized >>> placement: >>> if (!rs.is_reserved()) { >>> rs = Metaspace::reserve_address_space_for_compressed_classes(size); >>> } >>> >>> [7] argument.cpp checks this: >>> >>> if (UseSharedSpaces || DumpSharedSpaces) { >>> if >>> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >>> log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for >>> this platform, option will be ignored.", >>> p2i((address)SharedBaseAddress)); >>> FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >>> } >>> } >>> >>> but initialize_dumptime_shared_and_meta_spaces aligns up the >>> SharedBaseAddress before doing the assert. >>> >>> void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >>> assert(DumpSharedSpaces, "should be called for dump time only"); >>> >>> const size_t reserve_alignment = >>> MetaspaceShared::reserved_space_alignment(); >>> char* shared_base = (char*)align_up((char*)SharedBaseAddress, >>> reserve_alignment); >>> >>> #ifdef _LP64 >>> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >>> "Sanity"); >>> >>> So theoretically shared_base may no longer be is_valid_base after the >>> align_up. It probably won't happen, but I think the code will be much >>> clearer tofirst to align_up, then check for is_valid_base and reset to >>> default, and finally assert. >>> >>> [8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >>> following block can be simplified: >>> >>> if (!_shared_rs.is_reserved()) { >>> // Get a reserved space anywhere if attaching at the >>> SharedBaseAddress fails: >>> if (UseCompressedClassPointers) { >>> // If we need a compressed class space too, let the platform >>> handle the reservation. >>> _shared_rs = >>> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >>> } else { >>> // anywhere is fine. >>> _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >>> large */, (char*)NULL); >>> } >>> } >>> >>> ... if you change the declaration to: >>> >>> static ReservedSpace >>> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >>> ReservedSpace();}); >>> >>> [9] I think the #ifdef _LP64 is not necessary: >>> >>> #ifdef _LP64 >>> // Some sanity checks after reserving address spaces for archives >>> and class space. >>> assert(archive_space_rs.is_reserved(), "Sanity"); >>> if (Metaspace::using_class_space()) { >>> // Class space must closely follow the archive space. Both spaces >>> must be aligned correctly. >>> assert(class_space_rs.is_reserved(), "A class space should have >>> been reserved"); >>> assert(class_space_rs.base() >= archive_space_rs.end(), "class >>> space should follow the cds archive space"); >>> assert(is_aligned(archive_space_rs.base(), >>> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >>> assert(is_aligned(class_space_rs.base(), >>> Metaspace::reserve_alignment()), "class space misaligned"); >>> } >>> #endif >>> >>> The rest of the code looks OK to me, but I may take a look at it again >>> after getting more sleep :-) >>> >>> Thanks >>> - Ioi >>> >>> >>> >>> >>> On 5/7/20 7:21 AM, Thomas St?fe wrote: >>> >>> Hi all, >>> >>> please take a look at the third iteration of this change: >>> >>> >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >>> >>> Changes in this version: >>> >>> - at the request of Coleen (sorry I did not see your post earlier) I >>> removed all platform dependent files and put the platform dependent coding >>> back to metaspace.cpp and compressedOops.cpp to ease the review pain. I >>> used plain platform defines though (#ifdef AARCH64) instead of hiding them >>> behind another macro to make things clearer. Note that I still intent to >>> put this code away into the platform corners but will do so in a follow up >>> RFE. >>> >>> - I reinstated, in a fashion, the special handling of reservations on >>> Windows. On all platforms we reserve address space to map the archive files >>> in with a subsequent mapping operation. However, on Windows, we cannot use >>> MapViewOfFile() into an existing mapping. So I remove the mapping again >>> before mapping the archives - see comment in code for details. >>> >>> All CI tests at SAP run through without problems, including on Win64 and >>> aarch64, but I would be happy if others were to run test too. >>> >>> Thank you, Thomas >>> >>> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >>> wrote: >>> >>>> Hi all, >>>> >>>> Could I have reviews for the following proposal of reworking cds/class >>>> space reservation? >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >>>> >>>> (Many thanks to Ioi Lam for so patiently explaining CDS internals to >>>> me, and to Andrew Haley and Nick Gasson for help with aarch64!) >>>> >>>> Reservation of the compressed class space is needlessly complicated and >>>> has some minor issues. It can be simplified and made clearer. >>>> >>>> The complexity stems from the fact that this area lives at the >>>> intersection of two to three sub systems, depending on how one counts. >>>> Metaspace, CDS, and the platform which may or may not its own view of how >>>> to reserve class space. And this code has been growing organically over >>>> time. >>>> >>>> One small example: >>>> >>>> ReservedSpace Metaspace::reserve_preferred_space(size_t size, size_t >>>> alignment, >>>> bool large_pages, char >>>> *requested_addr, >>>> bool >>>> use_requested_addr) >>>> >>>> which I spent hours decoding, resulting in a very confused mail to >>>> hs-runtime and aarch64-port-dev [2]. >>>> >>>> This patch attempts to simplify cds and metaspace setup a bit; to >>>> comment implicit knowledge which is not immediately clear; to cleanly >>>> abstract platform concerns like optimized class space placement; and to >>>> disentangle cds from metaspace to solve issues which may bite us later with >>>> Elastic Metaspace [4]. >>>> >>>> --- >>>> >>>> The main change is the reworked reservation mechanism. This is based on >>>> Ioi's proposal [5]. >>>> >>>> When reserving class space, three things must happen: >>>> >>>> 1) reservation of the space obviously. If cds is active that space must >>>> be in the vicinity of cds archives to be covered by compressed class >>>> pointer encoding. >>>> 2) setting up the internal Metaspace structures atop of that space >>>> 3) setting up compressed class pointer encoding. >>>> >>>> In its current form, Metaspace may or may not do some or all of that in >>>> one function >>>> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>>> metaspace_rs, char* requested_addr, address cds_base);) - if cds is active, >>>> it will reserve the space for Metaspace and hand it in, otherwise it will >>>> create it itself. >>>> >>>> When discussing this in [2], Ioi proposed to move the reservation of >>>> the class space completely out of Metaspace and make it a responsibility of >>>> the caller always. This would reduce some complexity, and this patch >>>> follows the proposal. >>>> >>>> I removed >>>> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>>> metaspace_rs, char* requested_addr, address cds_base); and all its sub >>>> functions. >>>> >>>> (1) now has to always be done outside - a ReservedSpace for class space >>>> has to be provided by the caller. However, Metaspace now offers a utility >>>> function for reserving space at a "nice" location, and explicitly doing >>>> nothing else: >>>> >>>> ReservedSpace >>>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>>> >>>> this function can be redefined on a platform level for platform >>>> optimized reservation, see below for details. >>>> >>>> (2) is taken care of by a new function, >>>> Metaspace::initialize_class_space(ReservedSpace rs) >>>> >>>> (3) is taken care of a new function >>>> CompressedKlassPointers::initialize(), see below for details. >>>> >>>> >>>> So, class space now is set up three explicit steps: >>>> >>>> - First, reserve a suitable space by however means you want. For >>>> convenience you may use >>>> Metaspace::reserve_address_space_for_compressed_classes(), or you may roll >>>> your own reservation. >>>> - Next, tell Metaspace to use that range as backing storage for class >>>> space: Metaspace::initialize_class_space(ReservedSpace rs) >>>> - Finally, set up encoding. Encoding is independent from the concept of >>>> a ReservedSpace, it just gets an address range, see below for details. >>>> >>>> Separating these steps and moving them out of the responsibility of >>>> Metaspace makes this whole thing more flexible; it also removes unnecessary >>>> knowledge (e.g. Metaspace does not need to know anything about either ccp >>>> encoding or cds). >>>> >>>> --- >>>> >>>> How it comes together: >>>> >>>> If CDS is off, we just reserve a space using >>>> Metaspace::reserve_address_space_for_compressed_classes(), initialize it >>>> with Metaspace::initialize_class_space(ReservedSpace rs), then set up >>>> compressed class pointer encoding covering the range of this class space. >>>> >>>> If CDS is on (dump time), we reserve large 4G space, either at >>>> SharedBaseAddress or using >>>> Metaspace::reserve_address_space_for_compressed_classes(); we then split >>>> that into 3G archive space and 1G class space; we set up that space with >>>> Metaspace as class space; then we set up compressed class pointer encoding >>>> covering both archive space and cds. >>>> >>>> If CDS is on (run time), we reserve a large space, split it into >>>> archive space (large enough to hold both archives) and class space, then >>>> basically proceed as above. >>>> >>>> Note that this is almost exactly how things worked before (modulo some >>>> minor fixes, e.g. alignment issues), only the code is reformed and made >>>> more explicit. >>>> >>>> --- >>>> >>>> I moved compressed class pointer setup over to CompressedKlassPointers >>>> and changed the interface: >>>> >>>> -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >>>> metaspace_rs, address cds_base) >>>> +void CompressedKlassPointers::initialize(address addr, size_t len); >>>> >>>> Instead of feeding it a single ReservedSpace, which is supposed to >>>> represent class space, and an optional alternate base if cds is on, now we >>>> give it just an numeric address range. That range marks the limits to where >>>> Klass structures are to be expected, and is the implicit promise that >>>> outside that range no Klass structures will exist, so encoding has to cover >>>> only this range. >>>> >>>> This range may contain just the class space; or class space+cds; or >>>> whatever allocation scheme we come up with in the future. Encoding does not >>>> really care how the memory is organized as long as the input range covers >>>> all possible Klass locations. That way we remove knowledge about class >>>> space/cds from compressed class pointer encoding. >>>> >>>> Moving it away from metaspace.cpp into the CompressedKlassPointers >>>> class also mirrors CompressedOops::initialize(). >>>> >>>> --- >>>> >>>> I renamed _narrow_klass_range to just _range, because strictly speaking >>>> this is the range un-narrow Klass pointers can have. >>>> >>>> As for the implementation of >>>> CompressedKlassPointers::initialize(address addr, size_t len), I mimicked >>>> very closely what happened before, so there should be almost no >>>> differences. Since "almost no differences" sounds scary :) here are the >>>> differences: >>>> >>>> - When CDS is active (dump or run time) we now always, unconditionally, >>>> set the encoding range to 4G. This fixes a theoretical bug discussed on >>>> aarch64-port-dev [1]. >>>> >>>> - When CDS is not active, we set the encoding range to the minimum >>>> required length. Before, it was left at its default value of 4G. >>>> >>>> Both differences only affect aarch64, since they are currently the only >>>> one using the range field in CompressedKlassPointers. >>>> >>>> I wanted to add an assert somewhere to test encoding of the very last >>>> address of the CompressedKlassPointers range, again to prevent errors like >>>> [3]. But I did not come up with a good place for this assert which would >>>> cover also the encoding done by C1/C2. >>>> >>>> For the same reason I thought about introducing a mode where Klass >>>> structures would be allocated in reverse order, starting at the end of the >>>> ccs, but again left it out as too big a change. >>>> >>>> --- >>>> >>>> OS abstraction: platforms may have restrictions of what constitutes a >>>> valid compressed class pointer encoding base. Or if not, they may have at >>>> least preferences. There was logic like this in metaspace.cpp, which I >>>> removed and cleanly factored out into platform dependent files, giving each >>>> platform the option to add special logic. >>>> >>>> These are two new methods: >>>> >>>> - bool CompressedKlassPointers::is_valid_base(address p) >>>> >>>> to let the platform tell you whether it considers p to be a valid >>>> encoding base. The only platform having these restrictions currently is >>>> aarch64. >>>> >>>> - ReservedSpace >>>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>>> >>>> this hands over the process of allocating a range suitable for >>>> compressed class pointer encoding to the platform. Most platforms will >>>> allocate just anywhere, but some platforms may have a better strategy (e.g. >>>> trying low memory first, trying only correctly aligned addresses and so >>>> on). >>>> >>>> Beforehand, this coding existed in a similar form in metaspace.cpp for >>>> aarch64 and AIX. For now, I left the AIX part out - it seems only half >>>> done, and I want to check further if we even need it, if yes why not on >>>> Linux ppc, and C1 does not seem to support anything other than base+offset >>>> with shift either, but I may be mistaken. >>>> >>>> These two methods should give the platform enough control to implement >>>> their own scheme for optimized class space placement without bothering any >>>> shared code about it. >>>> >>>> Note about the form, I introduced two new platform dependent files, >>>> "metaspace_.cpp" and "compressedOops_.cpp". I am not happy about >>>> this but this seems to be what we generally do in hotspot, right? >>>> >>>> --- >>>> >>>> Metaspace reserve alignment vs cds alignment >>>> >>>> CDS was using Metaspace reserve alignment for CDS internal purposes. I >>>> guess this was just a copy paste issue. It never caused problems since >>>> Metaspace reserve alignment == page size, but that is not true anymore in >>>> the upcoming Elastic Metaspace where reserve alignment will be larger. This >>>> causes a number of issues. >>>> >>>> I separated those two cleanly. CDS now uses >>>> os::vm_allocation_granularity. Metaspace::reserve_alignment is only used in >>>> those two places where it is needed, when CDS creates the address space for >>>> class space on behalf of the Metaspace. >>>> >>>> --- >>>> >>>> Windows special handling in CDS >>>> >>>> To simplify coding I removed the windows specific handling which left >>>> out reservation of the archive. This was needed because windows cannot mmap >>>> files into reserved regions. But fallback code exists in filemap.cpp for >>>> this case which just reads in the region instead of mapping it. >>>> >>>> Should that turn out to be a performance problem, I will reinstate the >>>> feature. But a simpler way would be reserve the archive and later just >>>> before mmapping the archive file to release the archive space. That would >>>> not only be simpler but give us the best guarantee that that address space >>>> is actually available. But I'd be happy to leave that part out completely >>>> if we do not see any performance problems on windows x64. >>>> >>>> --- >>>> >>>> NMT cannot deal with spaces which are split. This problem manifests in >>>> that bookkeeping for class space is done under "Shared Classes", not >>>> "Classes" as it should. This problem exists today too at dump time and >>>> randomly at run time. But since I simplified the reservation, this problem >>>> now shows up always, whether or not we map at the SharedBaseAddress. >>>> While I could work around this problem, I'd prefer this problem to be >>>> solved at the core, and NMT to have an option to recognize reservation >>>> splits. So I'd rather not put a workaround for this into the patch but >>>> leave it for fixing as a separate issue. I opened this issue to track it >>>> [6]. >>>> >>>> --- >>>> >>>> Jtreg tests: >>>> >>>> I expanded the CompressedOops/CompressedClassPointers.java. I also >>>> extended them to Windows. The tests now optionally omit strict class space >>>> placement tests, since these tests heavily depend on ASLR and were the >>>> reason they were excluded on Windows. However I think even without checking >>>> for class space placement they make sense, just to see that the VM comes up >>>> and lives with the many different settings we can run in. >>>> >>>> --- >>>> >>>> Tests: >>>> >>>> - I ran the patch through Oracles submit repo >>>> - I ran tests manually for aarch64, zero, linux 32bit and windows x64 >>>> - The whole battery of nightly tests at SAP, including ppc, ppcle and >>>> aarch64, unfortunately excluding windows because of unrelated errors. >>>> Windows x64 tests will be redone tonight. >>>> >>>> >>>> Thank you, >>>> >>>> Thomas >>>> >>>> [1] >>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >>>> [2] >>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >>>> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >>>> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >>>> [5] >>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >>>> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >>>> >>>> >>> >> > From thomas.stuefe at gmail.com Thu May 21 17:35:37 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 21 May 2020 19:35:37 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> <8bf736b3-9ab1-86b7-fd22-230e449b1b9f@oracle.com> Message-ID: p.s. Just noted today that this patch will solve https://bugs.openjdk.java.net/browse/JDK-8245514 , so it has some real benefits beside being just a code cleanup. ..Thomas On Thu, May 21, 2020 at 8:26 AM Thomas St?fe wrote: > Hi all, > > tests went through successfully at SAP two nights in a row. Ioi gave his > okay, and I would like to push today or tomorrow. Any remaining objections`? > > The current, final version is: > > full: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ > delta: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ > > Thank you, > > Thomas > > On Wed, May 20, 2020 at 1:43 AM Ioi Lam wrote: > >> Hi Thomas, >> >> The testing took a little longer than I expected. I've run your previous >> version: >> >> >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ >> >> In our CI tiers 1/4. No failures. >> >> I didn't run your latest version. I would suggesting testing it with your >> own CI as well as in jdk-submit before pushing (after you get OKs from >> other reviewers). >> >> Thumbs up. >> - Ioi >> >> >> On 5/19/20 7:40 AM, Thomas St?fe wrote: >> >> Hi guys, >> >> new webrev: >> >> full: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ >> delta: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ >> >> Mostly format changes, breaking up at column 80 as Andrew requested. >> >> The only changes which matter is the removal of the unnecessary include >> in arguments.cpp as Coleen suggested; and to not validate SharedBaseAddress >> anymore at runtime, as Ioi requested. >> >> Also rebased to head. >> >> Thanks, Thomas >> >> >> >> >> On Tue, May 19, 2020 at 6:38 AM Ioi Lam wrote: >> >>> >>> >>> On 5/18/20 7:37 AM, Thomas St?fe wrote: >>> >>> Hi all, >>> >>> fourth webrev: >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ >>> >>> I rebased atop of >>> >>> changeset: 59325:1ba9a9a3f948 >>> summary: 8244433: Remove saving of RSP in Assembler::pusha_uncached() >>> >>> so, atop of "8241825: Make compressed oops and compressed class pointers >>> independent (x86_64, PPC, S390)". >>> >>> The changes are very small this time, see notes below: >>> >>> @Ioi: >>> >>> >Hi Thomas, >>> >>> >I am running your patch in out CI pipeline now. Some comments: >>> >>> >[2] Does JDK-8243535 exist with the current jdk/jdk repo? >>> >>> I assigned JDK-8243535 to me and I'm working on a patch. But I would >>> like to keep that separate from this work, since this patch is already >>> large. And I would like to fix JDK-8243535 after this remodel patch has >>> been pushed, which makes some minor things easier. That would leave a small >>> time window where NMT does mis-track class space as shared class space, but >>> that is not a big issue, it does not even disturb any test. Which in itself >>> is something we may want to fix, come to think about it. >>> >>> >>> > [3] I think this can be put in header file. >>> > >>> > bool Metaspace::class_space_is_initialized() { >>> > return _class_space_list != NULL; >>> > } >>> >>> Done. >>> >>> > [7] argument.cpp checks this: >>> > >>> > if (UseSharedSpaces || DumpSharedSpaces) { >>> > if >>> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >>> > log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid >>> for this platform, option will be ignored.", >>> > p2i((address)SharedBaseAddress)); >>> > FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >>> > } >>> > } >>> > >>> > but initialize_dumptime_shared_and_meta_spaces aligns up the >>> SharedBaseAddress before doing the assert. >>> > >>> > void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >>> > assert(DumpSharedSpaces, "should be called for dump time only"); >>> > >>> > const size_t reserve_alignment = >>> MetaspaceShared::reserved_space_alignment(); >>> > char* shared_base = (char*)align_up((char*)SharedBaseAddress, >>> reserve_alignment); >>> > >>> > #ifdef _LP64 >>> > >>> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >>> "Sanity"); >>> > >>> > So theoretically shared_base may no longer be is_valid_base after the >>> align_up. It probably won't happen, but I think the code will be > much >>> clearer tofirst to align_up, then check for is_valid_base and reset to >>> default, and finally assert. >>> >>> The problem with that is that at the time this coding runs os::init() >>> did not yet run and we do not know yet allocation granularity. >>> >>> So I changed verification of SharedBaseAddress: I added the alignment as >>> you suggested and moved it to a later time in VM initialization, to CDS >>> initialization. >>> >>> >>> Is the call to check_SharedBaseAddress() necessary inside >>> MetaspaceShared::initialize_runtime_shared_and_meta_spaces()? >>> SharedBaseAddress is currently ignored at run time. Does your patch change >>> that behavior? >>> >>> >[8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >>> following block can be simplified: >>> > >>> > if (!_shared_rs.is_reserved()) { >>> > // Get a reserved space anywhere if attaching at the >>> SharedBaseAddress fails: >>> > if (UseCompressedClassPointers) { >>> > // If we need a compressed class space too, let the platform >>> handle the reservation. >>> > _shared_rs = >>> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >>> > } else { >>> > // anywhere is fine. >>> > _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >>> large */, (char*)NULL); >>> > } >>> > } >>> > >>> > ... if you change the declaration to: >>> > >>> > static ReservedSpace >>> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >>> ReservedSpace();}); >>> >>> I'd like to avoid keep the method LP64 only, but I rewrote the section >>> somewhat to be more clearer. >>> >>> The new code looks good. >>> >>> > [9] I think the #ifdef _LP64 is not necessary: >>> > >>> > #ifdef _LP64 >>> > // Some sanity checks after reserving address spaces for archives >>> and class space. >>> > assert(archive_space_rs.is_reserved(), "Sanity"); >>> > if (Metaspace::using_class_space()) { >>> > // Class space must closely follow the archive space. Both >>> spaces must be aligned correctly. >>> > assert(class_space_rs.is_reserved(), "A class space should have >>> been reserved"); >>> > assert(class_space_rs.base() >= archive_space_rs.end(), "class >>> space should follow the cds archive space"); >>> > assert(is_aligned(archive_space_rs.base(), >>> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >>> > assert(is_aligned(class_space_rs.base(), >>> Metaspace::reserve_alignment()), "class space misaligned"); >>> > } >>> > #endif >>> >>> I removed the #ifdef, but replaced it with an #ifdef ASSERT. Which is >>> also not needed, technically, but makes it clear that this section is >>> debugging checks only. >>> >>> >>> OK. >>> >>> The rest of the code looks good to me. I am going to run your latest >>> patch in our CI and will report the results. >>> >>> Thanks >>> - Ioi >>> >>> ---- >>> >>> @Coleen: >>> >>> > >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/ >>> >virtualSpaceList.cpp.udiff.html >>> > >>> >Why did you make this change? Shouldn't the caller align it? >>> >>> >>> You are right, I removed the assert. >>> >>> --- >>> >>> Thanks all for your review work, >>> >>> ..Thomas >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Fri, May 8, 2020 at 7:37 AM Ioi Lam wrote: >>> >>>> Hi Thomas, >>>> >>>> I am running your patch in out CI pipeline now. Some comments: >>>> >>>> [1] Do we still need to use _class_space_list? It looks like we just >>>> have a single range. Is _class_space_list->current_virtual_space() always >>>> the same VS? >>>> >>>> if (_class_space_list != NULL) { >>>> address base = >>>> (address)_class_space_list->current_virtual_space()->bottom(); >>>> address top = base + compressed_class_space_size(); >>>> st->print("Compressed class space mapped at: " PTR_FORMAT "-" >>>> PTR_FORMAT ", size: " SIZE_FORMAT, >>>> p2i(base), p2i(top), top - base); >>>> >>>> [2] Does JDK-8243535 exist with the current jdk/jdk repo? >>>> >>>> // This does currently not work because rs may be the result of a >>>> split operation >>>> // and NMT seems not to be able to handle splits. >>>> // See JDK-8243535. >>>> // MemTracker::record_virtual_memory_type((address)rs.base(), >>>> mtClass); >>>> >>>> >>>> [3] I think this can be put in header file. >>>> >>>> bool Metaspace::class_space_is_initialized() { >>>> return _class_space_list != NULL; >>>> } >>>> >>>> [4] Why does the CCS need to follow UseLargePagesInMetaspace? This is >>>> the reason for the >>>> gap in the following chart: >>>> >>>> // We do this by reserving space for the ccs behind the archives. >>>> Note however that >>>> // ccs follows a different alignment >>>> (Metaspace::reserve_alignment), so there may >>>> // be a gap between ccs and cds. >>>> // We use a similar layout at runtime, see >>>> reserve_address_space_for_archives(). >>>> // >>>> // +-- SharedBaseAddress (default = >>>> 0x800000000) >>>> // v >>>> // +-..---------+---------+ ... >>>> +----+----+----+--------+--------------------+ >>>> // | Heap | Archive | | MC | RW | RO | [gap] | class >>>> space | >>>> // +-..---------+---------+ ... >>>> +----+----+----+--------+--------------------+ >>>> // |<-- MaxHeapSize -->| |<-- UnscaledClassSpaceMax = 4GB >>>> -->| >>>> >>>> >>>> _reserve_alignment is determined here: >>>> >>>> void Metaspace::ergo_initialize() { >>>> if (DumpSharedSpaces) { >>>> // Using large pages when dumping the shared archive is currently >>>> not implemented. >>>> FLAG_SET_ERGO(UseLargePagesInMetaspace, false); >>>> } >>>> >>>> size_t page_size = os::vm_page_size(); >>>> if (UseLargePages && UseLargePagesInMetaspace) { >>>> page_size = os::large_page_size(); >>>> } >>>> >>>> _commit_alignment = page_size; >>>> _reserve_alignment = MAX2(page_size, >>>> (size_t)os::vm_allocation_granularity()); >>>> >>>> But when CDS is enabled, the RS is reserved without large pages, so it >>>> should be possible to incrementally commit the CCS using just >>>> os::vm_allocation_granularity(). >>>> >>>> [5] I see the aarch64 code in >>>> Metaspace::reserve_address_space_for_compressed_classes has been changed. >>>> Is this necessary for this RFE, or is it a separate improvement that can be >>>> done in a different RFE? >>>> >>>> [6] For AARCH64, should we skip the part that reserves it right above >>>> the heap? Or does AARCH64 always allocate the heap such that it's a >>>> preferable address? >>>> >>>> // case (b) >>>> ReservedSpace rs; >>>> >>>> // Try right above the Java heap... >>>> address base = align_up(CompressedOops::end(), >>>> Metaspace::reserve_alignment()); >>>> assert(base != NULL, "Sanity"); >>>> >>>> const size_t size = align_up(CompressedClassSpaceSize, >>>> Metaspace::reserve_alignment()); >>>> if (CompressedKlassPointers::is_valid_base(base)) { >>>> rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* >>>> large */, (char*)base); >>>> } >>>> >>>> // ...failing that, reserve anywhere, but let platform do optimized >>>> placement: >>>> if (!rs.is_reserved()) { >>>> rs = >>>> Metaspace::reserve_address_space_for_compressed_classes(size); >>>> } >>>> >>>> [7] argument.cpp checks this: >>>> >>>> if (UseSharedSpaces || DumpSharedSpaces) { >>>> if >>>> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >>>> log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for >>>> this platform, option will be ignored.", >>>> p2i((address)SharedBaseAddress)); >>>> FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >>>> } >>>> } >>>> >>>> but initialize_dumptime_shared_and_meta_spaces aligns up the >>>> SharedBaseAddress before doing the assert. >>>> >>>> void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >>>> assert(DumpSharedSpaces, "should be called for dump time only"); >>>> >>>> const size_t reserve_alignment = >>>> MetaspaceShared::reserved_space_alignment(); >>>> char* shared_base = (char*)align_up((char*)SharedBaseAddress, >>>> reserve_alignment); >>>> >>>> #ifdef _LP64 >>>> >>>> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >>>> "Sanity"); >>>> >>>> So theoretically shared_base may no longer be is_valid_base after the >>>> align_up. It probably won't happen, but I think the code will be much >>>> clearer tofirst to align_up, then check for is_valid_base and reset to >>>> default, and finally assert. >>>> >>>> [8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >>>> following block can be simplified: >>>> >>>> if (!_shared_rs.is_reserved()) { >>>> // Get a reserved space anywhere if attaching at the >>>> SharedBaseAddress fails: >>>> if (UseCompressedClassPointers) { >>>> // If we need a compressed class space too, let the platform >>>> handle the reservation. >>>> _shared_rs = >>>> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >>>> } else { >>>> // anywhere is fine. >>>> _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >>>> large */, (char*)NULL); >>>> } >>>> } >>>> >>>> ... if you change the declaration to: >>>> >>>> static ReservedSpace >>>> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >>>> ReservedSpace();}); >>>> >>>> [9] I think the #ifdef _LP64 is not necessary: >>>> >>>> #ifdef _LP64 >>>> // Some sanity checks after reserving address spaces for archives >>>> and class space. >>>> assert(archive_space_rs.is_reserved(), "Sanity"); >>>> if (Metaspace::using_class_space()) { >>>> // Class space must closely follow the archive space. Both spaces >>>> must be aligned correctly. >>>> assert(class_space_rs.is_reserved(), "A class space should have >>>> been reserved"); >>>> assert(class_space_rs.base() >= archive_space_rs.end(), "class >>>> space should follow the cds archive space"); >>>> assert(is_aligned(archive_space_rs.base(), >>>> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >>>> assert(is_aligned(class_space_rs.base(), >>>> Metaspace::reserve_alignment()), "class space misaligned"); >>>> } >>>> #endif >>>> >>>> The rest of the code looks OK to me, but I may take a look at it again >>>> after getting more sleep :-) >>>> >>>> Thanks >>>> - Ioi >>>> >>>> >>>> >>>> >>>> On 5/7/20 7:21 AM, Thomas St?fe wrote: >>>> >>>> Hi all, >>>> >>>> please take a look at the third iteration of this change: >>>> >>>> >>>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >>>> >>>> Changes in this version: >>>> >>>> - at the request of Coleen (sorry I did not see your post earlier) I >>>> removed all platform dependent files and put the platform dependent coding >>>> back to metaspace.cpp and compressedOops.cpp to ease the review pain. I >>>> used plain platform defines though (#ifdef AARCH64) instead of hiding them >>>> behind another macro to make things clearer. Note that I still intent to >>>> put this code away into the platform corners but will do so in a follow up >>>> RFE. >>>> >>>> - I reinstated, in a fashion, the special handling of reservations on >>>> Windows. On all platforms we reserve address space to map the archive files >>>> in with a subsequent mapping operation. However, on Windows, we cannot use >>>> MapViewOfFile() into an existing mapping. So I remove the mapping again >>>> before mapping the archives - see comment in code for details. >>>> >>>> All CI tests at SAP run through without problems, including on Win64 >>>> and aarch64, but I would be happy if others were to run test too. >>>> >>>> Thank you, Thomas >>>> >>>> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Could I have reviews for the following proposal of reworking cds/class >>>>> space reservation? >>>>> >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >>>>> >>>>> (Many thanks to Ioi Lam for so patiently explaining CDS internals to >>>>> me, and to Andrew Haley and Nick Gasson for help with aarch64!) >>>>> >>>>> Reservation of the compressed class space is needlessly complicated >>>>> and has some minor issues. It can be simplified and made clearer. >>>>> >>>>> The complexity stems from the fact that this area lives at the >>>>> intersection of two to three sub systems, depending on how one counts. >>>>> Metaspace, CDS, and the platform which may or may not its own view of how >>>>> to reserve class space. And this code has been growing organically over >>>>> time. >>>>> >>>>> One small example: >>>>> >>>>> ReservedSpace Metaspace::reserve_preferred_space(size_t size, size_t >>>>> alignment, >>>>> bool large_pages, >>>>> char *requested_addr, >>>>> bool >>>>> use_requested_addr) >>>>> >>>>> which I spent hours decoding, resulting in a very confused mail to >>>>> hs-runtime and aarch64-port-dev [2]. >>>>> >>>>> This patch attempts to simplify cds and metaspace setup a bit; to >>>>> comment implicit knowledge which is not immediately clear; to cleanly >>>>> abstract platform concerns like optimized class space placement; and to >>>>> disentangle cds from metaspace to solve issues which may bite us later with >>>>> Elastic Metaspace [4]. >>>>> >>>>> --- >>>>> >>>>> The main change is the reworked reservation mechanism. This is based >>>>> on Ioi's proposal [5]. >>>>> >>>>> When reserving class space, three things must happen: >>>>> >>>>> 1) reservation of the space obviously. If cds is active that space >>>>> must be in the vicinity of cds archives to be covered by compressed class >>>>> pointer encoding. >>>>> 2) setting up the internal Metaspace structures atop of that space >>>>> 3) setting up compressed class pointer encoding. >>>>> >>>>> In its current form, Metaspace may or may not do some or all of that >>>>> in one function >>>>> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>>>> metaspace_rs, char* requested_addr, address cds_base);) - if cds is active, >>>>> it will reserve the space for Metaspace and hand it in, otherwise it will >>>>> create it itself. >>>>> >>>>> When discussing this in [2], Ioi proposed to move the reservation of >>>>> the class space completely out of Metaspace and make it a responsibility of >>>>> the caller always. This would reduce some complexity, and this patch >>>>> follows the proposal. >>>>> >>>>> I removed >>>>> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>>>> metaspace_rs, char* requested_addr, address cds_base); and all its sub >>>>> functions. >>>>> >>>>> (1) now has to always be done outside - a ReservedSpace for class >>>>> space has to be provided by the caller. However, Metaspace now offers a >>>>> utility function for reserving space at a "nice" location, and explicitly >>>>> doing nothing else: >>>>> >>>>> ReservedSpace >>>>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>>>> >>>>> this function can be redefined on a platform level for platform >>>>> optimized reservation, see below for details. >>>>> >>>>> (2) is taken care of by a new function, >>>>> Metaspace::initialize_class_space(ReservedSpace rs) >>>>> >>>>> (3) is taken care of a new function >>>>> CompressedKlassPointers::initialize(), see below for details. >>>>> >>>>> >>>>> So, class space now is set up three explicit steps: >>>>> >>>>> - First, reserve a suitable space by however means you want. For >>>>> convenience you may use >>>>> Metaspace::reserve_address_space_for_compressed_classes(), or you may roll >>>>> your own reservation. >>>>> - Next, tell Metaspace to use that range as backing storage for class >>>>> space: Metaspace::initialize_class_space(ReservedSpace rs) >>>>> - Finally, set up encoding. Encoding is independent from the concept >>>>> of a ReservedSpace, it just gets an address range, see below for details. >>>>> >>>>> Separating these steps and moving them out of the responsibility of >>>>> Metaspace makes this whole thing more flexible; it also removes unnecessary >>>>> knowledge (e.g. Metaspace does not need to know anything about either ccp >>>>> encoding or cds). >>>>> >>>>> --- >>>>> >>>>> How it comes together: >>>>> >>>>> If CDS is off, we just reserve a space using >>>>> Metaspace::reserve_address_space_for_compressed_classes(), initialize it >>>>> with Metaspace::initialize_class_space(ReservedSpace rs), then set up >>>>> compressed class pointer encoding covering the range of this class space. >>>>> >>>>> If CDS is on (dump time), we reserve large 4G space, either at >>>>> SharedBaseAddress or using >>>>> Metaspace::reserve_address_space_for_compressed_classes(); we then split >>>>> that into 3G archive space and 1G class space; we set up that space with >>>>> Metaspace as class space; then we set up compressed class pointer encoding >>>>> covering both archive space and cds. >>>>> >>>>> If CDS is on (run time), we reserve a large space, split it into >>>>> archive space (large enough to hold both archives) and class space, then >>>>> basically proceed as above. >>>>> >>>>> Note that this is almost exactly how things worked before (modulo some >>>>> minor fixes, e.g. alignment issues), only the code is reformed and made >>>>> more explicit. >>>>> >>>>> --- >>>>> >>>>> I moved compressed class pointer setup over to CompressedKlassPointers >>>>> and changed the interface: >>>>> >>>>> -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >>>>> metaspace_rs, address cds_base) >>>>> +void CompressedKlassPointers::initialize(address addr, size_t len); >>>>> >>>>> Instead of feeding it a single ReservedSpace, which is supposed to >>>>> represent class space, and an optional alternate base if cds is on, now we >>>>> give it just an numeric address range. That range marks the limits to where >>>>> Klass structures are to be expected, and is the implicit promise that >>>>> outside that range no Klass structures will exist, so encoding has to cover >>>>> only this range. >>>>> >>>>> This range may contain just the class space; or class space+cds; or >>>>> whatever allocation scheme we come up with in the future. Encoding does not >>>>> really care how the memory is organized as long as the input range covers >>>>> all possible Klass locations. That way we remove knowledge about class >>>>> space/cds from compressed class pointer encoding. >>>>> >>>>> Moving it away from metaspace.cpp into the CompressedKlassPointers >>>>> class also mirrors CompressedOops::initialize(). >>>>> >>>>> --- >>>>> >>>>> I renamed _narrow_klass_range to just _range, because strictly >>>>> speaking this is the range un-narrow Klass pointers can have. >>>>> >>>>> As for the implementation of >>>>> CompressedKlassPointers::initialize(address addr, size_t len), I mimicked >>>>> very closely what happened before, so there should be almost no >>>>> differences. Since "almost no differences" sounds scary :) here are the >>>>> differences: >>>>> >>>>> - When CDS is active (dump or run time) we now always, >>>>> unconditionally, set the encoding range to 4G. This fixes a theoretical bug >>>>> discussed on aarch64-port-dev [1]. >>>>> >>>>> - When CDS is not active, we set the encoding range to the minimum >>>>> required length. Before, it was left at its default value of 4G. >>>>> >>>>> Both differences only affect aarch64, since they are currently the >>>>> only one using the range field in CompressedKlassPointers. >>>>> >>>>> I wanted to add an assert somewhere to test encoding of the very last >>>>> address of the CompressedKlassPointers range, again to prevent errors like >>>>> [3]. But I did not come up with a good place for this assert which would >>>>> cover also the encoding done by C1/C2. >>>>> >>>>> For the same reason I thought about introducing a mode where Klass >>>>> structures would be allocated in reverse order, starting at the end of the >>>>> ccs, but again left it out as too big a change. >>>>> >>>>> --- >>>>> >>>>> OS abstraction: platforms may have restrictions of what constitutes a >>>>> valid compressed class pointer encoding base. Or if not, they may have at >>>>> least preferences. There was logic like this in metaspace.cpp, which I >>>>> removed and cleanly factored out into platform dependent files, giving each >>>>> platform the option to add special logic. >>>>> >>>>> These are two new methods: >>>>> >>>>> - bool CompressedKlassPointers::is_valid_base(address p) >>>>> >>>>> to let the platform tell you whether it considers p to be a valid >>>>> encoding base. The only platform having these restrictions currently is >>>>> aarch64. >>>>> >>>>> - ReservedSpace >>>>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>>>> >>>>> this hands over the process of allocating a range suitable for >>>>> compressed class pointer encoding to the platform. Most platforms will >>>>> allocate just anywhere, but some platforms may have a better strategy (e.g. >>>>> trying low memory first, trying only correctly aligned addresses and so >>>>> on). >>>>> >>>>> Beforehand, this coding existed in a similar form in metaspace.cpp for >>>>> aarch64 and AIX. For now, I left the AIX part out - it seems only half >>>>> done, and I want to check further if we even need it, if yes why not on >>>>> Linux ppc, and C1 does not seem to support anything other than base+offset >>>>> with shift either, but I may be mistaken. >>>>> >>>>> These two methods should give the platform enough control to implement >>>>> their own scheme for optimized class space placement without bothering any >>>>> shared code about it. >>>>> >>>>> Note about the form, I introduced two new platform dependent files, >>>>> "metaspace_.cpp" and "compressedOops_.cpp". I am not happy about >>>>> this but this seems to be what we generally do in hotspot, right? >>>>> >>>>> --- >>>>> >>>>> Metaspace reserve alignment vs cds alignment >>>>> >>>>> CDS was using Metaspace reserve alignment for CDS internal purposes. I >>>>> guess this was just a copy paste issue. It never caused problems since >>>>> Metaspace reserve alignment == page size, but that is not true anymore in >>>>> the upcoming Elastic Metaspace where reserve alignment will be larger. This >>>>> causes a number of issues. >>>>> >>>>> I separated those two cleanly. CDS now uses >>>>> os::vm_allocation_granularity. Metaspace::reserve_alignment is only used in >>>>> those two places where it is needed, when CDS creates the address space for >>>>> class space on behalf of the Metaspace. >>>>> >>>>> --- >>>>> >>>>> Windows special handling in CDS >>>>> >>>>> To simplify coding I removed the windows specific handling which left >>>>> out reservation of the archive. This was needed because windows cannot mmap >>>>> files into reserved regions. But fallback code exists in filemap.cpp for >>>>> this case which just reads in the region instead of mapping it. >>>>> >>>>> Should that turn out to be a performance problem, I will reinstate the >>>>> feature. But a simpler way would be reserve the archive and later just >>>>> before mmapping the archive file to release the archive space. That would >>>>> not only be simpler but give us the best guarantee that that address space >>>>> is actually available. But I'd be happy to leave that part out completely >>>>> if we do not see any performance problems on windows x64. >>>>> >>>>> --- >>>>> >>>>> NMT cannot deal with spaces which are split. This problem manifests in >>>>> that bookkeeping for class space is done under "Shared Classes", not >>>>> "Classes" as it should. This problem exists today too at dump time and >>>>> randomly at run time. But since I simplified the reservation, this problem >>>>> now shows up always, whether or not we map at the SharedBaseAddress. >>>>> While I could work around this problem, I'd prefer this problem to be >>>>> solved at the core, and NMT to have an option to recognize reservation >>>>> splits. So I'd rather not put a workaround for this into the patch but >>>>> leave it for fixing as a separate issue. I opened this issue to track it >>>>> [6]. >>>>> >>>>> --- >>>>> >>>>> Jtreg tests: >>>>> >>>>> I expanded the CompressedOops/CompressedClassPointers.java. I also >>>>> extended them to Windows. The tests now optionally omit strict class space >>>>> placement tests, since these tests heavily depend on ASLR and were the >>>>> reason they were excluded on Windows. However I think even without checking >>>>> for class space placement they make sense, just to see that the VM comes up >>>>> and lives with the many different settings we can run in. >>>>> >>>>> --- >>>>> >>>>> Tests: >>>>> >>>>> - I ran the patch through Oracles submit repo >>>>> - I ran tests manually for aarch64, zero, linux 32bit and windows x64 >>>>> - The whole battery of nightly tests at SAP, including ppc, ppcle and >>>>> aarch64, unfortunately excluding windows because of unrelated errors. >>>>> Windows x64 tests will be redone tonight. >>>>> >>>>> >>>>> Thank you, >>>>> >>>>> Thomas >>>>> >>>>> [1] >>>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >>>>> [2] >>>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >>>>> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >>>>> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >>>>> [5] >>>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >>>>> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >>>>> >>>>> >>>> >>> >> From kim.barrett at oracle.com Thu May 21 22:31:32 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Thu, 21 May 2020 18:31:32 -0400 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class Message-ID: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> Please review this change to eliminate volatile qualification in the member functions and parameters of the oop class and its subclasses. Removed volatile qualified member functions and parameters from class oop and its subclasses (instanceOop, arrayOop, objArrayOop, typeArrayOop). Changed the few remaining volatile oop accesses to instead use relaxed atomics. Changed constructors for MemoryManager and MemoryPool to use initializer list, so we don't need to use Atomic::store to initialize an oop-derived member. Eliminated ShenandoahVerifierTask's volatile-qualified assignment operator, which isn't used. Some tidying up oop class. Cleaned up constructors, eliminating public set_obj(). Use const-ref rather than by-value parameters for comparisons, to avoid unnecessary copies that go through register/deregister. CR: https://bugs.openjdk.java.net/browse/JDK-8245594 Webrev: https://cr.openjdk.java.net/~kbarrett/8245594/open.00/ Testing: mach5 tier1-5 Local (linux-x64) hotspot:tier1 with Shenandoah enabled. From kim.barrett at oracle.com Thu May 21 22:36:17 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Thu, 21 May 2020 18:36:17 -0400 Subject: RFR: 8228692: BitMap::reallocate might not clear some bits Message-ID: Please review this change to BitMap reallocation that ensures all newly added bits are cleared when requested. Previously, there were circumstances where some might not be. Includes a new gtest for the specific issue. This change also includes a fix for JDK-8235714. Contrary to previous comments and bug updates, it was not fixed by JDK-8241581 after all. This was noticed while testing the change for JDK-8228692. CR: https://bugs.openjdk.java.net/browse/JDK-8228692 https://bugs.openjdk.java.net/browse/JDK-8235714 Webrev: https://cr.openjdk.java.net/~kbarrett/8228692/open.00/ Testing: mach5 tier1 Local (linux-x64) BitMap tests, including the new test. The new test (1) failed before the changes, (2) still failed after either one but not both of the fix for relocate and the fix for count_one_bits, (3) and passes with both fixes. From kim.barrett at oracle.com Fri May 22 01:54:08 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Thu, 21 May 2020 21:54:08 -0400 Subject: RFR: 8240259: Disable -Wshift-negative-value warnings Message-ID: Please review this change which disables warnings for left shift of a negative value when compiling HotSpot with gcc or clang. This warning isn't helpful, given that all compilers seem to do the obvious thing, and that obvious thing will soon-ish (C++20) be the specified thing. See the CR for more details. CR: https://bugs.openjdk.java.net/browse/JDK-8240259 Webrev: https://cr.openjdk.java.net/~kbarrett/8240259/open.00/ Testing: mach5 tier1. Also tier1-5 with C++14 enabled. Note that there are already occurrences of left shift of negative values in our code, with no apparent consequences, even when testing with C++11/14 enabled. From david.holmes at oracle.com Fri May 22 02:39:30 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 22 May 2020 12:39:30 +1000 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> Message-ID: <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> Hi Kim, Not following the details 100% but the general gist seems okay, except for a couple of queries below. On 22/05/2020 8:31 am, Kim Barrett wrote: > Please review this change to eliminate volatile qualification in the > member functions and parameters of the oop class and its subclasses. > > Removed volatile qualified member functions and parameters from class > oop and its subclasses (instanceOop, arrayOop, objArrayOop, typeArrayOop). > Changed the few remaining volatile oop accesses to instead use relaxed > atomics. > > Changed constructors for MemoryManager and MemoryPool to use > initializer list, so we don't need to use Atomic::store to initialize > an oop-derived member. What is the connection between using an initializer-list and not needing the Atomic::Store? > Eliminated ShenandoahVerifierTask's volatile-qualified assignment > operator, which isn't used. > > Some tidying up oop class. Cleaned up constructors, eliminating public > set_obj(). Use const-ref rather than by-value parameters for comparisons, > to avoid unnecessary copies that go through register/deregister. > > CR: > https://bugs.openjdk.java.net/browse/JDK-8245594 > > Webrev: > https://cr.openjdk.java.net/~kbarrett/8245594/open.00/ In the following the referenced oops still retain their volatile qualifier - should it not now be removed? - src/hotspot/share/runtime/thread.hpp _exception_oop is still marked volatile - src/hotspot/share/services/memoryManager.hpp _memory_mgr_obj is still marked volatile - src/hotspot/share/services/memoryPool.hpp _memory_pool_obj is still marked volatile Thanks, David > Testing: > mach5 tier1-5 > Local (linux-x64) hotspot:tier1 with Shenandoah enabled. > From magnus.ihse.bursie at oracle.com Fri May 22 07:38:01 2020 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Fri, 22 May 2020 09:38:01 +0200 Subject: RFR: 8240259: Disable -Wshift-negative-value warnings In-Reply-To: References: Message-ID: <87d1ceb1-d18a-5875-73c3-9ae89f100f31@oracle.com> Looks good to me. /Magnus On 2020-05-22 03:54, Kim Barrett wrote: > Please review this change which disables warnings for left shift of a > negative value when compiling HotSpot with gcc or clang. This warning > isn't helpful, given that all compilers seem to do the obvious thing, > and that obvious thing will soon-ish (C++20) be the specified thing. > See the CR for more details. > > CR: > https://bugs.openjdk.java.net/browse/JDK-8240259 > > Webrev: > https://cr.openjdk.java.net/~kbarrett/8240259/open.00/ > > Testing: > mach5 tier1. Also tier1-5 with C++14 enabled. > > Note that there are already occurrences of left shift of negative > values in our code, with no apparent consequences, even when testing > with C++11/14 enabled. > From kim.barrett at oracle.com Fri May 22 08:23:05 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Fri, 22 May 2020 04:23:05 -0400 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> Message-ID: > On May 21, 2020, at 10:39 PM, David Holmes wrote: > > Hi Kim, > > Not following the details 100% but the general gist seems okay, except for a couple of queries below. > > On 22/05/2020 8:31 am, Kim Barrett wrote: >> Please review this change to eliminate volatile qualification in the >> member functions and parameters of the oop class and its subclasses. >> Removed volatile qualified member functions and parameters from class >> oop and its subclasses (instanceOop, arrayOop, objArrayOop, typeArrayOop). >> Changed the few remaining volatile oop accesses to instead use relaxed >> atomics. >> Changed constructors for MemoryManager and MemoryPool to use >> initializer list, so we don't need to use Atomic::store to initialize >> an oop-derived member. > > What is the connection between using an initializer-list and not needing the Atomic::Store? The volatile oop-derived member (instanceOop in both cases) needs to be initialized. If done in an initializer list, it's at construction time for the member object, where volatile isn't relevant. If done later (for example, in the containing object's constructor body, the member is now volatile and so involved a volatile assignment, which would now need to be an Atomic::store. >> Eliminated ShenandoahVerifierTask's volatile-qualified assignment >> operator, which isn't used. >> Some tidying up oop class. Cleaned up constructors, eliminating public >> set_obj(). Use const-ref rather than by-value parameters for comparisons, >> to avoid unnecessary copies that go through register/deregister. >> CR: >> https://bugs.openjdk.java.net/browse/JDK-8245594 >> Webrev: >> https://cr.openjdk.java.net/~kbarrett/8245594/open.00/ > > In the following the referenced oops still retain their volatile qualifier - should it not now be removed? > > - src/hotspot/share/runtime/thread.hpp > _exception_oop is still marked volatile > - src/hotspot/share/services/memoryManager.hpp > _memory_mgr_obj is still marked volatile > - src/hotspot/share/services/memoryPool.hpp > _memory_pool_obj is still marked volatile Access to these objects is via Atomic operations, so they should still be volatile qualified per current convention. It might be worth examining their use of Atomic. The MemoryManager case seems a bit odd, but I haven't studied it carefully. I haven't looked at the others yet. I think doing so is outside the scope of this change. > > Thanks, > David > >> Testing: >> mach5 tier1-5 >> Local (linux-x64) hotspot:tier1 with Shenandoah enabled. From kim.barrett at oracle.com Fri May 22 08:23:34 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Fri, 22 May 2020 04:23:34 -0400 Subject: RFR: 8240259: Disable -Wshift-negative-value warnings In-Reply-To: <87d1ceb1-d18a-5875-73c3-9ae89f100f31@oracle.com> References: <87d1ceb1-d18a-5875-73c3-9ae89f100f31@oracle.com> Message-ID: <7506FB40-FF15-44A8-9071-9AED46081CAC@oracle.com> > On May 22, 2020, at 3:38 AM, Magnus Ihse Bursie wrote: > > Looks good to me. Thanks. > > /Magnus > > On 2020-05-22 03:54, Kim Barrett wrote: >> Please review this change which disables warnings for left shift of a >> negative value when compiling HotSpot with gcc or clang. This warning >> isn't helpful, given that all compilers seem to do the obvious thing, >> and that obvious thing will soon-ish (C++20) be the specified thing. >> See the CR for more details. >> >> CR: >> https://bugs.openjdk.java.net/browse/JDK-8240259 >> >> Webrev: >> https://cr.openjdk.java.net/~kbarrett/8240259/open.00/ >> >> Testing: >> mach5 tier1. Also tier1-5 with C++14 enabled. >> >> Note that there are already occurrences of left shift of negative >> values in our code, with no apparent consequences, even when testing >> with C++11/14 enabled. From sgehwolf at redhat.com Fri May 22 09:10:07 2020 From: sgehwolf at redhat.com (Severin Gehwolf) Date: Fri, 22 May 2020 11:10:07 +0200 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: References: <7201E4D5-96AA-47C1-B0B7-A19E8A0F198B@tencent.com> <51519874123ae26a4d3ebf4863d80e682c13b989.camel@redhat.com> Message-ID: <54807978d02d8383a6b9804bfcd8d2ca5dc88194.camel@redhat.com> Hi Bob, On Wed, 2020-05-20 at 16:16 -0400, Bob Vandette wrote: > What do you think about fixing getTotalSwapSpaceSze and getFreeSwapSpaceSize to return 0L if either of > the underlying Metrics used to calculate the result return -1. Of course we only do this if containerMetrics != null. No, I don't think that would be appropriate. This would result in too many false positives for JDKs running in a container, say with a CPU limit, but without a memory limit. In general, the mode of operation for metrics is to use them if you can be sure they're present otherwise defer to values of the container host. Perhaps we should consider adding some traces when this happens, so that we at least point *something* out to the user. Even though, it might be a wrong warning in some cases. > Failure to read these values, means we have no swap. If we have no swap we also have no free swap. 0 seems > like an appropriate value to represent this state. I believe, -1 for certain container metrics can be returned for the "unlimited" case, too. In that case, returning 0 would be wrong. > It seems as though some kernels report 0 when kernel swap limit is not enabled while others don?t populate > the memory status files. Unfortunately, yes. Thanks, Severin From coleen.phillimore at oracle.com Fri May 22 12:14:58 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 22 May 2020 08:14:58 -0400 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> <8bf736b3-9ab1-86b7-fd22-230e449b1b9f@oracle.com> Message-ID: <2cb02a62-954d-c261-2553-02b7e2ab4efb@oracle.com> On 5/21/20 2:26 AM, Thomas St?fe wrote: > Hi all, > > tests went through successfully at SAP two nights in a row. Ioi gave > his okay, and I would like to push today or tomorrow. Any remaining > objections`? None from me.? Ship it! Coleen > > The current, final version is: > > full: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ > delta: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ > > Thank you, > > Thomas > > On Wed, May 20, 2020 at 1:43 AM Ioi Lam > wrote: > > Hi Thomas, > > The testing took a little longer than I expected. I've run your > previous version: > > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ > > In our CI tiers 1/4. No failures. > > I didn't run your latest version. I would suggesting testing it > with your own CI as well as in jdk-submit before pushing (after > you get OKs from other reviewers). > > Thumbs up. > - Ioi > > > On 5/19/20 7:40 AM, Thomas St?fe wrote: >> Hi guys, >> >> new webrev: >> >> full: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ >> delta: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ >> >> Mostly format changes, breaking up at column 80 as Andrew requested. >> >> The only changes which matter is the removal of the unnecessary >> include in arguments.cpp as Coleen suggested; and to not validate >> SharedBaseAddress anymore at runtime, as Ioi requested. >> >> Also rebased to head. >> >> Thanks, Thomas >> >> >> >> >> On Tue, May 19, 2020 at 6:38 AM Ioi Lam > > wrote: >> >> >> >> On 5/18/20 7:37 AM, Thomas St?fe wrote: >>> Hi all, >>> >>> fourth webrev: >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ >>> >>> I rebased atop of >>> >>> changeset: ? 59325:1ba9a9a3f948 >>> summary: ? ? 8244433: Remove saving of RSP in >>> Assembler::pusha_uncached() >>> >>> so, atop of "8241825: Make compressed oops and compressed >>> class pointers independent (x86_64, PPC, S390)". >>> >>> The changes are very small this time, see notes below: >>> >>> @Ioi: >>> >>> >Hi Thomas, >>> >>> >I am running your patch in out CI pipeline now. Some comments: >>> >>> >[2] Does JDK-8243535 exist with the current jdk/jdk repo? >>> >>> I assigned JDK-8243535 to me and I'm working on a patch. But >>> I would like to keep that separate from this work, since >>> this patch is already large. And I would like to fix >>> JDK-8243535 after this remodel patch has been pushed, which >>> makes some minor things easier. That would leave a small >>> time window where NMT does mis-track class space as shared >>> class space, but that is not a big issue, it does not even >>> disturb any test. Which in itself is something we may want >>> to fix, come to think about it. >>> >>> >>> > [3] I think this can be put in header file. >>> > >>> > bool Metaspace::class_space_is_initialized() { >>> > ? return _class_space_list != NULL; >>> > } >>> >>> Done. >>> >>> > [7] argument.cpp checks this: >>> > >>> > ? if (UseSharedSpaces || DumpSharedSpaces) { >>> > ? ? if >>> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) >>> { >>> > ? ? ? log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " >>> is invalid for this platform, option will be ignored.", >>> > ?p2i((address)SharedBaseAddress)); >>> > ? ? ? FLAG_SET_DEFAULT(SharedBaseAddress, >>> default_SharedBaseAddress()); >>> > ? ? } >>> > ? } >>> > >>> > but initialize_dumptime_shared_and_meta_spaces aligns up >>> the SharedBaseAddress before doing the assert. >>> > >>> > ? void >>> MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >>> > ? ? assert(DumpSharedSpaces, "should be called for dump >>> time only"); >>> > >>> > ? ? const size_t reserve_alignment = >>> MetaspaceShared::reserved_space_alignment(); >>> > ? ? char* shared_base = >>> (char*)align_up((char*)SharedBaseAddress, reserve_alignment); >>> > >>> > ? #ifdef _LP64 >>> > >>> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >>> "Sanity"); >>> > >>> > So theoretically shared_base may no longer be >>> is_valid_base after the align_up. It probably won't happen, >>> but I think the code will be > much clearer tofirst to >>> align_up, then check for is_valid_base and reset to default, >>> and finally assert. >>> >>> The problem with that is that at the time this coding runs >>> os::init() did not yet run and we do not know yet allocation >>> granularity. >>> >>> So I changed verification of SharedBaseAddress: I added the >>> alignment as you suggested and moved it to a later time in >>> VM initialization, to CDS initialization. >>> >> >> Is the call to? check_SharedBaseAddress() necessary inside >> MetaspaceShared::initialize_runtime_shared_and_meta_spaces()? >> SharedBaseAddress is currently ignored at run time. Does your >> patch change that behavior? >> >>> >[8] In >>> MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, >>> the following block can be simplified: >>> > >>> > ?if (!_shared_rs.is_reserved()) { >>> > ? ?// Get a reserved space anywhere if attaching at the >>> SharedBaseAddress fails: >>> > ? ?if (UseCompressedClassPointers) { >>> > ? ? ?// If we need a compressed class space too, let the >>> platform handle the reservation. >>> > ? ? ?_shared_rs = >>> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >>> > ? ?} else { >>> > ? ? ?// anywhere is fine. >>> > ? ? ?_shared_rs = ReservedSpace(cds_total, >>> reserve_alignment, false /* large */, (char*)NULL); >>> > ? ?} >>> > ?} >>> > >>> > ? ... if you change the declaration to: >>> > >>> > ?static ReservedSpace >>> reserve_address_space_for_compressed_classes(size_t size) >>> NOT_LP64({ return ReservedSpace();}); >>> >>> I'd like to avoid keep the method LP64 only, but I rewrote >>> the section somewhat to be more clearer. >>> >> The new code looks good. >> >>> > [9] I think the #ifdef _LP64 is not necessary: >>> > >>> > #ifdef _LP64 >>> > ? ? // Some sanity checks after reserving address spaces >>> for archives and class space. >>> > assert(archive_space_rs.is_reserved(), "Sanity"); >>> > ? ? if (Metaspace::using_class_space()) { >>> > ? ? ? // Class space must closely follow the archive >>> space. Both spaces must be aligned correctly. >>> > assert(class_space_rs.is_reserved(), "A class space should >>> have been reserved"); >>> > ? ? ? assert(class_space_rs.base() >= >>> archive_space_rs.end(), "class space should follow the cds >>> archive space"); >>> > assert(is_aligned(archive_space_rs.base(), >>> MetaspaceShared::reserved_space_alignment()), "Archive space >>> misaligned"); >>> > assert(is_aligned(class_space_rs.base(), >>> Metaspace::reserve_alignment()), "class space misaligned"); >>> > ? ? } >>> > #endif >>> >>> I removed the #ifdef, but replaced it with an #ifdef ASSERT. >>> Which is also not needed, technically, but makes it clear >>> that this section is debugging checks only. >>> >> >> OK. >> >> The rest of the code looks good to me. I am going to run your >> latest patch in our CI and will report the results. >> >> Thanks >> - Ioi >>> ---- >>> >>> @Coleen: >>> >>> >http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/>virtualSpaceList.cpp.udiff.html >>> > >>> >Why did you make this change? Shouldn't the caller align it? >>> >>> >>> You are right, I removed the assert. >>> >>> --- >>> >>> Thanks all for your review work, >>> >>> ..Thomas >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Fri, May 8, 2020 at 7:37 AM Ioi Lam >> > wrote: >>> >>> Hi Thomas, >>> >>> I am running your patch in out CI pipeline now. Some >>> comments: >>> >>> [1] Do we still need to use _class_space_list? It looks >>> like we just have a single range. Is >>> _class_space_list->current_virtual_space() always the >>> same VS? >>> >>> ? if (_class_space_list != NULL) { >>> ??? address base = >>> (address)_class_space_list->current_virtual_space()->bottom(); >>> ??? address top = base + compressed_class_space_size(); >>> ??? st->print("Compressed class space mapped at: " >>> PTR_FORMAT "-" PTR_FORMAT ", size: " SIZE_FORMAT, >>> ?????????????? p2i(base), p2i(top), top - base); >>> >>> [2] Does JDK-8243535 exist with the current jdk/jdk repo? >>> >>> ? // This does currently not work because rs may be the >>> result of a split operation >>> ? // and NMT seems not to be able to handle splits. >>> ? // See JDK-8243535. >>> ? // >>> MemTracker::record_virtual_memory_type((address)rs.base(), >>> mtClass); >>> >>> >>> [3] I think this can be put in header file. >>> >>> bool Metaspace::class_space_is_initialized() { >>> ? return _class_space_list != NULL; >>> } >>> >>> [4] Why does the CCS need to follow >>> UseLargePagesInMetaspace? This is the reason for the >>> ??? gap in the following chart: >>> >>> ??? // We do this by reserving space for the ccs behind >>> the archives. Note however that >>> ??? //? ccs follows a different alignment >>> (Metaspace::reserve_alignment), so there may >>> ??? //? be a gap between ccs and cds. >>> ??? // We use a similar layout at runtime, see >>> reserve_address_space_for_archives(). >>> ??? // >>> ??? //????????????????????????????? +-- >>> SharedBaseAddress (default = 0x800000000) >>> ??? //????????????????????????????? v >>> ??? // +-..---------+---------+ ... >>> +----+----+----+--------+--------------------+ >>> ??? // |??? Heap??? | Archive |???? | MC | RW | RO | >>> [gap]? |??? class space???? | >>> ??? // +-..---------+---------+ ... >>> +----+----+----+--------+--------------------+ >>> ??? // |<--?? MaxHeapSize? -->| |<-- >>> UnscaledClassSpaceMax = 4GB -->| >>> >>> >>> _reserve_alignment is determined here: >>> >>> void Metaspace::ergo_initialize() { >>> ? if (DumpSharedSpaces) { >>> ??? // Using large pages when dumping the shared archive >>> is currently not implemented. >>> FLAG_SET_ERGO(UseLargePagesInMetaspace, false); >>> ? } >>> >>> ? size_t page_size = os::vm_page_size(); >>> ? if (UseLargePages && UseLargePagesInMetaspace) { >>> ??? page_size = os::large_page_size(); >>> ? } >>> >>> ? _commit_alignment? = page_size; >>> ? _reserve_alignment = MAX2(page_size, >>> (size_t)os::vm_allocation_granularity()); >>> >>> But when CDS is enabled, the RS is reserved without >>> large pages, so it should be possible to incrementally >>> commit the CCS using just os::vm_allocation_granularity(). >>> >>> [5] I see the aarch64 code in >>> Metaspace::reserve_address_space_for_compressed_classes >>> has been changed. Is this necessary for this RFE, or is >>> it a separate improvement that can be done in a >>> different RFE? >>> >>> [6] For AARCH64, should we skip the part that reserves >>> it right above the heap? Or does AARCH64 always allocate >>> the heap such that it's a preferable address? >>> >>> ??? // case (b) >>> ??? ReservedSpace rs; >>> >>> ??? // Try right above the Java heap... >>> ??? address base = align_up(CompressedOops::end(), >>> Metaspace::reserve_alignment()); >>> ??? assert(base != NULL, "Sanity"); >>> >>> ??? const size_t size = >>> align_up(CompressedClassSpaceSize, >>> Metaspace::reserve_alignment()); >>> ??? if (CompressedKlassPointers::is_valid_base(base)) { >>> ????? rs = ReservedSpace(size, >>> Metaspace::reserve_alignment(), false /* large */, >>> (char*)base); >>> ??? } >>> >>> ??? // ...failing that, reserve anywhere, but let >>> platform do optimized placement: >>> ??? if (!rs.is_reserved()) { >>> ????? rs = >>> Metaspace::reserve_address_space_for_compressed_classes(size); >>> ??? } >>> >>> [7] argument.cpp checks this: >>> >>> ? if (UseSharedSpaces || DumpSharedSpaces) { >>> ??? if >>> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) >>> { >>> log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is >>> invalid for this platform, option will be ignored.", >>> p2i((address)SharedBaseAddress)); >>> ????? FLAG_SET_DEFAULT(SharedBaseAddress, >>> default_SharedBaseAddress()); >>> ??? } >>> ? } >>> >>> but initialize_dumptime_shared_and_meta_spaces aligns up >>> the SharedBaseAddress before doing the assert. >>> >>> ? void >>> MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() >>> { >>> ??? assert(DumpSharedSpaces, "should be called for dump >>> time only"); >>> >>> ??? const size_t reserve_alignment = >>> MetaspaceShared::reserved_space_alignment(); >>> ??? char* shared_base = >>> (char*)align_up((char*)SharedBaseAddress, >>> reserve_alignment); >>> >>> ? #ifdef _LP64 >>> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >>> "Sanity"); >>> >>> So theoretically shared_base may no longer be >>> is_valid_base after the align_up. It probably won't >>> happen, but I think the code will be much clearer >>> tofirst to align_up, then check for is_valid_base and >>> reset to default, and finally assert. >>> >>> [8] In >>> MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, >>> the following block can be simplified: >>> >>> ? if (!_shared_rs.is_reserved()) { >>> ??? // Get a reserved space anywhere if attaching at the >>> SharedBaseAddress fails: >>> ??? if (UseCompressedClassPointers) { >>> ????? // If we need a compressed class space too, let >>> the platform handle the reservation. >>> ????? _shared_rs = >>> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >>> ??? } else { >>> ????? // anywhere is fine. >>> ????? _shared_rs = ReservedSpace(cds_total, >>> reserve_alignment, false /* large */, (char*)NULL); >>> ??? } >>> ? } >>> >>> ?? ... if you change the declaration to: >>> >>> ? static ReservedSpace >>> reserve_address_space_for_compressed_classes(size_t >>> size) NOT_LP64({ return ReservedSpace();}); >>> >>> [9] I think the #ifdef _LP64 is not necessary: >>> >>> #ifdef _LP64 >>> ??? // Some sanity checks after reserving address spaces >>> for archives and class space. >>> ??? assert(archive_space_rs.is_reserved(), "Sanity"); >>> ??? if (Metaspace::using_class_space()) { >>> ????? // Class space must closely follow the archive >>> space. Both spaces must be aligned correctly. >>> ????? assert(class_space_rs.is_reserved(), "A class >>> space should have been reserved"); >>> ????? assert(class_space_rs.base() >= >>> archive_space_rs.end(), "class space should follow the >>> cds archive space"); >>> assert(is_aligned(archive_space_rs.base(), >>> MetaspaceShared::reserved_space_alignment()), "Archive >>> space misaligned"); >>> assert(is_aligned(class_space_rs.base(), >>> Metaspace::reserve_alignment()), "class space misaligned"); >>> ??? } >>> #endif >>> >>> The rest of the code looks OK to me, but I may take a >>> look at it again after getting more sleep :-) >>> >>> Thanks >>> - Ioi >>> >>> >>> >>> >>> On 5/7/20 7:21 AM, Thomas St?fe wrote: >>>> Hi all, >>>> >>>> please take a look at the third iteration of this change: >>>> >>>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >>>> >>>> Changes in this version: >>>> >>>> - at the request of Coleen (sorry I did not see your >>>> post earlier) I removed all platform dependent?files >>>> and put the platform dependent coding back to >>>> metaspace.cpp and compressedOops.cpp to ease the review >>>> pain. I used plain platform defines though (#ifdef >>>> AARCH64) instead of hiding them behind another macro to >>>> make things clearer. Note that I still intent to put >>>> this code away into the platform corners but will do so >>>> in a follow up RFE. >>>> >>>> - I reinstated, in a fashion, the special handling of >>>> reservations on Windows. On all platforms we reserve >>>> address space to map the archive files in with a >>>> subsequent mapping operation. However, on Windows, we >>>> cannot use MapViewOfFile() into an existing mapping. So >>>> I remove the mapping again before mapping the archives >>>> - see comment in code for details. >>>> >>>> All CI tests at SAP run through without problems, >>>> including on Win64 and aarch64, but I would be happy if >>>> others were to run test too. >>>> >>>> Thank you, Thomas >>>> >>>> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >>>> >>> > wrote: >>>> >>>> Hi all, >>>> >>>> Could I have reviews for the following proposal of >>>> reworking cds/class space reservation? >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >>>> >>>> (Many thanks to Ioi Lam for so patiently explaining >>>> CDS internals to me, and to Andrew Haley and Nick >>>> Gasson for help with aarch64!) >>>> >>>> Reservation of the compressed class space is >>>> needlessly complicated and has some minor issues. >>>> It can be simplified and made clearer. >>>> >>>> The complexity stems from the fact that this area >>>> lives at the intersection of two to three sub >>>> systems, depending on how one counts. Metaspace, >>>> CDS, and the platform which may or may not its own >>>> view of how to reserve class space. And this code >>>> has been growing organically over time. >>>> >>>> One small example: >>>> >>>> ReservedSpace >>>> Metaspace::reserve_preferred_space(size_t size, >>>> size_t alignment, >>>> ? ? ? ? ? ? ? ?bool large_pages, char *requested_addr, >>>> ? ? ? ? ? ? ? ?bool use_requested_addr) >>>> >>>> which I spent hours decoding, resulting in a very >>>> confused mail to hs-runtime and aarch64-port-dev [2]. >>>> >>>> This patch attempts to simplify cds and metaspace >>>> setup a bit; to comment implicit knowledge which is >>>> not immediately clear; to cleanly abstract platform >>>> concerns like optimized class space placement; and >>>> to disentangle cds from metaspace to solve issues >>>> which may bite us later with Elastic Metaspace [4]. >>>> >>>> --- >>>> >>>> The main change is the reworked reservation >>>> mechanism. This is based on Ioi's proposal [5]. >>>> >>>> When reserving class space, three things must happen: >>>> >>>> 1) reservation of the space obviously. If cds is >>>> active that space must be in the vicinity of cds >>>> archives to be covered by compressed class pointer >>>> encoding. >>>> 2) setting up the internal Metaspace structures >>>> atop of that space >>>> 3) setting up compressed class pointer encoding. >>>> >>>> In its current form, Metaspace may or may not do >>>> some or all of that in one function >>>> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>>> metaspace_rs, char* requested_addr, address >>>> cds_base);) - if cds is active, it will reserve the >>>> space for Metaspace and hand it in, otherwise it >>>> will create it itself. >>>> >>>> When discussing this in [2], Ioi proposed to move >>>> the reservation of the class space completely out >>>> of Metaspace and make it a responsibility of the >>>> caller always. This would reduce some complexity, >>>> and this patch follows the proposal. >>>> >>>> I removed >>>> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>>> metaspace_rs, char* requested_addr, address >>>> cds_base); and all its sub functions. >>>> >>>> (1) now has to always be done outside - a >>>> ReservedSpace for class space has to be provided by >>>> the caller. However, Metaspace now offers a utility >>>> function for reserving space at a "nice" location, >>>> and explicitly doing nothing else: >>>> >>>> ReservedSpace >>>> Metaspace::reserve_address_space_for_compressed_classes(size_t >>>> size); >>>> >>>> this function can be redefined on a platform level >>>> for platform optimized reservation, see below for >>>> details. >>>> >>>> (2) is taken care of by a new function, >>>> Metaspace::initialize_class_space(ReservedSpace rs) >>>> >>>> (3) is taken care of a new function >>>> CompressedKlassPointers::initialize(), see below >>>> for details. >>>> >>>> >>>> So, class space now is set up three explicit steps: >>>> >>>> - First, reserve a suitable space by however means >>>> you want. For convenience you may use >>>> Metaspace::reserve_address_space_for_compressed_classes(), >>>> or you may roll your own reservation. >>>> - Next, tell Metaspace to use that range as backing >>>> storage for class space: >>>> Metaspace::initialize_class_space(ReservedSpace rs) >>>> - Finally, set up encoding. Encoding is independent >>>> from the concept of a ReservedSpace, it just gets >>>> an address range, see below for details. >>>> >>>> Separating these steps and moving them out of the >>>> responsibility of Metaspace makes this whole thing >>>> more flexible; it also removes unnecessary >>>> knowledge (e.g. Metaspace does not need to know >>>> anything about either ccp encoding or cds). >>>> >>>> --- >>>> >>>> How it comes together: >>>> >>>> If CDS is off, we just reserve a space using >>>> Metaspace::reserve_address_space_for_compressed_classes(), >>>> initialize it with >>>> Metaspace::initialize_class_space(ReservedSpace >>>> rs), then set up compressed class pointer encoding >>>> covering the range of this class space. >>>> >>>> If CDS is on (dump time), we reserve large 4G >>>> space, either at SharedBaseAddress or using >>>> Metaspace::reserve_address_space_for_compressed_classes(); >>>> we then split that into 3G archive space and 1G >>>> class space; we set up that space with Metaspace as >>>> class space; then we set up?compressed class >>>> pointer encoding covering both archive space and cds. >>>> >>>> If CDS is on (run time), we reserve a large space, >>>> split it into archive space (large enough to hold >>>> both archives) and class space, then basically >>>> proceed as above. >>>> >>>> Note that this is almost exactly how things worked >>>> before (modulo some minor fixes, e.g. alignment >>>> issues), only the code is reformed and made more >>>> explicit. >>>> >>>> --- >>>> >>>> I moved compressed class pointer setup over to >>>> CompressedKlassPointers and changed the interface: >>>> >>>> -void >>>> Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >>>> metaspace_rs, address cds_base) >>>> +void CompressedKlassPointers::initialize(address >>>> addr, size_t len); >>>> >>>> Instead of feeding it a single ReservedSpace, which >>>> is supposed to represent class space, and an >>>> optional alternate base if cds is on, now we give >>>> it just an numeric address range. That range marks >>>> the limits to where Klass structures are to be >>>> expected, and is the implicit promise that outside >>>> that range no Klass structures will exist, so >>>> encoding has to cover only this range. >>>> >>>> This range may contain just the class space; or >>>> class space+cds; or whatever allocation scheme we >>>> come up with in the future. Encoding does not >>>> really care how the memory is organized as long as >>>> the input range covers all possible Klass >>>> locations. That way we remove knowledge about class >>>> space/cds from compressed class pointer encoding. >>>> >>>> Moving it away from metaspace.cpp into the >>>> CompressedKlassPointers class also mirrors >>>> CompressedOops::initialize(). >>>> >>>> --- >>>> >>>> I renamed _narrow_klass_range to just _range, >>>> because strictly speaking this is the range >>>> un-narrow Klass pointers can have. >>>> >>>> As for the implementation of >>>> CompressedKlassPointers::initialize(address addr, >>>> size_t len), I mimicked very closely what happened >>>> before, so there should be almost no differences. >>>> Since "almost no differences" sounds scary :) here >>>> are the differences: >>>> >>>> - When CDS is active (dump or run time) we now >>>> always, unconditionally, set the encoding range to >>>> 4G. This fixes a theoretical bug discussed on >>>> aarch64-port-dev [1]. >>>> >>>> - When CDS is not active, we set the encoding range >>>> to the minimum required length. Before, it was left >>>> at its default value of 4G. >>>> >>>> Both differences only affect aarch64, since they >>>> are currently the only one using the range field in >>>> CompressedKlassPointers. >>>> >>>> I wanted to add an assert somewhere to test >>>> encoding of the very last address of the >>>> CompressedKlassPointers range, again to prevent >>>> errors like [3]. But I did not come up with a good >>>> place for this assert which would cover also the >>>> encoding done by C1/C2. >>>> >>>> For the same reason I thought about introducing a >>>> mode where Klass structures would be allocated in >>>> reverse order, starting at the end of the ccs, but >>>> again left it out as too big a change. >>>> >>>> --- >>>> >>>> OS abstraction: platforms may have restrictions of >>>> what constitutes a valid compressed class pointer >>>> encoding base. Or if not, they may have at least >>>> preferences. There was logic like this in >>>> metaspace.cpp, which I removed and cleanly factored >>>> out into platform dependent files, giving each >>>> platform the option to add special logic. >>>> >>>> These are two new methods: >>>> >>>> - bool >>>> CompressedKlassPointers::is_valid_base(address p) >>>> >>>> to let the platform tell you whether it considers p >>>> to be a valid encoding base. The only platform >>>> having these restrictions currently is aarch64. >>>> >>>> - ReservedSpace >>>> Metaspace::reserve_address_space_for_compressed_classes(size_t >>>> size); >>>> >>>> this hands over the process of allocating a range >>>> suitable for compressed class pointer encoding to >>>> the platform. Most platforms will allocate just >>>> anywhere, but some platforms may have a better >>>> strategy (e.g. trying low memory first, trying only >>>> correctly aligned addresses and so on). >>>> >>>> Beforehand, this coding existed in a similar form >>>> in metaspace.cpp for aarch64 and AIX. For now, I >>>> left the AIX part out - it seems only half done, >>>> and I want to check further if we even need it, if >>>> yes why not on Linux ppc, and C1 does not seem to >>>> support anything other than base+offset with shift >>>> either, but I may be mistaken. >>>> >>>> These two methods should give the platform enough >>>> control to implement their own scheme for optimized >>>> class space placement without bothering any shared >>>> code about it. >>>> >>>> Note about the form, I introduced two new platform >>>> dependent files, "metaspace_.cpp" and >>>> "compressedOops_.cpp". I am not happy about >>>> this but this seems to be what we generally do in >>>> hotspot, right? >>>> >>>> --- >>>> >>>> Metaspace reserve alignment vs cds alignment >>>> >>>> CDS was using Metaspace reserve alignment for CDS >>>> internal purposes. I guess this was just a copy >>>> paste issue. It never caused problems since >>>> Metaspace reserve alignment == page size, but that >>>> is not true anymore in the upcoming Elastic >>>> Metaspace where reserve alignment will be larger. >>>> This causes a number of issues. >>>> >>>> I separated those two cleanly. CDS now uses >>>> os::vm_allocation_granularity. >>>> Metaspace::reserve_alignment is only used in those >>>> two places where it is needed, when CDS creates the >>>> address space for class space on behalf of the >>>> Metaspace. >>>> >>>> --- >>>> >>>> Windows special handling in CDS >>>> >>>> To simplify coding I removed the windows specific >>>> handling which left out reservation of the archive. >>>> This was needed because windows cannot mmap files >>>> into reserved regions. But fallback code exists in >>>> filemap.cpp for this case which just reads in the >>>> region instead of mapping?it. >>>> >>>> Should that turn out to be a performance problem, I >>>> will reinstate the feature. But a simpler way would >>>> be reserve the archive and later just before >>>> mmapping?the archive file to release the archive >>>> space. That would not only be simpler but give us >>>> the best guarantee that that address space is >>>> actually available. But I'd be happy to leave that >>>> part out completely if we do not see any >>>> performance problems on windows x64. >>>> >>>> --- >>>> >>>> NMT cannot deal with spaces which are split. This >>>> problem manifests in that bookkeeping for class >>>> space is done under "Shared Classes", not "Classes" >>>> as it should. This problem exists today too at >>>> dump?time and randomly at run time. But since I >>>> simplified the reservation, this problem now shows >>>> up always, whether or not we map at the >>>> SharedBaseAddress. >>>> While I could work around this problem, I'd prefer >>>> this problem to be solved at the core, and NMT to >>>> have an option to recognize reservation splits. So >>>> I'd rather not put a workaround for this into the >>>> patch but leave it for fixing as a separate issue. >>>> I opened this issue to track it [6]. >>>> >>>> --- >>>> >>>> Jtreg tests: >>>> >>>> I expanded the >>>> CompressedOops/CompressedClassPointers.java. I also >>>> extended them to Windows. The tests now optionally >>>> omit strict class space placement tests, since >>>> these tests heavily depend on ASLR and were the >>>> reason they were excluded on Windows. However I >>>> think even without checking for class space >>>> placement they make sense, just to see that the VM >>>> comes up and lives with the many different settings >>>> we can run in. >>>> >>>> --- >>>> >>>> Tests: >>>> >>>> - I ran the patch through Oracles submit repo >>>> - I ran tests manually for aarch64, zero, linux >>>> 32bit and windows x64 >>>> - The whole battery of nightly tests at SAP, >>>> including ppc, ppcle and aarch64, unfortunately >>>> excluding windows because of unrelated errors. >>>> Windows x64 tests will be redone tonight. >>>> >>>> >>>> Thank you, >>>> >>>> Thomas >>>> >>>> [1] >>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >>>> [2] >>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >>>> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >>>> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >>>> [5] >>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >>>> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >>>> >>> >> > From aph at redhat.com Fri May 22 13:00:58 2020 From: aph at redhat.com (Andrew Haley) Date: Fri, 22 May 2020 14:00:58 +0100 Subject: [aarch64-port-dev ] RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes In-Reply-To: References: Message-ID: <275eb57c-51c0-675e-c32a-91b198023559@redhat.com> On 5/19/20 11:55 AM, Yang Zhang wrote: > Following up on review requests of API [0], Java implementation and > test [1], General Hotspot changes[2] for Vector API and x86 backend > changes [3]. Here's a request for review of AArch64 backend changes > required for supporting the Vector API: > > JEP: https://openjdk.java.net/jeps/338 > JBS: https://bugs.openjdk.java.net/browse/JDK-8223347 > Webrev: http://cr.openjdk.java.net/~yzhang/vectorapi/vectorapi.rfr/aarch64_webrev/webrev.01/ > > Complete implementation resides in vector-unstable branch of panama/dev > repository [4]. This looks great, and it's very impressive. Unfortunately, there are few of us sufficiently knowledgeable about Panama to review it in the detail that perhaps it deserves. I'm happy with it. However, we need tests for the new assembly instructions, so please add some to aarch64_asmtest.py and update macroassemler.cpp. Also, aarch64.ad is getting to be far too large, and perhaps all the vector stuff should be moved into a new file. Thanks. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From matthias.baesken at sap.com Fri May 22 13:36:14 2020 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Fri, 22 May 2020 13:36:14 +0000 Subject: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java In-Reply-To: References: <7201E4D5-96AA-47C1-B0B7-A19E8A0F198B@tencent.com> <51519874123ae26a4d3ebf4863d80e682c13b989.camel@redhat.com> Message-ID: >Anyway, I think we should change the test code similar to this: >http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8244500/01/webrev/ Hi Severin, this gave me : Checking OperatingSystemMXBean Exception in thread "main" java.lang.IllegalAccessError: class CheckOperatingSystemMXBean (in unnamed module @0x5acf9800) cannot access class jdk.internal.platform.Container (in module java.base) because module java.base does not export jdk.internal.platform to unnamed module @0x5acf9800 at CheckOperatingSystemMXBean.main(CheckOperatingSystemMXBean.java:33) (a bit unexpected because you added it to the test, I though this would be sufficient !?) Guess adding the export to the module-info.java of java.base would also not be so good. (plus we still need Jie's adjustment to fix getFreeSwapSpaceSize ) Best regards, Matthias -----Original Message----- From: Severin Gehwolf Sent: Mittwoch, 20. Mai 2020 21:30 To: Baesken, Matthias ; jiefu(??) ; David Holmes ; 'hotspot-dev at openjdk.java.net' Cc: Bob Vandette Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java Hi Matthias, On Wed, 2020-05-20 at 11:07 +0000, Baesken, Matthias wrote: > Hi Severin , > > > FWIW, parts of this work-around got added by JDK-8236617. Looking at > > the bug it was observed that: > > > > containerMetrics.getMemoryAndSwapLimit() == 0 and > > containerMetrics.getMemoryLimit() returned a large value. > > > > So then with JDK-8231111 I suspect > > containerMetrics.getMemoryAndSwapLimit() now returns -1 and, thus, > > getTotalSwapSpaceSize0() is being used instead of the container values. > > I added some tracing output to OperatingSystemImpl.getTotalSwapSpaceSize() and get these values : > > containerMetrics.getMemoryAndSwapLimit():-1 containerMetrics.getMemoryLimit():-1 > ****** getTotalSwapSpaceSize0() returned 16106061824 Thanks for this. This seems to confirm my suspicion. So my working theory here is that the system where you are observing the issue now is similar to the one when JDK-8236617 was discovered. For some reason containerMetrics.getMemoryAndSwapLimit() returned 0 for the case when the necessary files were missing (which was wrong IMO). After JDK-8231111 there is a clear distinction for this case: -1 is being returned. That makes more sense to me, as there wouldn't be any container limit to take into account. Your trace output seems to confirm that: containerMetrics.getMemoryAndSwapLimit():-1 containerMetrics.getMemoryLimit():-1 > > > So far the theory, I'd like to gather more data before we come up with > > another patch, though. In JDK-8236617 it wasn't investigated why > > containerMetrics.getMemoryAndSwapLimit() returned 0 on such a system. > > It would be good to know why. Could it be that file > > 'memory.memsw.limit_in_bytes' doesn't exist on such a system? > > True, it does not exist : > > ls -alL /sys/fs/cgroup/memory/ > total 0 > drwxr-xr-x 3 root root 0 Mar 16 11:00 . > drwxr-xr-x 11 root root 260 Mar 16 11:00 .. > -rw-r--r-- 1 root root 0 Mar 16 11:00 cgroup.clone_children > --w--w--w- 1 root root 0 Mar 16 11:00 cgroup.event_control > -rw-r--r-- 1 root root 0 May 20 13:00 cgroup.procs > -r--r--r-- 1 root root 0 Mar 16 11:00 cgroup.sane_behavior > drwxr-xr-x 2 root root 0 May 20 12:52 docker > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.failcnt > --w------- 1 root root 0 Mar 16 11:00 memory.force_empty > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.limit_in_bytes > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.low_limit_in_bytes > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.max_usage_in_bytes > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.move_charge_at_immigrate > -r--r--r-- 1 root root 0 Mar 16 11:00 memory.numa_stat > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.oom_control > ---------- 1 root root 0 Mar 16 11:00 memory.pressure_level > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.soft_limit_in_bytes > -r--r--r-- 1 root root 0 Mar 16 11:00 memory.stat > -rw-r--r-- 1 root root 0 Mar 16 11:00 memory.swappiness > -r--r--r-- 1 root root 0 Mar 16 11:00 memory.usage_in_bytes > -rw-r--r-- 1 root root 0 May 20 13:00 memory.use_hierarchy > -rw-r--r-- 1 root root 0 Mar 16 11:00 notify_on_release > -rw-r--r-- 1 root root 0 Mar 16 11:00 release_agent > -rw-r--r-- 1 root root 0 Mar 16 11:00 tasks Hmm, is this inside a container on this system with --memory and -- memory-swap being specified on the docker command line? Anyway, I think we should change the test code similar to this: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8244500/01/webrev/ If this works for you, then that's what I'd suggest as a fix. Note that I haven't tested this patch. I need to recreate my set-up for proper docker testing here :-/ Thanks, Severin > Best regards, Matthias > > > > -----Original Message----- > From: Severin Gehwolf > Sent: Montag, 18. Mai 2020 12:09 > To: jiefu(??) ; David Holmes ; Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' > Cc: Bob Vandette > Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > Hi, > > On Mon, 2020-05-18 at 06:15 +0000, jiefu(??) wrote: > > Hi David, > > > > This bug should be introduced by JDK-8242480. > > I'm sorry for that. > > > > And I think the logic in src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java should be fixed. > > > > In my fix for JDK-8242480, I had ignored the case for kernels which don't support "--memory-swap 150M". > > In that case, containerMetrics.getMemoryAndSwapLimit()[1] will return -1. > > It appears so that containerMetrics.getMemoryAndSwapLimit() returned 0 > prior JDK-8231111 and -1 after. I'm not convinced JDK-8242480 > introduced this. > > > I didn't notice this bug because all our kernels support --memory-swap. > > Fortunately, Matthias found this bug and will fix it. > > I'm still not sure what's happening in Matthias' case on that system. > It would be best to first try to understand the actual problem better. > The info I have is: > > - It's a cgroup v1 system > - The system now returns large values for > OperatingSystemMXBean.getTotalSwapSpaceSize() and > OperatingSystemMXBean.getFreeSwapSpaceSize() > > It would be good to know what the actual return values are for the > container metrics on this system with swap limit disabled at kernel > level. And also whether or not relevant cgroup files exist on such a > system (within docker). > > FWIW, parts of this work-around got added by JDK-8236617. Looking at > the bug it was observed that: > > containerMetrics.getMemoryAndSwapLimit() == 0 and > containerMetrics.getMemoryLimit() returned a large value. > > So then with JDK-8231111 I suspect > containerMetrics.getMemoryAndSwapLimit() now returns -1 and, thus, > getTotalSwapSpaceSize0() is being used instead of the container values. > > So far the theory, I'd like to gather more data before we come up with > another patch, though. In JDK-8236617 it wasn't investigated why > containerMetrics.getMemoryAndSwapLimit() returned 0 on such a system. > It would be good to know why. Could it be that file > 'memory.memsw.limit_in_bytes' doesn't exist on such a system? > > Unfortunately, I don't know what config one needs in order to reproduce > so we'll have to rely on Matthias to gather the info. > > Thanks, > Severin > > > Thanks a lot. > > Best regards, > > Jie > > > > [1] http://hg.openjdk.java.net/jdk/jdk/file/a95911422005/src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java#l73 > > > > ?On 2020/5/18, 1:48 PM, "David Holmes" wrote: > > > > Hi Matthias, > > > > On 16/05/2020 12:59 am, Baesken, Matthias wrote: > > > Hello David , I get the warning always on this linux machine (I think it depends on machine setup and/or kernel version). > > > > > >> Surely it must be specified to return 0 or some such sentinel in such a case? > > > > > > The doc just says this : > > > > > > 62 /** > > > 63 * Returns the total amount of swap space in bytes. > > > 64 * > > > 65 * @return the total amount of swap space in bytes. > > > 66 */ > > > 67 public long getTotalSwapSpaceSize(); > > > > > > ( however it returned previously 0 before the latest changes in the container support code came in ). > > > > That sounds like a bug was introduced to me then. > > > > > Regarding the getFreeSwapSpaceSize issue , Jie provided a change to src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java > > > Because of this I could remove the try-construct . > > > > > > New webrev : > > > http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.3/ > > > > I'm quite confused by your changes. Your first version simply adjusted > > the test to check for the warning. But now you are also changing the > > main logic. I can't assess the validity of what you are doing sorry. > > > > I've cc'd Severin and Bob as I think they need to evaluate what the real > > problem is here and when it was introduced. > > > > David > > ---- > > > > > Best regards, Matthias > > > > > > > > > > > > -----Original Message----- > > > From: David Holmes > > > Sent: Donnerstag, 14. Mai 2020 07:02 > > > To: Baesken, Matthias ; 'hotspot-dev at openjdk.java.net' > > > Cc: jiefu(??) > > > Subject: Re: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > > > > > On 14/05/2020 12:28 am, Baesken, Matthias wrote: > > >> Hello, here is a new webrev removing the comment : > > >> > > >> http://cr.openjdk.java.net/~mbaesken/webrevs/8244500.1/ > > >> > > >> It removes the comment , and it also handles 0 for getFreeSwapSpaceSize > > >> ( this can occur after JDK-8242480 : Negative value may be returned by getFreeSwapSpaceSize() in the docker was pushed recently ). > > > > > > Sorry Matthias but I still see this partly as a bug in the underlying > > > container support code. If you get the warning then what does > > > getTotalSwapSpaceSize() return? Surely it must be specified to return 0 > > > or some such sentinel in such a case? > > > > > > Other comments: > > > > > > ! try { > > > ! > > > out.shouldMatch("OperatingSystemMXBean\\.getFreeSwapSpaceSize: > > > [1-9][0-9]+"); > > > ! } catch(RuntimeException ex) { // allow 0, see 8242480 > > > ! > > > out.shouldContain("OperatingSystemMXBean.getFreeSwapSpaceSize: 0"); > > > ! } > > > > > > can't you just change the original regex to match >= 0? > > > > > > Style nit: avoid historical comments like "see 8242480" and "since > > > 8231111 ...". > > > > > > Thanks, > > > David > > > > > >> Best regards, Matthias > > >> > > >> -----Original Message----- > > >> From: Baesken, Matthias > > >> Sent: Mittwoch, 13. Mai 2020 10:11 > > >> To: 'David Holmes' ; 'hotspot-dev at openjdk.java.net' > > >> Subject: RE: RFR [XS]: 8244500: jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java > > >> > > >> > > >>> But according to the comment immediately before your changes: > > >>> > > >>> // in case of warnings like : "Your kernel does not support swap limit > > >>> capabilities or the cgroup is not mounted. Memory limited without swap." > > >>> // the getTotalSwapSpaceSize does not return the expected result, but 0 > > >>> > > >>> we should be returning zero. So this seems to be a bug in the > > >>> implementation not in the test. > > >> > > >> > > >> Hi David , I think in case of "Your kernel does not support swap limit capabilities ... " we just do not get the values > > >> that are normally expected. > > >> Previously we indeed got "0" . > > >> Probably the comment should be deleted . > > >> > > >> Best regards, Matthias > > >> > > > > > > From david.holmes at oracle.com Fri May 22 13:59:01 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 22 May 2020 23:59:01 +1000 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> Message-ID: <49e9f865-639b-47cf-b02b-1fcada91c43c@oracle.com> On 22/05/2020 6:23 pm, Kim Barrett wrote: >> On May 21, 2020, at 10:39 PM, David Holmes wrote: >> >> Hi Kim, >> >> Not following the details 100% but the general gist seems okay, except for a couple of queries below. >> >> On 22/05/2020 8:31 am, Kim Barrett wrote: >>> Please review this change to eliminate volatile qualification in the >>> member functions and parameters of the oop class and its subclasses. >>> Removed volatile qualified member functions and parameters from class >>> oop and its subclasses (instanceOop, arrayOop, objArrayOop, typeArrayOop). >>> Changed the few remaining volatile oop accesses to instead use relaxed >>> atomics. >>> Changed constructors for MemoryManager and MemoryPool to use >>> initializer list, so we don't need to use Atomic::store to initialize >>> an oop-derived member. >> >> What is the connection between using an initializer-list and not needing the Atomic::Store? > > The volatile oop-derived member (instanceOop in both cases) needs to > be initialized. If done in an initializer list, it's at construction > time for the member object, where volatile isn't relevant. If done > later (for example, in the containing object's constructor body, the > member is now volatile and so involved a volatile assignment, which > would now need to be an Atomic::store. I still don't see any distinction there with regards to why we use volatile or equivalently Atomic::store. >>> Eliminated ShenandoahVerifierTask's volatile-qualified assignment >>> operator, which isn't used. >>> Some tidying up oop class. Cleaned up constructors, eliminating public >>> set_obj(). Use const-ref rather than by-value parameters for comparisons, >>> to avoid unnecessary copies that go through register/deregister. >>> CR: >>> https://bugs.openjdk.java.net/browse/JDK-8245594 >>> Webrev: >>> https://cr.openjdk.java.net/~kbarrett/8245594/open.00/ >> >> In the following the referenced oops still retain their volatile qualifier - should it not now be removed? >> >> - src/hotspot/share/runtime/thread.hpp >> _exception_oop is still marked volatile >> - src/hotspot/share/services/memoryManager.hpp >> _memory_mgr_obj is still marked volatile >> - src/hotspot/share/services/memoryPool.hpp >> _memory_pool_obj is still marked volatile > > Access to these objects is via Atomic operations, so they should still > be volatile qualified per current convention. The current convention allows for naked loads and stores on volatile variables. When we only have Atomic load/store plus other atomic ops then the "convention" no longer requires the variable to be volatile. Which in turn would I assume undo the initializer-list argument. David ----- > It might be worth examining their use of Atomic. The MemoryManager > case seems a bit odd, but I haven't studied it carefully. I haven't > looked at the others yet. I think doing so is outside the scope of > this change. > >> >> Thanks, >> David >> >>> Testing: >>> mach5 tier1-5 >>> Local (linux-x64) hotspot:tier1 with Shenandoah enabled. > > From sean.mullan at oracle.com Fri May 22 14:04:31 2020 From: sean.mullan at oracle.com (Sean Mullan) Date: Fri, 22 May 2020 10:04:31 -0400 Subject: RFR[15] JDK-8243114: Implement montgomery{Multiply, Square}intrinsics on Windows In-Reply-To: <725e8e5a-71b5-7d6e-4d3a-7c799a52ab56@redhat.com> References: <725e8e5a-71b5-7d6e-4d3a-7c799a52ab56@redhat.com> Message-ID: <2442a046-6640-f759-6330-e971405f0c78@oracle.com> Cross-posting to hotspot-dev for additional review since the code changes are in hotspot. --Sean On 5/21/20 1:24 PM, Simon Tooke wrote: > Hello, > > I'd like to request a review for: > > JBS: https://bugs.openjdk.java.net/browse/JDK-8243114 > > Webrev: http://cr.openjdk.java.net/~stooke/webrevs/jdk-8243114-jdk/00/00/ > > This change implements the given intrinsics on Windows. > > The Windows toolchain has no 128 bit integer types, and no inline asm > (on x64 and Arm).? In addition, 'long' is 4 bytes, not 8, as it is with > gcc.? This patch had to change some of the linux implementation to > account for these limitations. > > My gratitude for Andrew Haley for doing the heavy lifting at the core of > this patch. > > The patch, if accepted, will be offered to 11u as a potential backport. > The changes apply cleanly modulo some line number changes. > > As for the speedup, this test case: > > BigInteger base = BigInteger.ONE.shiftLeft(1024); > long count = LongStream.rangeClosed(2, 100_000) > ??? .mapToObj(n -> BigInteger.valueOf(n).add(base)) > ??? .filter(i -> i.isProbablePrime(50)) > ??? .count(); > > goes from 1 minute 20 seconds down to about 35 seconds om my VM, over > multiple runs.? As is the case on other platforms where the intrinsics > are supported, they will be enabled by default on Windows. > > Thank you for your time, > > -Simon Tooke > > > Principle Software Engineer, > > Red Hat Canada > > > From coleen.phillimore at oracle.com Fri May 22 16:10:11 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 22 May 2020 12:10:11 -0400 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> Message-ID: <1bbd8c26-add8-a4e7-54b8-b2aa5eadf391@oracle.com> On 5/21/20 6:31 PM, Kim Barrett wrote: > Please review this change to eliminate volatile qualification in the > member functions and parameters of the oop class and its subclasses. This is nice.? I thought more code would fail to compile if the volatile versions were removed, which is why they were added in the first place.? It does not look like this is the case since this changeset is small. > > Removed volatile qualified member functions and parameters from class > oop and its subclasses (instanceOop, arrayOop, objArrayOop, typeArrayOop). > Changed the few remaining volatile oop accesses to instead use relaxed > atomics. "relaxed atomics" means Atomic::load(&_field) ? So if the field is declared "volatile", will the compiler give you an error if you just store a value to it without Atomic::store(&_field) ?? But it will not give you an error if you fail to load the value without Atomic::load(&_field).? But keeping the "volatile" keyword on declarations is a good way to visually signal that this variable needs special access? Thanks for cleaning the oop class up. Coleen > > Changed constructors for MemoryManager and MemoryPool to use > initializer list, so we don't need to use Atomic::store to initialize > an oop-derived member. > > Eliminated ShenandoahVerifierTask's volatile-qualified assignment > operator, which isn't used. > > Some tidying up oop class. Cleaned up constructors, eliminating public > set_obj(). Use const-ref rather than by-value parameters for comparisons, > to avoid unnecessary copies that go through register/deregister. > > CR: > https://bugs.openjdk.java.net/browse/JDK-8245594 > > Webrev: > https://cr.openjdk.java.net/~kbarrett/8245594/open.00/ > > Testing: > mach5 tier1-5 > Local (linux-x64) hotspot:tier1 with Shenandoah enabled. > From paul.sandoz at oracle.com Fri May 22 16:12:03 2020 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 22 May 2020 09:12:03 -0700 Subject: [aarch64-port-dev ] RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes In-Reply-To: <275eb57c-51c0-675e-c32a-91b198023559@redhat.com> References: <275eb57c-51c0-675e-c32a-91b198023559@redhat.com> Message-ID: <719F9169-ABC4-408E-B732-F1BD9A84337F@oracle.com> HI Andrew, Thanks for looking. I am not terribly familiar with the AArch64 code, but I would note the Vector API comes with a bunch of unit tests should exercise the code gen, just not as directly as I presume you would like. To what extent do you feel we can follow up with additional issues and fix them after the initial integration? Paul. > On May 22, 2020, at 6:00 AM, Andrew Haley wrote: > > On 5/19/20 11:55 AM, Yang Zhang wrote: >> Following up on review requests of API [0], Java implementation and >> test [1], General Hotspot changes[2] for Vector API and x86 backend >> changes [3]. Here's a request for review of AArch64 backend changes >> required for supporting the Vector API: >> >> JEP: https://openjdk.java.net/jeps/338 >> JBS: https://bugs.openjdk.java.net/browse/JDK-8223347 >> Webrev: http://cr.openjdk.java.net/~yzhang/vectorapi/vectorapi.rfr/aarch64_webrev/webrev.01/ >> >> Complete implementation resides in vector-unstable branch of panama/dev >> repository [4]. > > This looks great, and it's very impressive. Unfortunately, there are few > of us sufficiently knowledgeable about Panama to review it in the detail > that perhaps it deserves. I'm happy with it. > > However, we need tests for the new assembly instructions, so please add some > to aarch64_asmtest.py and update macroassemler.cpp. > > Also, aarch64.ad is getting to be far too large, and perhaps all the vector > stuff should be moved into a new file. Thanks. > > -- > Andrew Haley (he/him) > Java Platform Lead Engineer > Red Hat UK Ltd. > https://keybase.io/andrewhaley > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 > From thomas.stuefe at gmail.com Fri May 22 17:02:13 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 22 May 2020 19:02:13 +0200 Subject: RFR(M): 8243392: Remodel CDS/Metaspace storage reservation In-Reply-To: <2cb02a62-954d-c261-2553-02b7e2ab4efb@oracle.com> References: <525df209-5bb6-47b5-474a-f0c0c44ca65e@oracle.com> <4e911098-ebdc-3222-5b6d-970a612b3884@oracle.com> <8bf736b3-9ab1-86b7-fd22-230e449b1b9f@oracle.com> <2cb02a62-954d-c261-2553-02b7e2ab4efb@oracle.com> Message-ID: Thank you Coleen! I wait until tomorrow, if I do not hear any objections I will push. ..Thomas On Fri, May 22, 2020 at 2:17 PM wrote: > > > On 5/21/20 2:26 AM, Thomas St?fe wrote: > > Hi all, > > tests went through successfully at SAP two nights in a row. Ioi gave his > okay, and I would like to push today or tomorrow. Any remaining objections`? > > > None from me. Ship it! > Coleen > > > The current, final version is: > > full: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ > delta: > http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ > > Thank you, > > Thomas > > On Wed, May 20, 2020 at 1:43 AM Ioi Lam wrote: > >> Hi Thomas, >> >> The testing took a little longer than I expected. I've run your previous >> version: >> >> >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ >> >> In our CI tiers 1/4. No failures. >> >> I didn't run your latest version. I would suggesting testing it with your >> own CI as well as in jdk-submit before pushing (after you get OKs from >> other reviewers). >> >> Thumbs up. >> - Ioi >> >> >> On 5/19/20 7:40 AM, Thomas St?fe wrote: >> >> Hi guys, >> >> new webrev: >> >> full: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.04/webrev/ >> delta: >> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev_delta.04/webrev/ >> >> Mostly format changes, breaking up at column 80 as Andrew requested. >> >> The only changes which matter is the removal of the unnecessary include >> in arguments.cpp as Coleen suggested; and to not validate SharedBaseAddress >> anymore at runtime, as Ioi requested. >> >> Also rebased to head. >> >> Thanks, Thomas >> >> >> >> >> On Tue, May 19, 2020 at 6:38 AM Ioi Lam wrote: >> >>> >>> >>> On 5/18/20 7:37 AM, Thomas St?fe wrote: >>> >>> Hi all, >>> >>> fourth webrev: >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.03/webrev/ >>> >>> I rebased atop of >>> >>> changeset: 59325:1ba9a9a3f948 >>> summary: 8244433: Remove saving of RSP in Assembler::pusha_uncached() >>> >>> so, atop of "8241825: Make compressed oops and compressed class pointers >>> independent (x86_64, PPC, S390)". >>> >>> The changes are very small this time, see notes below: >>> >>> @Ioi: >>> >>> >Hi Thomas, >>> >>> >I am running your patch in out CI pipeline now. Some comments: >>> >>> >[2] Does JDK-8243535 exist with the current jdk/jdk repo? >>> >>> I assigned JDK-8243535 to me and I'm working on a patch. But I would >>> like to keep that separate from this work, since this patch is already >>> large. And I would like to fix JDK-8243535 after this remodel patch has >>> been pushed, which makes some minor things easier. That would leave a small >>> time window where NMT does mis-track class space as shared class space, but >>> that is not a big issue, it does not even disturb any test. Which in itself >>> is something we may want to fix, come to think about it. >>> >>> >>> > [3] I think this can be put in header file. >>> > >>> > bool Metaspace::class_space_is_initialized() { >>> > return _class_space_list != NULL; >>> > } >>> >>> Done. >>> >>> > [7] argument.cpp checks this: >>> > >>> > if (UseSharedSpaces || DumpSharedSpaces) { >>> > if >>> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >>> > log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid >>> for this platform, option will be ignored.", >>> > p2i((address)SharedBaseAddress)); >>> > FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >>> > } >>> > } >>> > >>> > but initialize_dumptime_shared_and_meta_spaces aligns up the >>> SharedBaseAddress before doing the assert. >>> > >>> > void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >>> > assert(DumpSharedSpaces, "should be called for dump time only"); >>> > >>> > const size_t reserve_alignment = >>> MetaspaceShared::reserved_space_alignment(); >>> > char* shared_base = (char*)align_up((char*)SharedBaseAddress, >>> reserve_alignment); >>> > >>> > #ifdef _LP64 >>> > >>> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >>> "Sanity"); >>> > >>> > So theoretically shared_base may no longer be is_valid_base after the >>> align_up. It probably won't happen, but I think the code will be > much >>> clearer tofirst to align_up, then check for is_valid_base and reset to >>> default, and finally assert. >>> >>> The problem with that is that at the time this coding runs os::init() >>> did not yet run and we do not know yet allocation granularity. >>> >>> So I changed verification of SharedBaseAddress: I added the alignment as >>> you suggested and moved it to a later time in VM initialization, to CDS >>> initialization. >>> >>> >>> Is the call to check_SharedBaseAddress() necessary inside >>> MetaspaceShared::initialize_runtime_shared_and_meta_spaces()? >>> SharedBaseAddress is currently ignored at run time. Does your patch change >>> that behavior? >>> >>> >[8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >>> following block can be simplified: >>> > >>> > if (!_shared_rs.is_reserved()) { >>> > // Get a reserved space anywhere if attaching at the >>> SharedBaseAddress fails: >>> > if (UseCompressedClassPointers) { >>> > // If we need a compressed class space too, let the platform >>> handle the reservation. >>> > _shared_rs = >>> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >>> > } else { >>> > // anywhere is fine. >>> > _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >>> large */, (char*)NULL); >>> > } >>> > } >>> > >>> > ... if you change the declaration to: >>> > >>> > static ReservedSpace >>> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >>> ReservedSpace();}); >>> >>> I'd like to avoid keep the method LP64 only, but I rewrote the section >>> somewhat to be more clearer. >>> >>> The new code looks good. >>> >>> > [9] I think the #ifdef _LP64 is not necessary: >>> > >>> > #ifdef _LP64 >>> > // Some sanity checks after reserving address spaces for archives >>> and class space. >>> > assert(archive_space_rs.is_reserved(), "Sanity"); >>> > if (Metaspace::using_class_space()) { >>> > // Class space must closely follow the archive space. Both >>> spaces must be aligned correctly. >>> > assert(class_space_rs.is_reserved(), "A class space should have >>> been reserved"); >>> > assert(class_space_rs.base() >= archive_space_rs.end(), "class >>> space should follow the cds archive space"); >>> > assert(is_aligned(archive_space_rs.base(), >>> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >>> > assert(is_aligned(class_space_rs.base(), >>> Metaspace::reserve_alignment()), "class space misaligned"); >>> > } >>> > #endif >>> >>> I removed the #ifdef, but replaced it with an #ifdef ASSERT. Which is >>> also not needed, technically, but makes it clear that this section is >>> debugging checks only. >>> >>> >>> OK. >>> >>> The rest of the code looks good to me. I am going to run your latest >>> patch in our CI and will report the results. >>> >>> Thanks >>> - Ioi >>> >>> ---- >>> >>> @Coleen: >>> >>> > >>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/src/hotspot/share/memory/metaspace/ >>> >virtualSpaceList.cpp.udiff.html >>> > >>> >Why did you make this change? Shouldn't the caller align it? >>> >>> >>> You are right, I removed the assert. >>> >>> --- >>> >>> Thanks all for your review work, >>> >>> ..Thomas >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Fri, May 8, 2020 at 7:37 AM Ioi Lam wrote: >>> >>>> Hi Thomas, >>>> >>>> I am running your patch in out CI pipeline now. Some comments: >>>> >>>> [1] Do we still need to use _class_space_list? It looks like we just >>>> have a single range. Is _class_space_list->current_virtual_space() always >>>> the same VS? >>>> >>>> if (_class_space_list != NULL) { >>>> address base = >>>> (address)_class_space_list->current_virtual_space()->bottom(); >>>> address top = base + compressed_class_space_size(); >>>> st->print("Compressed class space mapped at: " PTR_FORMAT "-" >>>> PTR_FORMAT ", size: " SIZE_FORMAT, >>>> p2i(base), p2i(top), top - base); >>>> >>>> [2] Does JDK-8243535 exist with the current jdk/jdk repo? >>>> >>>> // This does currently not work because rs may be the result of a >>>> split operation >>>> // and NMT seems not to be able to handle splits. >>>> // See JDK-8243535. >>>> // MemTracker::record_virtual_memory_type((address)rs.base(), >>>> mtClass); >>>> >>>> >>>> [3] I think this can be put in header file. >>>> >>>> bool Metaspace::class_space_is_initialized() { >>>> return _class_space_list != NULL; >>>> } >>>> >>>> [4] Why does the CCS need to follow UseLargePagesInMetaspace? This is >>>> the reason for the >>>> gap in the following chart: >>>> >>>> // We do this by reserving space for the ccs behind the archives. >>>> Note however that >>>> // ccs follows a different alignment >>>> (Metaspace::reserve_alignment), so there may >>>> // be a gap between ccs and cds. >>>> // We use a similar layout at runtime, see >>>> reserve_address_space_for_archives(). >>>> // >>>> // +-- SharedBaseAddress (default = >>>> 0x800000000) >>>> // v >>>> // +-..---------+---------+ ... >>>> +----+----+----+--------+--------------------+ >>>> // | Heap | Archive | | MC | RW | RO | [gap] | class >>>> space | >>>> // +-..---------+---------+ ... >>>> +----+----+----+--------+--------------------+ >>>> // |<-- MaxHeapSize -->| |<-- UnscaledClassSpaceMax = 4GB >>>> -->| >>>> >>>> >>>> _reserve_alignment is determined here: >>>> >>>> void Metaspace::ergo_initialize() { >>>> if (DumpSharedSpaces) { >>>> // Using large pages when dumping the shared archive is currently >>>> not implemented. >>>> FLAG_SET_ERGO(UseLargePagesInMetaspace, false); >>>> } >>>> >>>> size_t page_size = os::vm_page_size(); >>>> if (UseLargePages && UseLargePagesInMetaspace) { >>>> page_size = os::large_page_size(); >>>> } >>>> >>>> _commit_alignment = page_size; >>>> _reserve_alignment = MAX2(page_size, >>>> (size_t)os::vm_allocation_granularity()); >>>> >>>> But when CDS is enabled, the RS is reserved without large pages, so it >>>> should be possible to incrementally commit the CCS using just >>>> os::vm_allocation_granularity(). >>>> >>>> [5] I see the aarch64 code in >>>> Metaspace::reserve_address_space_for_compressed_classes has been changed. >>>> Is this necessary for this RFE, or is it a separate improvement that can be >>>> done in a different RFE? >>>> >>>> [6] For AARCH64, should we skip the part that reserves it right above >>>> the heap? Or does AARCH64 always allocate the heap such that it's a >>>> preferable address? >>>> >>>> // case (b) >>>> ReservedSpace rs; >>>> >>>> // Try right above the Java heap... >>>> address base = align_up(CompressedOops::end(), >>>> Metaspace::reserve_alignment()); >>>> assert(base != NULL, "Sanity"); >>>> >>>> const size_t size = align_up(CompressedClassSpaceSize, >>>> Metaspace::reserve_alignment()); >>>> if (CompressedKlassPointers::is_valid_base(base)) { >>>> rs = ReservedSpace(size, Metaspace::reserve_alignment(), false /* >>>> large */, (char*)base); >>>> } >>>> >>>> // ...failing that, reserve anywhere, but let platform do optimized >>>> placement: >>>> if (!rs.is_reserved()) { >>>> rs = >>>> Metaspace::reserve_address_space_for_compressed_classes(size); >>>> } >>>> >>>> [7] argument.cpp checks this: >>>> >>>> if (UseSharedSpaces || DumpSharedSpaces) { >>>> if >>>> (!CompressedKlassPointers::is_valid_base((address)SharedBaseAddress)) { >>>> log_warning(cds)("SharedBaseAddress=" PTR_FORMAT " is invalid for >>>> this platform, option will be ignored.", >>>> p2i((address)SharedBaseAddress)); >>>> FLAG_SET_DEFAULT(SharedBaseAddress, default_SharedBaseAddress()); >>>> } >>>> } >>>> >>>> but initialize_dumptime_shared_and_meta_spaces aligns up the >>>> SharedBaseAddress before doing the assert. >>>> >>>> void MetaspaceShared::initialize_dumptime_shared_and_meta_spaces() { >>>> assert(DumpSharedSpaces, "should be called for dump time only"); >>>> >>>> const size_t reserve_alignment = >>>> MetaspaceShared::reserved_space_alignment(); >>>> char* shared_base = (char*)align_up((char*)SharedBaseAddress, >>>> reserve_alignment); >>>> >>>> #ifdef _LP64 >>>> >>>> assert(CompressedKlassPointers::is_valid_base((address)shared_base), >>>> "Sanity"); >>>> >>>> So theoretically shared_base may no longer be is_valid_base after the >>>> align_up. It probably won't happen, but I think the code will be much >>>> clearer tofirst to align_up, then check for is_valid_base and reset to >>>> default, and finally assert. >>>> >>>> [8] In MetaspaceShared::initialize_dumptime_shared_and_meta_spaces, the >>>> following block can be simplified: >>>> >>>> if (!_shared_rs.is_reserved()) { >>>> // Get a reserved space anywhere if attaching at the >>>> SharedBaseAddress fails: >>>> if (UseCompressedClassPointers) { >>>> // If we need a compressed class space too, let the platform >>>> handle the reservation. >>>> _shared_rs = >>>> Metaspace::reserve_address_space_for_compressed_classes(cds_total); >>>> } else { >>>> // anywhere is fine. >>>> _shared_rs = ReservedSpace(cds_total, reserve_alignment, false /* >>>> large */, (char*)NULL); >>>> } >>>> } >>>> >>>> ... if you change the declaration to: >>>> >>>> static ReservedSpace >>>> reserve_address_space_for_compressed_classes(size_t size) NOT_LP64({ return >>>> ReservedSpace();}); >>>> >>>> [9] I think the #ifdef _LP64 is not necessary: >>>> >>>> #ifdef _LP64 >>>> // Some sanity checks after reserving address spaces for archives >>>> and class space. >>>> assert(archive_space_rs.is_reserved(), "Sanity"); >>>> if (Metaspace::using_class_space()) { >>>> // Class space must closely follow the archive space. Both spaces >>>> must be aligned correctly. >>>> assert(class_space_rs.is_reserved(), "A class space should have >>>> been reserved"); >>>> assert(class_space_rs.base() >= archive_space_rs.end(), "class >>>> space should follow the cds archive space"); >>>> assert(is_aligned(archive_space_rs.base(), >>>> MetaspaceShared::reserved_space_alignment()), "Archive space misaligned"); >>>> assert(is_aligned(class_space_rs.base(), >>>> Metaspace::reserve_alignment()), "class space misaligned"); >>>> } >>>> #endif >>>> >>>> The rest of the code looks OK to me, but I may take a look at it again >>>> after getting more sleep :-) >>>> >>>> Thanks >>>> - Ioi >>>> >>>> >>>> >>>> >>>> On 5/7/20 7:21 AM, Thomas St?fe wrote: >>>> >>>> Hi all, >>>> >>>> please take a look at the third iteration of this change: >>>> >>>> >>>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.02/webrev/ >>>> >>>> Changes in this version: >>>> >>>> - at the request of Coleen (sorry I did not see your post earlier) I >>>> removed all platform dependent files and put the platform dependent coding >>>> back to metaspace.cpp and compressedOops.cpp to ease the review pain. I >>>> used plain platform defines though (#ifdef AARCH64) instead of hiding them >>>> behind another macro to make things clearer. Note that I still intent to >>>> put this code away into the platform corners but will do so in a follow up >>>> RFE. >>>> >>>> - I reinstated, in a fashion, the special handling of reservations on >>>> Windows. On all platforms we reserve address space to map the archive files >>>> in with a subsequent mapping operation. However, on Windows, we cannot use >>>> MapViewOfFile() into an existing mapping. So I remove the mapping again >>>> before mapping the archives - see comment in code for details. >>>> >>>> All CI tests at SAP run through without problems, including on Win64 >>>> and aarch64, but I would be happy if others were to run test too. >>>> >>>> Thank you, Thomas >>>> >>>> On Tue, Apr 28, 2020 at 4:54 PM Thomas St?fe >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Could I have reviews for the following proposal of reworking cds/class >>>>> space reservation? >>>>> >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8243392 >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~stuefe/webrevs/rework-cds-ccs-reservation/webrev.00/webrev/ >>>>> >>>>> (Many thanks to Ioi Lam for so patiently explaining CDS internals to >>>>> me, and to Andrew Haley and Nick Gasson for help with aarch64!) >>>>> >>>>> Reservation of the compressed class space is needlessly complicated >>>>> and has some minor issues. It can be simplified and made clearer. >>>>> >>>>> The complexity stems from the fact that this area lives at the >>>>> intersection of two to three sub systems, depending on how one counts. >>>>> Metaspace, CDS, and the platform which may or may not its own view of how >>>>> to reserve class space. And this code has been growing organically over >>>>> time. >>>>> >>>>> One small example: >>>>> >>>>> ReservedSpace Metaspace::reserve_preferred_space(size_t size, size_t >>>>> alignment, >>>>> bool large_pages, >>>>> char *requested_addr, >>>>> bool >>>>> use_requested_addr) >>>>> >>>>> which I spent hours decoding, resulting in a very confused mail to >>>>> hs-runtime and aarch64-port-dev [2]. >>>>> >>>>> This patch attempts to simplify cds and metaspace setup a bit; to >>>>> comment implicit knowledge which is not immediately clear; to cleanly >>>>> abstract platform concerns like optimized class space placement; and to >>>>> disentangle cds from metaspace to solve issues which may bite us later with >>>>> Elastic Metaspace [4]. >>>>> >>>>> --- >>>>> >>>>> The main change is the reworked reservation mechanism. This is based >>>>> on Ioi's proposal [5]. >>>>> >>>>> When reserving class space, three things must happen: >>>>> >>>>> 1) reservation of the space obviously. If cds is active that space >>>>> must be in the vicinity of cds archives to be covered by compressed class >>>>> pointer encoding. >>>>> 2) setting up the internal Metaspace structures atop of that space >>>>> 3) setting up compressed class pointer encoding. >>>>> >>>>> In its current form, Metaspace may or may not do some or all of that >>>>> in one function >>>>> (Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>>>> metaspace_rs, char* requested_addr, address cds_base);) - if cds is active, >>>>> it will reserve the space for Metaspace and hand it in, otherwise it will >>>>> create it itself. >>>>> >>>>> When discussing this in [2], Ioi proposed to move the reservation of >>>>> the class space completely out of Metaspace and make it a responsibility of >>>>> the caller always. This would reduce some complexity, and this patch >>>>> follows the proposal. >>>>> >>>>> I removed >>>>> Metaspace::allocate_metaspace_compressed_klass_ptrs(ReservedSpace >>>>> metaspace_rs, char* requested_addr, address cds_base); and all its sub >>>>> functions. >>>>> >>>>> (1) now has to always be done outside - a ReservedSpace for class >>>>> space has to be provided by the caller. However, Metaspace now offers a >>>>> utility function for reserving space at a "nice" location, and explicitly >>>>> doing nothing else: >>>>> >>>>> ReservedSpace >>>>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>>>> >>>>> this function can be redefined on a platform level for platform >>>>> optimized reservation, see below for details. >>>>> >>>>> (2) is taken care of by a new function, >>>>> Metaspace::initialize_class_space(ReservedSpace rs) >>>>> >>>>> (3) is taken care of a new function >>>>> CompressedKlassPointers::initialize(), see below for details. >>>>> >>>>> >>>>> So, class space now is set up three explicit steps: >>>>> >>>>> - First, reserve a suitable space by however means you want. For >>>>> convenience you may use >>>>> Metaspace::reserve_address_space_for_compressed_classes(), or you may roll >>>>> your own reservation. >>>>> - Next, tell Metaspace to use that range as backing storage for class >>>>> space: Metaspace::initialize_class_space(ReservedSpace rs) >>>>> - Finally, set up encoding. Encoding is independent from the concept >>>>> of a ReservedSpace, it just gets an address range, see below for details. >>>>> >>>>> Separating these steps and moving them out of the responsibility of >>>>> Metaspace makes this whole thing more flexible; it also removes unnecessary >>>>> knowledge (e.g. Metaspace does not need to know anything about either ccp >>>>> encoding or cds). >>>>> >>>>> --- >>>>> >>>>> How it comes together: >>>>> >>>>> If CDS is off, we just reserve a space using >>>>> Metaspace::reserve_address_space_for_compressed_classes(), initialize it >>>>> with Metaspace::initialize_class_space(ReservedSpace rs), then set up >>>>> compressed class pointer encoding covering the range of this class space. >>>>> >>>>> If CDS is on (dump time), we reserve large 4G space, either at >>>>> SharedBaseAddress or using >>>>> Metaspace::reserve_address_space_for_compressed_classes(); we then split >>>>> that into 3G archive space and 1G class space; we set up that space with >>>>> Metaspace as class space; then we set up compressed class pointer encoding >>>>> covering both archive space and cds. >>>>> >>>>> If CDS is on (run time), we reserve a large space, split it into >>>>> archive space (large enough to hold both archives) and class space, then >>>>> basically proceed as above. >>>>> >>>>> Note that this is almost exactly how things worked before (modulo some >>>>> minor fixes, e.g. alignment issues), only the code is reformed and made >>>>> more explicit. >>>>> >>>>> --- >>>>> >>>>> I moved compressed class pointer setup over to CompressedKlassPointers >>>>> and changed the interface: >>>>> >>>>> -void Metaspace::set_narrow_klass_base_and_shift(ReservedSpace >>>>> metaspace_rs, address cds_base) >>>>> +void CompressedKlassPointers::initialize(address addr, size_t len); >>>>> >>>>> Instead of feeding it a single ReservedSpace, which is supposed to >>>>> represent class space, and an optional alternate base if cds is on, now we >>>>> give it just an numeric address range. That range marks the limits to where >>>>> Klass structures are to be expected, and is the implicit promise that >>>>> outside that range no Klass structures will exist, so encoding has to cover >>>>> only this range. >>>>> >>>>> This range may contain just the class space; or class space+cds; or >>>>> whatever allocation scheme we come up with in the future. Encoding does not >>>>> really care how the memory is organized as long as the input range covers >>>>> all possible Klass locations. That way we remove knowledge about class >>>>> space/cds from compressed class pointer encoding. >>>>> >>>>> Moving it away from metaspace.cpp into the CompressedKlassPointers >>>>> class also mirrors CompressedOops::initialize(). >>>>> >>>>> --- >>>>> >>>>> I renamed _narrow_klass_range to just _range, because strictly >>>>> speaking this is the range un-narrow Klass pointers can have. >>>>> >>>>> As for the implementation of >>>>> CompressedKlassPointers::initialize(address addr, size_t len), I mimicked >>>>> very closely what happened before, so there should be almost no >>>>> differences. Since "almost no differences" sounds scary :) here are the >>>>> differences: >>>>> >>>>> - When CDS is active (dump or run time) we now always, >>>>> unconditionally, set the encoding range to 4G. This fixes a theoretical bug >>>>> discussed on aarch64-port-dev [1]. >>>>> >>>>> - When CDS is not active, we set the encoding range to the minimum >>>>> required length. Before, it was left at its default value of 4G. >>>>> >>>>> Both differences only affect aarch64, since they are currently the >>>>> only one using the range field in CompressedKlassPointers. >>>>> >>>>> I wanted to add an assert somewhere to test encoding of the very last >>>>> address of the CompressedKlassPointers range, again to prevent errors like >>>>> [3]. But I did not come up with a good place for this assert which would >>>>> cover also the encoding done by C1/C2. >>>>> >>>>> For the same reason I thought about introducing a mode where Klass >>>>> structures would be allocated in reverse order, starting at the end of the >>>>> ccs, but again left it out as too big a change. >>>>> >>>>> --- >>>>> >>>>> OS abstraction: platforms may have restrictions of what constitutes a >>>>> valid compressed class pointer encoding base. Or if not, they may have at >>>>> least preferences. There was logic like this in metaspace.cpp, which I >>>>> removed and cleanly factored out into platform dependent files, giving each >>>>> platform the option to add special logic. >>>>> >>>>> These are two new methods: >>>>> >>>>> - bool CompressedKlassPointers::is_valid_base(address p) >>>>> >>>>> to let the platform tell you whether it considers p to be a valid >>>>> encoding base. The only platform having these restrictions currently is >>>>> aarch64. >>>>> >>>>> - ReservedSpace >>>>> Metaspace::reserve_address_space_for_compressed_classes(size_t size); >>>>> >>>>> this hands over the process of allocating a range suitable for >>>>> compressed class pointer encoding to the platform. Most platforms will >>>>> allocate just anywhere, but some platforms may have a better strategy (e.g. >>>>> trying low memory first, trying only correctly aligned addresses and so >>>>> on). >>>>> >>>>> Beforehand, this coding existed in a similar form in metaspace.cpp for >>>>> aarch64 and AIX. For now, I left the AIX part out - it seems only half >>>>> done, and I want to check further if we even need it, if yes why not on >>>>> Linux ppc, and C1 does not seem to support anything other than base+offset >>>>> with shift either, but I may be mistaken. >>>>> >>>>> These two methods should give the platform enough control to implement >>>>> their own scheme for optimized class space placement without bothering any >>>>> shared code about it. >>>>> >>>>> Note about the form, I introduced two new platform dependent files, >>>>> "metaspace_.cpp" and "compressedOops_.cpp". I am not happy about >>>>> this but this seems to be what we generally do in hotspot, right? >>>>> >>>>> --- >>>>> >>>>> Metaspace reserve alignment vs cds alignment >>>>> >>>>> CDS was using Metaspace reserve alignment for CDS internal purposes. I >>>>> guess this was just a copy paste issue. It never caused problems since >>>>> Metaspace reserve alignment == page size, but that is not true anymore in >>>>> the upcoming Elastic Metaspace where reserve alignment will be larger. This >>>>> causes a number of issues. >>>>> >>>>> I separated those two cleanly. CDS now uses >>>>> os::vm_allocation_granularity. Metaspace::reserve_alignment is only used in >>>>> those two places where it is needed, when CDS creates the address space for >>>>> class space on behalf of the Metaspace. >>>>> >>>>> --- >>>>> >>>>> Windows special handling in CDS >>>>> >>>>> To simplify coding I removed the windows specific handling which left >>>>> out reservation of the archive. This was needed because windows cannot mmap >>>>> files into reserved regions. But fallback code exists in filemap.cpp for >>>>> this case which just reads in the region instead of mapping it. >>>>> >>>>> Should that turn out to be a performance problem, I will reinstate the >>>>> feature. But a simpler way would be reserve the archive and later just >>>>> before mmapping the archive file to release the archive space. That would >>>>> not only be simpler but give us the best guarantee that that address space >>>>> is actually available. But I'd be happy to leave that part out completely >>>>> if we do not see any performance problems on windows x64. >>>>> >>>>> --- >>>>> >>>>> NMT cannot deal with spaces which are split. This problem manifests in >>>>> that bookkeeping for class space is done under "Shared Classes", not >>>>> "Classes" as it should. This problem exists today too at dump time and >>>>> randomly at run time. But since I simplified the reservation, this problem >>>>> now shows up always, whether or not we map at the SharedBaseAddress. >>>>> While I could work around this problem, I'd prefer this problem to be >>>>> solved at the core, and NMT to have an option to recognize reservation >>>>> splits. So I'd rather not put a workaround for this into the patch but >>>>> leave it for fixing as a separate issue. I opened this issue to track it >>>>> [6]. >>>>> >>>>> --- >>>>> >>>>> Jtreg tests: >>>>> >>>>> I expanded the CompressedOops/CompressedClassPointers.java. I also >>>>> extended them to Windows. The tests now optionally omit strict class space >>>>> placement tests, since these tests heavily depend on ASLR and were the >>>>> reason they were excluded on Windows. However I think even without checking >>>>> for class space placement they make sense, just to see that the VM comes up >>>>> and lives with the many different settings we can run in. >>>>> >>>>> --- >>>>> >>>>> Tests: >>>>> >>>>> - I ran the patch through Oracles submit repo >>>>> - I ran tests manually for aarch64, zero, linux 32bit and windows x64 >>>>> - The whole battery of nightly tests at SAP, including ppc, ppcle and >>>>> aarch64, unfortunately excluding windows because of unrelated errors. >>>>> Windows x64 tests will be redone tonight. >>>>> >>>>> >>>>> Thank you, >>>>> >>>>> Thomas >>>>> >>>>> [1] >>>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008804.html >>>>> [2] >>>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008757.html >>>>> [3] https://bugs.openjdk.java.net/browse/JDK-8193266 >>>>> [4] https://bugs.openjdk.java.net/browse/JDK-8221173 >>>>> [5] >>>>> https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-April/008765.html >>>>> [6] https://bugs.openjdk.java.net/browse/JDK-8243535 >>>>> >>>>> >>>> >>> >> > From aph at redhat.com Fri May 22 17:40:12 2020 From: aph at redhat.com (Andrew Haley) Date: Fri, 22 May 2020 18:40:12 +0100 Subject: [aarch64-port-dev ] RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes In-Reply-To: <719F9169-ABC4-408E-B732-F1BD9A84337F@oracle.com> References: <275eb57c-51c0-675e-c32a-91b198023559@redhat.com> <719F9169-ABC4-408E-B732-F1BD9A84337F@oracle.com> Message-ID: <9a13f5df-d946-579d-4282-917dc7338dc8@redhat.com> On 5/22/20 5:12 PM, Paul Sandoz wrote: > I am not terribly familiar with the AArch64 code, but I would note > the Vector API comes with a bunch of unit tests should exercise the > code gen, just not as directly as I presume you would like. Yes, you've understood me: direct is what I want. The assembler tests are intended to make sure we generate exactly the right instructions, rather than having something painfully hard to debug later on. When a patch adds a lot of instructions to the assembler, that IMO is the right time to test that they generate correctly-encoded instructions. But yes, that can go into a follow-up patch, as long as it gets done fairly shortly. > To what extent do you feel we can follow up with additional issues > and fix them after the initial integration? We can do that. Note that after this patch, aarch64.ad is 21762 lines long. I know we don't have any hard-and-fast rule about this, but I'd rather it didn't get forgotten. Maybe I should do that one myself, but I guess I'd rather avoid the problem of version skew between the Panama repo and mainline. That'd make merging rather grim. Yang, against which repo is this webrev intended to be applied? -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From paul.sandoz at oracle.com Fri May 22 18:01:01 2020 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 22 May 2020 11:01:01 -0700 Subject: [aarch64-port-dev ] RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes In-Reply-To: <9a13f5df-d946-579d-4282-917dc7338dc8@redhat.com> References: <275eb57c-51c0-675e-c32a-91b198023559@redhat.com> <719F9169-ABC4-408E-B732-F1BD9A84337F@oracle.com> <9a13f5df-d946-579d-4282-917dc7338dc8@redhat.com> Message-ID: <09BC0693-80E0-4F87-855E-0B38A6F5EFA2@oracle.com> > On May 22, 2020, at 10:40 AM, Andrew Haley wrote: > > On 5/22/20 5:12 PM, Paul Sandoz wrote: > >> I am not terribly familiar with the AArch64 code, but I would note >> the Vector API comes with a bunch of unit tests should exercise the >> code gen, just not as directly as I presume you would like. > > Yes, you've understood me: direct is what I want. The assembler tests > are intended to make sure we generate exactly the right instructions, > rather than having something painfully hard to debug later on. When a > patch adds a lot of instructions to the assembler, that IMO is the > right time to test that they generate correctly-encoded > instructions. But yes, that can go into a follow-up patch, as long as > it gets done fairly shortly. Ok. > >> To what extent do you feel we can follow up with additional issues >> and fix them after the initial integration? > > We can do that. Note that after this patch, aarch64.ad is 21762 lines > long. I know we don't have any hard-and-fast rule about this, but I'd > rather it didn't get forgotten. We have made changes similar in spirit to the x64 ad file (reducing in size at least), so I think it reasonable request before integration to reduce the cognitive and maintenance burden. (FWIW I don?t know to what extent some functionality is utilized by the auto vectorizer and whether that impacts its location or not.) > Maybe I should do that one myself, but > I guess I'd rather avoid the problem of version skew between the > Panama repo and mainline. That'd make merging rather grim. > > Yang, against which repo is this webrev intended to be applied? > I cannot speak for Yang but we have been generating webrevs from the Panama dev repo, branch vector-unstable (unfortunate name I know!) and doing: hg diff -r default Where the default is regularly, but manually, pulled from jdk/jdk. More specifically: - code is generally pushed to the vectorIntrinsics branch - on a manual but regular basis vectorIntrinsics is synced up to jdk/jdk (via the default branch). - on a manual but regular basis vector-unstable is synced with vectorIntrinsics - occasionally there are fixes pushed directly to vector-unstable for the purposes of integration (e.g. removal of the perf test or the x64 SVML stubs). Hth, Paul. From igor.ignatyev at oracle.com Fri May 22 18:12:38 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Fri, 22 May 2020 11:12:38 -0700 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest Message-ID: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ > 132 lines changed: 80 ins; 36 del; 16 mod http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ > 57482 lines changed: 80 ins; 57385 del; 17 mod; Hi all, could you please review this small (if you ignore removal part) patch which removes in-tree copy of gtest (test/fmw/gtest) and updates makefiles to use one provided thru an added configure option `--with-gtest`? the previously used configure option `--enable-hotspot-gtest` gets depricated. the patch also compiles gtest and gmock source code into a static library so they can be compiled w/ all warnings disabled. from JBS: > w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all compilers used by OpenJDK became supported by gtest out-of-box, so there is no need to have our copy of gtest framework (including gmock) within OpenJDK source tree. instead, it can be treated like any other external dependencies, and a pointer to the directory w/ gtest code can be passed via configure options. JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 webrevs: - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ (meaningful changes) - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ (all changes) testing: - gtest tests on {linux,windows,macosx}-x64; - tier[1-5] builds which include but not limited to linux-aarch64, linux-arm32, linux-x64-zero Thanks, -- Igor From igor.ignatyev at oracle.com Fri May 22 18:27:34 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Fri, 22 May 2020 11:27:34 -0700 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest In-Reply-To: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> References: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> Message-ID: <37FB8413-69AB-42EE-9A12-8D2AEC23B8F6@oracle.com> somehow I forgot to include this although planned: this patch does *not* change the version of gtest we use in hotspot, that's to say if you want to build/run gtest tests, the source of gtest-1.8.1 should be provided to --with-gtest (unless you use jib which would download and pass source directory --with-gtest). one can download the source bundle from [1] or clone googletest, checkout release-1.8.1[2]. [1] https://github.com/google/googletest/releases/tag/release-1.8.1 [2] git clone https://github.com/google/googletest git checkout release-1.8.1 -- Igor > On May 22, 2020, at 11:12 AM, Igor Ignatyev wrote: > > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ >> 132 lines changed: 80 ins; 36 del; 16 mod > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ >> 57482 lines changed: 80 ins; 57385 del; 17 mod; > > Hi all, > > could you please review this small (if you ignore removal part) patch which removes in-tree copy of gtest (test/fmw/gtest) and updates makefiles to use one provided thru an added configure option `--with-gtest`? the previously used configure option `--enable-hotspot-gtest` gets depricated. > > the patch also compiles gtest and gmock source code into a static library so they can be compiled w/ all warnings disabled. > > from JBS: >> w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all compilers used by OpenJDK became supported by gtest out-of-box, so there is no need to have our copy of gtest framework (including gmock) within OpenJDK source tree. instead, it can be treated like any other external dependencies, and a pointer to the directory w/ gtest code can be passed via configure options. > > > JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 > webrevs: > - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ (meaningful changes) > - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ (all changes) > testing: > - gtest tests on {linux,windows,macosx}-x64; > - tier[1-5] builds which include but not limited to linux-aarch64, linux-arm32, linux-x64-zero > > Thanks, > -- Igor > From kim.barrett at oracle.com Fri May 22 18:42:40 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Fri, 22 May 2020 14:42:40 -0400 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <1bbd8c26-add8-a4e7-54b8-b2aa5eadf391@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <1bbd8c26-add8-a4e7-54b8-b2aa5eadf391@oracle.com> Message-ID: <825A3488-6B82-4464-9955-328E7C784DBF@oracle.com> > On May 22, 2020, at 12:10 PM, coleen.phillimore at oracle.com wrote: > > > > On 5/21/20 6:31 PM, Kim Barrett wrote: >> Please review this change to eliminate volatile qualification in the >> member functions and parameters of the oop class and its subclasses. > > This is nice. I thought more code would fail to compile if the volatile versions were removed, which is why they were added in the first place. It does not look like this is the case since this changeset is small. There were a number of preparatory changes. A couple, but not all, are linked as related to this one in JBS. >> >> Removed volatile qualified member functions and parameters from class >> oop and its subclasses (instanceOop, arrayOop, objArrayOop, typeArrayOop). >> Changed the few remaining volatile oop accesses to instead use relaxed >> atomics. > "relaxed atomics" means Atomic::load(&_field) ? And Atomic::store, yes. > So if the field is declared "volatile", will the compiler give you an error if you just store a value to it without Atomic::store(&_field) ? But it will not give you an error if you fail to load the value without Atomic::load(&_field). But keeping the "volatile" keyword on declarations is a good way to visually signal that this variable needs special access? It depends, but that is likely to produce a compiler error too. There are no operations in oop with a parameter that is a pointer/reference to volatile oop, so those aren't applicable. Another class could have operations that take a pointer/reference to volatile oop, but there don't appear to be any of those. So having the variable be volatile qualified provides some checking that we're using Atomic operations where we should be, at least for this class (and any other similar classes that don't have volatile qualifiers in their operations). It doesn't help for fundamental types though. Leo is working on something for that. > Thanks for cleaning the oop class up. Thanks for reviewing. From erik.joelsson at oracle.com Fri May 22 18:48:49 2020 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Fri, 22 May 2020 11:48:49 -0700 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest In-Reply-To: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> References: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> Message-ID: Hello Igor, The changes look good to me, but I think we should give Magnus a chance to look at this too. /Erik On 2020-05-22 11:12, Igor Ignatyev wrote: > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ >> 132 lines changed: 80 ins; 36 del; 16 mod > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ >> 57482 lines changed: 80 ins; 57385 del; 17 mod; > Hi all, > > could you please review this small (if you ignore removal part) patch which removes in-tree copy of gtest (test/fmw/gtest) and updates makefiles to use one provided thru an added configure option `--with-gtest`? the previously used configure option `--enable-hotspot-gtest` gets depricated. > > the patch also compiles gtest and gmock source code into a static library so they can be compiled w/ all warnings disabled. > > from JBS: >> w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all compilers used by OpenJDK became supported by gtest out-of-box, so there is no need to have our copy of gtest framework (including gmock) within OpenJDK source tree. instead, it can be treated like any other external dependencies, and a pointer to the directory w/ gtest code can be passed via configure options. > > JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 > webrevs: > - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ (meaningful changes) > - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ (all changes) > testing: > - gtest tests on {linux,windows,macosx}-x64; > - tier[1-5] builds which include but not limited to linux-aarch64, linux-arm32, linux-x64-zero > > Thanks, > -- Igor > From kim.barrett at oracle.com Fri May 22 19:04:59 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Fri, 22 May 2020 15:04:59 -0400 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <49e9f865-639b-47cf-b02b-1fcada91c43c@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> <49e9f865-639b-47cf-b02b-1fcada91c43c@oracle.com> Message-ID: <91871890-2C05-4D2B-89A5-8EE2E29BFB1A@oracle.com> > On May 22, 2020, at 9:59 AM, David Holmes wrote: > > On 22/05/2020 6:23 pm, Kim Barrett wrote: >>> On May 21, 2020, at 10:39 PM, David Holmes wrote: >>> >>> On 22/05/2020 8:31 am, Kim Barrett wrote: >>>> Changed constructors for MemoryManager and MemoryPool to use >>>> initializer list, so we don't need to use Atomic::store to initialize >>>> an oop-derived member. >>> >>> What is the connection between using an initializer-list and not needing the Atomic::Store? >> The volatile oop-derived member (instanceOop in both cases) needs to >> be initialized. If done in an initializer list, it's at construction >> time for the member object, where volatile isn't relevant. If done >> later (for example, in the containing object's constructor body, the >> member is now volatile and so involved a volatile assignment, which >> would now need to be an Atomic::store. > > I still don't see any distinction there with regards to why we use volatile or equivalently Atomic::store. Here's an example. A sufficient smaller change to the MemoryManager constructor would have been 41 MemoryManager::MemoryManager(const char* name) : _name(name) { 42 _num_pools = 0; - 43 (void)const_cast(_memory_mgr_obj = instanceOop(NULL)); + 43 Atomic::store(&_memory_manager, instanceOop(NULL)); 44 } The old code doesn't compile without the volatile qualified oop::operator=. A different, but not good (as in invokes UB), way to address the compilation failure const_cast(_memory_mgr_obj) = instanceOop(NULL); Instead, I changed it to 41 MemoryManager::MemoryManager(const char* name) : 42 _num_pools(0), _name(name), _memory_mgr_obj() {} No Atomic store needed here. >>> >>> In the following the referenced oops still retain their volatile qualifier - should it not now be removed? >>> >>> - src/hotspot/share/runtime/thread.hpp >>> _exception_oop is still marked volatile >>> - src/hotspot/share/services/memoryManager.hpp >>> _memory_mgr_obj is still marked volatile >>> - src/hotspot/share/services/memoryPool.hpp >>> _memory_pool_obj is still marked volatile >> Access to these objects is via Atomic operations, so they should still >> be volatile qualified per current convention. > > The current convention allows for naked loads and stores on volatile variables. When we only have Atomic load/store plus other atomic ops then the "convention" no longer requires the variable to be volatile. Which in turn would I assume undo the initializer-list argument. (See also my reply to Coleen in this thread.) The convention of volatile qualifying variables that we want to treat "atomically" only allows naked loads and stores for fundamental types, not for class types. The volatile qualifier for a member of type oop uses the compiler to help keep us honest, because naked loads and stores won't compile. (The same thing was done for markWord; JDK-8229838.) From igor.ignatyev at oracle.com Fri May 22 19:53:09 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Fri, 22 May 2020 12:53:09 -0700 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest In-Reply-To: References: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> Message-ID: Hi Erik, thanks for you review. sure, I'll hold up on pushing till Tue's evening, so Magnus and other will have a chance to look/comment. -- Igor > On May 22, 2020, at 11:48 AM, Erik Joelsson wrote: > > Hello Igor, > > The changes look good to me, but I think we should give Magnus a chance to look at this too. > > /Erik > > On 2020-05-22 11:12, Igor Ignatyev wrote: >> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ >>> 132 lines changed: 80 ins; 36 del; 16 mod >> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ >>> 57482 lines changed: 80 ins; 57385 del; 17 mod; >> Hi all, >> >> could you please review this small (if you ignore removal part) patch which removes in-tree copy of gtest (test/fmw/gtest) and updates makefiles to use one provided thru an added configure option `--with-gtest`? the previously used configure option `--enable-hotspot-gtest` gets depricated. >> >> the patch also compiles gtest and gmock source code into a static library so they can be compiled w/ all warnings disabled. >> >> from JBS: >>> w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all compilers used by OpenJDK became supported by gtest out-of-box, so there is no need to have our copy of gtest framework (including gmock) within OpenJDK source tree. instead, it can be treated like any other external dependencies, and a pointer to the directory w/ gtest code can be passed via configure options. >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 >> webrevs: >> - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ (meaningful changes) >> - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ (all changes) >> testing: >> - gtest tests on {linux,windows,macosx}-x64; >> - tier[1-5] builds which include but not limited to linux-aarch64, linux-arm32, linux-x64-zero >> >> Thanks, >> -- Igor >> From igor.ignatyev at oracle.com Fri May 22 22:42:18 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Fri, 22 May 2020 15:42:18 -0700 Subject: RFC : 8244614 : cleanup keywords used/available in hotspot testbase Message-ID: Hi all, I'm planning to clean up jtreg keywords that we have in test/hotspot/jtreg as most of them seem to be useless, confuse and make people to add these keywords in vain. After looking at how hotspot k/w are used, as well as which k/w are available and used by other test suites, I'd like to propose the following: - keep and document 'headful', 'stress' as they are useful for test selection; - keep and document 'intermittent', 'randomness' as they bring extra information about tests stability/reproducibility/determinism; - remove 'nmt', 'gc', 'metaspace' as they are used only in obvious places and aren't informative; - remove 'cte_test' as it's an artifact of old days and doesn't bring any useful information anymore; - remove 'jcmd', not all tests which depend on jcmd has this k/w, so it can't be used for test selection/filtering; there is also no obvious reasons to single out jcmd from other binaries bundled w/ JDK - remove 'regression' as the usage of at-bug is more common and convey the same meaning; and for two tests which have regression k/w, but don't have at-bug: - mark vmTestbase/nsk/jvmti/RedefineClasses/redefclass031/TestDescription.java w/ at-bug 6318850; - don't mark runtime/NMT/ReleaseNoCommit.java as it wasn't integrated as a true regression test, but rather as a function test for NMT w/ bunch of other NMT tests (where aren't marked w/ 'regression' k/w) by JDK-8054952. I'd like to receive some feedback on the proposed changes before I (more likely sed) remove keywords from 500+ tests and spend time testing that. [1] is the preliminary webrev just to illustrate the change, it only updates TEST.ROOT file and adds @bug to redefclass031 test. Thanks, -- Igor for reference, k/w available in other test suites: jdk : 2d dnd headful i18n intermittent printer randomness jfr langtools: intermittent randomness jaxp: intermittent randomness [1] http://cr.openjdk.java.net/~iignatyev//8244614/webrev.00 From aph at redhat.com Sat May 23 09:16:07 2020 From: aph at redhat.com (Andrew Haley) Date: Sat, 23 May 2020 10:16:07 +0100 Subject: [aarch64-port-dev ] RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes In-Reply-To: <09BC0693-80E0-4F87-855E-0B38A6F5EFA2@oracle.com> References: <275eb57c-51c0-675e-c32a-91b198023559@redhat.com> <719F9169-ABC4-408E-B732-F1BD9A84337F@oracle.com> <9a13f5df-d946-579d-4282-917dc7338dc8@redhat.com> <09BC0693-80E0-4F87-855E-0B38A6F5EFA2@oracle.com> Message-ID: <668e500e-f621-5a2c-a41e-f73536880f73@redhat.com> On 5/22/20 7:01 PM, Paul Sandoz wrote: > We have made changes similar in spirit to the x64 ad file (reducing in size at least), so I think it reasonable request before integration to reduce the cognitive and maintenance burden. So here's a question: can the changes to the AArch64 back end be made to mainline now? Or is there a problem in that the C2 patterns being used don't exist? -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From david.holmes at oracle.com Sat May 23 13:30:25 2020 From: david.holmes at oracle.com (David Holmes) Date: Sat, 23 May 2020 23:30:25 +1000 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <91871890-2C05-4D2B-89A5-8EE2E29BFB1A@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> <49e9f865-639b-47cf-b02b-1fcada91c43c@oracle.com> <91871890-2C05-4D2B-89A5-8EE2E29BFB1A@oracle.com> Message-ID: <2b7ba9bf-c63c-57c4-e7dc-9ff6d223b42c@oracle.com> Hi Kim, On 23/05/2020 5:04 am, Kim Barrett wrote: >> On May 22, 2020, at 9:59 AM, David Holmes wrote: >> >> On 22/05/2020 6:23 pm, Kim Barrett wrote: >>>> On May 21, 2020, at 10:39 PM, David Holmes wrote: >>>> >>>> On 22/05/2020 8:31 am, Kim Barrett wrote: >>>>> Changed constructors for MemoryManager and MemoryPool to use >>>>> initializer list, so we don't need to use Atomic::store to initialize >>>>> an oop-derived member. >>>> >>>> What is the connection between using an initializer-list and not needing the Atomic::Store? >>> The volatile oop-derived member (instanceOop in both cases) needs to >>> be initialized. If done in an initializer list, it's at construction >>> time for the member object, where volatile isn't relevant. If done >>> later (for example, in the containing object's constructor body, the >>> member is now volatile and so involved a volatile assignment, which >>> would now need to be an Atomic::store. >> >> I still don't see any distinction there with regards to why we use volatile or equivalently Atomic::store. > > Here's an example. > > A sufficient smaller change to the MemoryManager constructor would have been > > 41 MemoryManager::MemoryManager(const char* name) : _name(name) { > 42 _num_pools = 0; > - 43 (void)const_cast(_memory_mgr_obj = instanceOop(NULL)); > + 43 Atomic::store(&_memory_manager, instanceOop(NULL)); > 44 } > > The old code doesn't compile without the volatile qualified oop::operator=. > > A different, but not good (as in invokes UB), way to address the compilation failure > > const_cast(_memory_mgr_obj) = instanceOop(NULL); > > Instead, I changed it to > > 41 MemoryManager::MemoryManager(const char* name) : > 42 _num_pools(0), _name(name), _memory_mgr_obj() {} > > No Atomic store needed here. This is missing the point. If Atomic::store is presumed necessary for some kind of atomicity guarantee, then how does use of an initializer list achieve the same requirement? >>>> >>>> In the following the referenced oops still retain their volatile qualifier - should it not now be removed? >>>> >>>> - src/hotspot/share/runtime/thread.hpp >>>> _exception_oop is still marked volatile >>>> - src/hotspot/share/services/memoryManager.hpp >>>> _memory_mgr_obj is still marked volatile >>>> - src/hotspot/share/services/memoryPool.hpp >>>> _memory_pool_obj is still marked volatile >>> Access to these objects is via Atomic operations, so they should still >>> be volatile qualified per current convention. >> >> The current convention allows for naked loads and stores on volatile variables. When we only have Atomic load/store plus other atomic ops then the "convention" no longer requires the variable to be volatile. Which in turn would I assume undo the initializer-list argument. > > (See also my reply to Coleen in this thread.) > > The convention of volatile qualifying variables that we want to treat > "atomically" only allows naked loads and stores for fundamental types, > not for class types. The volatile qualifier for a member of type oop > uses the compiler to help keep us honest, because naked loads and > stores won't compile. (The same thing was done for markWord; JDK-8229838.) Hmmm. Granted "volatile" on a Class type has no meaning. But I always think of oop as really being a pointer type so volatile actually makes some sense there. By the same token Atomic::load/store have no meaning on class types either. So now I'm very unsure what we are actually doing here ?? David From kim.barrett at oracle.com Sat May 23 16:49:49 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Sat, 23 May 2020 12:49:49 -0400 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <2b7ba9bf-c63c-57c4-e7dc-9ff6d223b42c@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> <49e9f865-639b-47cf-b02b-1fcada91c43c@oracle.com> <91871890-2C05-4D2B-89A5-8EE2E29BFB1A@oracle.com> <2b7ba9bf-c63c-57c4-e7dc-9ff6d223b42c@oracle.com> Message-ID: <12A213F7-35AC-4F81-8EC7-30398EFF8895@oracle.com> > On May 23, 2020, at 9:30 AM, David Holmes wrote: > > Hi Kim, > > On 23/05/2020 5:04 am, Kim Barrett wrote: >>> On May 22, 2020, at 9:59 AM, David Holmes wrote: >>> >>> On 22/05/2020 6:23 pm, Kim Barrett wrote: >>>>> On May 21, 2020, at 10:39 PM, David Holmes wrote: >>>>> >>>>> On 22/05/2020 8:31 am, Kim Barrett wrote: >>>>>> Changed constructors for MemoryManager and MemoryPool to use >>>>>> initializer list, so we don't need to use Atomic::store to initialize >>>>>> an oop-derived member. >>>>> >>>>> What is the connection between using an initializer-list and not needing the Atomic::Store? >>>> The volatile oop-derived member (instanceOop in both cases) needs to >>>> be initialized. If done in an initializer list, it's at construction >>>> time for the member object, where volatile isn't relevant. If done >>>> later (for example, in the containing object's constructor body, the >>>> member is now volatile and so involved a volatile assignment, which >>>> would now need to be an Atomic::store. >>> >>> I still don't see any distinction there with regards to why we use volatile or equivalently Atomic::store. >> Here's an example. >> A sufficient smaller change to the MemoryManager constructor would have been >> 41 MemoryManager::MemoryManager(const char* name) : _name(name) { >> 42 _num_pools = 0; >> - 43 (void)const_cast(_memory_mgr_obj = instanceOop(NULL)); >> + 43 Atomic::store(&_memory_manager, instanceOop(NULL)); >> 44 } >> The old code doesn't compile without the volatile qualified oop::operator=. >> A different, but not good (as in invokes UB), way to address the compilation failure >> const_cast(_memory_mgr_obj) = instanceOop(NULL); >> Instead, I changed it to >> 41 MemoryManager::MemoryManager(const char* name) : >> 42 _num_pools(0), _name(name), _memory_mgr_obj() {} >> No Atomic store needed here. > > This is missing the point. If Atomic::store is presumed necessary for some kind of atomicity guarantee, then how does use of an initializer list achieve the same requirement? No atomicity guarantees are needed until the object is exposed to other threads than the one constructing the object. (That exposure needs to deal with atomicity and ordering requirements, of course. We wouldn't want to expose a partially constructed MemoryManager object to another thread, for example.) But other than at construction time for the member subobject, we don't have a way to say that (other than by casting away the volatile qualification or adding a volatile qualified assignment operator, neither of which we want). So we would end up needing to use Atomic::store for an "initialization" outside the initializer list. >>>>> In the following the referenced oops still retain their volatile qualifier - should it not now be removed? >>>>> >>>>> - src/hotspot/share/runtime/thread.hpp >>>>> _exception_oop is still marked volatile >>>>> - src/hotspot/share/services/memoryManager.hpp >>>>> _memory_mgr_obj is still marked volatile >>>>> - src/hotspot/share/services/memoryPool.hpp >>>>> _memory_pool_obj is still marked volatile >>>> Access to these objects is via Atomic operations, so they should still >>>> be volatile qualified per current convention. >>> >>> The current convention allows for naked loads and stores on volatile variables. When we only have Atomic load/store plus other atomic ops then the "convention" no longer requires the variable to be volatile. Which in turn would I assume undo the initializer-list argument. >> (See also my reply to Coleen in this thread.) >> The convention of volatile qualifying variables that we want to treat >> "atomically" only allows naked loads and stores for fundamental types, >> not for class types. The volatile qualifier for a member of type oop >> uses the compiler to help keep us honest, because naked loads and >> stores won't compile. (The same thing was done for markWord; JDK-8229838.) > > Hmmm. Granted "volatile" on a Class type has no meaning. But I always think of oop as really being a pointer type so volatile actually makes some sense there. > > By the same token Atomic::load/store have no meaning on class types either. > > So now I'm very unsure what we are actually doing here ?? Atomic operations have to be "taught" about classes that are thin wrappers over primitive types, like oop and markWord. That's done via PrimitiveConversions::Translate<> specializations. Atomic::load(const volatile oop* p) is translated into oop(Atomic::PlatformLoad()(reinterpret_cast(p))) Of course, in the case of oop that's only relevant in the CHECK_UNHANDLED_OOPS case, where oop is a class wrapper around an oopDesc*. In the !CHECK_UNHANDLED_OOPS case, that translation machinary isn't needed and doesn't apply, because Translate is only defined when oop is defined as a class. From david.holmes at oracle.com Mon May 25 00:01:08 2020 From: david.holmes at oracle.com (David Holmes) Date: Mon, 25 May 2020 10:01:08 +1000 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <12A213F7-35AC-4F81-8EC7-30398EFF8895@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> <49e9f865-639b-47cf-b02b-1fcada91c43c@oracle.com> <91871890-2C05-4D2B-89A5-8EE2E29BFB1A@oracle.com> <2b7ba9bf-c63c-57c4-e7dc-9ff6d223b42c@oracle.com> <12A213F7-35AC-4F81-8EC7-30398EFF8895@oracle.com> Message-ID: Hi Kim, On 24/05/2020 2:49 am, Kim Barrett wrote: >> On May 23, 2020, at 9:30 AM, David Holmes wrote: >> >> Hi Kim, >> >> On 23/05/2020 5:04 am, Kim Barrett wrote: >>>> On May 22, 2020, at 9:59 AM, David Holmes wrote: >>>> >>>> On 22/05/2020 6:23 pm, Kim Barrett wrote: >>>>>> On May 21, 2020, at 10:39 PM, David Holmes wrote: >>>>>> >>>>>> On 22/05/2020 8:31 am, Kim Barrett wrote: >>>>>>> Changed constructors for MemoryManager and MemoryPool to use >>>>>>> initializer list, so we don't need to use Atomic::store to initialize >>>>>>> an oop-derived member. >>>>>> >>>>>> What is the connection between using an initializer-list and not needing the Atomic::Store? >>>>> The volatile oop-derived member (instanceOop in both cases) needs to >>>>> be initialized. If done in an initializer list, it's at construction >>>>> time for the member object, where volatile isn't relevant. If done >>>>> later (for example, in the containing object's constructor body, the >>>>> member is now volatile and so involved a volatile assignment, which >>>>> would now need to be an Atomic::store. >>>> >>>> I still don't see any distinction there with regards to why we use volatile or equivalently Atomic::store. >>> Here's an example. >>> A sufficient smaller change to the MemoryManager constructor would have been >>> 41 MemoryManager::MemoryManager(const char* name) : _name(name) { >>> 42 _num_pools = 0; >>> - 43 (void)const_cast(_memory_mgr_obj = instanceOop(NULL)); >>> + 43 Atomic::store(&_memory_manager, instanceOop(NULL)); >>> 44 } >>> The old code doesn't compile without the volatile qualified oop::operator=. >>> A different, but not good (as in invokes UB), way to address the compilation failure >>> const_cast(_memory_mgr_obj) = instanceOop(NULL); >>> Instead, I changed it to >>> 41 MemoryManager::MemoryManager(const char* name) : >>> 42 _num_pools(0), _name(name), _memory_mgr_obj() {} >>> No Atomic store needed here. >> >> This is missing the point. If Atomic::store is presumed necessary for some kind of atomicity guarantee, then how does use of an initializer list achieve the same requirement? > > No atomicity guarantees are needed until the object is exposed to > other threads than the one constructing the object. (That exposure > needs to deal with atomicity and ordering requirements, of course. We > wouldn't want to expose a partially constructed MemoryManager object > to another thread, for example.) But other than at construction time > for the member subobject, we don't have a way to say that (other than > by casting away the volatile qualification or adding a volatile > qualified assignment operator, neither of which we want). So we would > end up needing to use Atomic::store for an "initialization" outside > the initializer list. Okay this is really nothing to do with Atomic::store at all. using the initializer list is the obviously correct and simple way to do this. >>>>>> In the following the referenced oops still retain their volatile qualifier - should it not now be removed? >>>>>> >>>>>> - src/hotspot/share/runtime/thread.hpp >>>>>> _exception_oop is still marked volatile >>>>>> - src/hotspot/share/services/memoryManager.hpp >>>>>> _memory_mgr_obj is still marked volatile >>>>>> - src/hotspot/share/services/memoryPool.hpp >>>>>> _memory_pool_obj is still marked volatile >>>>> Access to these objects is via Atomic operations, so they should still >>>>> be volatile qualified per current convention. >>>> >>>> The current convention allows for naked loads and stores on volatile variables. When we only have Atomic load/store plus other atomic ops then the "convention" no longer requires the variable to be volatile. Which in turn would I assume undo the initializer-list argument. >>> (See also my reply to Coleen in this thread.) >>> The convention of volatile qualifying variables that we want to treat >>> "atomically" only allows naked loads and stores for fundamental types, >>> not for class types. The volatile qualifier for a member of type oop >>> uses the compiler to help keep us honest, because naked loads and >>> stores won't compile. (The same thing was done for markWord; JDK-8229838.) >> >> Hmmm. Granted "volatile" on a Class type has no meaning. But I always think of oop as really being a pointer type so volatile actually makes some sense there. >> >> By the same token Atomic::load/store have no meaning on class types either. >> >> So now I'm very unsure what we are actually doing here ?? > > Atomic operations have to be "taught" about classes that are thin > wrappers over primitive types, like oop and markWord. That's done via > PrimitiveConversions::Translate<> specializations. > > Atomic::load(const volatile oop* p) is translated into > > oop(Atomic::PlatformLoad()(reinterpret_cast(p))) > > Of course, in the case of oop that's only relevant in the > CHECK_UNHANDLED_OOPS case, where oop is a class wrapper around an > oopDesc*. In the !CHECK_UNHANDLED_OOPS case, that translation > machinary isn't needed and doesn't apply, because Translate > is only defined when oop is defined as a class. Okay, but I still don't see why we would not remove the volatile qualifier for these oop cases. Thanks, David > From kim.barrett at oracle.com Mon May 25 00:53:50 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Sun, 24 May 2020 20:53:50 -0400 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> <49e9f865-639b-47cf-b02b-1fcada91c43c@oracle.com> <91871890-2C05-4D2B-89A5-8EE2E29BFB1A@oracle.com> <2b7ba9bf-c63c-57c4-e7dc-9ff6d223b42c@oracle.com> <12A213F7-35AC-4F81-8EC7-30398EFF8895@oracle.com> Message-ID: <0683741A-4FEA-4603-B7ED-63EF92BE695A@oracle.com> > On May 24, 2020, at 8:01 PM, David Holmes wrote: > > Okay this is really nothing to do with Atomic::store at all. using the initializer list is the obviously correct and simple way to do this. There are many reasons to prefer using initializer lists for initialization. In this case, one is not needing to have a weak atomic store, as those generally engender additional scrutiny. I guess my RFR description was overly terse and suggested more than was intended. >>> Hmmm. Granted "volatile" on a Class type has no meaning. But I always think of oop as really being a pointer type so volatile actually makes some sense there. >>> [?] > > Okay, but I still don't see why we would not remove the volatile qualifier for these oop cases. It "tags" the variable as being read/written by multiple threads and semantically atomic. And unlike similar usage with fundamental types, because we've eliminated the volatile qualifiers in member functions, the compiler will help us remember to use atomic accesses. From david.holmes at oracle.com Mon May 25 02:32:00 2020 From: david.holmes at oracle.com (David Holmes) Date: Mon, 25 May 2020 12:32:00 +1000 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <0683741A-4FEA-4603-B7ED-63EF92BE695A@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> <49e9f865-639b-47cf-b02b-1fcada91c43c@oracle.com> <91871890-2C05-4D2B-89A5-8EE2E29BFB1A@oracle.com> <2b7ba9bf-c63c-57c4-e7dc-9ff6d223b42c@oracle.com> <12A213F7-35AC-4F81-8EC7-30398EFF8895@oracle.com> <0683741A-4FEA-4603-B7ED-63EF92BE695A@oracle.com> Message-ID: <62455662-53fc-191d-7f4c-c4067e5ea41c@oracle.com> On 25/05/2020 10:53 am, Kim Barrett wrote: >> On May 24, 2020, at 8:01 PM, David Holmes wrote: >> >> Okay this is really nothing to do with Atomic::store at all. using the initializer list is the obviously correct and simple way to do this. > > There are many reasons to prefer using initializer lists for > initialization. In this case, one is not needing to have a weak atomic > store, as those generally engender additional scrutiny. I guess my > RFR description was overly terse and suggested more than was intended. > >>>> Hmmm. Granted "volatile" on a Class type has no meaning. But I always think of oop as really being a pointer type so volatile actually makes some sense there. >>>> [?] >> >> Okay, but I still don't see why we would not remove the volatile qualifier for these oop cases. > > It "tags" the variable as being read/written by multiple threads and > semantically atomic. And unlike similar usage with fundamental types, > because we've eliminated the volatile qualifiers in member functions, > the compiler will help us remember to use atomic accesses. I have no issue with using "volatile" as a tag but I'm confused that we now will have two distinct conventions in this area. Cheers, David From david.holmes at oracle.com Mon May 25 04:23:04 2020 From: david.holmes at oracle.com (David Holmes) Date: Mon, 25 May 2020 14:23:04 +1000 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <62455662-53fc-191d-7f4c-c4067e5ea41c@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> <49e9f865-639b-47cf-b02b-1fcada91c43c@oracle.com> <91871890-2C05-4D2B-89A5-8EE2E29BFB1A@oracle.com> <2b7ba9bf-c63c-57c4-e7dc-9ff6d223b42c@oracle.com> <12A213F7-35AC-4F81-8EC7-30398EFF8895@oracle.com> <0683741A-4FEA-4603-B7ED-63EF92BE695A@oracle.com> <62455662-53fc-191d-7f4c-c4067e5ea41c@oracle.com> Message-ID: <779fb3b6-eb82-3634-3665-47d55bdfda1e@oracle.com> After discussing with Kim off-line I need to clarify some comments ... On 25/05/2020 12:32 pm, David Holmes wrote: > On 25/05/2020 10:53 am, Kim Barrett wrote: >>> On May 24, 2020, at 8:01 PM, David Holmes >>> wrote: >>> >>> Okay this is really nothing to do with Atomic::store at all. using >>> the initializer list is the obviously correct and simple way to do this. >> >> There are many reasons to prefer using initializer lists for >> initialization. In this case, one is not needing to have a weak atomic >> store, as those generally engender additional scrutiny.? I guess my >> RFR description was overly terse and suggested more than was intended. >> >>>>> Hmmm. Granted "volatile" on a Class type has no meaning. But I >>>>> always think of oop as really being a pointer type so volatile >>>>> actually makes some sense there. >>>>> [?] >>> >>> Okay, but I still don't see why we would not remove the volatile >>> qualifier for these oop cases. >> >> It "tags" the variable as being read/written by multiple threads and >> semantically atomic. And unlike similar usage with fundamental types, >> because we've eliminated the volatile qualifiers in member functions, >> the compiler will help us remember to use atomic accesses. > > I have no issue with using "volatile" as a tag but I'm confused that we > now will have two distinct conventions in this area. The convention I'm referring to is one that has been discussed in the context of other code-reviews, is partly behind https://bugs.openjdk.java.net/browse/JDK-8234192 (though that also leads to a full blown AtomicVal style of API), and which is clearly demonstrated by the proposal in: https://bugs.openjdk.java.net/browse/JDK-8238174 and which involves removing use of the volatile qualifier once we have switched to using Atomic::load/store for accesses. Hence my confusion as to why the volatile qualifier is not only being retained in this code, but is seen as a good thing to have for documenting the fact this member is special. Anyway, as these "conventions" are still TBD and not widely endorsed yet, I won't delay Kim's current updates any further. Thanks, David > Cheers, > David > From kim.barrett at oracle.com Mon May 25 06:44:22 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Mon, 25 May 2020 02:44:22 -0400 Subject: RFR: 8245594: Remove volatile-qualified member functions and parameters from oop class In-Reply-To: <779fb3b6-eb82-3634-3665-47d55bdfda1e@oracle.com> References: <8014FE44-5B0C-4562-82F0-1138725E9615@oracle.com> <40d5e1e8-0401-6cb8-fe41-ba89feff55e2@oracle.com> <49e9f865-639b-47cf-b02b-1fcada91c43c@oracle.com> <91871890-2C05-4D2B-89A5-8EE2E29BFB1A@oracle.com> <2b7ba9bf-c63c-57c4-e7dc-9ff6d223b42c@oracle.com> <12A213F7-35AC-4F81-8EC7-30398EFF8895@oracle.com> <0683741A-4FEA-4603-B7ED-63EF92BE695A@oracle.com> <62455662-53fc-191d-7f4c-c4067e5ea41c@oracle.com> <779fb3b6-eb82-3634-3665-47d55bdfda1e@oracle.com> Message-ID: > On May 25, 2020, at 12:23 AM, David Holmes wrote: > > After discussing with Kim off-line I need to clarify some comments ... > > On 25/05/2020 12:32 pm, David Holmes wrote: >> On 25/05/2020 10:53 am, Kim Barrett wrote: >>>> On May 24, 2020, at 8:01 PM, David Holmes wrote: >>>> >>>> Okay this is really nothing to do with Atomic::store at all. using the initializer list is the obviously correct and simple way to do this. >>> >>> There are many reasons to prefer using initializer lists for >>> initialization. In this case, one is not needing to have a weak atomic >>> store, as those generally engender additional scrutiny. I guess my >>> RFR description was overly terse and suggested more than was intended. >>> >>>>>> Hmmm. Granted "volatile" on a Class type has no meaning. But I always think of oop as really being a pointer type so volatile actually makes some sense there. >>>>>> [?] >>>> >>>> Okay, but I still don't see why we would not remove the volatile qualifier for these oop cases. >>> >>> It "tags" the variable as being read/written by multiple threads and >>> semantically atomic. And unlike similar usage with fundamental types, >>> because we've eliminated the volatile qualifiers in member functions, >>> the compiler will help us remember to use atomic accesses. >> I have no issue with using "volatile" as a tag but I'm confused that we now will have two distinct conventions in this area. > > The convention I'm referring to is one that has been discussed in the context of other code-reviews, is partly behind > > https://bugs.openjdk.java.net/browse/JDK-8234192 > > (though that also leads to a full blown AtomicVal style of API), and which is clearly demonstrated by the proposal in: > > https://bugs.openjdk.java.net/browse/JDK-8238174 > > and which involves removing use of the volatile qualifier once we have switched to using Atomic::load/store for accesses. > > Hence my confusion as to why the volatile qualifier is not only being retained in this code, but is seen as a good thing to have for documenting the fact this member is special. > > Anyway, as these "conventions" are still TBD and not widely endorsed yet, I won't delay Kim's current updates any further. Thanks. > > Thanks, > David > >> Cheers, >> David From Yang.Zhang at arm.com Mon May 25 08:26:45 2020 From: Yang.Zhang at arm.com (Yang Zhang) Date: Mon, 25 May 2020 08:26:45 +0000 Subject: [aarch64-port-dev ] RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes In-Reply-To: <668e500e-f621-5a2c-a41e-f73536880f73@redhat.com> References: <275eb57c-51c0-675e-c32a-91b198023559@redhat.com> <719F9169-ABC4-408E-B732-F1BD9A84337F@oracle.com> <9a13f5df-d946-579d-4282-917dc7338dc8@redhat.com> <09BC0693-80E0-4F87-855E-0B38A6F5EFA2@oracle.com> <668e500e-f621-5a2c-a41e-f73536880f73@redhat.com> Message-ID: Hi Andrew Please check the following. >> We have made changes similar in spirit to the x64 ad file (reducing in size at least), so I think it reasonable request before integration to reduce the cognitive and maintenance burden. > So here's a question: can the changes to the AArch64 back end be made to mainline now? Or is there a problem in that the C2 patterns being used don't exist? X86 ad file has been refactored to reduce code size in a series of JBSs [1]. I also investigated how to make similar changes to AArch64 ad file in Aug 2019. In summary, these changes wouldn't have a great impact on AArch64. If making similar changes to AArch64, about ~200kb (1% of total code size) will be reduced [2]. So I don't think we need to make these changes to AArch64 for now. To reduce maintenance burden, I agree that all the vector stuff should be moved into a new file, just like AArch64 SVE has done [3]. All the SVE instructions are generated by m4 file and placed in aarch64_sve.ad. For newly added NEON instructions in Vector API, they are all generated by m4 file. In jdk master, what we need to do is that writing m4 file for existing vector instructions and placed them to a new file aarch64_neon.ad. If no question, I will do it right away. [1] https://bugs.openjdk.java.net/browse/JDK-8230015 [2] The data came from: I implement an example (merging vaddB/S/I/F) in AArch64 platform. The code size reduction in libjvm.so is ~15kb. If all the vector instructions are merged, the estimated size reduction would be ~200kb. The code size from vector support in AArch64 backend is ~450kb (2% of total size). Original libjvm.so is 20770256, while libjvm.so without vector support is 20317328. Based on the idea of generic vector operands, half of vector stuff can be removed. So the estimated size reduction is also ~200kb. [3] http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2020-March/037628.html Regards Yang -----Original Message----- From: Andrew Haley Sent: Saturday, May 23, 2020 5:16 PM To: Paul Sandoz Cc: Yang Zhang ; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; core-libs-dev at openjdk.java.net; aarch64-port-dev at openjdk.java.net; nd Subject: Re: [aarch64-port-dev ] RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes On 5/22/20 7:01 PM, Paul Sandoz wrote: > We have made changes similar in spirit to the x64 ad file (reducing in size at least), so I think it reasonable request before integration to reduce the cognitive and maintenance burden. So here's a question: can the changes to the AArch64 back end be made to mainline now? Or is there a problem in that the C2 patterns being used don't exist? -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From magnus.ihse.bursie at oracle.com Mon May 25 14:46:00 2020 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Mon, 25 May 2020 16:46:00 +0200 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest In-Reply-To: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> References: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> Message-ID: <0c5d41f4-eebd-a8ae-d537-e14a878a6663@oracle.com> Looks basically good to me. We need to document the dependency on gtest, and how to retrieve it. I recommend you add a section next to the JTReg information (under the "## Running Tests" heading) in doc/building.md. I think you should include basically the same information as you did in your follow-up mail, that was informative and clear. I assume the most suitable replacement for developers who are used to add a '--disable-hotspot-gtest' to e.g. a pre-definied jib configuration is to now use '--without-gtest'. This will need to be communicated, perhaps to both hotspot-dev and jdk-dev. /Magnus On 2020-05-22 20:12, Igor Ignatyev wrote: > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ >> 132 lines changed: 80 ins; 36 del; 16 mod > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ >> 57482 lines changed: 80 ins; 57385 del; 17 mod; > Hi all, > > could you please review this small (if you ignore removal part) patch which removes in-tree copy of gtest (test/fmw/gtest) and updates makefiles to use one provided thru an added configure option `--with-gtest`? the previously used configure option `--enable-hotspot-gtest` gets depricated. > > the patch also compiles gtest and gmock source code into a static library so they can be compiled w/ all warnings disabled. > > from JBS: >> w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all compilers used by OpenJDK became supported by gtest out-of-box, so there is no need to have our copy of gtest framework (including gmock) within OpenJDK source tree. instead, it can be treated like any other external dependencies, and a pointer to the directory w/ gtest code can be passed via configure options. > > JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 > webrevs: > - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ (meaningful changes) > - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ (all changes) > testing: > - gtest tests on {linux,windows,macosx}-x64; > - tier[1-5] builds which include but not limited to linux-aarch64, linux-arm32, linux-x64-zero > > Thanks, > -- Igor > From igor.ignatyev at oracle.com Mon May 25 17:53:31 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Mon, 25 May 2020 10:53:31 -0700 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest In-Reply-To: <0c5d41f4-eebd-a8ae-d537-e14a878a6663@oracle.com> References: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> <0c5d41f4-eebd-a8ae-d537-e14a878a6663@oracle.com> Message-ID: <7B996E95-B57D-4F38-97F1-65FFCA2DD01E@oracle.com> Hi Magnus, > On May 25, 2020, at 7:46 AM, Magnus Ihse Bursie wrote: > > Looks basically good to me. thanks for your review! > > We need to document the dependency on gtest, and how to retrieve it. I recommend you add a section next to the JTReg information (under the "## Running Tests" heading) in doc/building.md. I think you should include basically the same information as you did in your follow-up mail, that was informative and clear. that's a good suggestion, I've added a small paragraph to 'Running Tests' in doc/building.md and regenerated corresponding .html file -- http://cr.openjdk.java.net/~iignatyev/8245610/webrev.doc/ please let me know if you think something should be added or reworded. > > I assume the most suitable replacement for developers who are used to add a '--disable-hotspot-gtest' to e.g. a pre-definied jib configuration is to now use '--without-gtest'. This will need to be communicated, perhaps to both hotspot-dev and jdk-dev. sure, after this gets integrated, I'll let both hotspot-dev and jdk-dev aliases know which changes might be required to enable/disable hotspot gtest tests compilation. Thanks. -- Igor > > /Magnus > > On 2020-05-22 20:12, Igor Ignatyev wrote: >> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ >>> 132 lines changed: 80 ins; 36 del; 16 mod >> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ >>> 57482 lines changed: 80 ins; 57385 del; 17 mod; >> Hi all, >> >> could you please review this small (if you ignore removal part) patch which removes in-tree copy of gtest (test/fmw/gtest) and updates makefiles to use one provided thru an added configure option `--with-gtest`? the previously used configure option `--enable-hotspot-gtest` gets depricated. >> >> the patch also compiles gtest and gmock source code into a static library so they can be compiled w/ all warnings disabled. >> >> from JBS: >>> w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all compilers used by OpenJDK became supported by gtest out-of-box, so there is no need to have our copy of gtest framework (including gmock) within OpenJDK source tree. instead, it can be treated like any other external dependencies, and a pointer to the directory w/ gtest code can be passed via configure options. >> >> JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 >> webrevs: >> - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ (meaningful changes) >> - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ (all changes) >> testing: >> - gtest tests on {linux,windows,macosx}-x64; >> - tier[1-5] builds which include but not limited to linux-aarch64, linux-arm32, linux-x64-zero >> >> Thanks, >> -- Igor >> > From jiefu at tencent.com Tue May 26 01:24:39 2020 From: jiefu at tencent.com (=?utf-8?B?amllZnUo5YKF5p2wKQ==?=) Date: Tue, 26 May 2020 01:24:39 +0000 Subject: RFR: 8245761: [TESTBUG] runtime/logging/TestMethodHandlesVerbose.java fails with release VMs Message-ID: Hi all, May I get reviews for this fix? JBS: https://bugs.openjdk.java.net/browse/JDK-8245761 Webrev: http://cr.openjdk.java.net/~jiefu/8245761/webrev.00/ Thanks a lot. Best regards, Jie From david.holmes at oracle.com Tue May 26 03:01:42 2020 From: david.holmes at oracle.com (David Holmes) Date: Tue, 26 May 2020 13:01:42 +1000 Subject: RFR: 8245761: [TESTBUG] runtime/logging/TestMethodHandlesVerbose.java fails with release VMs In-Reply-To: References: Message-ID: Hi Jie, Looks good and trivial. Thanks for fixing it! Our testing only just hit this. David On 26/05/2020 11:24 am, jiefu(??) wrote: > Hi all, > > May I get reviews for this fix? > > JBS: https://bugs.openjdk.java.net/browse/JDK-8245761 > Webrev: http://cr.openjdk.java.net/~jiefu/8245761/webrev.00/ > > Thanks a lot. > Best regards, > Jie > From jiefu at tencent.com Tue May 26 03:08:08 2020 From: jiefu at tencent.com (=?utf-8?B?amllZnUo5YKF5p2wKQ==?=) Date: Tue, 26 May 2020 03:08:08 +0000 Subject: 8245761: [TESTBUG] runtime/logging/TestMethodHandlesVerbose.java fails with release VMs(Internet mail) Message-ID: <21F99E10-B13D-4706-829D-E8F91667E748@tencent.com> Thanks David for your review. Pushed. Best regards, Jie ?On 2020/5/26, 11:02 AM, "David Holmes" wrote: Hi Jie, Looks good and trivial. Thanks for fixing it! Our testing only just hit this. David On 26/05/2020 11:24 am, jiefu(??) wrote: > Hi all, > > May I get reviews for this fix? > > JBS: https://bugs.openjdk.java.net/browse/JDK-8245761 > Webrev: http://cr.openjdk.java.net/~jiefu/8245761/webrev.00/ > > Thanks a lot. > Best regards, > Jie > From aph at redhat.com Tue May 26 08:25:02 2020 From: aph at redhat.com (Andrew Haley) Date: Tue, 26 May 2020 09:25:02 +0100 Subject: [aarch64-port-dev ] RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes In-Reply-To: References: <275eb57c-51c0-675e-c32a-91b198023559@redhat.com> <719F9169-ABC4-408E-B732-F1BD9A84337F@oracle.com> <9a13f5df-d946-579d-4282-917dc7338dc8@redhat.com> <09BC0693-80E0-4F87-855E-0B38A6F5EFA2@oracle.com> <668e500e-f621-5a2c-a41e-f73536880f73@redhat.com> Message-ID: <1909fa9d-98bb-c2fb-45d8-540247d1ca8b@redhat.com> On 25/05/2020 09:26, Yang Zhang wrote: > In jdk master, what we need to do is that writing m4 file for existing > vector instructions and placed them to a new file aarch64_neon.ad. > If no question, I will do it right away. I'm not entirely sure that such a change is necessary now. In particular, reorganizing the existing vector instructions is IMO excessive, but I admit that it might be an improvement. But to my earlier question. please: can the new instructions be moved into jdk head first, and then merged into the Panama branch, or not? It'd help if this was possible. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From thomas.schatzl at oracle.com Tue May 26 11:56:43 2020 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 26 May 2020 13:56:43 +0200 Subject: RFR: 8228692: BitMap::reallocate might not clear some bits In-Reply-To: References: Message-ID: <9190cb98-9660-227f-9189-10c24d1950a4@oracle.com> Hi, On 22.05.20 00:36, Kim Barrett wrote: > Please review this change to BitMap reallocation that ensures all > newly added bits are cleared when requested. Previously, there were > circumstances where some might not be. Includes a new gtest for the > specific issue. > > This change also includes a fix for JDK-8235714. Contrary to previous > comments and bug updates, it was not fixed by JDK-8241581 after all. > This was noticed while testing the change for JDK-8228692. > > CR: > https://bugs.openjdk.java.net/browse/JDK-8228692 > https://bugs.openjdk.java.net/browse/JDK-8235714 > > Webrev: > https://cr.openjdk.java.net/~kbarrett/8228692/open.00/ > > Testing: > mach5 tier1 > > Local (linux-x64) BitMap tests, including the new test. The new test > (1) failed before the changes, (2) still failed after either one but > not both of the fix for relocate and the fix for count_one_bits, (3) > and passes with both fixes. > lgtm. Thomas From stefan.johansson at oracle.com Tue May 26 12:16:11 2020 From: stefan.johansson at oracle.com (stefan.johansson at oracle.com) Date: Tue, 26 May 2020 14:16:11 +0200 Subject: RFR: 8245723: Remove PrintCompressedOopsMode and gc+heap+coops info log to debug level Message-ID: <83ef9d38-cf42-f3b3-067d-891afb6c1572@oracle.com> Hi, Please review this change that removes the deprecated diagnostic option PrintCompressedOopsMode. Webrev: http://cr.openjdk.java.net/~sjohanss/8245723/00/ JBS: https://bugs.openjdk.java.net/browse/JDK-8245723 Summary After adding the new gc+init logging in JDK-8244817, the info logging on gc+heap+coops can be moved to debug level. When looking into this I realized that gc+heap+coops=info has an old alias flag, PrintCompressedOopsMode, connected to it. I see no reason keeping this flag around and from what I understand we don't need to do anything special since it has been deprecated since JDK 9. Testing Locally verified output and run the tests affected. Also Mach5 tier1-2. Thanks, Stefan From magnus.ihse.bursie at oracle.com Tue May 26 12:26:34 2020 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Tue, 26 May 2020 14:26:34 +0200 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest In-Reply-To: <7B996E95-B57D-4F38-97F1-65FFCA2DD01E@oracle.com> References: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> <0c5d41f4-eebd-a8ae-d537-e14a878a6663@oracle.com> <7B996E95-B57D-4F38-97F1-65FFCA2DD01E@oracle.com> Message-ID: On 2020-05-25 19:53, Igor Ignatyev wrote: > Hi Magnus, > >> On May 25, 2020, at 7:46 AM, Magnus Ihse Bursie >> > > wrote: >> >> Looks basically good to me. > thanks for your review! >> >> We need to document the dependency on gtest, and how to retrieve it. >> I recommend you add a section next to the JTReg information (under >> the "## Running Tests" heading) in doc/building.md. I think you >> should include basically the same information as you did in your >> follow-up mail, that was informative and clear. > that's a good suggestion, I've added a small paragraph to 'Running > Tests' in doc/building.md and regenerated corresponding .html file -- > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.doc/ > please let me know if you think something should be added or reworded. Looks great! Thank you. /Magnus > >> >> I assume the most suitable replacement for developers who are used to >> add a '--disable-hotspot-gtest' to e.g. a pre-definied jib >> configuration is to now use '--without-gtest'. This will need to be >> communicated, perhaps to both hotspot-dev and jdk-dev. > sure, after this gets integrated, I'll let both hotspot-dev and > jdk-dev aliases know which changes might be required to enable/disable > hotspot gtest tests compilation. > > Thanks. > -- Igor > >> >> /Magnus >> >> On 2020-05-22 20:12, Igor Ignatyev wrote: >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ >>>> 132 lines changed: 80 ins; 36 del; 16 mod >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ >>>> 57482 lines changed: 80 ins; 57385 del; 17 mod; >>> Hi all, >>> >>> could you please review this small (if you ignore removal part) >>> patch which removes in-tree copy of gtest (test/fmw/gtest) and >>> updates makefiles to use one provided thru an added configure option >>> `--with-gtest`? the previously used configure option >>> `--enable-hotspot-gtest` gets depricated. >>> >>> the patch also compiles gtest and gmock source code into a static >>> library so they can be compiled w/ all warnings disabled. >>> >>> from JBS: >>>> w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all >>>> compilers used by OpenJDK became supported by gtest out-of-box, so >>>> there is no need to have our copy of gtest framework (including >>>> gmock) within OpenJDK source tree. instead, it can be treated like >>>> any other external dependencies, and a pointer to the directory w/ >>>> gtest code can be passed via configure options. >>> >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 >>> webrevs: >>> ?- http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ >>> (meaningful changes) >>> ?- >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ >>> (all changes) >>> testing: >>> - gtest tests on {linux,windows,macosx}-x64; >>> - tier[1-5] builds which include but not limited to linux-aarch64, >>> linux-arm32, linux-x64-zero >>> >>> Thanks, >>> -- Igor >>> >> > From coleen.phillimore at oracle.com Tue May 26 12:39:44 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 26 May 2020 08:39:44 -0400 Subject: RFR 8245289: Clean up offset code in JavaClasses Message-ID: Summary: Use X macro to declare and define offset, and declare accessors when needed, also make names consistent. Changes include. 1. making all offset field names start with underscore. 2. renamed blah_offset_in_bytes to just blah_offset (they are all in bytes and would never be in "words") 3. Use X macro to declare then define the field offsets to avoid retyping all that 4. Use X macro to define accessors for classes that needed them, so that the accessors can check if the field has been initialized. Since these are member fields and static fields, their offsets can never be zero, so there's no need to have a test for -1.? I kept some _initialized members for a couple of classes.? They can be removed later if we want. ? - I didn't change offset field access internally in the class member functions everywhere because it wouldn't be a net improvements in lines of code 5. removed "private" keyword because in class declarations fields are private by default. Most of these classes have accessor patterns like void java_lang_Class::set_class_loader(oop java_class, oop loader) { ? assert(_class_loader_offset != 0, "offsets should have been initialized"); ? java_class->obj_field_put(_class_loader_offset, loader); } oop java_lang_Class::class_loader(oop java_class) { ? assert(_class_loader_offset != 0, "must be set"); ? return java_class->obj_field(_class_loader_offset); } So it seems like it'd be useful to wrap that up with the X macro too, but there are also a lot of classes that don't have this pattern for all of the fields, ie. they do some other things so there's be too many exceptions to make this a nice code savings change.? The jvmci code has this sort of giant X macro that is, in my opinion, completely unreadable. Anyway, this is an improvement for now and tested tier1-6.? What do you think? open webrev at http://cr.openjdk.java.net/~coleenp/2020/8245289.01/webrev bug link https://bugs.openjdk.java.net/browse/JDK-8245289 thanks, Coleen From coleen.phillimore at oracle.com Tue May 26 13:17:08 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 26 May 2020 09:17:08 -0400 Subject: RFR: 8245723: Remove PrintCompressedOopsMode and gc+heap+coops info log to debug level In-Reply-To: <83ef9d38-cf42-f3b3-067d-891afb6c1572@oracle.com> References: <83ef9d38-cf42-f3b3-067d-891afb6c1572@oracle.com> Message-ID: Looks good to me. thanks, Coleen On 5/26/20 8:16 AM, stefan.johansson at oracle.com wrote: > Hi, > > Please review this change that removes the deprecated diagnostic > option PrintCompressedOopsMode. > > Webrev: http://cr.openjdk.java.net/~sjohanss/8245723/00/ > JBS: https://bugs.openjdk.java.net/browse/JDK-8245723 > > Summary > After adding the new gc+init logging in JDK-8244817, the info logging > on gc+heap+coops can be moved to debug level. When looking into this I > realized that gc+heap+coops=info has an old alias flag, > PrintCompressedOopsMode, connected to it. I see no reason keeping this > flag around and from what I understand we don't need to do anything > special since it has been deprecated since JDK 9. > > Testing > Locally verified output and run the tests affected. Also Mach5 tier1-2. > > Thanks, > Stefan From david.holmes at oracle.com Tue May 26 13:22:04 2020 From: david.holmes at oracle.com (David Holmes) Date: Tue, 26 May 2020 23:22:04 +1000 Subject: RFR: 8245723: Remove PrintCompressedOopsMode and gc+heap+coops info log to debug level In-Reply-To: <83ef9d38-cf42-f3b3-067d-891afb6c1572@oracle.com> References: <83ef9d38-cf42-f3b3-067d-891afb6c1572@oracle.com> Message-ID: Hi Stefan, That all seems fine to me. Aside: I couldn't tell why runtime/CompressedOops/CompressedClassPointers.java was actually using the flag. Do you think it is just for additional diagnostics in case of failure? Thanks, David On 26/05/2020 10:16 pm, stefan.johansson at oracle.com wrote: > Hi, > > Please review this change that removes the deprecated diagnostic option > PrintCompressedOopsMode. > > Webrev: http://cr.openjdk.java.net/~sjohanss/8245723/00/ > JBS: https://bugs.openjdk.java.net/browse/JDK-8245723 > > Summary > After adding the new gc+init logging in JDK-8244817, the info logging on > gc+heap+coops can be moved to debug level. When looking into this I > realized that gc+heap+coops=info has an old alias flag, > PrintCompressedOopsMode, connected to it. I see no reason keeping this > flag around and from what I understand we don't need to do anything > special since it has been deprecated since JDK 9. > > Testing > Locally verified output and run the tests affected. Also Mach5 tier1-2. > > Thanks, > Stefan From stefan.johansson at oracle.com Tue May 26 13:38:41 2020 From: stefan.johansson at oracle.com (stefan.johansson at oracle.com) Date: Tue, 26 May 2020 15:38:41 +0200 Subject: RFR: 8245723: Remove PrintCompressedOopsMode and gc+heap+coops info log to debug level In-Reply-To: References: <83ef9d38-cf42-f3b3-067d-891afb6c1572@oracle.com> Message-ID: <47ef5810-5e74-78a5-7a81-1f39101226af@oracle.com> Thanks for the review David, On 2020-05-26 15:22, David Holmes wrote: > Hi Stefan, > > That all seems fine to me. > > Aside: I couldn't tell why > runtime/CompressedOops/CompressedClassPointers.java was actually using > the flag. Do you think it is just for additional diagnostics in case of > failure? Yes, I think so. I thought about skipping it, but on the other hand it might be helpful in case of a failure. Cheers, Stefan > > Thanks, > David > > On 26/05/2020 10:16 pm, stefan.johansson at oracle.com wrote: >> Hi, >> >> Please review this change that removes the deprecated diagnostic >> option PrintCompressedOopsMode. >> >> Webrev: http://cr.openjdk.java.net/~sjohanss/8245723/00/ >> JBS: https://bugs.openjdk.java.net/browse/JDK-8245723 >> >> Summary >> After adding the new gc+init logging in JDK-8244817, the info logging >> on gc+heap+coops can be moved to debug level. When looking into this I >> realized that gc+heap+coops=info has an old alias flag, >> PrintCompressedOopsMode, connected to it. I see no reason keeping this >> flag around and from what I understand we don't need to do anything >> special since it has been deprecated since JDK 9. >> >> Testing >> Locally verified output and run the tests affected. Also Mach5 tier1-2. >> >> Thanks, >> Stefan From stefan.johansson at oracle.com Tue May 26 13:39:05 2020 From: stefan.johansson at oracle.com (stefan.johansson at oracle.com) Date: Tue, 26 May 2020 15:39:05 +0200 Subject: RFR: 8245723: Remove PrintCompressedOopsMode and gc+heap+coops info log to debug level In-Reply-To: References: <83ef9d38-cf42-f3b3-067d-891afb6c1572@oracle.com> Message-ID: <5d4dbc00-5c87-e78c-7c66-42170cc62f38@oracle.com> Thanks for the review Coleen, Stefan On 2020-05-26 15:17, coleen.phillimore at oracle.com wrote: > Looks good to me. > thanks, > Coleen > > On 5/26/20 8:16 AM, stefan.johansson at oracle.com wrote: >> Hi, >> >> Please review this change that removes the deprecated diagnostic >> option PrintCompressedOopsMode. >> >> Webrev: http://cr.openjdk.java.net/~sjohanss/8245723/00/ >> JBS: https://bugs.openjdk.java.net/browse/JDK-8245723 >> >> Summary >> After adding the new gc+init logging in JDK-8244817, the info logging >> on gc+heap+coops can be moved to debug level. When looking into this I >> realized that gc+heap+coops=info has an old alias flag, >> PrintCompressedOopsMode, connected to it. I see no reason keeping this >> flag around and from what I understand we don't need to do anything >> special since it has been deprecated since JDK 9. >> >> Testing >> Locally verified output and run the tests affected. Also Mach5 tier1-2. >> >> Thanks, >> Stefan > From kim.barrett at oracle.com Tue May 26 13:43:36 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Tue, 26 May 2020 09:43:36 -0400 Subject: RFR: 8228692: BitMap::reallocate might not clear some bits In-Reply-To: <9190cb98-9660-227f-9189-10c24d1950a4@oracle.com> References: <9190cb98-9660-227f-9189-10c24d1950a4@oracle.com> Message-ID: <80BE5C25-4205-45DD-B47D-CE95C35FFDFE@oracle.com> > On May 26, 2020, at 7:56 AM, Thomas Schatzl wrote: > > Hi, > > On 22.05.20 00:36, Kim Barrett wrote: >> Please review this change to BitMap reallocation that ensures all >> newly added bits are cleared when requested. Previously, there were >> circumstances where some might not be. Includes a new gtest for the >> specific issue. >> This change also includes a fix for JDK-8235714. Contrary to previous >> comments and bug updates, it was not fixed by JDK-8241581 after all. >> This was noticed while testing the change for JDK-8228692. >> CR: >> https://bugs.openjdk.java.net/browse/JDK-8228692 >> https://bugs.openjdk.java.net/browse/JDK-8235714 >> Webrev: >> https://cr.openjdk.java.net/~kbarrett/8228692/open.00/ >> Testing: >> mach5 tier1 >> Local (linux-x64) BitMap tests, including the new test. The new test >> (1) failed before the changes, (2) still failed after either one but >> not both of the fix for relocate and the fix for count_one_bits, (3) >> and passes with both fixes. > > lgtm. > > Thomas Thanks. From coleen.phillimore at oracle.com Tue May 26 13:49:57 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Tue, 26 May 2020 09:49:57 -0400 Subject: RFR 8245289: Clean up offset code in JavaClasses In-Reply-To: References: Message-ID: I might have a better idea, so please disregard this for now. thanks, Coleen On 5/26/20 8:39 AM, coleen.phillimore at oracle.com wrote: > Summary: Use X macro to declare and define offset, and declare > accessors when needed, also make names consistent. > > Changes include. > > 1. making all offset field names start with underscore. > 2. renamed blah_offset_in_bytes to just blah_offset (they are all in > bytes and would never be in "words") > 3. Use X macro to declare then define the field offsets to avoid > retyping all that > 4. Use X macro to define accessors for classes that needed them, so > that the accessors can check if the field has been initialized. Since > these are member fields and static fields, their offsets can never be > zero, so there's no need to have a test for -1.? I kept some > _initialized members for a couple of classes.? They can be removed > later if we want. > ? - I didn't change offset field access internally in the class member > functions everywhere because it wouldn't be a net improvements in > lines of code > 5. removed "private" keyword because in class declarations fields are > private by default. > > Most of these classes have accessor patterns like > > void java_lang_Class::set_class_loader(oop java_class, oop loader) { > ? assert(_class_loader_offset != 0, "offsets should have been > initialized"); > ? java_class->obj_field_put(_class_loader_offset, loader); > } > > oop java_lang_Class::class_loader(oop java_class) { > ? assert(_class_loader_offset != 0, "must be set"); > ? return java_class->obj_field(_class_loader_offset); > } > > So it seems like it'd be useful to wrap that up with the X macro too, > but there are also a lot of classes that don't have this pattern for > all of the fields, ie. they do some other things so there's be too > many exceptions to make this a nice code savings change.? The jvmci > code has this sort of giant X macro that is, in my opinion, completely > unreadable. > > Anyway, this is an improvement for now and tested tier1-6.? What do > you think? > > open webrev at http://cr.openjdk.java.net/~coleenp/2020/8245289.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8245289 > > thanks, > Coleen > From stefan.johansson at oracle.com Tue May 26 13:55:27 2020 From: stefan.johansson at oracle.com (stefan.johansson at oracle.com) Date: Tue, 26 May 2020 15:55:27 +0200 Subject: RFR: 8228692: BitMap::reallocate might not clear some bits In-Reply-To: References: Message-ID: Hi Kim, On 2020-05-22 00:36, Kim Barrett wrote: > Please review this change to BitMap reallocation that ensures all > newly added bits are cleared when requested. Previously, there were > circumstances where some might not be. Includes a new gtest for the > specific issue. > > This change also includes a fix for JDK-8235714. Contrary to previous > comments and bug updates, it was not fixed by JDK-8241581 after all. > This was noticed while testing the change for JDK-8228692. > > CR: > https://bugs.openjdk.java.net/browse/JDK-8228692 > https://bugs.openjdk.java.net/browse/JDK-8235714 > > Webrev: > https://cr.openjdk.java.net/~kbarrett/8228692/open.00/ Looks good! Thanks, Stefan > > Testing: > mach5 tier1 > > Local (linux-x64) BitMap tests, including the new test. The new test > (1) failed before the changes, (2) still failed after either one but > not both of the fix for relocate and the fix for count_one_bits, (3) > and passes with both fixes. > From thomas.schatzl at oracle.com Tue May 26 13:56:34 2020 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 26 May 2020 15:56:34 +0200 Subject: RFR: 8245723: Remove PrintCompressedOopsMode and gc+heap+coops info log to debug level In-Reply-To: <83ef9d38-cf42-f3b3-067d-891afb6c1572@oracle.com> References: <83ef9d38-cf42-f3b3-067d-891afb6c1572@oracle.com> Message-ID: <8ad66cd4-26fd-9e7b-685d-982e93080b0e@oracle.com> Hi, On 26.05.20 14:16, stefan.johansson at oracle.com wrote: > Hi, > > Please review this change that removes the deprecated diagnostic option > PrintCompressedOopsMode. > > Webrev: http://cr.openjdk.java.net/~sjohanss/8245723/00/ > JBS: https://bugs.openjdk.java.net/browse/JDK-8245723 > > Summary > After adding the new gc+init logging in JDK-8244817, the info logging on > gc+heap+coops can be moved to debug level. When looking into this I > realized that gc+heap+coops=info has an old alias flag, > PrintCompressedOopsMode, connected to it. I see no reason keeping this > flag around and from what I understand we don't need to do anything > special since it has been deprecated since JDK 9. > > Testing > Locally verified output and run the tests affected. Also Mach5 tier1-2. > looks good. Please add a verb to the second part of the CR subject before pushing because as is the sentence seems kind of incomprehensible :) Thomas From stefan.johansson at oracle.com Tue May 26 14:09:41 2020 From: stefan.johansson at oracle.com (stefan.johansson at oracle.com) Date: Tue, 26 May 2020 16:09:41 +0200 Subject: RFR: 8245723: Remove PrintCompressedOopsMode and gc+heap+coops info log to debug level In-Reply-To: <8ad66cd4-26fd-9e7b-685d-982e93080b0e@oracle.com> References: <83ef9d38-cf42-f3b3-067d-891afb6c1572@oracle.com> <8ad66cd4-26fd-9e7b-685d-982e93080b0e@oracle.com> Message-ID: <93f2c5cf-5e33-129d-4ef4-cf28eaaedab1@oracle.com> Hi Thomas, On 2020-05-26 15:56, Thomas Schatzl wrote: > Hi, > > On 26.05.20 14:16, stefan.johansson at oracle.com wrote: >> Hi, >> >> Please review this change that removes the deprecated diagnostic >> option PrintCompressedOopsMode. >> >> Webrev: http://cr.openjdk.java.net/~sjohanss/8245723/00/ >> JBS: https://bugs.openjdk.java.net/browse/JDK-8245723 >> >> Summary >> After adding the new gc+init logging in JDK-8244817, the info logging >> on gc+heap+coops can be moved to debug level. When looking into this I >> realized that gc+heap+coops=info has an old alias flag, >> PrintCompressedOopsMode, connected to it. I see no reason keeping this >> flag around and from what I understand we don't need to do anything >> special since it has been deprecated since JDK 9. >> >> Testing >> Locally verified output and run the tests affected. Also Mach5 tier1-2. >> > > ? looks good. > > Please add a verb to the second part of the CR subject before pushing > because as is the sentence seems kind of incomprehensible :) Added "change", thanks for the review, Stefan > > Thomas From kim.barrett at oracle.com Tue May 26 14:43:35 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Tue, 26 May 2020 10:43:35 -0400 Subject: RFR: 8228692: BitMap::reallocate might not clear some bits In-Reply-To: References: Message-ID: <4AB9A6BF-4B59-4458-BEE1-5E08907AD9FF@oracle.com> > On May 26, 2020, at 9:55 AM, stefan.johansson at oracle.com wrote: > > Hi Kim, > > On 2020-05-22 00:36, Kim Barrett wrote: >> Please review this change to BitMap reallocation that ensures all >> newly added bits are cleared when requested. Previously, there were >> circumstances where some might not be. Includes a new gtest for the >> specific issue. >> This change also includes a fix for JDK-8235714. Contrary to previous >> comments and bug updates, it was not fixed by JDK-8241581 after all. >> This was noticed while testing the change for JDK-8228692. >> CR: >> https://bugs.openjdk.java.net/browse/JDK-8228692 >> https://bugs.openjdk.java.net/browse/JDK-8235714 >> Webrev: >> https://cr.openjdk.java.net/~kbarrett/8228692/open.00/ > Looks good! Thanks. > Thanks, > Stefan >> Testing: >> mach5 tier1 >> Local (linux-x64) BitMap tests, including the new test. The new test >> (1) failed before the changes, (2) still failed after either one but >> not both of the fix for relocate and the fix for count_one_bits, (3) >> and passes with both fixes. From igor.ignatyev at oracle.com Tue May 26 20:42:13 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Tue, 26 May 2020 13:42:13 -0700 Subject: Changes in gtest building after 8245610(remove in-tree copy on gtest) Message-ID: <786CD25F-36F5-435B-B731-AA0DEF664715@oracle.com> Dear all, The integration of 8245610[1-3] has slightly changed how to enable/disable hotspot Gtest test suite, which might affect some of you and require changes in your build/configure scripts. This email is to notify you about the possible impact and provide with the suggested changes for your `configure` option-set. The patch removed the copy of Gtest framework which was located in `test/fmw/gtest`, now Gtest source code is to be stored separately and provided via `--with-gtest` configure options. Building the JDK[4] documentation has been updated to reflect this new dependency. The patch also deprecated `--(enable|disable)-hotspot-gtest` flags and changed defaults: unless `--with-gtest` points to the source code of Gtest, the test suite won't be built. jib profiles were updated to have Gtest as dependency, so `jib` automatically downloads Gtest source code and properly passes it to `--with-gtest` option. The impacted people can be grouped by their needs to build Gtest test suite, and whether `jib` is used; the suggested changes for each group: - use `jib` and want Gtest test suite to be built: you are all set, no changes are required; - use `jib` and don't need Gtest test suite, i.e. you used to run `jib configure -- --disable-hotspot-gtest`: you need to replace `--disable-hotspot-gtest` by `--without-gtest`; - don't use `jib` and don't need Gtest test suite, i.e. you used to run `./configure --disable-hotspot-gtest`: you just need to remove `--disable-hotspot-gtest`; - don't use `jib` and want Gtest test suite to be built: you need to get Gtest source code and provide its path via `--with-gtest`. Running Tests[5] section gives suggestions on how the source code can be obtained. Thanks, -- Igor [1] https://bugs.openjdk.java.net/browse/JDK-8245610 [2] https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-May/041900.html [3] https://hg.openjdk.java.net/jdk/jdk/rev/21c94b6f23e7 [4] http://hg.openjdk.java.net/jdk/jdk/raw-file/21c94b6f23e7/doc/building.html [5] http://hg.openjdk.java.net/jdk/jdk/raw-file/21c94b6f23e7/doc/building.html#running-tests From Yang.Zhang at arm.com Wed May 27 02:59:11 2020 From: Yang.Zhang at arm.com (Yang Zhang) Date: Wed, 27 May 2020 02:59:11 +0000 Subject: [aarch64-port-dev ] RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes In-Reply-To: <1909fa9d-98bb-c2fb-45d8-540247d1ca8b@redhat.com> References: <275eb57c-51c0-675e-c32a-91b198023559@redhat.com> <719F9169-ABC4-408E-B732-F1BD9A84337F@oracle.com> <9a13f5df-d946-579d-4282-917dc7338dc8@redhat.com> <09BC0693-80E0-4F87-855E-0B38A6F5EFA2@oracle.com> <668e500e-f621-5a2c-a41e-f73536880f73@redhat.com> <1909fa9d-98bb-c2fb-45d8-540247d1ca8b@redhat.com> Message-ID: > But to my earlier question. please: can the new instructions be moved into jdk head first, and then merged into the Panama branch, or not? The new instructions can be classified as: 1. Instructions that can be matched with NEON instructions directly. MulVB and SqrtVF have been merged into jdk master already. The patch of AbsV is in review [1]. 2. Instructions that Jdk master has middle end support for, but they cannot be matched with NEON instructions directly. Such as AddReductionVL, MulReductionVL, And/Or/XorReductionV These new instructions can be moved into jdk master first, but for auto-vectorization, the performance might not get improved. May I have a new patch for these? 3. Panama/Vector API specific instructions Such as Load/StoreVector ( 16 bits), VectorReinterpret, VectorMaskCmp, MaxV/MinV, VectorBlend etc. These instructions cannot be moved into jdk master first because there isn't middle-end support. Regards Yang [1] https://mail.openjdk.java.net/pipermail/aarch64-port-dev/2020-May/008861.html -----Original Message----- From: Andrew Haley Sent: Tuesday, May 26, 2020 4:25 PM To: Yang Zhang ; Paul Sandoz Cc: hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; core-libs-dev at openjdk.java.net; aarch64-port-dev at openjdk.java.net; nd Subject: Re: [aarch64-port-dev ] RFR (XXL): 8223347: Integration of Vector API (Incubator): AArch64 backend changes On 25/05/2020 09:26, Yang Zhang wrote: > In jdk master, what we need to do is that writing m4 file for existing > vector instructions and placed them to a new file aarch64_neon.ad. > If no question, I will do it right away. I'm not entirely sure that such a change is necessary now. In particular, reorganizing the existing vector instructions is IMO excessive, but I admit that it might be an improvement. But to my earlier question. please: can the new instructions be moved into jdk head first, and then merged into the Panama branch, or not? It'd help if this was possible. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From igor.ignatyev at oracle.com Wed May 27 17:25:01 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Wed, 27 May 2020 10:25:01 -0700 Subject: RFR(S/T) : 8245874 : requires.extraPropDefns.vmOpts doesn't need -Xbootclasspath/a:bootClasses Message-ID: http://cr.openjdk.java.net/~iignatyev//8245874/webrev.00 > 8 lines changed: 2 ins; 0 del; 6 mod; Hi all, could you please review this small and trivial cleanup in TEST.ROOT files of test/jdk/ and test/hotspot/jtreg test suites? from JBS: > to make sure that classes listed in `requires.extraPropDefns.bootlibs` are in BCP, jtreg automatically adds corresponding `-Xbootclasspath/a:` to the options list, so `requires.extraPropDefns.vmOpts` doesn't have to have `-Xbootclasspath/a:bootClasses` the patch - removes -Xbootclasspath/a:bootClasses from requires.extraPropDefns.vmOpts - moves jdk.test.lib.Platform and Container classes from requires.extraPropDefns.bootlibs to requires.extraPropDefns.libs as there is no need for them to be in BCP JBS: https://bugs.openjdk.java.net/browse/JDK-8245874 testing: tier1 webrev: http://cr.openjdk.java.net/~iignatyev//8245874/webrev.00 Thanks, -- Igor From david.holmes at oracle.com Mon May 4 06:51:28 2020 From: david.holmes at oracle.com (David Holmes) Date: Mon, 04 May 2020 06:51:28 -0000 Subject: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant In-Reply-To: References: <3c59b9f9-ec38-18c9-8f24-e1186a08a04a@oracle.com> <410eed04-e2ef-0f4f-1c56-19e6734a10f6@oracle.com> <81d7caa8-4244-85f3-4d4e-78117fe5e25b@oss.nttdata.com> <550b95ac-8b29-1eb8-a507-533e81d02322@oracle.com> Message-ID: <9c49ea2d-e3b8-b576-1d17-d18ad87cd6ed@oracle.com> Hi Richard, On 28/04/2020 12:09 am, Reingruber, Richard wrote: > Hi David, > >> Not a review but some general commentary ... > > That's welcome. Having had to take an even closer look now I have a review comment too :) src/hotspot/share/prims/jvmtiThreadState.cpp void JvmtiThreadState::invalidate_cur_stack_depth() { ! assert(SafepointSynchronize::is_at_safepoint() || ! (Thread::current()->is_VM_thread() && get_thread()->is_vmthread_processing_handshake()) || (JavaThread *)Thread::current() == get_thread(), "must be current thread or at safepoint"); The message needs updating to include handshakes. More below ... >> On 25/04/2020 2:08 am, Reingruber, Richard wrote: >>> Hi Yasumasa, Patricio, >>> >>>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>>> Does it help you? I think it gives you to remove workaround. >>>>> >>>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >>> >>>> Thanks for your information. >>>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>>> I will modify and will test it after yours. >>> >>> Thanks :) >>> >>>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>>> to me, how this has to be handled. >>> >>>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >>> >>> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >>> also I'm unsure if a thread should do safepoint checks while executing a handshake. > >> I'm growing increasingly concerned that use of direct handshakes to >> replace VM operations needs a much greater examination for correctness >> than might initially be thought. I see a number of issues: > > I agree. I'll address your concerns in the context of this review thread for JDK-8238585 below. > > In addition I would suggest to take the general part of the discussion to a dedicated thread or to > the review thread for JDK-8242427. I would like to keep this thread closer to its subject. I will focus on the issues in the context of this particular change then, though the issues themselves are applicable to all handshake situations (and more so with direct handshakes). This is mostly just discussion. >> First, the VMThread executes (most) VM operations with a clean stack in >> a clean state, so it has lots of room to work. If we now execute the >> same logic in a JavaThread then we risk hitting stackoverflows if >> nothing else. But we are also now executing code in a JavaThread and so >> we have to be sure that code is not going to act differently (in a bad >> way) if executed by a JavaThread rather than the VMThread. For example, >> may it be possible that if executing in the VMThread we defer some >> activity that might require execution of Java code, or else hand it off >> to one of the service threads? If we execute that code directly in the >> current JavaThread instead we may not be in a valid state (e.g. consider >> re-entrancy to various subsystems that is not allowed). > > It is not too complex, what EnterInterpOnlyModeClosure::do_thread() is doing. I already added a > paragraph to the JBS-Item [1] explaining why the direct handshake is sufficient from a > synchronization point of view. Just to be clear, your proposed change is not using a direct handshake. > Furthermore the stack is walked and the return pc of compiled frames is replaced with the address of > the deopt handler. > > I can't see why this cannot be done with a direct handshake. Something very similar is already done > in JavaThread::deoptimize_marked_methods() which is executed as part of an ordinary handshake. Note that existing non-direct handshakes may also have issues that not have been fully investigated. > The demand on stack-space should be very modest. I would not expect a higher risk for stackoverflow. For the target thread if you use more stack than would be used stopping at a safepoint then you are at risk. For the thread initiating the direct handshake if you use more stack than would be used enqueuing a VM operation, then you are at risk. As we have not quantified these numbers, nor have any easy way to establish the stack use of the actual code to be executed, we're really just hoping for the best. This is a general problem with handshakes that needs to be investigated more deeply. As a simple, general, example just imagine if the code involves logging that might utilise an on-stack buffer. >> Second, we have this question mark over what happens if the operation >> hits further safepoint or handshake polls/checks? Are there constraints >> on what is allowed here? How can we recognise this problem may exist and >> so deal with it? > > The thread in EnterInterpOnlyModeClosure::do_thread() can't become safepoint/handshake safe. I > tested locally test/hotspot/jtreg:vmTestbase_nsk_jvmti with a NoSafepointVerifier. That's good to hear but such tests are not exhaustive, they will detect if you do reach a safepoint/handshake but they can't prove that you cannot reach one. What you have done is necessary but may not be sufficient. Plus you didn't actually add the NSV to the code - is there a reason we can't actually keep it in do_thread? (I'm not sure if the NSV also acts as a NoHandshakeVerifier?) >> Third, while we are generally considering what appear to be >> single-thread operations, which should be amenable to a direct >> handshake, we also have to be careful that some of the code involved >> doesn't already expect/assume we are at a safepoint - e.g. a VM op may >> not need to take a lock where a direct handshake might! > > See again my arguments in the JBS item [1]. Yes I see the reasoning and that is good. My point is a general one as it may not be obvious when such assumptions exist in the current code. Thanks, David > Thanks, > Richard. > > [1] https://bugs.openjdk.java.net/browse/JDK-8238585 > > -----Original Message----- > From: David Holmes > Sent: Montag, 27. April 2020 07:16 > To: Reingruber, Richard ; Yasumasa Suenaga ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net > Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant > > Hi all, > > Not a review but some general commentary ... > > On 25/04/2020 2:08 am, Reingruber, Richard wrote: >> Hi Yasumasa, Patricio, >> >>>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>>> Does it help you? I think it gives you to remove workaround. >>>> >>>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >> >>> Thanks for your information. >>> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >>> I will modify and will test it after yours. >> >> Thanks :) >> >>>> Also my first impression was that it won't be that easy from a synchronization point of view to >>>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>>> to me, how this has to be handled. >> >>> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >> >> Yes. To me it is unclear what synchronization is necessary, if it is called during a handshake. And >> also I'm unsure if a thread should do safepoint checks while executing a handshake. > > I'm growing increasingly concerned that use of direct handshakes to > replace VM operations needs a much greater examination for correctness > than might initially be thought. I see a number of issues: > > First, the VMThread executes (most) VM operations with a clean stack in > a clean state, so it has lots of room to work. If we now execute the > same logic in a JavaThread then we risk hitting stackoverflows if > nothing else. But we are also now executing code in a JavaThread and so > we have to be sure that code is not going to act differently (in a bad > way) if executed by a JavaThread rather than the VMThread. For example, > may it be possible that if executing in the VMThread we defer some > activity that might require execution of Java code, or else hand it off > to one of the service threads? If we execute that code directly in the > current JavaThread instead we may not be in a valid state (e.g. consider > re-entrancy to various subsystems that is not allowed). > > Second, we have this question mark over what happens if the operation > hits further safepoint or handshake polls/checks? Are there constraints > on what is allowed here? How can we recognise this problem may exist and > so deal with it? > > Third, while we are generally considering what appear to be > single-thread operations, which should be amenable to a direct > handshake, we also have to be careful that some of the code involved > doesn't already expect/assume we are at a safepoint - e.g. a VM op may > not need to take a lock where a direct handshake might! > > Cheers, > David > ----- > >> @Patricio, coming back to my question [1]: >> >> In the example you gave in your answer [2]: the java thread would execute a vm operation during a >> direct handshake operation, while the VMThread is actually in the middle of a VM_HandshakeAllThreads >> operation, waiting to handshake the same handshakee: why can't the VMThread just proceed? The >> handshakee would be safepoint safe, wouldn't it? >> >> Thanks, Richard. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301677&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301677 >> >> [2] https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14301763&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14301763 >> >> -----Original Message----- >> From: Yasumasa Suenaga >> Sent: Freitag, 24. April 2020 17:23 >> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >> >> Hi Richard, >> >> On 2020/04/24 23:44, Reingruber, Richard wrote: >>> Hi Yasumasa, >>> >>>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>>> Does it help you? I think it gives you to remove workaround. >>> >>> I think it would not help that much. Note that when replacing VM_SetFramePop with a direct handshake >>> you could not just execute VM_EnterInterpOnlyMode as a nested vm operation [1]. So you would have to >>> change/replace VM_EnterInterpOnlyMode and I would have to adapt to these changes. >> >> Thanks for your information. >> I tested my patch with both vmTestbase/nsk/jvmti/PopFrame and vmTestbase/nsk/jvmti/NotifyFramePop. >> I will modify and will test it after yours. >> >> >>> Also my first impression was that it won't be that easy from a synchronization point of view to >>> replace VM_SetFramePop with a direct handshake. E.g. VM_SetFramePop::doit() indirectly calls >>> JvmtiEventController::set_frame_pop(JvmtiEnvThreadState *ets, JvmtiFramePop fpop) where >>> JvmtiThreadState_lock is acquired with safepoint check, if not at safepoint. It's not directly clear >>> to me, how this has to be handled. >> >> I think JvmtiEventController::set_frame_pop() should hold JvmtiThreadState_lock because it affects other JVMTI operation especially FramePop event. >> >> >> Thanks, >> >> Yasumasa >> >> >>> So it appears to me that it would be easier to push JDK-8242427 after this (JDK-8238585). >>> >>>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>> >>> Would be interesting to see how you handled the issues above :) >>> >>> Thanks, Richard. >>> >>> [1] See question in comment https://bugs.openjdk.java.net/browse/JDK-8230594?focusedCommentId=14302030&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14302030 >>> >>> -----Original Message----- >>> From: Yasumasa Suenaga >>> Sent: Freitag, 24. April 2020 13:34 >>> To: Reingruber, Richard ; Patricio Chilano ; serguei.spitsyn at oracle.com; Vladimir Ivanov ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>> >>> Hi Richard, >>> >>> I will send review request to replace VM_SetFramePop to handshake in early next week in JDK-8242427. >>> Does it help you? I think it gives you to remove workaround. >>> >>> (The patch is available, but I want to see the result of PIT in this weekend whether JDK-8242425 works fine.) >>> >>> >>> Thanks, >>> >>> Yasumasa >>> >>> >>> On 2020/04/24 17:18, Reingruber, Richard wrote: >>>> Hi Patricio, Vladimir, and Serguei, >>>> >>>> now that direct handshakes are available, I've updated the patch to make use of them. >>>> >>>> In addition I have done some clean-up changes I missed in the first webrev. >>>> >>>> Finally I have implemented the workaround suggested by Patricio to avoid nesting the handshake >>>> into the vm operation VM_SetFramePop [1] >>>> >>>> Kindly review again: >>>> >>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1/ >>>> Webrev(delta): http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.1.inc/ >>>> >>>> I updated the JBS item explaining why the vm operation VM_EnterInterpOnlyMode can be replaced with a >>>> direct handshake: >>>> >>>> JBS: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>> >>>> Testing: >>>> >>>> * JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>> >>>> * Submit-repo: mach5-one-rrich-JDK-8238585-20200423-1436-10441737 >>>> >>>> Thanks, >>>> Richard. >>>> >>>> [1] An assertion in Handshake::execute_direct() fails, if called be VMThread, because it is no JavaThread. >>>> >>>> -----Original Message----- >>>> From: hotspot-dev On Behalf Of Reingruber, Richard >>>> Sent: Freitag, 14. Februar 2020 19:47 >>>> To: Patricio Chilano ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>> Subject: RE: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>> >>>> Hi Patricio, >>>> >>>> > > I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>> > > handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>> > > Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>> > > >>>> > > > Alternatively I think you could do something similar to what we do in >>>> > > > Deoptimization::deoptimize_all_marked(): >>>> > > > >>>> > > > EnterInterpOnlyModeClosure hs; >>>> > > > if (SafepointSynchronize::is_at_safepoint()) { >>>> > > > hs.do_thread(state->get_thread()); >>>> > > > } else { >>>> > > > Handshake::execute(&hs, state->get_thread()); >>>> > > > } >>>> > > > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>> > > > HandshakeClosure() constructor) >>>> > > >>>> > > Maybe this could be used also in the Handshake::execute() methods as general solution? >>>> > Right, we could also do that. Avoiding to clear the polling page in >>>> > HandshakeState::clear_handshake() should be enough to fix this issue and >>>> > execute a handshake inside a safepoint, but adding that "if" statement >>>> > in Hanshake::execute() sounds good to avoid all the extra code that we >>>> > go through when executing a handshake. I filed 8239084 to make that change. >>>> >>>> Thanks for taking care of this and creating the RFE. >>>> >>>> > >>>> > > > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>> > > > always called in a nested operation or just sometimes. >>>> > > >>>> > > At least one execution path without vm operation exists: >>>> > > >>>> > > JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>> > > JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>> > > JvmtiEventControllerPrivate::recompute_enabled() : void >>>> > > JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>> > > JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>> > > JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>> > > jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>> > > >>>> > > I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>> > > handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>> > > encouraged to do it with a handshake :) >>>> > Ah! I think you can still do it with a handshake with the >>>> > Deoptimization::deoptimize_all_marked() like solution. I can change the >>>> > if-else statement with just the Handshake::execute() call in 8239084. >>>> > But up to you. : ) >>>> >>>> Well, I think that's enough encouragement :) >>>> I'll wait for 8239084 and try then again. >>>> (no urgency and all) >>>> >>>> Thanks, >>>> Richard. >>>> >>>> -----Original Message----- >>>> From: Patricio Chilano >>>> Sent: Freitag, 14. Februar 2020 15:54 >>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>> >>>> Hi Richard, >>>> >>>> On 2/14/20 9:58 AM, Reingruber, Richard wrote: >>>>> Hi Patricio, >>>>> >>>>> thanks for having a look. >>>>> >>>>> > I?m only commenting on the handshake changes. >>>>> > I see that operation VM_EnterInterpOnlyMode can be called inside >>>>> > operation VM_SetFramePop which also allows nested operations. Here is a >>>>> > comment in VM_SetFramePop definition: >>>>> > >>>>> > // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>> > // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>> > >>>>> > So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>> > could have a handshake inside a safepoint operation. The issue I see >>>>> > there is that at the end of the handshake the polling page of the target >>>>> > thread could be disarmed. So if the target thread happens to be in a >>>>> > blocked state just transiently and wakes up then it will not stop for >>>>> > the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>> > polling page is armed at the beginning of disarm_safepoint(). >>>>> >>>>> I'm really glad you noticed the problematic nesting. This seems to be a general issue: currently a >>>>> handshake cannot be nested in a vm operation. Maybe it should be asserted in the >>>>> Handshake::execute() methods that they are not called by the vm thread evaluating a vm operation? >>>>> >>>>> > Alternatively I think you could do something similar to what we do in >>>>> > Deoptimization::deoptimize_all_marked(): >>>>> > >>>>> > EnterInterpOnlyModeClosure hs; >>>>> > if (SafepointSynchronize::is_at_safepoint()) { >>>>> > hs.do_thread(state->get_thread()); >>>>> > } else { >>>>> > Handshake::execute(&hs, state->get_thread()); >>>>> > } >>>>> > (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>> > HandshakeClosure() constructor) >>>>> >>>>> Maybe this could be used also in the Handshake::execute() methods as general solution? >>>> Right, we could also do that. Avoiding to clear the polling page in >>>> HandshakeState::clear_handshake() should be enough to fix this issue and >>>> execute a handshake inside a safepoint, but adding that "if" statement >>>> in Hanshake::execute() sounds good to avoid all the extra code that we >>>> go through when executing a handshake. I filed 8239084 to make that change. >>>> >>>>> > I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>> > always called in a nested operation or just sometimes. >>>>> >>>>> At least one execution path without vm operation exists: >>>>> >>>>> JvmtiEventControllerPrivate::enter_interp_only_mode(JvmtiThreadState *) : void >>>>> JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState *) : jlong >>>>> JvmtiEventControllerPrivate::recompute_enabled() : void >>>>> JvmtiEventControllerPrivate::change_field_watch(jvmtiEvent, bool) : void (2 matches) >>>>> JvmtiEventController::change_field_watch(jvmtiEvent, bool) : void >>>>> JvmtiEnv::SetFieldAccessWatch(fieldDescriptor *) : jvmtiError >>>>> jvmti_SetFieldAccessWatch(jvmtiEnv *, jclass, jfieldID) : jvmtiError >>>>> >>>>> I tend to revert back to VM_EnterInterpOnlyMode as it wasn't my main intent to replace it with a >>>>> handshake, but to avoid making the compiled methods on stack not_entrant.... unless I'm further >>>>> encouraged to do it with a handshake :) >>>> Ah! I think you can still do it with a handshake with the >>>> Deoptimization::deoptimize_all_marked() like solution. I can change the >>>> if-else statement with just the Handshake::execute() call in 8239084. >>>> But up to you.? : ) >>>> >>>> Thanks, >>>> Patricio >>>>> Thanks again, >>>>> Richard. >>>>> >>>>> -----Original Message----- >>>>> From: Patricio Chilano >>>>> Sent: Donnerstag, 13. Februar 2020 18:47 >>>>> To: Reingruber, Richard ; serviceability-dev at openjdk.java.net; hotspot-compiler-dev at openjdk.java.net; hotspot-dev at openjdk.java.net; hotspot-runtime-dev at openjdk.java.net; hotspot-gc-dev at openjdk.java.net >>>>> Subject: Re: RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant >>>>> >>>>> Hi Richard, >>>>> >>>>> I?m only commenting on the handshake changes. >>>>> I see that operation VM_EnterInterpOnlyMode can be called inside >>>>> operation VM_SetFramePop which also allows nested operations. Here is a >>>>> comment in VM_SetFramePop definition: >>>>> >>>>> // Nested operation must be allowed for the VM_EnterInterpOnlyMode that is >>>>> // called from the JvmtiEventControllerPrivate::recompute_thread_enabled. >>>>> >>>>> So if we change VM_EnterInterpOnlyMode to be a handshake, then now we >>>>> could have a handshake inside a safepoint operation. The issue I see >>>>> there is that at the end of the handshake the polling page of the target >>>>> thread could be disarmed. So if the target thread happens to be in a >>>>> blocked state just transiently and wakes up then it will not stop for >>>>> the ongoing safepoint. Maybe I can file an RFE to assert that the >>>>> polling page is armed at the beginning of disarm_safepoint(). >>>>> >>>>> I think one option could be to remove >>>>> SafepointMechanism::disarm_if_needed() in >>>>> HandshakeState::clear_handshake() and let each JavaThread disarm itself >>>>> for the handshake case. >>>>> >>>>> Alternatively I think you could do something similar to what we do in >>>>> Deoptimization::deoptimize_all_marked(): >>>>> >>>>> ? EnterInterpOnlyModeClosure hs; >>>>> ? if (SafepointSynchronize::is_at_safepoint()) { >>>>> ??? hs.do_thread(state->get_thread()); >>>>> ? } else { >>>>> ??? Handshake::execute(&hs, state->get_thread()); >>>>> ? } >>>>> (you could pass ?EnterInterpOnlyModeClosure? directly to the >>>>> HandshakeClosure() constructor) >>>>> >>>>> I don?t know JVMTI code so I?m not sure if VM_EnterInterpOnlyMode is >>>>> always called in a nested operation or just sometimes. >>>>> >>>>> Thanks, >>>>> Patricio >>>>> >>>>> On 2/12/20 7:23 AM, Reingruber, Richard wrote: >>>>>> // Repost including hotspot runtime and gc lists. >>>>>> // Dean Long suggested to do so, because the enhancement replaces a vm operation >>>>>> // with a handshake. >>>>>> // Original thread: http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-February/030359.html >>>>>> >>>>>> Hi, >>>>>> >>>>>> could I please get reviews for this small enhancement in hotspot's jvmti implementation: >>>>>> >>>>>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.0/ >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8238585 >>>>>> >>>>>> The change avoids making all compiled methods on stack not_entrant when switching a java thread to >>>>>> interpreter only execution for jvmti purposes. It is sufficient to deoptimize the compiled frames on stack. >>>>>> >>>>>> Additionally a handshake is used instead of a vm operation to walk the stack and do the deoptimizations. >>>>>> >>>>>> Testing: JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms. >>>>>> >>>>>> Thanks, Richard. >>>>>> >>>>>> See also my question if anyone knows a reason for making the compiled methods not_entrant: >>>>>> http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-January/030339.html >>>> From thomas.stuefe at gmail.com Thu May 28 07:31:32 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 28 May 2020 09:31:32 +0200 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest In-Reply-To: References: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> <0c5d41f4-eebd-a8ae-d537-e14a878a6663@oracle.com> <7B996E95-B57D-4F38-97F1-65FFCA2DD01E@oracle.com> Message-ID: Hi, I know the boat has sailed, I missed this one. But I am a bit unhappy about this. I always build with gtests; I think it makes no sense to not build with gtest. Even if you do not want to run the gtests (and it makes sense to always do since its a good first-base validity test), hotspot developers should always build them since changes in the hotspot sources may break hotspot gtests. hotspot source and gtests are a unity. So if it makes no sense to not build gtest, I should not need a special option to specify gtest location - I'd argue that the default case should not require special options. The JBS issue states that "it can be treated like any other external dependencies" but this comparison does not hold - almost all other dependencies come in the form of well established libraries with standard headers and standard installation locations which can be coded as default values. On a recent mainstream platform I do not have to specify any paths to libraries to build OpenJDK. This is now different, I always have to manually download gtests and specify gtest location. This is regrettable. Thanks, Thomas On Tue, May 26, 2020 at 2:27 PM Magnus Ihse Bursie < magnus.ihse.bursie at oracle.com> wrote: > > > On 2020-05-25 19:53, Igor Ignatyev wrote: > > Hi Magnus, > > > >> On May 25, 2020, at 7:46 AM, Magnus Ihse Bursie > >> >> > wrote: > >> > >> Looks basically good to me. > > thanks for your review! > >> > >> We need to document the dependency on gtest, and how to retrieve it. > >> I recommend you add a section next to the JTReg information (under > >> the "## Running Tests" heading) in doc/building.md. I think you > >> should include basically the same information as you did in your > >> follow-up mail, that was informative and clear. > > that's a good suggestion, I've added a small paragraph to 'Running > > Tests' in doc/building.md and regenerated corresponding .html file -- > > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.doc/ > > please let me know if you think something should be added or reworded. > Looks great! Thank you. > > /Magnus > > > >> > >> I assume the most suitable replacement for developers who are used to > >> add a '--disable-hotspot-gtest' to e.g. a pre-definied jib > >> configuration is to now use '--without-gtest'. This will need to be > >> communicated, perhaps to both hotspot-dev and jdk-dev. > > sure, after this gets integrated, I'll let both hotspot-dev and > > jdk-dev aliases know which changes might be required to enable/disable > > hotspot gtest tests compilation. > > > > Thanks. > > -- Igor > > > >> > >> /Magnus > >> > >> On 2020-05-22 20:12, Igor Ignatyev wrote: > >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ > >>>> 132 lines changed: 80 ins; 36 del; 16 mod > >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ > >>>> 57482 lines changed: 80 ins; 57385 del; 17 mod; > >>> Hi all, > >>> > >>> could you please review this small (if you ignore removal part) > >>> patch which removes in-tree copy of gtest (test/fmw/gtest) and > >>> updates makefiles to use one provided thru an added configure option > >>> `--with-gtest`? the previously used configure option > >>> `--enable-hotspot-gtest` gets depricated. > >>> > >>> the patch also compiles gtest and gmock source code into a static > >>> library so they can be compiled w/ all warnings disabled. > >>> > >>> from JBS: > >>>> w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all > >>>> compilers used by OpenJDK became supported by gtest out-of-box, so > >>>> there is no need to have our copy of gtest framework (including > >>>> gmock) within OpenJDK source tree. instead, it can be treated like > >>>> any other external dependencies, and a pointer to the directory w/ > >>>> gtest code can be passed via configure options. > >>> > >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 > >>> webrevs: > >>> - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ > >>> (meaningful changes) > >>> - > >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ > >>> (all changes) > >>> testing: > >>> - gtest tests on {linux,windows,macosx}-x64; > >>> - tier[1-5] builds which include but not limited to linux-aarch64, > >>> linux-arm32, linux-x64-zero > >>> > >>> Thanks, > >>> -- Igor > >>> > >> > > > > From coleen.phillimore at oracle.com Thu May 28 12:48:53 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 28 May 2020 08:48:53 -0400 Subject: RFR 8245289: Clean up offset code in JavaClasses Message-ID: Summary: Make offset member names consistent and private, move static initializations near owning classes This one is better.? I gave up on the X macros because they didn't save typing.? The changes here were to add underscores to offset field names, remove _in_bytes for offset accessors and add asserts that the offset is non-zero, made java_lang_ref_Reference offset fields private, and moved static member definitions closer to the class instead of some random place in javaClasses.cpp. See discussion for 8243996 Remove hardcoded field offsets for more details. https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-May/041732.html Tested with tier1-6. open webrev at http://cr.openjdk.java.net/~coleenp/2020/8245289.01/webrev bug link https://bugs.openjdk.java.net/browse/JDK-8245289 Thanks, Coleen From frederic.parain at oracle.com Thu May 28 16:57:31 2020 From: frederic.parain at oracle.com (Frederic Parain) Date: Thu, 28 May 2020 12:57:31 -0400 Subject: RFR 8245289: Clean up offset code in JavaClasses In-Reply-To: References: Message-ID: Coleen, This is a huge patch, fortunately most of the changes are related to the removal of the _in_bytes suffix and the encapsulation of fields and were easy to review. javaClasses.hpp/javaClasses.hpp: a lot of changes and code refactoring there, that are not easy to track with a webrev. Most changes make the code cleaner. However, there?re some inconsistencies on static fields declaration: some are initialized with an explicit value, others are not, and I?m not seeing a clear rule why. Otherwise, looks good. Regards, Fred > On May 28, 2020, at 08:48, coleen.phillimore at oracle.com wrote: > > Summary: Make offset member names consistent and private, move static initializations near owning classes > > This one is better. I gave up on the X macros because they didn't save typing. The changes here were to add underscores to offset field names, remove _in_bytes for offset accessors and add asserts that the offset is non-zero, made java_lang_ref_Reference offset fields private, and moved static member definitions closer to the class instead of some random place in javaClasses.cpp. > > See discussion for 8243996 Remove hardcoded field offsets for more details. https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-May/041732.html > > Tested with tier1-6. > > open webrev at http://cr.openjdk.java.net/~coleenp/2020/8245289.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8245289 > > Thanks, > Coleen From coleen.phillimore at oracle.com Thu May 28 17:09:11 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Thu, 28 May 2020 13:09:11 -0400 Subject: RFR 8245289: Clean up offset code in JavaClasses In-Reply-To: References: Message-ID: On 5/28/20 12:57 PM, Frederic Parain wrote: > Coleen, > > This is a huge patch, fortunately most of the changes are > related to the removal of the _in_bytes suffix and the encapsulation > of fields and were easy to review. > > javaClasses.hpp/javaClasses.hpp: a lot of changes and code refactoring > there, that are not easy to track with a webrev. Most changes make the > code cleaner. However, there?re some inconsistencies on static fields > declaration: some are initialized with an explicit value, others are > not, and I?m not seeing a clear rule why. Fred, Thank you for looking at the patch.? It is unfortunately hard to read with webrev by moving things.? I did move the static definitions and didn't remove the initializations in cases that had them.? The initialization to zero should be unnecessary, and I guess I can remove them.? I could also add initializations to the ones missing.? I don't have a preference to which to do, so I left them as they were.? What do you think? thanks, Coleen > > Otherwise, looks good. > > Regards, > > Fred > > >> On May 28, 2020, at 08:48, coleen.phillimore at oracle.com wrote: >> >> Summary: Make offset member names consistent and private, move static initializations near owning classes >> >> This one is better. I gave up on the X macros because they didn't save typing. The changes here were to add underscores to offset field names, remove _in_bytes for offset accessors and add asserts that the offset is non-zero, made java_lang_ref_Reference offset fields private, and moved static member definitions closer to the class instead of some random place in javaClasses.cpp. >> >> See discussion for 8243996 Remove hardcoded field offsets for more details. https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-May/041732.html >> >> Tested with tier1-6. >> >> open webrev at http://cr.openjdk.java.net/~coleenp/2020/8245289.01/webrev >> bug link https://bugs.openjdk.java.net/browse/JDK-8245289 >> >> Thanks, >> Coleen From frederic.parain at oracle.com Thu May 28 17:32:10 2020 From: frederic.parain at oracle.com (Frederic Parain) Date: Thu, 28 May 2020 13:32:10 -0400 Subject: RFR 8245289: Clean up offset code in JavaClasses In-Reply-To: References: Message-ID: > On May 28, 2020, at 13:09, coleen.phillimore at oracle.com wrote: > > > > On 5/28/20 12:57 PM, Frederic Parain wrote: >> Coleen, >> >> This is a huge patch, fortunately most of the changes are >> related to the removal of the _in_bytes suffix and the encapsulation >> of fields and were easy to review. >> >> javaClasses.hpp/javaClasses.hpp: a lot of changes and code refactoring >> there, that are not easy to track with a webrev. Most changes make the >> code cleaner. However, there?re some inconsistencies on static fields >> declaration: some are initialized with an explicit value, others are >> not, and I?m not seeing a clear rule why. > > Fred, Thank you for looking at the patch. It is unfortunately hard to read with webrev by moving things. I did move the static definitions and didn't remove the initializations in cases that had them. The initialization to zero should be unnecessary, and I guess I can remove them. I could also add initializations to the ones missing. I don't have a preference to which to do, so I left them as they were. What do you think? I don?t have a preference either, so I would ask: what is the recommended way to code this in HotSpot? The inconsistencies are not causing issues, it?s more of a style question, so I?m fine if you push your patch as is. Thank you, Fred > > thanks, > Coleen >> >> Otherwise, looks good. >> >> Regards, >> >> Fred >> >> >>> On May 28, 2020, at 08:48, coleen.phillimore at oracle.com wrote: >>> >>> Summary: Make offset member names consistent and private, move static initializations near owning classes >>> >>> This one is better. I gave up on the X macros because they didn't save typing. The changes here were to add underscores to offset field names, remove _in_bytes for offset accessors and add asserts that the offset is non-zero, made java_lang_ref_Reference offset fields private, and moved static member definitions closer to the class instead of some random place in javaClasses.cpp. >>> >>> See discussion for 8243996 Remove hardcoded field offsets for more details. https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-May/041732.html >>> >>> Tested with tier1-6. >>> >>> open webrev at http://cr.openjdk.java.net/~coleenp/2020/8245289.01/webrev >>> bug link https://bugs.openjdk.java.net/browse/JDK-8245289 >>> >>> Thanks, >>> Coleen > From joe.darcy at oracle.com Thu May 28 21:51:34 2020 From: joe.darcy at oracle.com (Joe Darcy) Date: Thu, 28 May 2020 14:51:34 -0700 Subject: JDK 16 RFR of JDK-8235496 : "Start of release updates for JDK 16" and related work Message-ID: Hello, The start of JDK 16 is right around the corner with JDK 15 forking off from jdk/jdk at ramp down 1 [1]. Please review the usual hotspot-related portions of ??? http://cr.openjdk.java.net/~darcy/8235496.4/ The hotspot src and test updates were as expected; thanks to David H. and Mikael V. for early assistance. Related review threads started on compiler-dev and build-dev. Thanks, -Joe [1] http://openjdk.java.net/projects/jdk/15/ From mikael.vidstedt at oracle.com Thu May 28 22:02:16 2020 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Thu, 28 May 2020 15:02:16 -0700 Subject: JDK 16 RFR of JDK-8235496 : "Start of release updates for JDK 16" and related work In-Reply-To: References: Message-ID: <2E62962B-7D74-43A5-B736-037A2111A665@oracle.com> (Still) looks good. Thanks for coordinating! Cheers, Mikael > On May 28, 2020, at 2:51 PM, Joe Darcy wrote: > > Hello, > > The start of JDK 16 is right around the corner with JDK 15 forking off from jdk/jdk at ramp down 1 [1]. > > Please review the usual hotspot-related portions of > > http://cr.openjdk.java.net/~darcy/8235496.4/ > > The hotspot src and test updates were as expected; thanks to David H. and Mikael V. for early assistance. > > Related review threads started on compiler-dev and build-dev. > > Thanks, > > -Joe > > [1] http://openjdk.java.net/projects/jdk/15/ > From david.holmes at oracle.com Thu May 28 23:47:00 2020 From: david.holmes at oracle.com (David Holmes) Date: Fri, 29 May 2020 09:47:00 +1000 Subject: JDK 16 RFR of JDK-8235496 : "Start of release updates for JDK 16" and related work In-Reply-To: <2E62962B-7D74-43A5-B736-037A2111A665@oracle.com> References: <2E62962B-7D74-43A5-B736-037A2111A665@oracle.com> Message-ID: <8d671c7d-9a51-3132-cf2e-1d596548b828@oracle.com> +1 Thanks, David On 29/05/2020 8:02 am, Mikael Vidstedt wrote: > > (Still) looks good. Thanks for coordinating! > > Cheers, > Mikael > >> On May 28, 2020, at 2:51 PM, Joe Darcy wrote: >> >> Hello, >> >> The start of JDK 16 is right around the corner with JDK 15 forking off from jdk/jdk at ramp down 1 [1]. >> >> Please review the usual hotspot-related portions of >> >> http://cr.openjdk.java.net/~darcy/8235496.4/ >> >> The hotspot src and test updates were as expected; thanks to David H. and Mikael V. for early assistance. >> >> Related review threads started on compiler-dev and build-dev. >> >> Thanks, >> >> -Joe >> >> [1] http://openjdk.java.net/projects/jdk/15/ >> > From ioi.lam at oracle.com Fri May 29 00:23:53 2020 From: ioi.lam at oracle.com (Ioi Lam) Date: Thu, 28 May 2020 17:23:53 -0700 Subject: RFR: 8240259: Disable -Wshift-negative-value warnings In-Reply-To: <7506FB40-FF15-44A8-9071-9AED46081CAC@oracle.com> References: <87d1ceb1-d18a-5875-73c3-9ae89f100f31@oracle.com> <7506FB40-FF15-44A8-9071-9AED46081CAC@oracle.com> Message-ID: <01bdcdc1-e4a0-02ae-6d78-57cc9fa4c868@oracle.com> Looks good to me. When I enabled c++14 in gcc, I had to do a few of these manually and didn't see any test failures, so I think the gcc warning isn't really catching anything bad in our code. Thanks - Ioi Node* RShiftINode::Identity(PhaseGVN* phase) { ?? ... #ifdef __GNUC__ #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wshift-negative-value" #endif ??? int lo = (-1 << (BitsPerJavaInteger - ((uint)shift)-1)); // FFFF8000 #ifdef __GNUC__ #pragma GCC diagnostic pop #endif and the VM passed On 5/22/20 1:23 AM, Kim Barrett wrote: >> On May 22, 2020, at 3:38 AM, Magnus Ihse Bursie wrote: >> >> Looks good to me. > Thanks. > >> /Magnus >> >> On 2020-05-22 03:54, Kim Barrett wrote: >>> Please review this change which disables warnings for left shift of a >>> negative value when compiling HotSpot with gcc or clang. This warning >>> isn't helpful, given that all compilers seem to do the obvious thing, >>> and that obvious thing will soon-ish (C++20) be the specified thing. >>> See the CR for more details. >>> >>> CR: >>> https://bugs.openjdk.java.net/browse/JDK-8240259 >>> >>> Webrev: >>> https://cr.openjdk.java.net/~kbarrett/8240259/open.00/ >>> >>> Testing: >>> mach5 tier1. Also tier1-5 with C++14 enabled. >>> >>> Note that there are already occurrences of left shift of negative >>> values in our code, with no apparent consequences, even when testing >>> with C++11/14 enabled. > From kim.barrett at oracle.com Fri May 29 01:33:30 2020 From: kim.barrett at oracle.com (Kim Barrett) Date: Thu, 28 May 2020 21:33:30 -0400 Subject: RFR: 8240259: Disable -Wshift-negative-value warnings In-Reply-To: <01bdcdc1-e4a0-02ae-6d78-57cc9fa4c868@oracle.com> References: <87d1ceb1-d18a-5875-73c3-9ae89f100f31@oracle.com> <7506FB40-FF15-44A8-9071-9AED46081CAC@oracle.com> <01bdcdc1-e4a0-02ae-6d78-57cc9fa4c868@oracle.com> Message-ID: > On May 28, 2020, at 8:23 PM, Ioi Lam wrote: > > Looks good to me. Thanks. > When I enabled c++14 in gcc, I had to do a few of these manually and didn't see any test failures, so I think the gcc warning isn't really catching anything bad in our code. > > Thanks > - Ioi > > Node* RShiftINode::Identity(PhaseGVN* phase) { > ... > #ifdef __GNUC__ > #pragma GCC diagnostic push > #pragma GCC diagnostic ignored "-Wshift-negative-value" > #endif > int lo = (-1 << (BitsPerJavaInteger - ((uint)shift)-1)); // FFFF8000 > #ifdef __GNUC__ > #pragma GCC diagnostic pop > #endif Yes, that one looks familiar. I thought there were a couple more, but maybe they disappeared in other code changes. I haven?t tried doing a C++14 build without that warning disabled for quite a while. From igor.ignatyev at oracle.com Fri May 29 03:20:17 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Thu, 28 May 2020 20:20:17 -0700 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest In-Reply-To: References: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> <0c5d41f4-eebd-a8ae-d537-e14a878a6663@oracle.com> <7B996E95-B57D-4F38-97F1-65FFCA2DD01E@oracle.com> Message-ID: <2A2FD345-F641-4D6A-85DC-6379E8582C6F@oracle.com> Hi Thomas, I regret that this patch made you unhappier. I fully agree that all hotspot developers are to always build *and* run gtest tests, yet not all people who work on OpenJDK project are hotspot developers. I also believe that all OpenJDK developers are to run tests related to the area they are changing. The vast majority of such tests are jtreg tests. And jtreg, for better or worse, is a counterexample to your last paragraph -- it's an essential part of OpenJDK, but it doesn't come in a form of well established library, and none of known to me platforms have jtreg in their package manager, so you do have to manually download jtreg and specify its location one way or another. I understand that there is a difference b/w gtest and jtreg, as jtreg isn't really need at build time. However the proper way to run any of OpenJDK tests is by `make test` and for it to work you need to either executed configure w/ --with-jtreg or specify JT_HOME on each invocation of `make test`, where the latter is a preferred way. From that point of view, gtest and jtreg aren't different, you need to download both manually and specify their location at configure-time. with that being said, I truly understand the inconvenience it causes to you, yet given you need to download gtest only once and it's fairly easy to hide `with-gtest` behind a script or an alias like configure_openjdk='bash ./configure --with-gtest=$GTEST_HOME', I hope it won't cause problems for you and won't discourage you in anyways. Thanks, -- Igor > On May 28, 2020, at 12:31 AM, Thomas St?fe wrote: > > Hi, > > I know the boat has sailed, I missed this one. But I am a bit unhappy about this. > > I always build with gtests; I think it makes no sense to not build with gtest. Even if you do not want to run the gtests (and it makes sense to always do since its a good first-base validity test), hotspot developers should always build them since changes in the hotspot sources may break hotspot gtests. hotspot source and gtests are a unity. > > So if it makes no sense to not build gtest, I should not need a special option to specify gtest location - I'd argue that the default case should not require special options. > > The JBS issue states that "it can be treated like any other external dependencies" but this comparison does not hold - almost all other dependencies come in the form of well established libraries with standard headers and standard installation locations which can be coded as default values. On a recent mainstream platform I do not have to specify any paths to libraries to build OpenJDK. This is now different, I always have to manually download gtests and specify gtest location. This is regrettable. > > Thanks, Thomas > > > On Tue, May 26, 2020 at 2:27 PM Magnus Ihse Bursie > wrote: > > > On 2020-05-25 19:53, Igor Ignatyev wrote: > > Hi Magnus, > > > >> On May 25, 2020, at 7:46 AM, Magnus Ihse Bursie > >> > >> >> wrote: > >> > >> Looks basically good to me. > > thanks for your review! > >> > >> We need to document the dependency on gtest, and how to retrieve it. > >> I recommend you add a section next to the JTReg information (under > >> the "## Running Tests" heading) in doc/building.md. I think you > >> should include basically the same information as you did in your > >> follow-up mail, that was informative and clear. > > that's a good suggestion, I've added a small paragraph to 'Running > > Tests' in doc/building.md and regenerated corresponding .html file -- > > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.doc/ > > please let me know if you think something should be added or reworded. > Looks great! Thank you. > > /Magnus > > > >> > >> I assume the most suitable replacement for developers who are used to > >> add a '--disable-hotspot-gtest' to e.g. a pre-definied jib > >> configuration is to now use '--without-gtest'. This will need to be > >> communicated, perhaps to both hotspot-dev and jdk-dev. > > sure, after this gets integrated, I'll let both hotspot-dev and > > jdk-dev aliases know which changes might be required to enable/disable > > hotspot gtest tests compilation. > > > > Thanks. > > -- Igor > > > >> > >> /Magnus > >> > >> On 2020-05-22 20:12, Igor Ignatyev wrote: > >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ > >>>> 132 lines changed: 80 ins; 36 del; 16 mod > >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ > >>>> 57482 lines changed: 80 ins; 57385 del; 17 mod; > >>> Hi all, > >>> > >>> could you please review this small (if you ignore removal part) > >>> patch which removes in-tree copy of gtest (test/fmw/gtest) and > >>> updates makefiles to use one provided thru an added configure option > >>> `--with-gtest`? the previously used configure option > >>> `--enable-hotspot-gtest` gets depricated. > >>> > >>> the patch also compiles gtest and gmock source code into a static > >>> library so they can be compiled w/ all warnings disabled. > >>> > >>> from JBS: > >>>> w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all > >>>> compilers used by OpenJDK became supported by gtest out-of-box, so > >>>> there is no need to have our copy of gtest framework (including > >>>> gmock) within OpenJDK source tree. instead, it can be treated like > >>>> any other external dependencies, and a pointer to the directory w/ > >>>> gtest code can be passed via configure options. > >>> > >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 > >>> webrevs: > >>> - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ > >>> (meaningful changes) > >>> - > >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ > >>> (all changes) > >>> testing: > >>> - gtest tests on {linux,windows,macosx}-x64; > >>> - tier[1-5] builds which include but not limited to linux-aarch64, > >>> linux-arm32, linux-x64-zero > >>> > >>> Thanks, > >>> -- Igor > >>> > >> > > > From thomas.stuefe at gmail.com Fri May 29 05:04:18 2020 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 29 May 2020 07:04:18 +0200 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest In-Reply-To: <2A2FD345-F641-4D6A-85DC-6379E8582C6F@oracle.com> References: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> <0c5d41f4-eebd-a8ae-d537-e14a878a6663@oracle.com> <7B996E95-B57D-4F38-97F1-65FFCA2DD01E@oracle.com> <2A2FD345-F641-4D6A-85DC-6379E8582C6F@oracle.com> Message-ID: Hi Igor, thank you for taking the time to answer my grumblings. Yes, comparison with jtreg is a bit crooked - it is not needed to get a valid build. But jtreg is also maintained in the official OpenJDK repositories; I can clone codetools/jtreg and have the correct version. With gtests, I have to know which version OpenJDK needs, which is not the latest version, and have to download that from outside our repositories. Getting it wrong version may yield me difficult to analyze build errors (plattform zoo, handicapped C++ compilers etc). Also, updating to a new gtest version will now involve a lot more communicatio (a version check in configure would help with that). But this is a minor complaint, I could live with that. I most dislike the fact that I have to specify this option *every single time*, and that omitting it is objectively wrong and may give me a false sense of security. Omitting it gives no warning, but if my changes break gtests I will only notice it hours later when jdk/submit results are back. Yes, I can hide this behind an alias or script, but I think this is the wrong way. You guys did such a good job in making the build dead easy: first time, it tells me exactly which Debian packets to install, I always loved that special touch. I specify boot jdk and maybe debug level and I am done. In fact, the build is so easy that until recently I did not even know we had a build documentation :) The new --with-gtest option is a jarring break from that. In my mind, gtest is in the same ballpark as the freetype library on Windows. That has always been a bit annoying but that was only Windows. Something you cannot rely the OS library mechanism to deliver but have to download and place yourself and tell the build about it. I wonder whether we can find a compromise somehow. Maybe something like an agreed on structure and place for a "companion directory", containing build dependencies like these. Its location could be by convention in a clear relation to the OpenJDK sources (e.g. just beside it) and configure would look for the libraries in there by default. Like this: - openjdk-source - configure - Makefile - .. - openjdk-builddeps - google-test - freetype - .. This would also lend itself very well to a maintained repository of build dependencies somewhere (not necessarily maintained by Oracle). We would have similar ease of use as with platform libraries, which by default are also expected in a standard place in the file system. Thanks, Thomas On Fri, May 29, 2020 at 5:20 AM Igor Ignatyev wrote: > Hi Thomas, > > I regret that this patch made you unhappier. I fully agree that all > hotspot developers are to always build *and* run gtest tests, yet not all > people who work on OpenJDK project are hotspot developers. > > I also believe that all OpenJDK developers are to run tests related to the > area they are changing. The vast majority of such tests are jtreg tests. > And jtreg, for better or worse, is a counterexample to your last paragraph > -- it's an essential part of OpenJDK, but it doesn't come in a form of well > established library, and none of known to me platforms have jtreg in their > package manager, so you do have to manually download jtreg and specify its > location one way or another. I understand that there is a difference b/w > gtest and jtreg, as jtreg isn't really need at build time. However the > proper way to run any of OpenJDK tests is by `make test` and for it to work > you need to either executed configure w/ --with-jtreg or specify JT_HOME on > each invocation of `make test`, where the latter is a preferred way. From > that point of view, gtest and jtreg aren't different, you need to download > both manually and specify their location at configure-time. > > with that being said, I truly understand the inconvenience it causes to > you, yet given you need to download gtest only once and it's fairly easy to > hide `with-gtest` behind a script or an alias like configure_openjdk='bash > ./configure --with-gtest=$GTEST_HOME', I hope it won't cause problems for > you and won't discourage you in anyways. > > Thanks, > -- Igor > > On May 28, 2020, at 12:31 AM, Thomas St?fe > wrote: > > Hi, > > I know the boat has sailed, I missed this one. But I am a bit unhappy > about this. > > I always build with gtests; I think it makes no sense to not build with > gtest. Even if you do not want to run the gtests (and it makes sense to > always do since its a good first-base validity test), hotspot developers > should always build them since changes in the hotspot sources may break > hotspot gtests. hotspot source and gtests are a unity. > > So if it makes no sense to not build gtest, I should not need a special > option to specify gtest location - I'd argue that the default case should > not require special options. > > The JBS issue states that "it can be treated like any other external > dependencies" but this comparison does not hold - almost all other > dependencies come in the form of well established libraries with standard > headers and standard installation locations which can be coded as default > values. On a recent mainstream platform I do not have to specify any paths > to libraries to build OpenJDK. This is now different, I always have to > manually download gtests and specify gtest location. This is regrettable. > > Thanks, Thomas > > > On Tue, May 26, 2020 at 2:27 PM Magnus Ihse Bursie < > magnus.ihse.bursie at oracle.com> wrote: > >> >> >> On 2020-05-25 19:53, Igor Ignatyev wrote: >> > Hi Magnus, >> > >> >> On May 25, 2020, at 7:46 AM, Magnus Ihse Bursie >> >> > >> > wrote: >> >> >> >> Looks basically good to me. >> > thanks for your review! >> >> >> >> We need to document the dependency on gtest, and how to retrieve it. >> >> I recommend you add a section next to the JTReg information (under >> >> the "## Running Tests" heading) in doc/building.md. I think you >> >> should include basically the same information as you did in your >> >> follow-up mail, that was informative and clear. >> > that's a good suggestion, I've added a small paragraph to 'Running >> > Tests' in doc/building.md and regenerated corresponding .html file -- >> > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.doc/ >> > please let me know if you think something should be added or reworded. >> Looks great! Thank you. >> >> /Magnus >> > >> >> >> >> I assume the most suitable replacement for developers who are used to >> >> add a '--disable-hotspot-gtest' to e.g. a pre-definied jib >> >> configuration is to now use '--without-gtest'. This will need to be >> >> communicated, perhaps to both hotspot-dev and jdk-dev. >> > sure, after this gets integrated, I'll let both hotspot-dev and >> > jdk-dev aliases know which changes might be required to enable/disable >> > hotspot gtest tests compilation. >> > >> > Thanks. >> > -- Igor >> > >> >> >> >> /Magnus >> >> >> >> On 2020-05-22 20:12, Igor Ignatyev wrote: >> >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ >> >>>> 132 lines changed: 80 ins; 36 del; 16 mod >> >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ >> >>>> 57482 lines changed: 80 ins; 57385 del; 17 mod; >> >>> Hi all, >> >>> >> >>> could you please review this small (if you ignore removal part) >> >>> patch which removes in-tree copy of gtest (test/fmw/gtest) and >> >>> updates makefiles to use one provided thru an added configure option >> >>> `--with-gtest`? the previously used configure option >> >>> `--enable-hotspot-gtest` gets depricated. >> >>> >> >>> the patch also compiles gtest and gmock source code into a static >> >>> library so they can be compiled w/ all warnings disabled. >> >>> >> >>> from JBS: >> >>>> w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all >> >>>> compilers used by OpenJDK became supported by gtest out-of-box, so >> >>>> there is no need to have our copy of gtest framework (including >> >>>> gmock) within OpenJDK source tree. instead, it can be treated like >> >>>> any other external dependencies, and a pointer to the directory w/ >> >>>> gtest code can be passed via configure options. >> >>> >> >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 >> >>> webrevs: >> >>> - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ >> >>> (meaningful changes) >> >>> - >> >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ >> >>> (all changes) >> >>> testing: >> >>> - gtest tests on {linux,windows,macosx}-x64; >> >>> - tier[1-5] builds which include but not limited to linux-aarch64, >> >>> linux-arm32, linux-x64-zero >> >>> >> >>> Thanks, >> >>> -- Igor >> >>> >> >> >> > >> >> > From rene.schuenemann at gmail.com Fri May 29 11:05:14 2020 From: rene.schuenemann at gmail.com (=?UTF-8?B?UmVuw6kgU2Now7xuZW1hbm4=?=) Date: Fri, 29 May 2020 13:05:14 +0200 Subject: RFR(S) : 8245610 : remove in-tree copy on gtest In-Reply-To: References: <93E621FF-580C-4FF9-8D8E-4BB33CB93A90@oracle.com> <0c5d41f4-eebd-a8ae-d537-e14a878a6663@oracle.com> <7B996E95-B57D-4F38-97F1-65FFCA2DD01E@oracle.com> <2A2FD345-F641-4D6A-85DC-6379E8582C6F@oracle.com> Message-ID: Hi all, I agree with Thomas. I think it would be beneficial to have a fallback when there is no gtest directory explicitly set. Also, running make target "run-test-gtest" now without "--with-gtest" results in a not so descriptive error message. make run-test-gtest Building target 'run-test-gtest' in configuration 'linux-x86_64-server-release' Unknown test selection: 'gtest' See doc/testing.[md|html] for help RunTests.gmk:539: *** Cannot continue. Stop. make[2]: *** [test-gtest] Error 1 I think it would be good to have the error message say that one tries to run the gtest suite without having the gtest sources set. The documentation under doc/testing.[md|html] doesn't describe the new behavior, so hotspot developers that missed this change might be a bit confused with this error message. Thanks, Rene On Fri, May 29, 2020 at 7:05 AM Thomas St?fe wrote: > > Hi Igor, > > thank you for taking the time to answer my grumblings. > > Yes, comparison with jtreg is a bit crooked - it is not needed to get a > valid build. But jtreg is also maintained in the official OpenJDK > repositories; I can clone codetools/jtreg and have the correct version. > With gtests, I have to know which version OpenJDK needs, which is not the > latest version, and have to download that from outside our repositories. > Getting it wrong version may yield me difficult to analyze build errors > (plattform zoo, handicapped C++ compilers etc). Also, updating to a new > gtest version will now involve a lot more communicatio (a version check in > configure would help with that). > > But this is a minor complaint, I could live with that. I most dislike the > fact that I have to specify this option *every single time*, and that > omitting it is objectively wrong and may give me a false sense of security. > Omitting it gives no warning, but if my changes break gtests I will only > notice it hours later when jdk/submit results are back. > > Yes, I can hide this behind an alias or script, but I think this is the > wrong way. You guys did such a good job in making the build dead easy: > first time, it tells me exactly which Debian packets to install, I always > loved that special touch. I specify boot jdk and maybe debug level and I am > done. In fact, the build is so easy that until recently I did not even know > we had a build documentation :) The new --with-gtest option is a jarring > break from that. > > In my mind, gtest is in the same ballpark as the freetype library on > Windows. That has always been a bit annoying but that was only Windows. > Something you cannot rely the OS library mechanism to deliver but have to > download and place yourself and tell the build about it. > > I wonder whether we can find a compromise somehow. Maybe something like > an agreed on structure and place for a "companion directory", containing > build dependencies like these. Its location could be by convention in a > clear relation to the OpenJDK sources (e.g. just beside it) and configure > would look for the libraries in there by default. Like this: > > - openjdk-source > - configure > - Makefile > - .. > - openjdk-builddeps > - google-test > - freetype > - .. > > This would also lend itself very well to a maintained repository of build > dependencies somewhere (not necessarily maintained by Oracle). We would > have similar ease of use as with platform libraries, which by default are > also expected in a standard place in the file system. > > Thanks, Thomas > > > On Fri, May 29, 2020 at 5:20 AM Igor Ignatyev > wrote: > > > Hi Thomas, > > > > I regret that this patch made you unhappier. I fully agree that all > > hotspot developers are to always build *and* run gtest tests, yet not all > > people who work on OpenJDK project are hotspot developers. > > > > I also believe that all OpenJDK developers are to run tests related to the > > area they are changing. The vast majority of such tests are jtreg tests. > > And jtreg, for better or worse, is a counterexample to your last paragraph > > -- it's an essential part of OpenJDK, but it doesn't come in a form of well > > established library, and none of known to me platforms have jtreg in their > > package manager, so you do have to manually download jtreg and specify its > > location one way or another. I understand that there is a difference b/w > > gtest and jtreg, as jtreg isn't really need at build time. However the > > proper way to run any of OpenJDK tests is by `make test` and for it to work > > you need to either executed configure w/ --with-jtreg or specify JT_HOME on > > each invocation of `make test`, where the latter is a preferred way. From > > that point of view, gtest and jtreg aren't different, you need to download > > both manually and specify their location at configure-time. > > > > with that being said, I truly understand the inconvenience it causes to > > you, yet given you need to download gtest only once and it's fairly easy to > > hide `with-gtest` behind a script or an alias like configure_openjdk='bash > > ./configure --with-gtest=$GTEST_HOME', I hope it won't cause problems for > > you and won't discourage you in anyways. > > > > Thanks, > > -- Igor > > > > On May 28, 2020, at 12:31 AM, Thomas St?fe > > wrote: > > > > Hi, > > > > I know the boat has sailed, I missed this one. But I am a bit unhappy > > about this. > > > > I always build with gtests; I think it makes no sense to not build with > > gtest. Even if you do not want to run the gtests (and it makes sense to > > always do since its a good first-base validity test), hotspot developers > > should always build them since changes in the hotspot sources may break > > hotspot gtests. hotspot source and gtests are a unity. > > > > So if it makes no sense to not build gtest, I should not need a special > > option to specify gtest location - I'd argue that the default case should > > not require special options. > > > > The JBS issue states that "it can be treated like any other external > > dependencies" but this comparison does not hold - almost all other > > dependencies come in the form of well established libraries with standard > > headers and standard installation locations which can be coded as default > > values. On a recent mainstream platform I do not have to specify any paths > > to libraries to build OpenJDK. This is now different, I always have to > > manually download gtests and specify gtest location. This is regrettable. > > > > Thanks, Thomas > > > > > > On Tue, May 26, 2020 at 2:27 PM Magnus Ihse Bursie < > > magnus.ihse.bursie at oracle.com> wrote: > > > >> > >> > >> On 2020-05-25 19:53, Igor Ignatyev wrote: > >> > Hi Magnus, > >> > > >> >> On May 25, 2020, at 7:46 AM, Magnus Ihse Bursie > >> >> >> >> > wrote: > >> >> > >> >> Looks basically good to me. > >> > thanks for your review! > >> >> > >> >> We need to document the dependency on gtest, and how to retrieve it. > >> >> I recommend you add a section next to the JTReg information (under > >> >> the "## Running Tests" heading) in doc/building.md. I think you > >> >> should include basically the same information as you did in your > >> >> follow-up mail, that was informative and clear. > >> > that's a good suggestion, I've added a small paragraph to 'Running > >> > Tests' in doc/building.md and regenerated corresponding .html file -- > >> > http://cr.openjdk.java.net/~iignatyev/8245610/webrev.doc/ > >> > please let me know if you think something should be added or reworded. > >> Looks great! Thank you. > >> > >> /Magnus > >> > > >> >> > >> >> I assume the most suitable replacement for developers who are used to > >> >> add a '--disable-hotspot-gtest' to e.g. a pre-definied jib > >> >> configuration is to now use '--without-gtest'. This will need to be > >> >> communicated, perhaps to both hotspot-dev and jdk-dev. > >> > sure, after this gets integrated, I'll let both hotspot-dev and > >> > jdk-dev aliases know which changes might be required to enable/disable > >> > hotspot gtest tests compilation. > >> > > >> > Thanks. > >> > -- Igor > >> > > >> >> > >> >> /Magnus > >> >> > >> >> On 2020-05-22 20:12, Igor Ignatyev wrote: > >> >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ > >> >>>> 132 lines changed: 80 ins; 36 del; 16 mod > >> >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ > >> >>>> 57482 lines changed: 80 ins; 57385 del; 17 mod; > >> >>> Hi all, > >> >>> > >> >>> could you please review this small (if you ignore removal part) > >> >>> patch which removes in-tree copy of gtest (test/fmw/gtest) and > >> >>> updates makefiles to use one provided thru an added configure option > >> >>> `--with-gtest`? the previously used configure option > >> >>> `--enable-hotspot-gtest` gets depricated. > >> >>> > >> >>> the patch also compiles gtest and gmock source code into a static > >> >>> library so they can be compiled w/ all warnings disabled. > >> >>> > >> >>> from JBS: > >> >>>> w/ JEP 381 (JDK-8241787 / JDK-8244224) being integrated, all > >> >>>> compilers used by OpenJDK became supported by gtest out-of-box, so > >> >>>> there is no need to have our copy of gtest framework (including > >> >>>> gmock) within OpenJDK source tree. instead, it can be treated like > >> >>>> any other external dependencies, and a pointer to the directory w/ > >> >>>> gtest code can be passed via configure options. > >> >>> > >> >>> JBS: https://bugs.openjdk.java.net/browse/JDK-8245610 > >> >>> webrevs: > >> >>> - http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00/ > >> >>> (meaningful changes) > >> >>> - > >> >>> http://cr.openjdk.java.net/~iignatyev/8245610/webrev.00%2bremoval/ > >> >>> (all changes) > >> >>> testing: > >> >>> - gtest tests on {linux,windows,macosx}-x64; > >> >>> - tier[1-5] builds which include but not limited to linux-aarch64, > >> >>> linux-arm32, linux-x64-zero > >> >>> > >> >>> Thanks, > >> >>> -- Igor > >> >>> > >> >> > >> > > >> > >> > > From lois.foltan at oracle.com Fri May 29 15:32:41 2020 From: lois.foltan at oracle.com (Lois Foltan) Date: Fri, 29 May 2020 11:32:41 -0400 Subject: RFR 8245289: Clean up offset code in JavaClasses In-Reply-To: References: Message-ID: <92f8f586-b97c-9df8-c1ab-e62157a8ff69@oracle.com> On 5/28/2020 8:48 AM, coleen.phillimore at oracle.com wrote: > Summary: Make offset member names consistent and private, move static > initializations near owning classes > > This one is better.? I gave up on the X macros because they didn't > save typing.? The changes here were to add underscores to offset field > names, remove _in_bytes for offset accessors and add asserts that the > offset is non-zero, made java_lang_ref_Reference offset fields > private, and moved static member definitions closer to the class > instead of some random place in javaClasses.cpp. > > See discussion for 8243996 Remove hardcoded field offsets for more > details. > https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-May/041732.html > > Tested with tier1-6. > > open webrev at http://cr.openjdk.java.net/~coleenp/2020/8245289.01/webrev > bug link https://bugs.openjdk.java.net/browse/JDK-8245289 > > Thanks, > Coleen Hi Coleen, Looks good.? Minor nits: share/classfile/javaClasses.hpp - line #239, can you add the "private" keyword back into the java_lang_Class class definition.? Having the "private" keyword is consistent with how every other class is defined in the file. - line #887, since you are making changes to the value_offset() method can you change the type check for T_LONG or T_DOUBLE to call is_double_word_type(). I don't have a preference concerning the discussion on the initialization with explicit value vs. no value for the static field declarations.? I'm fine with pushing your patch as is. Thanks, Lois From igor.ignatyev at oracle.com Fri May 29 17:14:20 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Fri, 29 May 2020 10:14:20 -0700 Subject: RFR(S/L) : 8244614 : cleanup keywords used/available in hotspot testbase In-Reply-To: References: Message-ID: <939A28A1-5CEA-438D-A2EE-E547E9C7D57E@oracle.com> http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01.manual > 5 lines changed: 4 ins; 0 del; 1 mod; http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01 > 930 lines changed: 3 ins; 204 del; 723 mod; Hi all, given the absense of comments, I assume that nobody has any strong feeling these to-be-removed keywords , so I'm repurposing this thread as RFR. the biggest part of the patch was done by sed: - removal of $key : gsed -ri "s/(@key.*) $key/\1/g" {} ; where $key is in nmt cte_test, jcmd, regression - removal of empty @key : gsed -ri '/^ \* @key\s*$/d' the manually done changes are limited to: - removal of keywords from TEST.ROOT - adding k/w description to TEST.ROOT - adding @bug to redefclass031 JBS: https://bugs.openjdk.java.net/browse/JDK-8244614 webrevs: - part done by sed: http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01.sed > 594 lines changed: 0 ins; 205 del; 389 mod; - part done by me: http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01.manual > 5 lines changed: 4 ins; 0 del; 1 mod; - whole (incl. copyright year update) : http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01 > 930 lines changed: 3 ins; 204 del; 723 mod; testing: - grep-ed for removed keywords - jtreg -l - tier1 (just for the good luck) Thanks, -- Igor > On May 22, 2020, at 3:42 PM, Igor Ignatyev wrote: > > Hi all, > > I'm planning to clean up jtreg keywords that we have in test/hotspot/jtreg as most of them seem to be useless, confuse and make people to add these keywords in vain. After looking at how hotspot k/w are used, as well as which k/w are available and used by other test suites, I'd like to propose the following: > - keep and document 'headful', 'stress' as they are useful for test selection; > - keep and document 'intermittent', 'randomness' as they bring extra information about tests stability/reproducibility/determinism; > - remove 'nmt', 'gc', 'metaspace' as they are used only in obvious places and aren't informative; > - remove 'cte_test' as it's an artifact of old days and doesn't bring any useful information anymore; > - remove 'jcmd', not all tests which depend on jcmd has this k/w, so it can't be used for test selection/filtering; there is also no obvious reasons to single out jcmd from other binaries bundled w/ JDK > - remove 'regression' as the usage of at-bug is more common and convey the same meaning; and for two tests which have regression k/w, but don't have at-bug: > - mark vmTestbase/nsk/jvmti/RedefineClasses/redefclass031/TestDescription.java w/ at-bug 6318850; > - don't mark runtime/NMT/ReleaseNoCommit.java as it wasn't integrated as a true regression test, but rather as a function test for NMT w/ bunch of other NMT tests (where aren't marked w/ 'regression' k/w) by JDK-8054952. > > I'd like to receive some feedback on the proposed changes before I (more likely sed) remove keywords from 500+ tests and spend time testing that. > > [1] is the preliminary webrev just to illustrate the change, it only updates TEST.ROOT file and adds @bug to redefclass031 test. > > Thanks, > -- Igor > > for reference, k/w available in other test suites: > jdk : 2d dnd headful i18n intermittent printer randomness jfr > langtools: intermittent randomness > jaxp: intermittent randomness > > [1] http://cr.openjdk.java.net/~iignatyev//8244614/webrev.00 > > From igor.ignatyev at oracle.com Fri May 29 17:49:00 2020 From: igor.ignatyev at oracle.com (Igor Ignatyev) Date: Fri, 29 May 2020 10:49:00 -0700 Subject: RFR(S/L) : 8244614 : cleanup keywords used/available in hotspot testbase In-Reply-To: <939A28A1-5CEA-438D-A2EE-E547E9C7D57E@oracle.com> References: <939A28A1-5CEA-438D-A2EE-E547E9C7D57E@oracle.com> Message-ID: <8F94C0EC-3470-4252-A18B-72F68FDA3BC5@oracle.com> ouch,.. it appears that this tier1 execution wasn't in vain. I realized that all my queries were limited to the tests `jtreg -l` gave me, which excludes tests which can't be w/ my default build on my localhost... http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01-02 updates 93 more tests. -- Igor > On May 29, 2020, at 10:14 AM, Igor Ignatyev wrote: > > http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01.manual >> 5 lines changed: 4 ins; 0 del; 1 mod; > http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01 >> 930 lines changed: 3 ins; 204 del; 723 mod; > > > Hi all, > > given the absense of comments, I assume that nobody has any strong feeling these to-be-removed keywords , so I'm repurposing this thread as RFR. > > the biggest part of the patch was done by sed: > - removal of $key : gsed -ri "s/(@key.*) $key/\1/g" {} ; where $key is in nmt cte_test, jcmd, regression > - removal of empty @key : gsed -ri '/^ \* @key\s*$/d' > > the manually done changes are limited to: > - removal of keywords from TEST.ROOT > - adding k/w description to TEST.ROOT > - adding @bug to redefclass031 > > JBS: https://bugs.openjdk.java.net/browse/JDK-8244614 > webrevs: > - part done by sed: http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01.sed >> 594 lines changed: 0 ins; 205 del; 389 mod; > > - part done by me: http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01.manual >> 5 lines changed: 4 ins; 0 del; 1 mod; > > - whole (incl. copyright year update) : http://cr.openjdk.java.net/~iignatyev//8244614/webrev.01 >> 930 lines changed: 3 ins; 204 del; 723 mod; > > > testing: > - grep-ed for removed keywords > - jtreg -l > - tier1 (just for the good luck) > > Thanks, > -- Igor > > > >> On May 22, 2020, at 3:42 PM, Igor Ignatyev wrote: >> >> Hi all, >> >> I'm planning to clean up jtreg keywords that we have in test/hotspot/jtreg as most of them seem to be useless, confuse and make people to add these keywords in vain. After looking at how hotspot k/w are used, as well as which k/w are available and used by other test suites, I'd like to propose the following: >> - keep and document 'headful', 'stress' as they are useful for test selection; >> - keep and document 'intermittent', 'randomness' as they bring extra information about tests stability/reproducibility/determinism; >> - remove 'nmt', 'gc', 'metaspace' as they are used only in obvious places and aren't informative; >> - remove 'cte_test' as it's an artifact of old days and doesn't bring any useful information anymore; >> - remove 'jcmd', not all tests which depend on jcmd has this k/w, so it can't be used for test selection/filtering; there is also no obvious reasons to single out jcmd from other binaries bundled w/ JDK >> - remove 'regression' as the usage of at-bug is more common and convey the same meaning; and for two tests which have regression k/w, but don't have at-bug: >> - mark vmTestbase/nsk/jvmti/RedefineClasses/redefclass031/TestDescription.java w/ at-bug 6318850; >> - don't mark runtime/NMT/ReleaseNoCommit.java as it wasn't integrated as a true regression test, but rather as a function test for NMT w/ bunch of other NMT tests (where aren't marked w/ 'regression' k/w) by JDK-8054952. >> >> I'd like to receive some feedback on the proposed changes before I (more likely sed) remove keywords from 500+ tests and spend time testing that. >> >> [1] is the preliminary webrev just to illustrate the change, it only updates TEST.ROOT file and adds @bug to redefclass031 test. >> >> Thanks, >> -- Igor >> >> for reference, k/w available in other test suites: >> jdk : 2d dnd headful i18n intermittent printer randomness jfr >> langtools: intermittent randomness >> jaxp: intermittent randomness >> >> [1] http://cr.openjdk.java.net/~iignatyev//8244614/webrev.00 >> >> > From coleen.phillimore at oracle.com Fri May 29 18:19:51 2020 From: coleen.phillimore at oracle.com (coleen.phillimore at oracle.com) Date: Fri, 29 May 2020 14:19:51 -0400 Subject: RFR 8245289: Clean up offset code in JavaClasses In-Reply-To: <92f8f586-b97c-9df8-c1ab-e62157a8ff69@oracle.com> References: <92f8f586-b97c-9df8-c1ab-e62157a8ff69@oracle.com> Message-ID: <4e089610-8fc4-a5dc-ebf8-dd5bc5112c8d@oracle.com> Lois, thank you for reviewing! On 5/29/20 11:32 AM, Lois Foltan wrote: > On 5/28/2020 8:48 AM, coleen.phillimore at oracle.com wrote: >> Summary: Make offset member names consistent and private, move static >> initializations near owning classes >> >> This one is better.? I gave up on the X macros because they didn't >> save typing.? The changes here were to add underscores to offset >> field names, remove _in_bytes for offset accessors and add asserts >> that the offset is non-zero, made java_lang_ref_Reference offset >> fields private, and moved static member definitions closer to the >> class instead of some random place in javaClasses.cpp. >> >> See discussion for 8243996 Remove hardcoded field offsets for more >> details. >> https://mail.openjdk.java.net/pipermail/hotspot-dev/2020-May/041732.html >> >> Tested with tier1-6. >> >> open webrev at >> http://cr.openjdk.java.net/~coleenp/2020/8245289.01/webrev >> bug link https://bugs.openjdk.java.net/browse/JDK-8245289 >> >> Thanks, >> Coleen > > Hi Coleen, > > Looks good.? Minor nits: > > share/classfile/javaClasses.hpp > - line #239, can you add the "private" keyword back into the > java_lang_Class class definition.? Having the "private" keyword is > consistent with how every other class is defined in the file. I had originally taken all the 'private' out because in its absent the data members are by default private, but then ended up only taking this one out.? I've restored it to be consistent. > - line #887, since you are making changes to the value_offset() method > can you change the type check for T_LONG or T_DOUBLE to call > is_double_word_type(). Fixed! > > I don't have a preference concerning the discussion on the > initialization with explicit value vs. no value for the static field > declarations.? I'm fine with pushing your patch as is. So there were more static offset definitions *without* initializers (default initialized) than with initializers, so I took out the initializers, and am doing a sanity tier1 test. Thanks, Coleen > > Thanks, > Lois