From erik.osterlund at oracle.com Mon Sep 3 08:37:04 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Mon, 3 Sep 2018 10:37:04 +0200 Subject: RFR: JDK-8207169: X86: Modularize cmpxchg-oop assembler for C1 and C2 In-Reply-To: References: <55efde87-1da8-e644-f45d-51dfbb3b934c@redhat.com> <5B86B7CE.3030507@oracle.com> <9c22a683-9065-6a2e-8f7c-202cc30cb378@redhat.com> <5B86BBB6.7000401@oracle.com> <026c2eef-fa68-cc74-0763-c3c54018342a@redhat.com> <4D4C2A84-7FB2-4E51-8EAF-51EBF73D0F14@oracle.com> Message-ID: <5B8CF2B0.2060509@oracle.com> Hi Roland, First of all, I apologize for getting your name wrong in the last email. On 2018-08-31 16:46, Roland Westrelin wrote: >> Well... C1 uses CAS in the heap only for the Unsafe CAS intrinsic, >> which is indeed inserted at parse time. And all other GCs alter the >> CFG for the GC barriers in their CAS barriers, using LIR. Except >> Epsilon I suppose. > Are you talking about for instance G1BarrierSetC1::pre_barrier()? That > method adds control flow withing a basic block. It doesn't hack the CFG > (it doesn't add new basic blocks). How can the register allocator > compute liveness without a correct CFG? Either > G1BarrierSetC1::pre_barrier() is a simple enough case that register > allocation is correct or there are some nasty bugs in there. In any > case, building control flow within a block like > G1BarrierSetC1::pre_barrier() does is an ugly hack. Doing anything more > complicated that way is asking for trouble. The C1 basic blocks are built and optimized as part of the HIR and are not to be changed after that. Once the HIR is generated, the LIR inserts operations required for lowering this optimized HIR to machine code. After IR::compute_code() of the HIR, those basic blocks are set in stone. That means that any control flow alterations needed by the LIRGenerator, which comes into play after that, is going to use branches within the HIR basic block instead (as we promised not to change the HIR basic blocks after the HIR is built and optimized). I can see how that might feel like a hack, but that is kind of the way that things are currently done in C1. It is used this way for all barrier sets today (UseCondCardMark for card marking GCs, for G1, ZGC), and it's also used by T_BOOLEAN normalization, switch statements, checking for referents in unsafe intrinsics etc. I suppose the stubs inserted at the LIR level also similarly break the basic block abstraction of the HIR level. These are things that can of course be changed into a more strict basic block model even at the LIR level. But I don't know how much that would help given that this is just the pass before lowering to machine code. But that is a whole different discussion. I do not propose to move the GC barriers into the HIR - it is too early. I propose to insert it at the LIR level like all the other GCs, in a similar way to all the other GCs, using the same mechanisms used by all the other GCs. @Roman: If you feel more comfortable using your own LIR_Op with your own lowering or stubs instead because you want this written in assembly for whatever reason, then I am fine with that too as long as it is contained in the shenandoah folders. What I do have reservations against is to change the API that everybody else uses to make the LIRGenerator raw CAS get lowered into a not raw Access call to the macro assembler, passing in temporary registers used by Shenandoah from above into the raw cas used by the not raw macro assembler access CAS. For example, in ZGC we have a class LIR_OpZLoadBarrierTest LIR_Op defined in zBarrierSetC1.cpp, which allows us to do custom machine dependent lowering of the test itself, which can be inserted into the LIR list. I hope we are on the same page here! Thanks, /Erik > Roland. From rwestrel at redhat.com Mon Sep 3 08:41:21 2018 From: rwestrel at redhat.com (Roland Westrelin) Date: Mon, 03 Sep 2018 10:41:21 +0200 Subject: RFR: JDK-8207169: X86: Modularize cmpxchg-oop assembler for C1 and C2 In-Reply-To: <5B8CF2B0.2060509@oracle.com> References: <55efde87-1da8-e644-f45d-51dfbb3b934c@redhat.com> <5B86B7CE.3030507@oracle.com> <9c22a683-9065-6a2e-8f7c-202cc30cb378@redhat.com> <5B86BBB6.7000401@oracle.com> <026c2eef-fa68-cc74-0763-c3c54018342a@redhat.com> <4D4C2A84-7FB2-4E51-8EAF-51EBF73D0F14@oracle.com> <5B8CF2B0.2060509@oracle.com> Message-ID: Hi Erik, > The C1 basic blocks are built and optimized as part of the HIR and are > not to be changed after that. Once the HIR is generated, the LIR inserts > operations required for lowering this optimized HIR to machine code. > After IR::compute_code() of the HIR, those basic blocks are set in > stone. That means that any control flow alterations needed by the > LIRGenerator, which comes into play after that, is going to use branches > within the HIR basic block instead (as we promised not to change the HIR > basic blocks after the HIR is built and optimized). I can see how that > might feel like a hack, but that is kind of the way that things are > currently done in C1. It is used this way for all barrier sets today > (UseCondCardMark for card marking GCs, for G1, ZGC), and it's also used > by T_BOOLEAN normalization, switch statements, checking for referents in > unsafe intrinsics etc. I suppose the stubs inserted at the LIR level > also similarly break the basic block abstraction of the HIR level. These > are things that can of course be changed into a more strict basic block > model even at the LIR level. But I don't know how much that would help > given that this is just the pass before lowering to machine code. But > that is a whole different discussion. Adding a loop within a basic block is simply not possible. The register allocator won't know it's a loop and has no way to know operands are live across iterations. So it's not like we even have a choice. Roland. From erik.osterlund at oracle.com Mon Sep 3 08:54:34 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Mon, 3 Sep 2018 10:54:34 +0200 Subject: RFR: JDK-8207169: X86: Modularize cmpxchg-oop assembler for C1 and C2 In-Reply-To: References: <55efde87-1da8-e644-f45d-51dfbb3b934c@redhat.com> <5B86B7CE.3030507@oracle.com> <9c22a683-9065-6a2e-8f7c-202cc30cb378@redhat.com> <5B86BBB6.7000401@oracle.com> <026c2eef-fa68-cc74-0763-c3c54018342a@redhat.com> <4D4C2A84-7FB2-4E51-8EAF-51EBF73D0F14@oracle.com> <5B8CF2B0.2060509@oracle.com> Message-ID: <5B8CF6CA.1010600@oracle.com> Hi Roman, Who would clobber those registers between iterations though in your tight loop? /Erik On 2018-09-03 10:41, Roland Westrelin wrote: > Hi Erik, > >> The C1 basic blocks are built and optimized as part of the HIR and are >> not to be changed after that. Once the HIR is generated, the LIR inserts >> operations required for lowering this optimized HIR to machine code. >> After IR::compute_code() of the HIR, those basic blocks are set in >> stone. That means that any control flow alterations needed by the >> LIRGenerator, which comes into play after that, is going to use branches >> within the HIR basic block instead (as we promised not to change the HIR >> basic blocks after the HIR is built and optimized). I can see how that >> might feel like a hack, but that is kind of the way that things are >> currently done in C1. It is used this way for all barrier sets today >> (UseCondCardMark for card marking GCs, for G1, ZGC), and it's also used >> by T_BOOLEAN normalization, switch statements, checking for referents in >> unsafe intrinsics etc. I suppose the stubs inserted at the LIR level >> also similarly break the basic block abstraction of the HIR level. These >> are things that can of course be changed into a more strict basic block >> model even at the LIR level. But I don't know how much that would help >> given that this is just the pass before lowering to machine code. But >> that is a whole different discussion. > Adding a loop within a basic block is simply not possible. The register > allocator won't know it's a loop and has no way to know operands are > live across iterations. So it's not like we even have a choice. > > Roland. From rwestrel at redhat.com Mon Sep 3 08:58:00 2018 From: rwestrel at redhat.com (Roland Westrelin) Date: Mon, 03 Sep 2018 10:58:00 +0200 Subject: RFR: JDK-8207169: X86: Modularize cmpxchg-oop assembler for C1 and C2 In-Reply-To: <5B8CF6CA.1010600@oracle.com> References: <55efde87-1da8-e644-f45d-51dfbb3b934c@redhat.com> <5B86B7CE.3030507@oracle.com> <9c22a683-9065-6a2e-8f7c-202cc30cb378@redhat.com> <5B86BBB6.7000401@oracle.com> <026c2eef-fa68-cc74-0763-c3c54018342a@redhat.com> <4D4C2A84-7FB2-4E51-8EAF-51EBF73D0F14@oracle.com> <5B8CF2B0.2060509@oracle.com> <5B8CF6CA.1010600@oracle.com> Message-ID: > Who would clobber those registers between iterations though in your > tight loop? Ignoring cas, but with a simple example: input = 0; loop_entry: input++; array[i] = input; // some other code goto loop_entry; input is live across iterations but given the loop is hidden in a basic block, the register allocator expects it to be live from its initialization to the store in the array. So it's free to assign a register to input and reuse it in whatever code is in the rest of the loop body. Roland. From rkennke at redhat.com Mon Sep 3 08:59:13 2018 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 03 Sep 2018 10:59:13 +0200 Subject: RFR: JDK-8207169: X86: Modularize cmpxchg-oop assembler for C1 and C2 In-Reply-To: <5B8CF2B0.2060509@oracle.com> References: <55efde87-1da8-e644-f45d-51dfbb3b934c@redhat.com> <5B86B7CE.3030507@oracle.com> <9c22a683-9065-6a2e-8f7c-202cc30cb378@redhat.com> <5B86BBB6.7000401@oracle.com> <026c2eef-fa68-cc74-0763-c3c54018342a@redhat.com> <4D4C2A84-7FB2-4E51-8EAF-51EBF73D0F14@oracle.com> <5B8CF2B0.2060509@oracle.com> Message-ID: I wasn't sure that the BarrierSetC1 interface allows to define custom ops. This sounds like a good natural solution. Ditto for C2. Let's see if we can make that work. Roman Am 3. September 2018 10:37:04 MESZ schrieb "Erik ?sterlund" : >Hi Roland, > >First of all, I apologize for getting your name wrong in the last >email. > >On 2018-08-31 16:46, Roland Westrelin wrote: >>> Well... C1 uses CAS in the heap only for the Unsafe CAS intrinsic, >>> which is indeed inserted at parse time. And all other GCs alter the >>> CFG for the GC barriers in their CAS barriers, using LIR. Except >>> Epsilon I suppose. >> Are you talking about for instance G1BarrierSetC1::pre_barrier()? >That >> method adds control flow withing a basic block. It doesn't hack the >CFG >> (it doesn't add new basic blocks). How can the register allocator >> compute liveness without a correct CFG? Either >> G1BarrierSetC1::pre_barrier() is a simple enough case that register >> allocation is correct or there are some nasty bugs in there. In any >> case, building control flow within a block like >> G1BarrierSetC1::pre_barrier() does is an ugly hack. Doing anything >more >> complicated that way is asking for trouble. > >The C1 basic blocks are built and optimized as part of the HIR and are >not to be changed after that. Once the HIR is generated, the LIR >inserts >operations required for lowering this optimized HIR to machine code. >After IR::compute_code() of the HIR, those basic blocks are set in >stone. That means that any control flow alterations needed by the >LIRGenerator, which comes into play after that, is going to use >branches >within the HIR basic block instead (as we promised not to change the >HIR >basic blocks after the HIR is built and optimized). I can see how that >might feel like a hack, but that is kind of the way that things are >currently done in C1. It is used this way for all barrier sets today >(UseCondCardMark for card marking GCs, for G1, ZGC), and it's also used > >by T_BOOLEAN normalization, switch statements, checking for referents >in >unsafe intrinsics etc. I suppose the stubs inserted at the LIR level >also similarly break the basic block abstraction of the HIR level. >These >are things that can of course be changed into a more strict basic block > >model even at the LIR level. But I don't know how much that would help >given that this is just the pass before lowering to machine code. But >that is a whole different discussion. > >I do not propose to move the GC barriers into the HIR - it is too >early. >I propose to insert it at the LIR level like all the other GCs, in a >similar way to all the other GCs, using the same mechanisms used by all > >the other GCs. > >@Roman: If you feel more comfortable using your own LIR_Op with your >own >lowering or stubs instead because you want this written in assembly for > >whatever reason, then I am fine with that too as long as it is >contained >in the shenandoah folders. What I do have reservations against is to >change the API that everybody else uses to make the LIRGenerator raw >CAS >get lowered into a not raw Access call to the macro assembler, passing >in temporary registers used by Shenandoah from above into the raw cas >used by the not raw macro assembler access CAS. > >For example, in ZGC we have a class LIR_OpZLoadBarrierTest LIR_Op >defined in zBarrierSetC1.cpp, which allows us to do custom machine >dependent lowering of the test itself, which can be inserted into the >LIR list. > >I hope we are on the same page here! > >Thanks, >/Erik > >> Roland. From erik.osterlund at oracle.com Mon Sep 3 09:25:12 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Mon, 3 Sep 2018 11:25:12 +0200 Subject: RFR: JDK-8207169: X86: Modularize cmpxchg-oop assembler for C1 and C2 In-Reply-To: References: <55efde87-1da8-e644-f45d-51dfbb3b934c@redhat.com> <5B86B7CE.3030507@oracle.com> <9c22a683-9065-6a2e-8f7c-202cc30cb378@redhat.com> <5B86BBB6.7000401@oracle.com> <026c2eef-fa68-cc74-0763-c3c54018342a@redhat.com> <4D4C2A84-7FB2-4E51-8EAF-51EBF73D0F14@oracle.com> <5B8CF2B0.2060509@oracle.com> Message-ID: <5B8CFDF8.9040700@oracle.com> Hi Roman, It did not use to be possible as it needed its own enum switches all over the place. But as part of my C1 barrier set interface work, I wanted to make it possible to make your own LIR_Ops in the barrier set as well without cluttering the switches and inserted appropriate virutal calls to the LIR_Ops allowing you to do that. Now, basically, if your LIR_Op id is lir_none (which the default constructor sets it to), then it will use virtual calls into your LIR_Op in the switch statements. I see how inserting LIR loops in the HIR basic block in the general case can go horribly wrong as Roland showed in his example. So if you feel like defining your own LIR_Op and lower it in your barrier set is the more natural solution for Shenandoah, you can use that mechanism of course. It sounds like we have reached an agreement? Thanks, /Erik On 2018-09-03 10:59, Roman Kennke wrote: > I wasn't sure that the BarrierSetC1 interface allows to define custom ops. This sounds like a good natural solution. Ditto for C2. Let's see if we can make that work. > > Roman > > Am 3. September 2018 10:37:04 MESZ schrieb "Erik ?sterlund" : >> Hi Roland, >> >> First of all, I apologize for getting your name wrong in the last >> email. >> >> On 2018-08-31 16:46, Roland Westrelin wrote: >>>> Well... C1 uses CAS in the heap only for the Unsafe CAS intrinsic, >>>> which is indeed inserted at parse time. And all other GCs alter the >>>> CFG for the GC barriers in their CAS barriers, using LIR. Except >>>> Epsilon I suppose. >>> Are you talking about for instance G1BarrierSetC1::pre_barrier()? >> That >>> method adds control flow withing a basic block. It doesn't hack the >> CFG >>> (it doesn't add new basic blocks). How can the register allocator >>> compute liveness without a correct CFG? Either >>> G1BarrierSetC1::pre_barrier() is a simple enough case that register >>> allocation is correct or there are some nasty bugs in there. In any >>> case, building control flow within a block like >>> G1BarrierSetC1::pre_barrier() does is an ugly hack. Doing anything >> more >>> complicated that way is asking for trouble. >> The C1 basic blocks are built and optimized as part of the HIR and are >> not to be changed after that. Once the HIR is generated, the LIR >> inserts >> operations required for lowering this optimized HIR to machine code. >> After IR::compute_code() of the HIR, those basic blocks are set in >> stone. That means that any control flow alterations needed by the >> LIRGenerator, which comes into play after that, is going to use >> branches >> within the HIR basic block instead (as we promised not to change the >> HIR >> basic blocks after the HIR is built and optimized). I can see how that >> might feel like a hack, but that is kind of the way that things are >> currently done in C1. It is used this way for all barrier sets today >> (UseCondCardMark for card marking GCs, for G1, ZGC), and it's also used >> >> by T_BOOLEAN normalization, switch statements, checking for referents >> in >> unsafe intrinsics etc. I suppose the stubs inserted at the LIR level >> also similarly break the basic block abstraction of the HIR level. >> These >> are things that can of course be changed into a more strict basic block >> >> model even at the LIR level. But I don't know how much that would help >> given that this is just the pass before lowering to machine code. But >> that is a whole different discussion. >> >> I do not propose to move the GC barriers into the HIR - it is too >> early. >> I propose to insert it at the LIR level like all the other GCs, in a >> similar way to all the other GCs, using the same mechanisms used by all >> >> the other GCs. >> >> @Roman: If you feel more comfortable using your own LIR_Op with your >> own >> lowering or stubs instead because you want this written in assembly for >> >> whatever reason, then I am fine with that too as long as it is >> contained >> in the shenandoah folders. What I do have reservations against is to >> change the API that everybody else uses to make the LIRGenerator raw >> CAS >> get lowered into a not raw Access call to the macro assembler, passing >> in temporary registers used by Shenandoah from above into the raw cas >> used by the not raw macro assembler access CAS. >> >> For example, in ZGC we have a class LIR_OpZLoadBarrierTest LIR_Op >> defined in zBarrierSetC1.cpp, which allows us to do custom machine >> dependent lowering of the test itself, which can be inserted into the >> LIR list. >> >> I hope we are on the same page here! >> >> Thanks, >> /Erik >> >>> Roland. From rkennke at redhat.com Mon Sep 3 09:57:51 2018 From: rkennke at redhat.com (Roman Kennke) Date: Mon, 3 Sep 2018 11:57:51 +0200 Subject: RFR: JDK-8207169: X86: Modularize cmpxchg-oop assembler for C1 and C2 In-Reply-To: <5B8CFDF8.9040700@oracle.com> References: <55efde87-1da8-e644-f45d-51dfbb3b934c@redhat.com> <5B86B7CE.3030507@oracle.com> <9c22a683-9065-6a2e-8f7c-202cc30cb378@redhat.com> <5B86BBB6.7000401@oracle.com> <026c2eef-fa68-cc74-0763-c3c54018342a@redhat.com> <4D4C2A84-7FB2-4E51-8EAF-51EBF73D0F14@oracle.com> <5B8CF2B0.2060509@oracle.com> <5B8CFDF8.9040700@oracle.com> Message-ID: <40ef7c2e-faad-ce6e-3fda-3e1c66aaf517@redhat.com> Hi Erik, > It did not use to be possible as it needed its own enum switches all > over the place. But as part of my C1 barrier set interface work, I > wanted to make it possible to make your own LIR_Ops in the barrier set > as well without cluttering the switches and inserted appropriate virutal > calls to the LIR_Ops allowing you to do that. Now, basically, if your > LIR_Op id is lir_none (which the default constructor sets it to), then > it will use virtual calls into your LIR_Op in the switch statements. > > I see how inserting LIR loops in the HIR basic block in the general case > can go horribly wrong as Roland showed in his example. So if you feel > like defining your own LIR_Op and lower it in your barrier set is the > more natural solution for Shenandoah, you can use that mechanism of course. > > It sounds like we have reached an agreement? I think so, at least for now. We'll try to turn our cmpxchg-oop problem into LIR_Op and C2 node and see how that goes. I withdraw this RFR. Thanks a lot, Roman > > Thanks, > /Erik > > On 2018-09-03 10:59, Roman Kennke wrote: >> I wasn't sure that the BarrierSetC1 interface allows to define custom >> ops. This sounds like a good natural solution. Ditto for C2. Let's see >> if we can make that work. >> >> Roman >> >> Am 3. September 2018 10:37:04 MESZ schrieb "Erik ?sterlund" >> : >>> Hi Roland, >>> >>> First of all, I apologize for getting your name wrong in the last >>> email. >>> >>> On 2018-08-31 16:46, Roland Westrelin wrote: >>>>> Well... C1 uses CAS in the heap only for the Unsafe CAS intrinsic, >>>>> which is indeed inserted at parse time. And all other GCs alter the >>>>> CFG for the GC barriers in their CAS barriers, using LIR. Except >>>>> Epsilon I suppose. >>>> Are you talking about for instance G1BarrierSetC1::pre_barrier()? >>> That >>>> method adds control flow withing a basic block. It doesn't hack the >>> CFG >>>> (it doesn't add new basic blocks). How can the register allocator >>>> compute liveness without a correct CFG? Either >>>> G1BarrierSetC1::pre_barrier() is a simple enough case that register >>>> allocation is correct or there are some nasty bugs in there. In any >>>> case, building control flow within a block like >>>> G1BarrierSetC1::pre_barrier() does is an ugly hack. Doing anything >>> more >>>> complicated that way is asking for trouble. >>> The C1 basic blocks are built and optimized as part of the HIR and are >>> not to be changed after that. Once the HIR is generated, the LIR >>> inserts >>> operations required for lowering this optimized HIR to machine code. >>> After IR::compute_code() of the HIR, those basic blocks are set in >>> stone. That means that any control flow alterations needed by the >>> LIRGenerator, which comes into play after that, is going to use >>> branches >>> within the HIR basic block instead (as we promised not to change the >>> HIR >>> basic blocks after the HIR is built and optimized). I can see how that >>> might feel like a hack, but that is kind of the way that things are >>> currently done in C1. It is used this way for all barrier sets today >>> (UseCondCardMark for card marking GCs, for G1, ZGC), and it's also used >>> >>> by T_BOOLEAN normalization, switch statements, checking for referents >>> in >>> unsafe intrinsics etc. I suppose the stubs inserted at the LIR level >>> also similarly break the basic block abstraction of the HIR level. >>> These >>> are things that can of course be changed into a more strict basic block >>> >>> model even at the LIR level. But I don't know how much that would help >>> given that this is just the pass before lowering to machine code. But >>> that is a whole different discussion. >>> >>> I do not propose to move the GC barriers into the HIR - it is too >>> early. >>> I propose to insert it at the LIR level like all the other GCs, in a >>> similar way to all the other GCs, using the same mechanisms used by all >>> >>> the other GCs. >>> >>> @Roman: If you feel more comfortable using your own LIR_Op with your >>> own >>> lowering or stubs instead because you want this written in assembly for >>> >>> whatever reason, then I am fine with that too as long as it is >>> contained >>> in the shenandoah folders. What I do have reservations against is to >>> change the API that everybody else uses to make the LIRGenerator raw >>> CAS >>> get lowered into a not raw Access call to the macro assembler, passing >>> in temporary registers used by Shenandoah from above into the raw cas >>> used by the not raw macro assembler access CAS. >>> >>> For example, in ZGC we have a class LIR_OpZLoadBarrierTest LIR_Op >>> defined in zBarrierSetC1.cpp, which allows us to do custom machine >>> dependent lowering of the test itself, which can be inserted into the >>> LIR list. >>> >>> I hope we are on the same page here! >>> >>> Thanks, >>> /Erik >>> >>>> Roland. > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From thomas.schatzl at oracle.com Mon Sep 3 10:56:42 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 03 Sep 2018 12:56:42 +0200 Subject: RFR (S): 8210265: Crash in HSpaceCounters::update_used() Message-ID: Hi all, can I have reviews for this change that fixes a crash when G1 is started with -XX:-UsePerfData? The reason has been that recent refactoring removed one "if (UsePerfData)" check too many. CR: https://bugs.openjdk.java.net/browse/JDK-8210265 Webrev: http://cr.openjdk.java.net/~tschatzl/8210265/webrev/ Testing: new test succeeds Thanks, Thomas From shade at redhat.com Mon Sep 3 10:59:53 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 3 Sep 2018 12:59:53 +0200 Subject: RFR (S): 8210265: Crash in HSpaceCounters::update_used() In-Reply-To: References: Message-ID: <4ad20a74-d4f5-016a-b2c7-bbcf3e21ef61@redhat.com> On 09/03/2018 12:56 PM, Thomas Schatzl wrote: > CR: > https://bugs.openjdk.java.net/browse/JDK-8210265 > Webrev: > http://cr.openjdk.java.net/~tschatzl/8210265/webrev/ Looks trivial and good. -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From stefan.johansson at oracle.com Mon Sep 3 11:43:36 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Mon, 3 Sep 2018 13:43:36 +0200 Subject: RFR (S): 8210265: Crash in HSpaceCounters::update_used() In-Reply-To: References: Message-ID: On 2018-09-03 12:56, Thomas Schatzl wrote: > Hi all, > > can I have reviews for this change that fixes a crash when G1 is > started with -XX:-UsePerfData? > The reason has been that recent refactoring removed one "if > (UsePerfData)" check too many. > > CR: > https://bugs.openjdk.java.net/browse/JDK-8210265 > Webrev: > http://cr.openjdk.java.net/~tschatzl/8210265/webrev/ > Testing: > new test succeeds > Looks good, thanks for fixing so quickly. Cheers, Stefan > Thanks, > Thomas > From thomas.schatzl at oracle.com Mon Sep 3 11:45:49 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 03 Sep 2018 13:45:49 +0200 Subject: RFR (S): 8210265: Crash in HSpaceCounters::update_used() In-Reply-To: <4ad20a74-d4f5-016a-b2c7-bbcf3e21ef61@redhat.com> References: <4ad20a74-d4f5-016a-b2c7-bbcf3e21ef61@redhat.com> Message-ID: <0c70695e6cb4c6971968be10bac171e0a3d70be1.camel@oracle.com> Hi, On Mon, 2018-09-03 at 12:59 +0200, Aleksey Shipilev wrote: > On 09/03/2018 12:56 PM, Thomas Schatzl wrote: > > CR: > > https://bugs.openjdk.java.net/browse/JDK-8210265 > > Webrev: > > http://cr.openjdk.java.net/~tschatzl/8210265/webrev/ > > Looks trivial and good. > > -Aleksey > thanks for your review. Thomas From thomas.schatzl at oracle.com Mon Sep 3 12:32:15 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 03 Sep 2018 14:32:15 +0200 Subject: Question about JDK-8192969 In-Reply-To: References: <8d3b30c13cd41468c28083dc259e367095118bcb.camel@oracle.com> Message-ID: <651db2bfb59cecab620262c78807d618fbd5593e.camel@oracle.com> Hi, On Fri, 2018-08-31 at 15:43 -0700, Man Cao wrote: > Thanks for the answer before vacation, Thomas! > > I checked and it is indeed not tracking old-to-old copied bytes in > mixed GC. The reason is that in G1Policy::update_ihop_prediction(), > the call to _ihop_control->update_allocation_info() is guarded by > "this_gc_was_young_only", which is collector_state()- > >in_young_only_phase(). > So adaptive IHOP is only considering > _bytes_allocated_in_old_since_last_gc for young-only collections. > This would only include promoted bytes and humongous allocations. > The code was initially introduce in https://bugs.openjdk.java.net/bro > wse/JDK-8136681. > > My follow-up questions are: > (a) Should adaptive IHOP take the promoted bytes during mixed > collections into account? Currently it seems omitting these bytes > altogether. Looking at the code, it is not that straightforward to > separate young-to-old promoted bytes and old-to-old copied bytes > during a mixed collection, because they use the same PLAB mechanism. > Nevertheless, the benefit is that it would make adaptive IHOP's > statistics more accurate, When implementing adaptive IHOP there have iirc been two reasons to not add information during mixed gc to the estimation: - it would take some effort to extract young-to-old promoted bytes from old-to-old promoted bytes which would require some care regarding performance impact. - the young-to-old promotion rate during mixed gc may be significantly different due to the much more frequent GCs because of the smaller young gen (ie. objects have typically significantly "less time to die" during mixed gc). Not sure if that helps the estimation. Another point is that during mixed gc you typically want that the amount of heap occupancy decreases (i.e. young-to-old promotion < freed space). Any mixed gc that does not free more space in old gen than it adds to old gen (due to young-to-old promotion) is kind of wasted effort. This means given this assumption holds, you can't go out of memory due to inexact estimation during the mixed phase. If this assumption does not hold, G1 would need to decide if it should continue with that mixed phase or not (or keep intensifying its efforts to clean up old gen, potentially blowing the pause time goal), which is unfortunately not trivial. I think this has been the main motivation to not do that at that time. It may be interesting to revisit that. > and makes it easier for monitoring tools (e.g. JFR or hsperf > counters) to accurately track promoted bytes. That is indeed true - there is actually an RFE to fix this JDK-8073051. > (b) For humongous allocations, I think it may be better to track them > separately and develop better heuristics for initiating concurrent > mark at humongous allocations. > The reason is that we have tried G1 on several production teams, and > 2 teams have seen many humongous allocations that triggers back-to- > back concurrent marks without reclaiming much memory, resulting in a > situation similarly to GC thrashing and wasting precious CPU time. > The region sizes were 2MB and 4MB. JDK-8163579 may be one instance of what you are experiencing: due to humongous objects G1 frequently crosses the IHOP threshold, causing an initial mark, but due to eager reclaim it never actually increases the IHOP because it never starts mixed gcs as there are no (or almost none) regular objects to reclaim. I think I still have a crude prototype change that implements what is suggested there to share. > I'm able to reproduce the back-to-back concurrent marks issue by > modifying BigRamTester to allocate humongous objects. I'll probably > ask JC to help me create an RFE for this thrashing issue. > In general, humongous objects are probably not that rare in our > workload, and there is still room for improvement in how G1 handles > humongous objects. There are a few open CRs for them, feel free to pick up some :) You might want to look through all CRs with the "gc-g1" label and the term "humongous" in the bug tracker. Depending on your problem, and what you are interested to fix in this area, browsing through them might give you some ideas. Keep us updated with your efforts. Thanks, Thomas From thomas.schatzl at oracle.com Mon Sep 3 14:48:10 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 03 Sep 2018 16:48:10 +0200 Subject: Storing both "fasttime" and time in Ticks and Tickspan In-Reply-To: References: <9f542de58eac66a07312da45fb4023893ab6e2e3.camel@oracle.com> Message-ID: Hi Kim, On Tue, 2018-08-28 at 20:36 -0400, Kim Barrett wrote: > > On Aug 24, 2018, at 8:03 AM, Thomas Schatzl > com> wrote: > > > > Hi all, > [...] > So I think the FastUnorderedElapsedCounterSource and everything > related to it should be removed. (I think this includes > cpu/x86/rdtsc_x86.[ch]pp and UseFastUnorderedTimeStamps.) > > Removing a few milliseconds of sleeping during the initialization of > this feature is a nice little startup benefit. > I agree with all that, but I would like to have an opinion from the JFR team too about this before doing this. The alternative I can see would be having GC specific Ticks/Tickspan which I would like to avoid for obvious reasons. Thanks, Thomas From kim.barrett at oracle.com Tue Sep 4 01:11:28 2018 From: kim.barrett at oracle.com (Kim Barrett) Date: Mon, 3 Sep 2018 21:11:28 -0400 Subject: Storing both "fasttime" and time in Ticks and Tickspan In-Reply-To: References: <9f542de58eac66a07312da45fb4023893ab6e2e3.camel@oracle.com> Message-ID: <5F8AF025-116B-4D0B-9EB9-9A37B3A3E51D@oracle.com> > On Sep 3, 2018, at 10:48 AM, Thomas Schatzl wrote: > > Hi Kim, > > On Tue, 2018-08-28 at 20:36 -0400, Kim Barrett wrote: >>> On Aug 24, 2018, at 8:03 AM, Thomas Schatzl >> com> wrote: >>> >>> Hi all, > >> [...] >> So I think the FastUnorderedElapsedCounterSource and everything >> related to it should be removed. (I think this includes >> cpu/x86/rdtsc_x86.[ch]pp and UseFastUnorderedTimeStamps.) >> >> Removing a few milliseconds of sleeping during the initialization of >> this feature is a nice little startup benefit. >> > > I agree with all that, but I would like to have an opinion from the > JFR team too about this before doing this. Agreed. > The alternative I can see would be having GC specific Ticks/Tickspan > which I would like to avoid for obvious reasons. Or JFR-specific variants, and let everyone else do what we think is right. > Thanks, > Thomas From kim.barrett at oracle.com Tue Sep 4 01:15:07 2018 From: kim.barrett at oracle.com (Kim Barrett) Date: Mon, 3 Sep 2018 21:15:07 -0400 Subject: RFR (S): 8210265: Crash in HSpaceCounters::update_used() In-Reply-To: References: Message-ID: > On Sep 3, 2018, at 6:56 AM, Thomas Schatzl wrote: > > Hi all, > > can I have reviews for this change that fixes a crash when G1 is > started with -XX:-UsePerfData? > The reason has been that recent refactoring removed one "if > (UsePerfData)" check too many. > > CR: > https://bugs.openjdk.java.net/browse/JDK-8210265 > Webrev: > http://cr.openjdk.java.net/~tschatzl/8210265/webrev/ > Testing: > new test succeeds > > Thanks, > Thomas Looks good. However, I was wondering why we?re making these unusable HSpaceCounters at all. From manc at google.com Tue Sep 4 01:37:14 2018 From: manc at google.com (Man Cao) Date: Mon, 3 Sep 2018 18:37:14 -0700 Subject: RFR (XS) 8210192: Hsperf counter ParNew::CMS should be ParNew:CMS In-Reply-To: <280d09df-3051-0f39-52bb-2a9f6ad7ff04@oracle.com> References: <280d09df-3051-0f39-52bb-2a9f6ad7ff04@oracle.com> Message-ID: Hi all, Thanks, Stefan! I have added a test for the hsperf counter. The test is based on test/hotspot/jtreg/gc/TestGenerationPerfCounter.java. New webrev: http://cr.openjdk.java.net/~jcbeyler/8210192/webrev.01/ I have run some basic tests with "make run-test", and those related to test/hotspot/jtreg/gc/testlibrary/PerfCounter.java are still passing. Thanks, Man On Friday, August 31, 2018, Stefan Johansson wrote: > On 2018-08-31 02:07, Man Cao wrote: > >> Thanks for the review! >> Should we wait for another "looks good" before JC can submit it? >> >> Yes, we should wait to get a second review. > > Since the testing infrastructure did not catch this change, should we add >> a test for this hsperf counter? >> > That would probably be good, please look at the other counter tests and > create something similar to that. Once you have the test I can apply the > patch and run it through our test system to make sure it passes on all > platforms. > > Thanks, > Stefan > > >> PS: I found a better command to test for its value: >> $ jstat -J-Djstat.showUnsupported=true -snap file:///$(pwd)/test.hsperf | >> grep sun.gc.policy.name >> >> -Man >> >> >> On Thu, Aug 30, 2018 at 1:52 AM Stefan Johansson < >> stefan.johansson at oracle.com > wrote: >> >> >> >> On 2018-08-30 02:29, Man Cao wrote: >> > Hi all, >> > >> > I noticed a possible slight naming bug for the HSPerf >> > counter "sun.gc.policy.name >> ", introduced by >> > http://hg.openjdk.java.net/jdk/jdk/rev/170c7b36aea6. >> > >> > Basically instead of "ParNew:CMS", the counter's value is now >> "ParNew::CMS". >> > >> > Here is a fix for the bug, could someone review it? >> > >> > Webrev: http://cr.openjdk.java.net/~jcbeyler/8210192/webrev.00/ >> >> > >> > Bug: https://bugs.openjdk.java.net/browse/JDK-8210192 >> >> Looks good and since the bug hasn't shown up in our testing I don't >> think fixing it back should be a problem testing wise. >> >> Thanks, >> StefanJ >> >> > >> > PS: Thanks JC for creating the bug and hosting the webrev! >> > >> > Thanks! >> > Man >> >> -- -Man -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Tue Sep 4 08:29:30 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 04 Sep 2018 10:29:30 +0200 Subject: RFR (S): 8210265: Crash in HSpaceCounters::update_used() In-Reply-To: References: Message-ID: <8d86d7fefd5fef0f2ad7af3e24dcafdc42bf9d0c.camel@oracle.com> Hi, On Mon, 2018-09-03 at 21:15 -0400, Kim Barrett wrote: > > On Sep 3, 2018, at 6:56 AM, Thomas Schatzl > om> wrote: > > > > Hi all, > > > > can I have reviews for this change that fixes a crash when G1 is > > started with -XX:-UsePerfData? > > The reason has been that recent refactoring removed one "if > > (UsePerfData)" check too many. > > > > CR: > > https://bugs.openjdk.java.net/browse/JDK-8210265 > > Webrev: > > http://cr.openjdk.java.net/~tschatzl/8210265/webrev/ > > Testing: > > new test succeeds > > > > Thanks, > > Thomas > > Looks good. > thanks for your review. > However, I was wondering why we?re making these unusable > HSpaceCounters at all. the _from_space_counters is there for backwards compatibility only afaik. Thanks, Thomas From thomas.schatzl at oracle.com Tue Sep 4 08:40:45 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 04 Sep 2018 10:40:45 +0200 Subject: RFR (M): 8207200: Committed > max memory usage when getting MemoryUsage In-Reply-To: <77380ea7-86ae-0418-17d2-5b96defad0e1@oracle.com> References: <77380ea7-86ae-0418-17d2-5b96defad0e1@oracle.com> Message-ID: Hi Sangheon, On Fri, 2018-08-24 at 11:42 -0700, sangheon.kim at oracle.com wrote: > Hi Thomas, > > On 8/24/18 3:17 AM, Thomas Schatzl wrote: > > Hi, > > > > ping! :) > > > > On Wed, 2018-08-08 at 17:15 +0200, Thomas Schatzl wrote: > > > Hi all, > > > > > > can I have reviews for this change that fixes some races when > > > reading from G1MonitoringSupport members, causing some assertion > > > failing after weeks of running some programs that poll the G1 > > > MemoryPools. > > > > > > It does so by adding a mutex that is held during reading and > > > writing them. > > > > > > There is also a new explicit call in CollectedHeap to get a > > > consistent overall MemoryUsage information instead of the > > > existing code looping over the heap's memory pools. That one is > > > problematic too otherwise. > > > > > > There is still no change in the calculation of the values of the > > > MemoryUsage members. > > > > > > CR: > > > https://bugs.openjdk.java.net/browse/JDK-8207200 > > > Webrev: > > > http://cr.openjdk.java.net/~tschatzl/8207200/webrev > > > Testing: > > > hs-tier1-4,jdk-tier1-3 > > > > > > This change is based on the changes for JDK-8209062. > > > > these changes have been pushed by now. > > Looks good as is. > > One minor nit(you can skip) at > src/hotspot/share/gc/g1/g1MonitoringSupport.cpp. > As you are repeating below pattern, you can make a common method and > use > it for 4 locations. > G1MonitoringSupport::memory_usage(), eden_space_memory_usage(), > survivor_space_memory_usage() and > old_space_memory_usage(). > > MemoryUsage memory_usage_locked(size_t initial, used, committed, max) > { > MutexLockerEx x(&_update_mutex, > Mutex::_no_safepoint_check_flag); > return MemoryUsage(initial, used, committed, max); > } this won't work as we need to make sure that the values copied to MemoryUsage (initial, used, committed, max) are read within the scope of the Mutex. Otherwise they may again be inconsistent. > BTW, how did you verify this patch? Or have you test this patch with > the reporter? > I think this patch will fix the problem, but just curious. There has been no extended testing as I never got to reproduce it locally. In hindsight the problem and its fix are "obvious" though. Thanks for your review, Thomas From thomas.schatzl at oracle.com Tue Sep 4 08:41:14 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 04 Sep 2018 10:41:14 +0200 Subject: RFR (M): 8207200: Committed > max memory usage when getting MemoryUsage In-Reply-To: References: Message-ID: Hi Mandy, On Fri, 2018-08-24 at 09:06 -0500, mandy chung wrote: > > On 8/24/18 5:17 AM, Thomas Schatzl wrote: > > On Wed, 2018-08-08 at 17:15 +0200, Thomas Schatzl wrote: > > > Hi all, > > > > > > can I have reviews for this change that fixes some races when > > > reading from G1MonitoringSupport members, causing some assertion > > > failing after weeks of running some programs that poll the G1 > > > MemoryPools. > > > > > > It does so by adding a mutex that is held during reading and > > > writing them. > > > > > > There is also a new explicit call in CollectedHeap to get a > > > consistent overall MemoryUsage information instead of the > > > existing > > > code looping over the heap's memory pools. That one is > > > problematic > > > too otherwise. > > > > > > There is still no change in the calculation of the values of the > > > MemoryUsage members. > > > > > > CR: > > > https://bugs.openjdk.java.net/browse/JDK-8207200 > > > Webrev: > > > http://cr.openjdk.java.net/~tschatzl/8207200/webrev > > This patch looks good to me AFAICT. thanks for your review. Thomas From erik.osterlund at oracle.com Tue Sep 4 09:39:07 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 4 Sep 2018 11:39:07 +0200 Subject: RFR: 8210064: ZGC: Introduce ZConcurrentRootsIterator for scanning a subset of strong IN_NATIVE roots concurrently In-Reply-To: References: <28b4de13-e167-67b2-8dec-3948c1e8c8c8@oracle.com> <123ab70e-f1e3-e83b-a03c-671899386fee@oracle.com> Message-ID: <5B8E52BB.10700@oracle.com> Hi, Coleen pushed 8210155 before I pushed this. So now I need to wrap the CLDG iterations in a CLDG lock. I made a patch that adds the lock and relaxes the locking assert appropriately so that the thread invoking the worker gang can take the lock. This implies that the worker threads will not own the lock, but that is okay. Incremental: http://cr.openjdk.java.net/~eosterlund/8210064/webrev.01_02/ Full: http://cr.openjdk.java.net/~eosterlund/8210064/webrev.02/ Thanks, /Erik On 2018-08-31 10:37, Erik ?sterlund wrote: > Hi Per, > > On 2018-08-31 09:57, Per Liden wrote: >> Hi Erik, >> >> On 08/30/2018 10:46 AM, Erik ?sterlund wrote: >>> Hi, >>> >>> We now have enough load barriers to support scanning of CLDs and JNI >>> handles concurrently. I propose to do that and move these root sets >>> out from ZRootsIterator, and hence the GC pause. They will be >>> scanned during concurrent marking (and heap iteration), but no >>> longer during relocation. >>> >>> I still perform ClassLoaderDataGraph::clear_claimed_marks() in the >>> pause because it seems cheap. But it can be moved out of the pause >>> when Coleen gets her new cool CLDG lock in. >>> >>> Webrev: >>> http://cr.openjdk.java.net/~eosterlund/8210064/webrev.00/ >> >> I have some minor requests. Instead of listing them all, I attached a >> patch which addresses those. >> >> The main thing is that I don't think ZDriver should know about >> "concurrent roots", just that it's doing "mark" or "mark continue", >> so I suggest we turn that into a "bool initial" argument to mark() >> instead of exposing a mark_concurrent_roots() function. > > Sure, that makes sense. > > Thanks for the review! > > /Erik > >> The other things are minor style adjustments. >> /Per >> >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8210064 >>> >>> Tested through hs-tier1-3. >>> >>> Thanks, >>> /Erik > From per.liden at oracle.com Tue Sep 4 09:47:05 2018 From: per.liden at oracle.com (Per Liden) Date: Tue, 4 Sep 2018 11:47:05 +0200 Subject: RFR: 8210064: ZGC: Introduce ZConcurrentRootsIterator for scanning a subset of strong IN_NATIVE roots concurrently In-Reply-To: <5B8E52BB.10700@oracle.com> References: <28b4de13-e167-67b2-8dec-3948c1e8c8c8@oracle.com> <123ab70e-f1e3-e83b-a03c-671899386fee@oracle.com> <5B8E52BB.10700@oracle.com> Message-ID: <1044166e-7581-c3a7-ce84-eb363188a000@oracle.com> Hi Erik, On 09/04/2018 11:39 AM, Erik ?sterlund wrote: > Hi, > > Coleen pushed 8210155 before I pushed this. So now I need to wrap the > CLDG iterations in a CLDG lock. > I made a patch that adds the lock and relaxes the locking assert > appropriately so that the thread invoking the worker gang can take the > lock. This implies that the worker threads will not own the lock, but > that is okay. > > Incremental: > http://cr.openjdk.java.net/~eosterlund/8210064/webrev.01_02/ > > Full: > http://cr.openjdk.java.net/~eosterlund/8210064/webrev.02/ Looks good. Just one small thing. It seems the old assert_locked_or_safepoint() can now be expressed as assert_locked_or_safepoint_weak() plus some extra conditions. /Per > > Thanks, > /Erik > > On 2018-08-31 10:37, Erik ?sterlund wrote: >> Hi Per, >> >> On 2018-08-31 09:57, Per Liden wrote: >>> Hi Erik, >>> >>> On 08/30/2018 10:46 AM, Erik ?sterlund wrote: >>>> Hi, >>>> >>>> We now have enough load barriers to support scanning of CLDs and JNI >>>> handles concurrently. I propose to do that and move these root sets >>>> out from ZRootsIterator, and hence the GC pause. They will be >>>> scanned during concurrent marking (and heap iteration), but no >>>> longer during relocation. >>>> >>>> I still perform ClassLoaderDataGraph::clear_claimed_marks() in the >>>> pause because it seems cheap. But it can be moved out of the pause >>>> when Coleen gets her new cool CLDG lock in. >>>> >>>> Webrev: >>>> http://cr.openjdk.java.net/~eosterlund/8210064/webrev.00/ >>> >>> I have some minor requests. Instead of listing them all, I attached a >>> patch which addresses those. >>> >>> The main thing is that I don't think ZDriver should know about >>> "concurrent roots", just that it's doing "mark" or "mark continue", >>> so I suggest we turn that into a "bool initial" argument to mark() >>> instead of exposing a mark_concurrent_roots() function. >> >> Sure, that makes sense. >> >> Thanks for the review! >> >> /Erik >> >>> The other things are minor style adjustments. >>> /Per >>> >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8210064 >>>> >>>> Tested through hs-tier1-3. >>>> >>>> Thanks, >>>> /Erik >> > From erik.osterlund at oracle.com Tue Sep 4 10:17:39 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Tue, 4 Sep 2018 12:17:39 +0200 Subject: RFR: 8210064: ZGC: Introduce ZConcurrentRootsIterator for scanning a subset of strong IN_NATIVE roots concurrently In-Reply-To: <1044166e-7581-c3a7-ce84-eb363188a000@oracle.com> References: <28b4de13-e167-67b2-8dec-3948c1e8c8c8@oracle.com> <123ab70e-f1e3-e83b-a03c-671899386fee@oracle.com> <5B8E52BB.10700@oracle.com> <1044166e-7581-c3a7-ce84-eb363188a000@oracle.com> Message-ID: <5B8E5BC3.7090106@oracle.com> Hi Per, On 2018-09-04 11:47, Per Liden wrote: > Hi Erik, > > On 09/04/2018 11:39 AM, Erik ?sterlund wrote: >> Hi, >> >> Coleen pushed 8210155 before I pushed this. So now I need to wrap the >> CLDG iterations in a CLDG lock. >> I made a patch that adds the lock and relaxes the locking assert >> appropriately so that the thread invoking the worker gang can take >> the lock. This implies that the worker threads will not own the lock, >> but that is okay. >> >> Incremental: >> http://cr.openjdk.java.net/~eosterlund/8210064/webrev.01_02/ >> >> Full: >> http://cr.openjdk.java.net/~eosterlund/8210064/webrev.02/ > > Looks good. Just one small thing. It seems the old > assert_locked_or_safepoint() can now be expressed as > assert_locked_or_safepoint_weak() plus some extra conditions. Thanks for the review. Unfortunately it looks like I can't express the strong one with the weak one as it has early exit code, and I need to change the weak one to return a boolean to make that work. Oh well. Thanks, /Erik > /Per > >> >> Thanks, >> /Erik >> >> On 2018-08-31 10:37, Erik ?sterlund wrote: >>> Hi Per, >>> >>> On 2018-08-31 09:57, Per Liden wrote: >>>> Hi Erik, >>>> >>>> On 08/30/2018 10:46 AM, Erik ?sterlund wrote: >>>>> Hi, >>>>> >>>>> We now have enough load barriers to support scanning of CLDs and >>>>> JNI handles concurrently. I propose to do that and move these root >>>>> sets out from ZRootsIterator, and hence the GC pause. They will be >>>>> scanned during concurrent marking (and heap iteration), but no >>>>> longer during relocation. >>>>> >>>>> I still perform ClassLoaderDataGraph::clear_claimed_marks() in the >>>>> pause because it seems cheap. But it can be moved out of the pause >>>>> when Coleen gets her new cool CLDG lock in. >>>>> >>>>> Webrev: >>>>> http://cr.openjdk.java.net/~eosterlund/8210064/webrev.00/ >>>> >>>> I have some minor requests. Instead of listing them all, I attached >>>> a patch which addresses those. >>>> >>>> The main thing is that I don't think ZDriver should know about >>>> "concurrent roots", just that it's doing "mark" or "mark continue", >>>> so I suggest we turn that into a "bool initial" argument to mark() >>>> instead of exposing a mark_concurrent_roots() function. >>> >>> Sure, that makes sense. >>> >>> Thanks for the review! >>> >>> /Erik >>> >>>> The other things are minor style adjustments. >>>> /Per >>>> >>>>> >>>>> Bug: >>>>> https://bugs.openjdk.java.net/browse/JDK-8210064 >>>>> >>>>> Tested through hs-tier1-3. >>>>> >>>>> Thanks, >>>>> /Erik >>> >> From per.liden at oracle.com Tue Sep 4 10:23:46 2018 From: per.liden at oracle.com (Per Liden) Date: Tue, 4 Sep 2018 12:23:46 +0200 Subject: RFR: 8210064: ZGC: Introduce ZConcurrentRootsIterator for scanning a subset of strong IN_NATIVE roots concurrently In-Reply-To: <5B8E5BC3.7090106@oracle.com> References: <28b4de13-e167-67b2-8dec-3948c1e8c8c8@oracle.com> <123ab70e-f1e3-e83b-a03c-671899386fee@oracle.com> <5B8E52BB.10700@oracle.com> <1044166e-7581-c3a7-ce84-eb363188a000@oracle.com> <5B8E5BC3.7090106@oracle.com> Message-ID: <66825cea-9e7f-2736-1656-6bf7c8b9cf80@oracle.com> On 09/04/2018 12:17 PM, Erik ?sterlund wrote: > Hi Per, > > On 2018-09-04 11:47, Per Liden wrote: >> Hi Erik, >> >> On 09/04/2018 11:39 AM, Erik ?sterlund wrote: >>> Hi, >>> >>> Coleen pushed 8210155 before I pushed this. So now I need to wrap the >>> CLDG iterations in a CLDG lock. >>> I made a patch that adds the lock and relaxes the locking assert >>> appropriately so that the thread invoking the worker gang can take >>> the lock. This implies that the worker threads will not own the lock, >>> but that is okay. >>> >>> Incremental: >>> http://cr.openjdk.java.net/~eosterlund/8210064/webrev.01_02/ >>> >>> Full: >>> http://cr.openjdk.java.net/~eosterlund/8210064/webrev.02/ >> >> Looks good. Just one small thing. It seems the old >> assert_locked_or_safepoint() can now be expressed as >> assert_locked_or_safepoint_weak() plus some extra conditions. > > Thanks for the review. Unfortunately it looks like I can't express the > strong one with the weak one as it has early exit code, and I need to > change the weak one to return a boolean to make that work. Oh well. Ah, you're right of course. Ignore that. /Per > > Thanks, > /Erik > >> /Per >> >>> >>> Thanks, >>> /Erik >>> >>> On 2018-08-31 10:37, Erik ?sterlund wrote: >>>> Hi Per, >>>> >>>> On 2018-08-31 09:57, Per Liden wrote: >>>>> Hi Erik, >>>>> >>>>> On 08/30/2018 10:46 AM, Erik ?sterlund wrote: >>>>>> Hi, >>>>>> >>>>>> We now have enough load barriers to support scanning of CLDs and >>>>>> JNI handles concurrently. I propose to do that and move these root >>>>>> sets out from ZRootsIterator, and hence the GC pause. They will be >>>>>> scanned during concurrent marking (and heap iteration), but no >>>>>> longer during relocation. >>>>>> >>>>>> I still perform ClassLoaderDataGraph::clear_claimed_marks() in the >>>>>> pause because it seems cheap. But it can be moved out of the pause >>>>>> when Coleen gets her new cool CLDG lock in. >>>>>> >>>>>> Webrev: >>>>>> http://cr.openjdk.java.net/~eosterlund/8210064/webrev.00/ >>>>> >>>>> I have some minor requests. Instead of listing them all, I attached >>>>> a patch which addresses those. >>>>> >>>>> The main thing is that I don't think ZDriver should know about >>>>> "concurrent roots", just that it's doing "mark" or "mark continue", >>>>> so I suggest we turn that into a "bool initial" argument to mark() >>>>> instead of exposing a mark_concurrent_roots() function. >>>> >>>> Sure, that makes sense. >>>> >>>> Thanks for the review! >>>> >>>> /Erik >>>> >>>>> The other things are minor style adjustments. >>>>> /Per >>>>> >>>>>> >>>>>> Bug: >>>>>> https://bugs.openjdk.java.net/browse/JDK-8210064 >>>>>> >>>>>> Tested through hs-tier1-3. >>>>>> >>>>>> Thanks, >>>>>> /Erik >>>> >>> > From thomas.schatzl at oracle.com Tue Sep 4 10:35:39 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 04 Sep 2018 12:35:39 +0200 Subject: RFR (S): 8209843: Optimize oop scan closure closures wrt to reference processing in G1 In-Reply-To: References: <74dab00bcc1885f04b04378a9120e854dabe5bfc.camel@oracle.com> <0734c7b2-31bf-c741-0efd-36729fceff6c@oracle.com> Message-ID: Hi Kim, On Sun, 2018-08-26 at 19:38 -0400, Kim Barrett wrote: > > On Aug 24, 2018, at 10:24 AM, Per Liden > > wrote: > > > > Hi Thomas, > > > > On 08/24/2018 03:16 PM, Thomas Schatzl wrote: > > > CR: > > > https://bugs.openjdk.java.net/browse/JDK-8209843 > > > Webrev: > > > http://cr.openjdk.java.net/~tschatzl/8209843/webrev/ > > > > About the new assert in iteration.hpp. Isn't it better to have it > > somewhere in the actual oop_oop_iterate path so that we also catch > > those cases that were using the constructor? Something like: > > > > --- a/src/hotspot/share/oops/instanceRefKlass.inline.hpp > > +++ b/src/hotspot/share/oops/instanceRefKlass.inline.hpp > > @@ -98,12 +98,14 @@ > > > > template > > void InstanceRefKlass::oop_oop_iterate_fields(oop obj, > > OopClosureType* closure, Contains& contains) { > > + assert(closure->ref_discoverer() == NULL, "ReferenceDiscoverer > > should not be set"); > > do_referent(obj, closure, contains); > > do_discovered(obj, closure, contains); > > } > > > > template > > void InstanceRefKlass::oop_oop_iterate_fields_except_referent(oop > > obj, OopClosureType* closure, Contains& contains) { > > + assert(closure->ref_discoverer() == NULL, "ReferenceDiscoverer > > should not be set"); > > do_discovered(obj, closure, contains); > > } > > > > If you do this, I'm thinking you also don't need the asserts you > > added in g1OopClosures.inline.hpp? > > > > cheers, > > Per > > I was wondering something similar, but thinking a place for such a > check might be in the reference_iteration_mode definitions. That is, > each implementation of that function should verify the discoverer > state is appropriate for the mode that is going to be returned. > > Though that wouldn't work if there are places that are disabling > discovery via a NULL discoverer, rather than via disable_discovery(). > But nobody would do that, right? :) G1 and ZGC use this shortcut to have a NULL discoverer if discovery is disabled (G1 makes sure both are consistent, ZGC's discoverer does not have any other way to disable discovery afaict). There is some code in the serial gc mark sweep compact algorithm to set the discoverer to NULL, but apparently only as a way to somehow catch bugs (which eludes me how this should work). Changing things either way (let G1 never use a NULL discoverer with an iteration_mode() indicating discovery or change other collectors to use a NULL discoverer to disable discovery) seems to go beyond what this RFE is set out to do. As for asserting, this leaves us with Per's suggestion that I implemented at http://cr.openjdk.java.net/~tschatzl/8209843/webrev.1/ (full only, sorry, but the change is very small) Thanks, Thomas From per.liden at oracle.com Tue Sep 4 12:34:56 2018 From: per.liden at oracle.com (Per Liden) Date: Tue, 4 Sep 2018 14:34:56 +0200 Subject: RFR (S): 8209843: Optimize oop scan closure closures wrt to reference processing in G1 In-Reply-To: References: <74dab00bcc1885f04b04378a9120e854dabe5bfc.camel@oracle.com> <0734c7b2-31bf-c741-0efd-36729fceff6c@oracle.com> Message-ID: <0e112c9a-7ec0-3e89-c78c-e4411d9a9281@oracle.com> Hi Thomas, On 09/04/2018 12:35 PM, Thomas Schatzl wrote: [...] > As for asserting, this leaves us with Per's suggestion that I > implemented at > > http://cr.openjdk.java.net/~tschatzl/8209843/webrev.1/ (full only, > sorry, but the change is very small) Looks good to me. /Per From stefan.johansson at oracle.com Tue Sep 4 12:47:13 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Tue, 4 Sep 2018 14:47:13 +0200 Subject: RFR (XS) 8210192: Hsperf counter ParNew::CMS should be ParNew:CMS In-Reply-To: References: <280d09df-3051-0f39-52bb-2a9f6ad7ff04@oracle.com> Message-ID: <35da9f09-b5da-ae7b-73d7-4c319f1993d7@oracle.com> Thanks for providing a test Man, I've run the test through our test system to make sure it runs ok on our core platforms and it did. This still looks good, but we still need a second reviewer. Cheers, Stefan On 2018-09-04 03:37, Man Cao wrote: > Hi all, > > Thanks, Stefan! I have added a test for the hsperf counter. The test is > based on test/hotspot/jtreg/gc/TestGenerationPerfCounter.java. > New webrev: > http://cr.openjdk.java.net/~jcbeyler/8210192/webrev.01/ > > > I have run some basic tests with "make run-test", and those related > to?test/hotspot/jtreg/gc/testlibrary/PerfCounter.java are still passing. > > Thanks, > Man > > On Friday, August 31, 2018, Stefan Johansson > > wrote: > > On 2018-08-31 02:07, Man Cao wrote: > > Thanks for the review! > Should we wait for another "looks good" before JC can submit it? > > Yes, we should wait to get a second review. > > Since the testing infrastructure did not catch this change, > should we add a test for this hsperf counter? > > That would probably be good, please look at the other counter tests > and create something similar to that. Once you have the test I can > apply the patch and run it through our test system to make sure it > passes on all platforms. > > Thanks, > Stefan > > > PS: I found a better command to test for its value: > $ jstat -J-Djstat.showUnsupported=true -snap > file:///$(pwd)/test.hsperf | grep sun.gc.policy.name > > > -Man > > > On Thu, Aug 30, 2018 at 1:52 AM Stefan Johansson > > >> wrote: > > > > ? ? On 2018-08-30 02:29, Man Cao wrote: > ? ? ?> Hi all, > ? ? ?> > ? ? ?> I noticed a possible slight naming bug for the HSPerf > ? ? ?> counter?"sun.gc.policy.name > > ? ? ", introduced by > ? ? ?> http://hg.openjdk.java.net/jdk/jdk/rev/170c7b36aea6 > . > ? ? ?> > ? ? ?> Basically instead of "ParNew:CMS", the counter's value > is now > ? ? "ParNew::CMS". > ? ? ?> > ? ? ?> Here is a fix for the bug, could someone review it? > ? ? ?> > ? ? ?> Webrev: > http://cr.openjdk.java.net/~jcbeyler/8210192/webrev.00/ > > ? ? > > ? ? ?> > > > ? ? ?> Bug: https://bugs.openjdk.java.net/browse/JDK-8210192 > > > ? ? Looks good and since the bug hasn't shown up in our testing > I don't > ? ? think fixing it back should be a problem testing wise. > > ? ? Thanks, > ? ? StefanJ > > ? ? ?> > ? ? ?> PS: Thanks JC for creating the bug and hosting the webrev! > ? ? ?> > ? ? ?> Thanks! > ? ? ?> Man > > > > -- > -Man > From thomas.schatzl at oracle.com Tue Sep 4 13:24:26 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 04 Sep 2018 15:24:26 +0200 Subject: RFR (XS) 8210192: Hsperf counter ParNew::CMS should be ParNew:CMS In-Reply-To: References: <280d09df-3051-0f39-52bb-2a9f6ad7ff04@oracle.com> Message-ID: Hi, On Mon, 2018-09-03 at 18:37 -0700, Man Cao wrote: > Hi all, > > Thanks, Stefan! I have added a test for the hsperf counter. The test > is based on test/hotspot/jtreg/gc/TestGenerationPerfCounter.java. > New webrev: > http://cr.openjdk.java.net/~jcbeyler/8210192/webrev.01/ > > I have run some basic tests with "make run-test", and those related > to test/hotspot/jtreg/gc/testlibrary/PerfCounter.java are still > passing. > looks good. Thomas From rbruno at gsd.inesc-id.pt Tue Sep 4 15:04:42 2018 From: rbruno at gsd.inesc-id.pt (Rodrigo Bruno) Date: Tue, 4 Sep 2018 15:04:42 +0000 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> Message-ID: Hi Thomas, Did you ever notice an improvement in heap usage after the first such > attempt? > There should be none except for memory held by j.l.ref.References. > > What I want to get at, as a later improvement, we could think of just > skipping those then (as except as mentioned for the j.l.ref.References > issue there would be "no" difference). > > I didn't devise any experiments to measure memory usage. However, we can obviously do that. The new version of the patch regarding http://openjdk.java.net/jeps/8204089 can be found at http://www.gsd.inesc-id.pt/~rbruno/webrev.zip This version already includes idle compaction (Full GC is optional) and your previous comments as well. Let me know what you think. cheers, rodrigo -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Tue Sep 4 15:20:31 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 04 Sep 2018 17:20:31 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> Message-ID: <7b000ee7dfcf4c004140dc52138cdc83c8b46ab2.camel@oracle.com> Hi Rodrigo, I extracted your webrev to http://cr.openjdk.java.net/~tschatzl/jelastic/pgc-webrev.1/ for easier review. Thanks, Thomas On Tue, 2018-09-04 at 15:04 +0000, Rodrigo Bruno wrote: > Hi Thomas, > > > Did you ever notice an improvement in heap usage after the first > > such > > attempt? > > There should be none except for memory held by j.l.ref.References. > > > > What I want to get at, as a later improvement, we could think of > > just > > skipping those then (as except as mentioned for the > > j.l.ref.References > > issue there would be "no" difference). > > > > I didn't devise any experiments to measure memory usage. However, we > can obviously do that. > > The new version of the patch regarding > > http://openjdk.java.net/jeps/8204089 > > can be found at > > http://www.gsd.inesc-id.pt/~rbruno/webrev.zip > > This version already includes idle compaction (Full GC is optional) > and your previous comments as well. > > Let me know what you think. > > cheers, > rodrigo > From jcbeyler at google.com Tue Sep 4 21:25:49 2018 From: jcbeyler at google.com (JC Beyler) Date: Tue, 4 Sep 2018 14:25:49 -0700 Subject: RFR (XS) 8210192: Hsperf counter ParNew::CMS should be ParNew:CMS In-Reply-To: References: <280d09df-3051-0f39-52bb-2a9f6ad7ff04@oracle.com> Message-ID: Hi all, I created the webrev with the metadata here: http://cr.openjdk.java.net/~jcbeyler/8210192/webrev.02/ I pushed the fix since Thomas & Stefan reviewed it. I added Man as the contributor and ensured it built and passed the new test on my side (which with Stefan's testing seemed to be sufficient). Thanks Man for the fix and Thomas and Stefan for the reviews :) Jc On Tue, Sep 4, 2018 at 6:25 AM Thomas Schatzl wrote: > Hi, > > On Mon, 2018-09-03 at 18:37 -0700, Man Cao wrote: > > Hi all, > > > > Thanks, Stefan! I have added a test for the hsperf counter. The test > > is based on test/hotspot/jtreg/gc/TestGenerationPerfCounter.java. > > New webrev: > > http://cr.openjdk.java.net/~jcbeyler/8210192/webrev.01/ > > > > I have run some basic tests with "make run-test", and those related > > to test/hotspot/jtreg/gc/testlibrary/PerfCounter.java are still > > passing. > > > > looks good. > > Thomas > > -- Thanks, Jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From kishor.kharbas at intel.com Wed Sep 5 05:41:02 2018 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Wed, 5 Sep 2018 05:41:02 +0000 Subject: RFR(M): 8204908: NVDIMM for POGC and G1GC - ReserveSpace.cpp changes are mostly eliminated/no collector specific code. In-Reply-To: References: <5df6d080894cfad5e6486a00f28b6ccfc5ca633f.camel@oracle.com> <87e5373f-a2e0-083e-6421-0209f519dca5@oracle.com> <177b91b2-c0ed-d7ad-84ab-ce479f07474a@oracle.com> Message-ID: Hi, I have uploaded implementation for parallel scavenge at http://cr.openjdk.java.net/~kkharbas/8204908/webrev-8204908.00 Majority of the implementation is handled in two new files - adjoiningGenerationsForHeteroHeap.cpp and psFileBackedVirtualspace.cpp. Would love to hear your feedback and suggestions. I will post G1 GC implementation in a few days. Thanks Kishor. > -----Original Message----- > From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] > Sent: Thursday, August 30, 2018 4:06 PM > To: Kharbas, Kishor ; Thomas St?fe > ; Thomas Schatzl > > Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime runtime-dev at openjdk.java.net>; Viswanathan, Sandhya > ; Aundhe, Shirish > > Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > ReserveSpace.cpp changes are mostly eliminated/no collector specific code. > > Hi Kishor, > > On 8/30/18 11:55 AM, Kharbas, Kishor wrote: > > Hi Sangheon, > > > > So far I don?t see a need to do so. I will post my implementation code > today or tomorrow, if we see a need or any simplification by changing > classes in VirtualSpace.hpp, we can discuss that. > Okay. > > Thanks, > Sangheon > > > > > > Regards, > > -Kishor > > > >> -----Original Message----- > >> From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] > >> Sent: Wednesday, August 29, 2018 10:17 AM > >> To: Kharbas, Kishor ; Thomas St?fe > >> ; Thomas Schatzl > > >> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime >> runtime-dev at openjdk.java.net>; Viswanathan, Sandhya > >> ; Aundhe, Shirish > >> > >> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > >> ReserveSpace.cpp changes are mostly eliminated/no collector specific > code. > >> > >> Hi Kishor, > >> > >> On 8/29/18 9:52 AM, Kharbas, Kishor wrote: > >>> Hi Sangheon, > >>> > >>> Thanks for reviewing the design. > >>> Since the collectors do not use them for heap memory, I did not have > >>> to override VirtualSpace > >> Sorry, I meant ReservedSpace and its friends at > >> share/memory/virtualspace.hpp. > >> > >> Thanks, > >> Sangheon > >> > >> > >>> -Kishor > >>>> -----Original Message----- > >>>> From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] > >>>> Sent: Tuesday, August 28, 2018 2:38 PM > >>>> To: Kharbas, Kishor ; Thomas St?fe > >>>> ; Thomas Schatzl > >> > >>>> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime >>>> runtime-dev at openjdk.java.net>; Viswanathan, Sandhya > >>>> ; Aundhe, Shirish > >>>> > >>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > >>>> ReserveSpace.cpp changes are mostly eliminated/no collector > >>>> specific > >> code. > >>>> Hi Kishor, > >>>> > >>>> On 8/21/18 10:57 AM, Kharbas, Kishor wrote: > >>>>> Hi All, > >>>>> > >>>>> Thank you for your valuable comments and feedback in this thread > >>>>> so > >> far. > >>>>> After taken in all the comments and reading thoroughly through the > >>>>> code, I > >>>> feel that the complexity added to virtualSpace.cpp is due to lack > >>>> of abstraction in VirtualSpace and at GC level. NV-DIMM size and > >>>> file paths are being passed all the way to OS calls. > >>>>> So I went back to the drawing board and created a high level > >>>>> design to remove all the pain points of current implementation. I > >>>>> have uploaded the design at > >>>>> > >> > http://cr.openjdk.java.net/~kkharbas/8202286/Design%20for%20JEP%20JDK > >>>> - > >>>>> 8202286.pdf I would love to hear your feedback and suggestions. > >>>>> Once we get confidence in the design, I will work on the > implementation. > >>>> The design looks good to me. > >>>> If you are planning to change VirtualSpace at > >>>> share/memory/virtualspace.hpp, including it on the design document > >>>> would be helpful too. > >>>> > >>>> Probably folks involved in the previous discussions would say > >>>> whether the design well covers their concerns or not. > >>>> > >>>> Thanks, > >>>> Sangheon > >>>> > >>>> > >>>>> PS: > >>>>> 1. Vinay has transitioned to another team in Intel, so he won't be > >>>>> able to > >>>> continue on this jep. > >>>>> 2. If you feel this should be part of JEP discussion thread, we > >>>>> can take it > >>>> there. > >>>>> Thanks and regards, > >>>>> Kishor > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Thomas St?fe [mailto:thomas.stuefe at gmail.com] > >>>>>> Sent: Friday, June 22, 2018 9:25 AM > >>>>>> To: Thomas Schatzl > >>>>>> Cc: Awasthi, Vinay K ; Paul Su > >>>>>> ; hotspot-gc-dev at openjdk.java.net; Hotspot > >> dev > >>>>>> runtime ; Viswanathan, > >>>> Sandhya > >>>>>> ; Aundhe, Shirish > >>>>>> ; Kharbas, Kishor > >>>>>> > >>>>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > >>>>>> ReserveSpace.cpp changes are mostly eliminated/no collector > >>>>>> specific > >>>> code. > >>>>>> Hi Thomas, > >>>>>> > >>>>>> On Fri, Jun 22, 2018 at 4:44 PM, Thomas Schatzl > >>>>>> wrote: > >>>>>>> Hi Thomas, > >>>>>>> > >>>>>>> On Tue, 2018-06-19 at 13:40 +0200, Thomas St?fe wrote: > >>>>>>>> Hi Vinay, > >>>>>>>> > >>>>>>>> On Mon, Jun 18, 2018 at 6:47 PM, Awasthi, Vinay K > >>>>>>>> wrote: > >>>>>>>>> Hi Thomas, > >>>>>>>>> > >>>>>>>>> Os::commit_memory calls map_memory_to_file which is same > as > >>>>>>>>> os::reserve_memory. > >>>>>>>>> > >>>>>>>>> I am failing to see why os::reserve_memory can call > >>>>>>>>> map_memory_to_file (i.e. tie it to mmap) but commit_memory > >>>> can't... > >>>>>>>>> Before this patch, commit_memory never dealt with > >>>>>>>>> incrementally committing pages to device so there has to be a > >>>>>>>>> way to pass file descriptor and offset. Windows has no such > >>>>>>>>> capability to manage incremental commits. All other OSes do > >>>>>>>>> and that is why map_memory_to_file is used (which by the way > >>>>>>>>> also works on Windows). > >>>>>>>> AIX uses System V shared memory by default, which follows a > >>>>>>>> different allocation scheme (semantics more like Windows > >>>> VirtualAlloc... > >>>>>>>> calls). > >>>>>>>> > >>>>>>>> But my doubts are not limited to that one, see my earlier > >>>>>>>> replies and those of others. It really makes sense to step back > >>>>>>>> one step and discuss the JEP first. > >>>>>>>> > >>>>>>> There is already a discussion thread as David mentioned > >>>>>>> (http://mail.op > >>>>>>> enjdk.java.net/pipermail/hotspot-gc-dev/2018-May/022092.html) > >> that > >>>>>>> so far nobody answered to. > >>>>>>> > >>>>>> Ah, I thought he wanted to have the JEP discussed in the comments > >>>>>> section of the JEP itself. > >>>>>> > >>>>>>> I believe discussion about contents the JEP and the > >>>>>>> implementation should be separate. > >>>>>>> > >>>>>> makes sense. > >>>>>> > >>>>>>> So far what I gathered from the responses to the implementation, > >>>>>>> the proposed idea itself is not the issue here (allowing the use > >>>>>>> of NVDIMM memory for parts of the heap for allowing the use of > >>>>>>> larger heaps to improve overall performance; I am not saying > >>>>>>> that the text doesn't need a bit more work :) ), but rather how > >>>>>>> an implementation of this JEP should proceed. > >>>>>> I have no problem with adding NVDIMM support. I think it is a > >>>>>> cool > >>>> feature. > >>>>>> Also, the awkwardness off the memory management abstraction > layer > >>>>>> in hotspot has always been a sore point with me (I originally > >>>>>> implemented the AIX mm layer in os_aix.cpp, and those are painful > >>>>>> memories). So, I have a lot of sympathy for Vinays struggles. > >>>>>> Unfortunately not much time atm, but I will respond to your mail. > >>>>>> > >>>>>>> Let's discuss the non-implementation stuff in that thread. Vinay > >>>>>>> or I can repost the proposal email if you do not have it any > >>>>>>> more so that answers will be in-thread. > >>>>>>> > >>>>>> Okay, sounds good. > >>>>>> > >>>>>> Thanks, > >>>>>> Thomas > >>>>>> > >>>>>> (one of us should change his name to make this less confusing :-) > >>>>>> > >>>>>>> Thanks, > >>>>>>> Thomas > >>>>>>> From leo.korinth at oracle.com Wed Sep 5 12:27:49 2018 From: leo.korinth at oracle.com (Leo Korinth) Date: Wed, 5 Sep 2018 14:27:49 +0200 Subject: RFR: 8196341: Add JFR events for parallel phases of G1 Message-ID: <8c0e4d82-0c87-da99-328e-548b8d7c12dd@oracle.com> Hi, I have added JFR events for the parallel phases of G1. This change adds JFR events for most values of the enumeration GCParPhases. I have also added a test case (TestG1ParallelPhases.java). This is a rebase/rework of a previous post I did 31th of January that never got pushed. Enhancement: https://bugs.openjdk.java.net/browse/JDK-8196341 Webrev: http://cr.openjdk.java.net/~lkorinth/8196341/00/ Testing: - mach5 hs-tier1,hs-tier2,hs-tier3 - brief visual inspection of result in JMC - new test TestG1ParallelPhases.java (100 runs) Thanks, Leo From fairoz.matte at oracle.com Wed Sep 5 13:55:05 2018 From: fairoz.matte at oracle.com (Fairoz Matte) Date: Wed, 5 Sep 2018 06:55:05 -0700 (PDT) Subject: RFR: 8175375: MemoryPoolMXBean.getCollectionUsage() does not works with -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent Message-ID: <24a9924f-c6eb-4309-80e7-892ac43b8d8a@default> Hi, Please review this change. MemoryPoolMXBean.getCollectionUsage() does not works in G1 with "-XX:+ExplicitGCInvokesConcurrent", it always returns 0. In case of "+ExplicitGCInvokesConcurrent" call to heap usage collection doesn't honor full gc. With the change of 8175375, it will check for full GC request and set the heap usage collection accordingly. JBS issue - https://bugs.openjdk.java.net/browse/JDK-8175375 Webrev - http://cr.openjdk.java.net/~fmatte/8175375/webrev.00/ Testing: mach5 tier1-3 Thanks, Fairoz From thomas.schatzl at oracle.com Wed Sep 5 15:09:57 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 05 Sep 2018 17:09:57 +0200 Subject: RFR: 8175375: MemoryPoolMXBean.getCollectionUsage() does not works with -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent In-Reply-To: <24a9924f-c6eb-4309-80e7-892ac43b8d8a@default> References: <24a9924f-c6eb-4309-80e7-892ac43b8d8a@default> Message-ID: <43b0e838c33c31945ad0208f8026ce314a7c9475.camel@oracle.com> Hi Fairoz, On Wed, 2018-09-05 at 06:55 -0700, Fairoz Matte wrote: > Hi, > > Please review this change. > MemoryPoolMXBean.getCollectionUsage() does not works in G1 with "- > XX:+ExplicitGCInvokesConcurrent", it always returns 0. > In case of "+ExplicitGCInvokesConcurrent" call to heap usage > collection doesn't honor full gc. > With the change of 8175375, it will check for full GC request and > set the heap usage collection accordingly. > > JBS issue - https://bugs.openjdk.java.net/browse/JDK-8175375 > Webrev - http://cr.openjdk.java.net/~fmatte/8175375/webrev.00/ I think this change breaks actual accounting: the G1MonitoringScopes will immediately be destructed at the end of the if/else blocks, so they will not be measuring values correctly. I think you need to set the full_gc parameter of the scope depending on the gc cause instead. I am kind of surprised that gccause is GCCause::_java_lang_system_gc for this young collection, but that may be correct. Thanks, Thomas From thomas.schatzl at oracle.com Wed Sep 5 18:23:43 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 05 Sep 2018 20:23:43 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> Message-ID: Hi Rodrigo, On Tue, 2018-09-04 at 15:04 +0000, Rodrigo Bruno wrote: > Hi Thomas, > > > Did you ever notice an improvement in heap usage after the first > > such attempt? > > There should be none except for memory held by j.l.ref.References. > > > > What I want to get at, as a later improvement, we could think of > > just skipping those then (as except as mentioned for the > > j.l.ref.References issue there would be "no" difference). > > > > I didn't devise any experiments to measure memory usage. However, we > can obviously do that. > > The new version of the patch regarding > > http://openjdk.java.net/jeps/8204089 > > can be found at > > http://www.gsd.inesc-id.pt/~rbruno/webrev.zip > > This version already includes idle compaction (Full GC is optional) > and your previous comments as well. > > Let me know what you think. - small tidbit: the change did not update the type of the _last_heap_resize variable; the webrev does not compile as is because of the recent introduction to use -Wreorder. - the test crashes here with some assert during resizing (after concurrent marking) in our regression testing - it does not give any new errors though, so that is good at least. I will definitely need to look into this - this may be an existing issue with (in this case heap shrinking) :) - it would have been a lot nicer if the sleep_before_next_cycle() returned some integer/enum which can be acted upon outside of it. I.e. it is imho much more readable to separate control/decision making from acting on it compared to separating both in this case. The current sleep_before_next_cycle() is kind of hacky :) I will look into this as well and propose an alternative. This may take a day or two. Thanks, Thomas From rbruno at gsd.inesc-id.pt Wed Sep 5 23:52:41 2018 From: rbruno at gsd.inesc-id.pt (Rodrigo Bruno) Date: Wed, 5 Sep 2018 23:52:41 +0000 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> Message-ID: Hi Thomas, thank you for your fast reply! Thomas Schatzl escreveu no dia quarta, 5/09/2018 ?(s) 20:13: > Hi Rodrigo, > > On Tue, 2018-09-04 at 15:04 +0000, Rodrigo Bruno wrote: > > Hi Thomas, > > > > > Did you ever notice an improvement in heap usage after the first > > > such attempt? > > > There should be none except for memory held by j.l.ref.References. > > > > > > What I want to get at, as a later improvement, we could think of > > > just skipping those then (as except as mentioned for the > > > j.l.ref.References issue there would be "no" difference). > > > > > > > I didn't devise any experiments to measure memory usage. However, we > > can obviously do that. > > > > The new version of the patch regarding > > > > http://openjdk.java.net/jeps/8204089 > > > > can be found at > > > > http://www.gsd.inesc-id.pt/~rbruno/webrev.zip > > > > This version already includes idle compaction (Full GC is optional) > > and your previous comments as well. > > > > Let me know what you think. > > - small tidbit: the change did not update the type of the > _last_heap_resize variable; the webrev does not compile as is because > of the recent introduction to use -Wreorder. > > Oh... I will fix that and send a new version very soon. > - the test crashes here with some assert during resizing (after > concurrent marking) in our regression testing - it does not give any > new errors though, so that is good at least. I will definitely need to look into this - this may be an existing > issue with (in this case heap shrinking) :) > > Which test? I was running the new test in Linux x86_64 slowdebug build and not hitting any assert. How did you run it? Still regardind tests, it there a way to run the same test with multiple (different) command line arguments? > - it would have been a lot nicer if the sleep_before_next_cycle() > returned some integer/enum which can be acted upon outside of it. > I.e. it is imho much more readable to separate control/decision making > from acting on it compared to separating both in this case. The current > sleep_before_next_cycle() is kind of hacky :) > I will look into this as well and propose an alternative. This may take > a day or two. > Okay, we can definitely improve the current version of this method. Thanks, rodrigo > > Thanks, > Thomas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fairoz.matte at oracle.com Thu Sep 6 04:59:06 2018 From: fairoz.matte at oracle.com (Fairoz Matte) Date: Thu, 6 Sep 2018 04:59:06 +0000 (UTC) Subject: RFR: 8175375: MemoryPoolMXBean.getCollectionUsage() does not works with -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent In-Reply-To: <43b0e838c33c31945ad0208f8026ce314a7c9475.camel@oracle.com> References: <24a9924f-c6eb-4309-80e7-892ac43b8d8a@default> <43b0e838c33c31945ad0208f8026ce314a7c9475.camel@oracle.com> Message-ID: <41f5d2a6-8398-4d1b-b963-946c14ccad6f@default> Hi Thomas, Thanks for looking into this. > -----Original Message----- > From: Thomas Schatzl > Sent: Wednesday, September 05, 2018 8:40 PM > To: Fairoz Matte ; hotspot-gc- > dev at openjdk.java.net > Subject: Re: RFR: 8175375: MemoryPoolMXBean.getCollectionUsage() does > not works with -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent > > Hi Fairoz, > > On Wed, 2018-09-05 at 06:55 -0700, Fairoz Matte wrote: > > Hi, > > > > Please review this change. > > MemoryPoolMXBean.getCollectionUsage() does not works in G1 with "- > > XX:+ExplicitGCInvokesConcurrent", it always returns 0. > > In case of "+ExplicitGCInvokesConcurrent" call to heap usage > > collection doesn't honor full gc. > > With the change of 8175375, it will check for full GC request and set > > the heap usage collection accordingly. > > > > JBS issue - https://bugs.openjdk.java.net/browse/JDK-8175375 > > Webrev - http://cr.openjdk.java.net/~fmatte/8175375/webrev.00/ > > I think this change breaks actual accounting: the G1MonitoringScopes will > immediately be destructed at the end of the if/else blocks, so they will not > be measuring values correctly. > > I think you need to set the full_gc parameter of the scope depending on the > gc cause instead. This call is to only initialize G1MonitoringScopes, so when we call G1MonitoringSupport::update_sizes() it still in same scoping level as an active TraceMemoryManagerStats object. Just forgot to mention that, I have added new test case "TestTenuredGenPoolCollectionUsage.java", which gives expected results before and after the patch. But it is good idea to keep the scope depending on the gc cause. I have updated the patch accordingly. Please find the updated webrev - http://cr.openjdk.java.net/~fmatte/8175375/webrev.01/ Thanks, Fairoz > > I am kind of surprised that gccause is GCCause::_java_lang_system_gc for > this young collection, but that may be correct. > > Thanks, > Thomas > From thomas.schatzl at oracle.com Thu Sep 6 06:14:22 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 06 Sep 2018 08:14:22 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> Message-ID: Hi, On Wed, 2018-09-05 at 23:52 +0000, Rodrigo Bruno wrote: > Hi Thomas, > > thank you for your fast reply! > > Thomas Schatzl escreveu no dia quarta, > 5/09/2018 ?(s) 20:13: > > Hi Rodrigo, > > > > On Tue, 2018-09-04 at 15:04 +0000, Rodrigo Bruno wrote: > > > Hi Thomas, > > [...] > > > The new version of the patch regarding > > > > > > http://openjdk.java.net/jeps/8204089 > > > > > > can be found at > > > > > > http://www.gsd.inesc-id.pt/~rbruno/webrev.zip > > > > > > This version already includes idle compaction (Full GC is > > optional) > > > and your previous comments as well. > > > > > > Let me know what you think. > > [...] > > - the test crashes here with some assert during resizing (after > > concurrent marking) in our regression testing - it does not give > > any > > new errors though, so that is good at least. > > I will definitely need to look into this - this may be an existing > > issue with (in this case heap shrinking) :) > > > > Which test? I was running the new test in Linux x86_64 slowdebug > build and not hitting any assert. How did you run it? The new TestTimelyCompaction test. There were no particular additoinal options; however since the @run statement did not contain e.g. heap size specification the issue might have been resulting from the fact that we run on different machines than you. > > Still regardind tests, it there a way to run the same test with > multiple (different) command line arguments? You can use multiple @run statements in a single test. > > - it would have been a lot nicer if the sleep_before_next_cycle() > > returned some integer/enum which can be acted upon outside of it. > > I.e. it is imho much more readable to separate control/decision > > making from acting on it compared to separating both in this case. > > The current sleep_before_next_cycle() is kind of hacky :) > > I will look into this as well and propose an alternative. This may > > take a day or two. > > Okay, we can definitely improve the current version of this method. :) Thanks, Thomas From thomas.schatzl at oracle.com Thu Sep 6 14:01:29 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 06 Sep 2018 16:01:29 +0200 Subject: RFR: 8175375: MemoryPoolMXBean.getCollectionUsage() does not works with -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent In-Reply-To: <41f5d2a6-8398-4d1b-b963-946c14ccad6f@default> References: <24a9924f-c6eb-4309-80e7-892ac43b8d8a@default> <43b0e838c33c31945ad0208f8026ce314a7c9475.camel@oracle.com> <41f5d2a6-8398-4d1b-b963-946c14ccad6f@default> Message-ID: <57dcc2a4b3c2ed1dfd1c69ddad0091f82068baeb.camel@oracle.com> Hi, On Thu, 2018-09-06 at 04:59 +0000, Fairoz Matte wrote: > Hi Thomas, > > Thanks for looking into this. > > > > [...] > > > JBS issue - https://bugs.openjdk.java.net/browse/JDK-8175375 > > > Webrev - http://cr.openjdk.java.net/~fmatte/8175375/webrev.00/ > > > > I think this change breaks actual accounting: the > > G1MonitoringScopes will > > immediately be destructed at the end of the if/else blocks, so they > > will not > > be measuring values correctly. > > > > I think you need to set the full_gc parameter of the scope > > depending on the > > gc cause instead. > > This call is to only initialize G1MonitoringScopes, so when we call > G1MonitoringSupport::update_sizes() it still in same scoping level as > an active TraceMemoryManagerStats object. > Just forgot to mention that, I have added new test case > "TestTenuredGenPoolCollectionUsage.java", which gives expected > results before and after the patch. But the MonitoringScope also implicitly starts and ends the embedded TraceCollectorStats/TraceMemoryManagerStats, i.e. calls their destructors, which e.g. save some current time stamp. Which will be completely off compared to before. > But it is good idea to keep the scope depending on the gc cause. I > have updated the patch accordingly. > Please find the updated webrev - > http://cr.openjdk.java.net/~fmatte/8175375/webrev.01/ The fix is good, but looking at the CR I am not sure that this fix addresses the entire problem, or I believe the problem has not been understood rightly. In this case the user wanted to have the old pool MemoryMXBean updated with the concurrent start gc. I assume that person is moving from CMS noting this inconsistency. CMS seems to update the old pool for any concurrent start gc. So this change only fixes the case when calling system.gc(), not any other start of marking gc. (Probably the predicate collector_state()- >in_initial_mark_gc() is correct here?) I believe further investigation of what MXBeans/pool(s)/jstat counters CMS updates in any initial mark GC would be warranted to get expected behavior, and fix this appropriately. I personally do not have an opinion, as an concurrent start could be considered both part of a "young" and "old" gc. Actually I would be really happy if somebody wrote down the expectations of what MemoryMXBean, pools and the CollectionCounters (jstat?) should be affected when (in form of a unit test preferably). Because changing this "full_gc" parameter depending on type of young gc or gc cause of the G1MonitoringScope also has the subtle effect that instead of the "young" collection counters (G1MonitoringSupport::_incremental_collection_counters) instead of the "old" collection counters (G1MonitoringSupport::_full_collection_counters) are updated. Instead of always the former. Cc'ed phh and ysuenaga because they seem to be users of these :) Thanks, Thomas From thomas.schatzl at oracle.com Thu Sep 6 15:51:09 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 06 Sep 2018 17:51:09 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> Message-ID: <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> Hi again, On Wed, 2018-09-05 at 23:52 +0000, Rodrigo Bruno wrote: > Hi Thomas, > > thank you for your fast reply! > > Thomas Schatzl escreveu no dia quarta, > 5/09/2018 ?(s) 20:13: > > Hi Rodrigo, > > > > On Tue, 2018-09-04 at 15:04 +0000, Rodrigo Bruno wrote: > > > Hi Thomas, > > > > > > > [...] > > > > > > > > I didn't devise any experiments to measure memory usage. However, > > > we can obviously do that. That's not required. > > > > > > The new version of the patch regarding > > > > > > http://openjdk.java.net/jeps/8204089 > > > > > > can be found at > > > > > > http://www.gsd.inesc-id.pt/~rbruno/webrev.zip > > > > > > This version already includes idle compaction (Full GC is > > > optional) and your previous comments as well. > > > > > > Let me know what you think. > > > > - small tidbit: the change did not update the type of the > > _last_heap_resize variable; the webrev does not compile as is > > because of the recent introduction to use -Wreorder. > > > > Oh... I will fix that and send a new version very soon. Don't worry. When trying this myselves, I found that we are lacking some helper methods to the Ticks/Tickspan classes that I intend to introduce in a bit... so I kept it. > > - the test crashes here with some assert during resizing (after > > concurrent marking) in our regression testing - it does not give > > any new errors though, so that is good at least. > > I will definitely need to look into this - this may be an existing > > issue with (in this case heap shrinking) :) > > > > Which test? I was running the new test in Linux x86_64 slowdebug > build and not hitting any assert. How did you run it? > > Still regardind tests, it there a way to run the same test with > multiple (different) command line arguments? The problem is some assert in the resizing code that checks whether some cached value of amount of bytes used in the heap is equal to the actual bytes used. This does not match if there is some region we are currently allocating into, as we are only updating that cached value when that region is full for various regions. The resizing (resize_if_necessary_after_full_collection()) had the implict assumption that there is no such regions, so that cached value did not contain it. It triggers e.g. with your test when running with -Xcomp. > > > - it would have been a lot nicer if the sleep_before_next_cycle() > > returned some integer/enum which can be acted upon outside of it. > > I.e. it is imho much more readable to separate control/decision > > making from acting on it compared to separating both in this case. > > The current sleep_before_next_cycle() is kind of hacky :) > > I will look into this as well and propose an alternative. This may > > take a day or two. > > Okay, we can definitely improve the current version of this method. So there is a new version of the change at http://cr.openjdk.java.net/~ tschatzl/jelastic/pgc-webrev.2/ (full) and http://cr.openjdk.java.net/~ tschatzl/jelastic/pgc-webrev.1_to_2 (diff) that fixes a few bugs, and also makes that method nicer, actually reverting it to the former. Let me explain what I changed: - there is a race between execution of the VM operation to start the concurrent phase (calling the VM_G1CollectForAllocation) and the actual work done. I.e. actual work outside of G1ConcurrentMarkThread::sleep_before_next_cycle() could start before the actual VM operation started. Finally, it is possible (but really unlikely) that the work outside completes before returning to G1ConcurrentMarkThread::sleep_before_next_cycle(), where the CGC_lock is grabbed again. However the VM operation itself also needs to CGC_lock to notify G1ConcurrentMarkThread, deadlocking (at least some code I had had that problem :)) So the only way out here unfortunately is to have a separate thread doing idle detection and scheduling of the necessary VM operation. That thread is only started if idle detection is active. I think the code was sort-of aware of this issue by break'ing out of the while loop and that ominous "TODO" comment :) - the mentioned failing assert. - I think something bad would happen if the idle timeout triggered while an existing concurrent marking would be under way (which could be if you set the "load" threshold high enough). I did not actually try so I may be wrong, but nevertheless. - instead of the additional VM operation you can use Universe::collect() and then using the UseFullGCForIdleCompaction flag to schedule the correct VM operation (either full or concurrent start), and simply always resize the heap in the concurrent cycle (in this case in the Remark pause). There is a RFE for that out in the bug tracker, and we've been asked about this just recently. I will split it out from this webrev later. That removes a lot of code. - I added a second test case that tests UseFullGCForIdleCompaction to the junit test. The webrev I've uploaded is probably not complete, in fact I only just submitted it to our test system. At least the following work needs to be done: - move G1DetectIdleThread out of the g1COncurrentMarkThread files into separate ones. - naming of the flags: I think at least GCFrequency is too generic, and should be in ms too. - debug messages for the idle detection to see in detail which condition triggered and why. - the current implementation for the lazy idle thread initialization is probably wrong at the moment: GCFrequency that is used is manageable, i.e. the current implementation does not consider that. If you turn it on during runtime after it has initially been off, it does not start. Btw, this should be tested too. Probably there are some bugs still left, but please have a look. Would you mind looking into the issues above? Thanks, Thomas From hohensee at amazon.com Thu Sep 6 17:57:46 2018 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 6 Sep 2018 17:57:46 +0000 Subject: 8175375: MemoryPoolMXBean.getCollectionUsage() does not works with -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent Message-ID: <52B034F7-F78C-465B-A55D-BEC6CE219A1A@amazon.com> I don't think this is a bug. It makes perfect sense to me that the G1 Old Gen pool may not be updated by a concurrent cycle. If the heap is big enough (and your test doesn't set -Xmx, so it is), there will be no mixed collection, so nothing will be promoted to the old gen, so the G1 Old Gen pool will correctly read as empty. Run with GC logging enabled to see if that's what's happening. I suggest writing a test that uses the white box API and a small -Xmx to force a mixed collection (see test/hotspot/jtreg/gc/g1/mixedgc/TestOldGenCollectionUsage.java for an example). You should see that the G1 Old Gen pool is updated after a concurrent cycle if a mixed collection occurs. The fix for https://bugs.openjdk.java.net/browse/JDK-8195115 made the G1 Old Gen pool correctly reflect the result of a mixed collection. I'm in the process of modifying the patch for https://bugs.openjdk.java.net/browse/JDK-8196989 to fit with Thomas' recent G1 serviceability refactoring. It details the way things will work once the patch is done. Thanks, Paul ?On 9/6/18, 7:02 AM, "Thomas Schatzl" wrote: Hi, On Thu, 2018-09-06 at 04:59 +0000, Fairoz Matte wrote: > Hi Thomas, > > Thanks for looking into this. > > > > [...] > > > JBS issue - https://bugs.openjdk.java.net/browse/JDK-8175375 > > > Webrev - http://cr.openjdk.java.net/~fmatte/8175375/webrev.00/ > > > > I think this change breaks actual accounting: the > > G1MonitoringScopes will > > immediately be destructed at the end of the if/else blocks, so they > > will not > > be measuring values correctly. > > > > I think you need to set the full_gc parameter of the scope > > depending on the > > gc cause instead. > > This call is to only initialize G1MonitoringScopes, so when we call > G1MonitoringSupport::update_sizes() it still in same scoping level as > an active TraceMemoryManagerStats object. > Just forgot to mention that, I have added new test case > "TestTenuredGenPoolCollectionUsage.java", which gives expected > results before and after the patch. But the MonitoringScope also implicitly starts and ends the embedded TraceCollectorStats/TraceMemoryManagerStats, i.e. calls their destructors, which e.g. save some current time stamp. Which will be completely off compared to before. > But it is good idea to keep the scope depending on the gc cause. I > have updated the patch accordingly. > Please find the updated webrev - > http://cr.openjdk.java.net/~fmatte/8175375/webrev.01/ The fix is good, but looking at the CR I am not sure that this fix addresses the entire problem, or I believe the problem has not been understood rightly. In this case the user wanted to have the old pool MemoryMXBean updated with the concurrent start gc. I assume that person is moving from CMS noting this inconsistency. CMS seems to update the old pool for any concurrent start gc. So this change only fixes the case when calling system.gc(), not any other start of marking gc. (Probably the predicate collector_state()- >in_initial_mark_gc() is correct here?) I believe further investigation of what MXBeans/pool(s)/jstat counters CMS updates in any initial mark GC would be warranted to get expected behavior, and fix this appropriately. I personally do not have an opinion, as an concurrent start could be considered both part of a "young" and "old" gc. Actually I would be really happy if somebody wrote down the expectations of what MemoryMXBean, pools and the CollectionCounters (jstat?) should be affected when (in form of a unit test preferably). Because changing this "full_gc" parameter depending on type of young gc or gc cause of the G1MonitoringScope also has the subtle effect that instead of the "young" collection counters (G1MonitoringSupport::_incremental_collection_counters) instead of the "old" collection counters (G1MonitoringSupport::_full_collection_counters) are updated. Instead of always the former. Cc'ed phh and ysuenaga because they seem to be users of these :) Thanks, Thomas From hohensee at amazon.com Thu Sep 6 18:05:12 2018 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 6 Sep 2018 18:05:12 +0000 Subject: 8175375: MemoryPoolMXBean.getCollectionUsage() does not works with -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent In-Reply-To: <52B034F7-F78C-465B-A55D-BEC6CE219A1A@amazon.com> References: <52B034F7-F78C-465B-A55D-BEC6CE219A1A@amazon.com> Message-ID: <37E08E63-46DD-4728-B966-3F64B9B1381F@amazon.com> Also, forgot to note that there's a jstat counter for concurrent collection cycles, see in g1MonitoringSupport.hpp "CollectorCounters* _conc_collection_counters;" and associated code. ?On 9/6/18, 10:59 AM, "hotspot-gc-dev on behalf of Hohensee, Paul" wrote: I don't think this is a bug. It makes perfect sense to me that the G1 Old Gen pool may not be updated by a concurrent cycle. If the heap is big enough (and your test doesn't set -Xmx, so it is), there will be no mixed collection, so nothing will be promoted to the old gen, so the G1 Old Gen pool will correctly read as empty. Run with GC logging enabled to see if that's what's happening. I suggest writing a test that uses the white box API and a small -Xmx to force a mixed collection (see test/hotspot/jtreg/gc/g1/mixedgc/TestOldGenCollectionUsage.java for an example). You should see that the G1 Old Gen pool is updated after a concurrent cycle if a mixed collection occurs. The fix for https://bugs.openjdk.java.net/browse/JDK-8195115 made the G1 Old Gen pool correctly reflect the result of a mixed collection. I'm in the process of modifying the patch for https://bugs.openjdk.java.net/browse/JDK-8196989 to fit with Thomas' recent G1 serviceability refactoring. It details the way things will work once the patch is done. Thanks, Paul On 9/6/18, 7:02 AM, "Thomas Schatzl" wrote: Hi, On Thu, 2018-09-06 at 04:59 +0000, Fairoz Matte wrote: > Hi Thomas, > > Thanks for looking into this. > > > > [...] > > > JBS issue - https://bugs.openjdk.java.net/browse/JDK-8175375 > > > Webrev - http://cr.openjdk.java.net/~fmatte/8175375/webrev.00/ > > > > I think this change breaks actual accounting: the > > G1MonitoringScopes will > > immediately be destructed at the end of the if/else blocks, so they > > will not > > be measuring values correctly. > > > > I think you need to set the full_gc parameter of the scope > > depending on the > > gc cause instead. > > This call is to only initialize G1MonitoringScopes, so when we call > G1MonitoringSupport::update_sizes() it still in same scoping level as > an active TraceMemoryManagerStats object. > Just forgot to mention that, I have added new test case > "TestTenuredGenPoolCollectionUsage.java", which gives expected > results before and after the patch. But the MonitoringScope also implicitly starts and ends the embedded TraceCollectorStats/TraceMemoryManagerStats, i.e. calls their destructors, which e.g. save some current time stamp. Which will be completely off compared to before. > But it is good idea to keep the scope depending on the gc cause. I > have updated the patch accordingly. > Please find the updated webrev - > http://cr.openjdk.java.net/~fmatte/8175375/webrev.01/ The fix is good, but looking at the CR I am not sure that this fix addresses the entire problem, or I believe the problem has not been understood rightly. In this case the user wanted to have the old pool MemoryMXBean updated with the concurrent start gc. I assume that person is moving from CMS noting this inconsistency. CMS seems to update the old pool for any concurrent start gc. So this change only fixes the case when calling system.gc(), not any other start of marking gc. (Probably the predicate collector_state()- >in_initial_mark_gc() is correct here?) I believe further investigation of what MXBeans/pool(s)/jstat counters CMS updates in any initial mark GC would be warranted to get expected behavior, and fix this appropriately. I personally do not have an opinion, as an concurrent start could be considered both part of a "young" and "old" gc. Actually I would be really happy if somebody wrote down the expectations of what MemoryMXBean, pools and the CollectionCounters (jstat?) should be affected when (in form of a unit test preferably). Because changing this "full_gc" parameter depending on type of young gc or gc cause of the G1MonitoringScope also has the subtle effect that instead of the "young" collection counters (G1MonitoringSupport::_incremental_collection_counters) instead of the "old" collection counters (G1MonitoringSupport::_full_collection_counters) are updated. Instead of always the former. Cc'ed phh and ysuenaga because they seem to be users of these :) Thanks, Thomas From hohensee at amazon.com Thu Sep 6 18:33:20 2018 From: hohensee at amazon.com (Hohensee, Paul) Date: Thu, 6 Sep 2018 18:33:20 +0000 Subject: RFR (S): 8210265: Crash in HSpaceCounters::update_used() In-Reply-To: <8d86d7fefd5fef0f2ad7af3e24dcafdc42bf9d0c.camel@oracle.com> References: <8d86d7fefd5fef0f2ad7af3e24dcafdc42bf9d0c.camel@oracle.com> Message-ID: True. All the jstat tests expect two survivor spaces for each collector. Paul ?On 9/4/18, 1:31 AM, "hotspot-gc-dev on behalf of Thomas Schatzl" wrote: Hi, On Mon, 2018-09-03 at 21:15 -0400, Kim Barrett wrote: > > On Sep 3, 2018, at 6:56 AM, Thomas Schatzl > om> wrote: > > > > Hi all, > > > > can I have reviews for this change that fixes a crash when G1 is > > started with -XX:-UsePerfData? > > The reason has been that recent refactoring removed one "if > > (UsePerfData)" check too many. > > > > CR: > > https://bugs.openjdk.java.net/browse/JDK-8210265 > > Webrev: > > http://cr.openjdk.java.net/~tschatzl/8210265/webrev/ > > Testing: > > new test succeeds > > > > Thanks, > > Thomas > > Looks good. > thanks for your review. > However, I was wondering why we?re making these unusable > HSpaceCounters at all. the _from_space_counters is there for backwards compatibility only afaik. Thanks, Thomas From thomas.schatzl at oracle.com Thu Sep 6 18:46:06 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 06 Sep 2018 20:46:06 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> Message-ID: <65063e5f31702649e12b485b570ecab9ebd32d01.camel@oracle.com> Hi, some additional comment: The specific issue with having a new additional thread is its (stack) memory usage which will show up in memory usage benchmarks. We could get away without an additional thread if we changed the G1ConcurrentMarkThread to do some kind of (regular) polling instead of signalling. I.e. the G1ConcurrentMarkThread would poll a few status variables every now and then, with sleeps inbetween. Then it would be possible to start VM operations/GCs within that loop easily as both G1ConcurrentMarkThread and VM thread (which runs the VM operations) try to lock the same lock. This would also allow easy disable/enable of the idle-uncommit feature at runtime. Btw, if I am not mistaken the current implementation of the idle triggering might be a bit surprising: since the _last_heap_resized counter is not reset for every gc, there may be unforeseen situations where idle is detected while (infrequent) GCs are running. Is the current implementation in this way what you expected? Thanks, Thomas On Thu, 2018-09-06 at 17:51 +0200, Thomas Schatzl wrote: > Hi again, > > On Wed, 2018-09-05 at 23:52 +0000, Rodrigo Bruno wrote: > > Hi Thomas, > > > > thank you for your fast reply! > > > > Thomas Schatzl escreveu no dia quarta, > > 5/09/2018 ?(s) 20:13: > > > Hi Rodrigo, > > > > > > On Tue, 2018-09-04 at 15:04 +0000, Rodrigo Bruno wrote: > > > > Hi Thomas, > > > > > > > > > > [...] > > > > > > > > > > > I didn't devise any experiments to measure memory usage. > > > > However, > > > > we can obviously do that. > > That's not required. > > > > > > > > > The new version of the patch regarding > > > > > > > > http://openjdk.java.net/jeps/8204089 > > > > > > > > can be found at > > > > > > > > http://www.gsd.inesc-id.pt/~rbruno/webrev.zip > > > > > > > > This version already includes idle compaction (Full GC is > > > > optional) and your previous comments as well. > > > > > > > > Let me know what you think. > > > > > > - small tidbit: the change did not update the type of the > > > _last_heap_resize variable; the webrev does not compile as is > > > because of the recent introduction to use -Wreorder. > > > > > > > Oh... I will fix that and send a new version very soon. > > Don't worry. When trying this myselves, I found that we are lacking > some helper methods to the Ticks/Tickspan classes that I intend to > introduce in a bit... so I kept it. > > > > - the test crashes here with some assert during resizing (after > > > concurrent marking) in our regression testing - it does not give > > > any new errors though, so that is good at least. > > > I will definitely need to look into this - this may be an > > > existing > > > issue with (in this case heap shrinking) :) > > > > > > > Which test? I was running the new test in Linux x86_64 slowdebug > > build and not hitting any assert. How did you run it? > > > > Still regardind tests, it there a way to run the same test with > > multiple (different) command line arguments? > > The problem is some assert in the resizing code that checks whether > some cached value of amount of bytes used in the heap is equal to the > actual bytes used. > > This does not match if there is some region we are currently > allocating > into, as we are only updating that cached value when that region is > full for various regions. > > The resizing (resize_if_necessary_after_full_collection()) had the > implict assumption that there is no such regions, so that cached > value > did not contain it. > > It triggers e.g. with your test when running with -Xcomp. > > > > > > - it would have been a lot nicer if the sleep_before_next_cycle() > > > returned some integer/enum which can be acted upon outside of it. > > > I.e. it is imho much more readable to separate control/decision > > > making from acting on it compared to separating both in this > > > case. > > > The current sleep_before_next_cycle() is kind of hacky :) > > > I will look into this as well and propose an alternative. This > > > may > > > take a day or two. > > > > Okay, we can definitely improve the current version of this method. > > So there is a new version of the change at http://cr.openjdk.java.net > /~ > tschatzl/jelastic/pgc-webrev.2/ (full) and http://cr.openjdk.java.net > /~ > tschatzl/jelastic/pgc-webrev.1_to_2 (diff) that fixes a few bugs, and > also makes that method nicer, actually reverting it to the former. > > Let me explain what I changed: > > - there is a race between execution of the VM operation to start the > concurrent phase (calling the VM_G1CollectForAllocation) and the > actual > work done. > I.e. actual work outside of > G1ConcurrentMarkThread::sleep_before_next_cycle() could start before > the actual VM operation started. > > Finally, it is possible (but really unlikely) that the work outside > completes before returning to > G1ConcurrentMarkThread::sleep_before_next_cycle(), where the CGC_lock > is grabbed again. However the VM operation itself also needs to > CGC_lock to notify G1ConcurrentMarkThread, deadlocking (at least some > code I had had that problem :)) > > So the only way out here unfortunately is to have a separate thread > doing idle detection and scheduling of the necessary VM operation. > That > thread is only started if idle detection is active. > > I think the code was sort-of aware of this issue by break'ing out of > the while loop and that ominous "TODO" comment :) > > - the mentioned failing assert. > > - I think something bad would happen if the idle timeout triggered > while an existing concurrent marking would be under way (which could > be > if you set the "load" threshold high enough). I did not actually try > so > I may be wrong, but nevertheless. > > - instead of the additional VM operation you can use > Universe::collect() and then using the UseFullGCForIdleCompaction > flag > to schedule the correct VM operation (either full or concurrent > start), > and simply always resize the heap in the concurrent cycle (in this > case > in the Remark pause). There is a RFE for that out in the bug tracker, > and we've been asked about this just recently. I will split it out > from > this webrev later. > > That removes a lot of code. > > - I added a second test case that tests UseFullGCForIdleCompaction to > the junit test. > > The webrev I've uploaded is probably not complete, in fact I only > just > submitted it to our test system. > > At least the following work needs to be done: > > - move G1DetectIdleThread out of the g1COncurrentMarkThread files > into > separate ones. > > - naming of the flags: I think at least GCFrequency is too generic, > and > should be in ms too. > > - debug messages for the idle detection to see in detail which > condition triggered and why. > > - the current implementation for the lazy idle thread initialization > is > probably wrong at the moment: GCFrequency that is used is manageable, > i.e. the current implementation does not consider that. If you turn > it > on during runtime after it has initially been off, it does not start. > Btw, this should be tested too. > > Probably there are some bugs still left, but please have a look. > > Would you mind looking into the issues above? > > Thanks, > Thomas > From yasuenag at gmail.com Fri Sep 7 01:40:18 2018 From: yasuenag at gmail.com (Yasumasa Suenaga) Date: Fri, 7 Sep 2018 10:40:18 +0900 Subject: RFR: 8175375: MemoryPoolMXBean.getCollectionUsage() does not works with -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent In-Reply-To: <57dcc2a4b3c2ed1dfd1c69ddad0091f82068baeb.camel@oracle.com> References: <24a9924f-c6eb-4309-80e7-892ac43b8d8a@default> <43b0e838c33c31945ad0208f8026ce314a7c9475.camel@oracle.com> <41f5d2a6-8398-4d1b-b963-946c14ccad6f@default> <57dcc2a4b3c2ed1dfd1c69ddad0091f82068baeb.camel@oracle.com> Message-ID: Hi, 2018?9?6?(?) 23:01 Thomas Schatzl : > > Hi, > > On Thu, 2018-09-06 at 04:59 +0000, Fairoz Matte wrote: > > Hi Thomas, > > > > Thanks for looking into this. > > > > > > [...] > > > > JBS issue - https://bugs.openjdk.java.net/browse/JDK-8175375 > > > > Webrev - http://cr.openjdk.java.net/~fmatte/8175375/webrev.00/ > > > > > > I think this change breaks actual accounting: the > > > G1MonitoringScopes will > > > immediately be destructed at the end of the if/else blocks, so they > > > will not > > > be measuring values correctly. > > > > > > I think you need to set the full_gc parameter of the scope > > > depending on the > > > gc cause instead. > > > > This call is to only initialize G1MonitoringScopes, so when we call > > G1MonitoringSupport::update_sizes() it still in same scoping level as > > an active TraceMemoryManagerStats object. > > Just forgot to mention that, I have added new test case > > "TestTenuredGenPoolCollectionUsage.java", which gives expected > > results before and after the patch. > > But the MonitoringScope also implicitly starts and ends the embedded > TraceCollectorStats/TraceMemoryManagerStats, i.e. calls their > destructors, which e.g. save some current time stamp. Which will be > completely off compared to before. > > > But it is good idea to keep the scope depending on the gc cause. I > > have updated the patch accordingly. > > Please find the updated webrev - > > http://cr.openjdk.java.net/~fmatte/8175375/webrev.01/ > > The fix is good, but looking at the CR I am not sure that this fix > addresses the entire problem, or I believe the problem has not been > understood rightly. IMHO we need to treat the following gc causes as same as _java_lang_system_gc: _full_gc_alot _jvmti_force_gc _heap_inspection _heap_dump _wb_full_gc _metadata_GC_threshold _metadata_GC_clear_soft_refs _dcmd_gc_run > In this case the user wanted to have the old pool MemoryMXBean updated > with the concurrent start gc. I assume that person is moving from CMS > noting this inconsistency. CMS seems to update the old pool for any > concurrent start gc. > > So this change only fixes the case when calling system.gc(), not any > other start of marking gc. (Probably the predicate collector_state()- > >in_initial_mark_gc() is correct here?) > > I believe further investigation of what MXBeans/pool(s)/jstat counters > CMS updates in any initial mark GC would be warranted to get expected > behavior, and fix this appropriately. > > I personally do not have an opinion, as an concurrent start could be > considered both part of a "young" and "old" gc. Actually I would be > really happy if somebody wrote down the expectations of what > MemoryMXBean, pools and the CollectionCounters (jstat?) should be > affected when (in form of a unit test preferably). > > Because changing this "full_gc" parameter depending on type of young gc > or gc cause of the G1MonitoringScope also has the subtle effect that > instead of the "young" collection counters > (G1MonitoringSupport::_incremental_collection_counters) instead of the > "old" collection counters > (G1MonitoringSupport::_full_collection_counters) are updated. Instead > of always the former. > > Cc'ed phh and ysuenaga because they seem to be users of these :) I added _cgc_counters to record STW phases in concurrent GC in JDK-8153333. I think this is not a bug because Full GC is invoked as concurrent GC if you pass -XX:+ExplicitGCInvokesConcurrent to commandline arguments. So it should be recorded in _cgc_counters. In case of CMS, it has two STW phase: Initial-Mark and Remark. But they just mark live objects only. So usage is not be changed. In case of G1, Remark and Cleanup are STW phases, they are also recorded in _cgc_counters. Thanks, Yasumasa > Thanks, > Thomas > From fairoz.matte at oracle.com Fri Sep 7 04:11:53 2018 From: fairoz.matte at oracle.com (Fairoz Matte) Date: Thu, 6 Sep 2018 21:11:53 -0700 (PDT) Subject: RFR: 8175375: MemoryPoolMXBean.getCollectionUsage() does not works with -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent In-Reply-To: References: <24a9924f-c6eb-4309-80e7-892ac43b8d8a@default> <43b0e838c33c31945ad0208f8026ce314a7c9475.camel@oracle.com> <41f5d2a6-8398-4d1b-b963-946c14ccad6f@default> <57dcc2a4b3c2ed1dfd1c69ddad0091f82068baeb.camel@oracle.com> Message-ID: <871fe981-41d8-4ab5-90d9-9486ca2d9bec@default> Hi Paul, Yasumasa and Thomas, By now I am convinced that it is not an issue. Thanks for providing detailed analysis on this. Closing 8175375 as not an issue. Thanks, Fairoz > -----Original Message----- > From: Yasumasa Suenaga > Sent: Friday, September 07, 2018 7:10 AM > To: Thomas Schatzl > Cc: Fairoz Matte ; hotspot-gc-dev dev at openjdk.java.net>; hohensee at amazon.com > Subject: Re: RFR: 8175375: MemoryPoolMXBean.getCollectionUsage() does > not works with -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent > > Hi, > > 2018?9?6?(?) 23:01 Thomas Schatzl : > > > > Hi, > > > > On Thu, 2018-09-06 at 04:59 +0000, Fairoz Matte wrote: > > > Hi Thomas, > > > > > > Thanks for looking into this. > > > > > > > > [...] > > > > > JBS issue - https://bugs.openjdk.java.net/browse/JDK-8175375 > > > > > Webrev - http://cr.openjdk.java.net/~fmatte/8175375/webrev.00/ > > > > > > > > I think this change breaks actual accounting: the > > > > G1MonitoringScopes will immediately be destructed at the end of > > > > the if/else blocks, so they will not be measuring values > > > > correctly. > > > > > > > > I think you need to set the full_gc parameter of the scope > > > > depending on the gc cause instead. > > > > > > This call is to only initialize G1MonitoringScopes, so when we call > > > G1MonitoringSupport::update_sizes() it still in same scoping level > > > as an active TraceMemoryManagerStats object. > > > Just forgot to mention that, I have added new test case > > > "TestTenuredGenPoolCollectionUsage.java", which gives expected > > > results before and after the patch. > > > > But the MonitoringScope also implicitly starts and ends the embedded > > TraceCollectorStats/TraceMemoryManagerStats, i.e. calls their > > destructors, which e.g. save some current time stamp. Which will be > > completely off compared to before. > > > > > But it is good idea to keep the scope depending on the gc cause. I > > > have updated the patch accordingly. > > > Please find the updated webrev - > > > http://cr.openjdk.java.net/~fmatte/8175375/webrev.01/ > > > > The fix is good, but looking at the CR I am not sure that this fix > > addresses the entire problem, or I believe the problem has not been > > understood rightly. > > IMHO we need to treat the following gc causes as same as > _java_lang_system_gc: > > _full_gc_alot > _jvmti_force_gc > _heap_inspection > _heap_dump > _wb_full_gc > _metadata_GC_threshold > _metadata_GC_clear_soft_refs > _dcmd_gc_run > > > In this case the user wanted to have the old pool MemoryMXBean updated > > with the concurrent start gc. I assume that person is moving from CMS > > noting this inconsistency. CMS seems to update the old pool for any > > concurrent start gc. > > > > So this change only fixes the case when calling system.gc(), not any > > other start of marking gc. (Probably the predicate collector_state()- > > >in_initial_mark_gc() is correct here?) > > > > I believe further investigation of what MXBeans/pool(s)/jstat counters > > CMS updates in any initial mark GC would be warranted to get expected > > behavior, and fix this appropriately. > > > > I personally do not have an opinion, as an concurrent start could be > > considered both part of a "young" and "old" gc. Actually I would be > > really happy if somebody wrote down the expectations of what > > MemoryMXBean, pools and the CollectionCounters (jstat?) should be > > affected when (in form of a unit test preferably). > > > > Because changing this "full_gc" parameter depending on type of young > > gc or gc cause of the G1MonitoringScope also has the subtle effect > > that instead of the "young" collection counters > > (G1MonitoringSupport::_incremental_collection_counters) instead of the > > "old" collection counters > > (G1MonitoringSupport::_full_collection_counters) are updated. Instead > > of always the former. > > > > Cc'ed phh and ysuenaga because they seem to be users of these :) > > I added _cgc_counters to record STW phases in concurrent GC in JDK- > 8153333. > I think this is not a bug because Full GC is invoked as concurrent GC if you > pass -XX:+ExplicitGCInvokesConcurrent to commandline arguments. > So it should be recorded in _cgc_counters. > > In case of CMS, it has two STW phase: Initial-Mark and Remark. But they just > mark live objects only. So usage is not be changed. > In case of G1, Remark and Cleanup are STW phases, they are also recorded in > _cgc_counters. > > > Thanks, > > Yasumasa > > > > Thanks, > > Thomas > > From thomas.schatzl at oracle.com Fri Sep 7 08:10:40 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 07 Sep 2018 10:10:40 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: <65063e5f31702649e12b485b570ecab9ebd32d01.camel@oracle.com> References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> <65063e5f31702649e12b485b570ecab9ebd32d01.camel@oracle.com> Message-ID: <66c536399b2f218af0d85bd3bced44f204e3dbf4.camel@oracle.com> Hi again, On Thu, 2018-09-06 at 20:46 +0200, Thomas Schatzl wrote: > Hi, > > some additional comment: > > The specific issue with having a new additional thread is its (stack) > memory usage which will show up in memory usage benchmarks. > > We could get away without an additional thread if we changed the > G1ConcurrentMarkThread to do some kind of (regular) polling instead > of signalling. > > I.e. the G1ConcurrentMarkThread would poll a few status variables > every now and then, with sleeps inbetween. > > Then it would be possible to start VM operations/GCs within that loop > easily as both G1ConcurrentMarkThread and VM thread (which runs the > VM operations) try to lock the same lock. > > This would also allow easy disable/enable of the idle-uncommit > feature at runtime. > > Btw, if I am not mistaken the current implementation of the idle > triggering might be a bit surprising: since the _last_heap_resized > counter is not reset for every gc, there may be unforeseen situations > where idle is detected while (infrequent) GCs are running. > > Is the current implementation in this way what you expected? overnight I thought a bit of the implementation, and given the problem with heap usage of the new thread, and the requirement of being able to turn on/off that feature by a managed variable, the best change would probably reusing the service thread as you did in the initial change. Exposing/using another callback to the collector would be least bad; however a call to some method to the CollectedHeap (or G1CollectedHeap) should be the only thing that should be in the service thread imho. Rodrigo, would that be okay for you, and could you take that over (everything else like putting the functionality in its own class and files, and making the decisions visible in the log, stays the same)? Sorry for changing my mind again, but yesterday I just wanted to get something out for you to look at even with that issue unresolved. :( Thanks, Thomas From per.liden at oracle.com Fri Sep 7 08:20:41 2018 From: per.liden at oracle.com (Per Liden) Date: Fri, 7 Sep 2018 10:20:41 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> Message-ID: <116f4e61-8a5b-30ed-565c-c3987c534369@oracle.com> Hi, On 09/06/2018 05:51 PM, Thomas Schatzl wrote: [...] > - naming of the flags: I think at least GCFrequency is too generic, and > should be in ms too. I'm also thinking we need to think through the flags names a bit. First question is if these should be prefixed with G1, or if we think they are general enough to be applicable to the other GC (or at least some of them). I'm leaning towards having a G1 prefix. The name "GCFrequency" suggests that the unit is Hz, but it's seconds, so something like "Interval" would probably be better. I'm also thinking that "UseFullGCForIdleCompaction" could be "IdleGCInvokesConcurrent" to better map with the exiting "ExplicitGCInvokesConcurrent". Here's a naming suggestion, just to kick of the discussion. GCFrequency -> G1IdleGCInterval UseFullGCForIdleCompaction -> G1IdleGCInvokesConcurrent MaxLoadGC -> G1IdleGCMaxLoad MaxOverCommittedMem -> G1IdleGCMaxUnused MinCommittedMem -> G1IdleGCMinCapacity Btw, the comments attached to these if-statements are not reflecting what they are actually doing. + // Check if the used memory is above a threshold. + if ((MinCommittedMem > 0) && + (Universe::heap()->capacity() < MinCommittedMem)) { + return false; + } s/used memory/capacity/ + // Check if the difference between max capacity and current capacity is + // above a threshold. + if ((MaxOverCommittedMem > 0) && + (Universe::heap()->capacity() - Universe::heap()->used() < MaxOverCommittedMem)) { + return false; + } s/max capacity and current capacity/capacity and used/ cheers, Per From per.liden at oracle.com Fri Sep 7 08:46:06 2018 From: per.liden at oracle.com (Per Liden) Date: Fri, 7 Sep 2018 10:46:06 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: <66c536399b2f218af0d85bd3bced44f204e3dbf4.camel@oracle.com> References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> <65063e5f31702649e12b485b570ecab9ebd32d01.camel@oracle.com> <66c536399b2f218af0d85bd3bced44f204e3dbf4.camel@oracle.com> Message-ID: <2821cf70-c4d5-016f-b976-5ecc850f377c@oracle.com> Hi Thomas, On 09/07/2018 10:10 AM, Thomas Schatzl wrote: [...] > overnight I thought a bit of the implementation, and given the > problem with heap usage of the new thread, and the requirement of being > able to turn on/off that feature by a managed variable, the best change > would probably reusing the service thread as you did in the initial > change. I'm not convinced that this should be handled outside of G1. If there's a need to have the flag manageable at runtime (is that really the case?), you could just always start the G1DetectIdleThread and have it check the flag. I wouldn't worry too much about the memory overhead for the stack. cheers, Per From thomas.schatzl at oracle.com Fri Sep 7 13:34:20 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 07 Sep 2018 15:34:20 +0200 Subject: RFR (T): 8210467: Remove unused G1CollectedHeap::_max_heap_capacity Message-ID: Hi, can I have a review for this trivial change that removes an unused member of G1CollectedHeap? CR: https://bugs.openjdk.java.net/browse/JDK-8210467 Webrev: http://cr.openjdk.java.net/~tschatzl/8210467/webrev/ testing: local compilation Thanks, THomas From thomas.schatzl at oracle.com Fri Sep 7 13:38:36 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 07 Sep 2018 15:38:36 +0200 Subject: RFR (XXS): 8210463: Recalculate_used() always sets time taken in G1GCPhaseTimes Message-ID: <954dc4f7a9aec0f56238007b292871d91b4d8820.camel@oracle.com> Hi, can I have reviews for this refactoring of G1CollectedHeap::recalculate_used() to move out time measurement from this method to the only place where we intend to measure the time that operation takes? In the other cases we use recalculate_used() only in asserts. While it does not really hurt because the wrong values are never read from, it's still a matter of cleaning up the code imho. CR: https://bugs.openjdk.java.net/browse/JDK-8210463 Webrev: http://cr.openjdk.java.net/~tschatzl/8210463/webrev/ Testing: manual checking that time is still reported for the only case we care Thanks, Thomas From hohensee at amazon.com Fri Sep 7 13:52:55 2018 From: hohensee at amazon.com (Hohensee, Paul) Date: Fri, 7 Sep 2018 13:52:55 +0000 Subject: RFR (T): 8210467: Remove unused G1CollectedHeap::_max_heap_capacity In-Reply-To: References: Message-ID: <1111B0F1-8357-441C-AFB0-5CB991C63B70@amazon.com> Lgtm. Paul ?On 9/7/18, 6:36 AM, "hotspot-gc-dev on behalf of Thomas Schatzl" wrote: Hi, can I have a review for this trivial change that removes an unused member of G1CollectedHeap? CR: https://bugs.openjdk.java.net/browse/JDK-8210467 Webrev: http://cr.openjdk.java.net/~tschatzl/8210467/webrev/ testing: local compilation Thanks, THomas From stefan.johansson at oracle.com Fri Sep 7 13:54:07 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Fri, 7 Sep 2018 15:54:07 +0200 Subject: RFR (T): 8210467: Remove unused G1CollectedHeap::_max_heap_capacity In-Reply-To: References: Message-ID: <6c45de08-909f-6363-d5e6-ceeb573846b6@oracle.com> Hi Thomas, On 2018-09-07 15:34, Thomas Schatzl wrote: > Hi, > > can I have a review for this trivial change that removes an unused > member of G1CollectedHeap? > > CR: > https://bugs.openjdk.java.net/browse/JDK-8210467 > Webrev: > http://cr.openjdk.java.net/~tschatzl/8210467/webrev/ Looks good, nice cleanup. Stefan > testing: > local compilation > > Thanks, > THomas From stefan.johansson at oracle.com Fri Sep 7 13:56:52 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Fri, 7 Sep 2018 15:56:52 +0200 Subject: RFR (XXS): 8210463: Recalculate_used() always sets time taken in G1GCPhaseTimes In-Reply-To: <954dc4f7a9aec0f56238007b292871d91b4d8820.camel@oracle.com> References: <954dc4f7a9aec0f56238007b292871d91b4d8820.camel@oracle.com> Message-ID: Hi, On 2018-09-07 15:38, Thomas Schatzl wrote: > Hi, > > can I have reviews for this refactoring of > G1CollectedHeap::recalculate_used() to move out time measurement from > this method to the only place where we intend to measure the time that > operation takes? > In the other cases we use recalculate_used() only in asserts. > > While it does not really hurt because the wrong values are never read > from, it's still a matter of cleaning up the code imho. I agree, nice cleanup. > CR: > https://bugs.openjdk.java.net/browse/JDK-8210463 > Webrev: > http://cr.openjdk.java.net/~tschatzl/8210463/webrev/ Looks good! Thanks, Stefan > Testing: > manual checking that time is still reported for the only case we care > > Thanks, > Thomas > From hohensee at amazon.com Fri Sep 7 13:59:10 2018 From: hohensee at amazon.com (Hohensee, Paul) Date: Fri, 7 Sep 2018 13:59:10 +0000 Subject: RFR (XXS): 8210463: Recalculate_used() always sets time taken in G1GCPhaseTimes In-Reply-To: <954dc4f7a9aec0f56238007b292871d91b4d8820.camel@oracle.com> References: <954dc4f7a9aec0f56238007b292871d91b4d8820.camel@oracle.com> Message-ID: <10CD20EF-9E38-4272-A5BB-D82476AC4235@amazon.com> The only other uses of recalculate_used() that I see are in asserts, so this lgtm. Paul ?On 9/7/18, 6:40 AM, "hotspot-gc-dev on behalf of Thomas Schatzl" wrote: Hi, can I have reviews for this refactoring of G1CollectedHeap::recalculate_used() to move out time measurement from this method to the only place where we intend to measure the time that operation takes? In the other cases we use recalculate_used() only in asserts. While it does not really hurt because the wrong values are never read from, it's still a matter of cleaning up the code imho. CR: https://bugs.openjdk.java.net/browse/JDK-8210463 Webrev: http://cr.openjdk.java.net/~tschatzl/8210463/webrev/ Testing: manual checking that time is still reported for the only case we care Thanks, Thomas From rbruno at gsd.inesc-id.pt Fri Sep 7 15:37:54 2018 From: rbruno at gsd.inesc-id.pt (Rodrigo Bruno) Date: Fri, 7 Sep 2018 15:37:54 +0000 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: <2821cf70-c4d5-016f-b976-5ecc850f377c@oracle.com> References: <244535da19fb696e8e48345f72c29a8f7478432a.camel@oracle.com> <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> <65063e5f31702649e12b485b570ecab9ebd32d01.camel@oracle.com> <66c536399b2f218af0d85bd3bced44f204e3dbf4.camel@oracle.com> <2821cf70-c4d5-016f-b976-5ecc850f377c@oracle.com> Message-ID: Hi Per and Thomas, thank you for your comments. I think it is possible to implement this feature using the service thread or using a separate thread. I see some pros and cons of having a separate thread: Pros: - using the service thread exposes something that is G1 specific to the rest of the JVM. Thus, using a separate thread, hides this feature from the outsite. Cons: - Having a manageable timeout is a bit more tricky to implement in a separate/dedicated thread. We need to be able to handle switch on and off. It might require some variable pooling. - It requires some more memory. Regardless of the path taken, I can prepare a new version of the patch whenever we decide on this. cheers, rodrigo Per Liden escreveu no dia sexta, 7/09/2018 ?(s) 11:58: > Hi Thomas, > > On 09/07/2018 10:10 AM, Thomas Schatzl wrote: > [...] > > overnight I thought a bit of the implementation, and given the > > problem with heap usage of the new thread, and the requirement of being > > able to turn on/off that feature by a managed variable, the best change > > would probably reusing the service thread as you did in the initial > > change. > > I'm not convinced that this should be handled outside of G1. If there's > a need to have the flag manageable at runtime (is that really the > case?), you could just always start the G1DetectIdleThread and have it > check the flag. I wouldn't worry too much about the memory overhead for > the stack. > > cheers, > Per > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Tue Sep 11 12:59:03 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 11 Sep 2018 14:59:03 +0200 Subject: RFR: 8196341: Add JFR events for parallel phases of G1 In-Reply-To: <8c0e4d82-0c87-da99-328e-548b8d7c12dd@oracle.com> References: <8c0e4d82-0c87-da99-328e-548b8d7c12dd@oracle.com> Message-ID: <239f211ba88bfccff3d743956f534710a6c7f7d3.camel@oracle.com> Hi Leo, On Wed, 2018-09-05 at 14:27 +0200, Leo Korinth wrote: > Hi, > > I have added JFR events for the parallel phases of G1. > > This change adds JFR events for most values of the enumeration > GCParPhases. I have also added a test case > (TestG1ParallelPhases.java). > > This is a rebase/rework of a previous post I did 31th of January > that > never got pushed. > > Enhancement: > https://bugs.openjdk.java.net/browse/JDK-8196341 > > Webrev: > http://cr.openjdk.java.net/~lkorinth/8196341/00/ > > Testing: > - mach5 hs-tier1,hs-tier2,hs-tier3 > - brief visual inspection of result in JMC > - new test TestG1ParallelPhases.java (100 runs) looks good. We discussed this in private earlier, please consider implementing some way to wrap JFR events in scoped objects. That would make the code much simpler to read, and easier to add JFR events in the future too. However this seems out of scope for this particular change. Thanks, Thomas From leo.korinth at oracle.com Tue Sep 11 13:59:37 2018 From: leo.korinth at oracle.com (Leo Korinth) Date: Tue, 11 Sep 2018 15:59:37 +0200 Subject: RFR: 8196341: Add JFR events for parallel phases of G1 In-Reply-To: <239f211ba88bfccff3d743956f534710a6c7f7d3.camel@oracle.com> References: <8c0e4d82-0c87-da99-328e-548b8d7c12dd@oracle.com> <239f211ba88bfccff3d743956f534710a6c7f7d3.camel@oracle.com> Message-ID: On 11/09/18 14:59, Thomas Schatzl wrote: > Hi Leo, > > On Wed, 2018-09-05 at 14:27 +0200, Leo Korinth wrote: >> Hi, >> >> I have added JFR events for the parallel phases of G1. >> >> This change adds JFR events for most values of the enumeration >> GCParPhases. I have also added a test case >> (TestG1ParallelPhases.java). >> >> This is a rebase/rework of a previous post I did 31th of January >> that >> never got pushed. >> >> Enhancement: >> https://bugs.openjdk.java.net/browse/JDK-8196341 >> >> Webrev: >> http://cr.openjdk.java.net/~lkorinth/8196341/00/ >> >> Testing: >> - mach5 hs-tier1,hs-tier2,hs-tier3 >> - brief visual inspection of result in JMC >> - new test TestG1ParallelPhases.java (100 runs) > > looks good. > > We discussed this in private earlier, please consider implementing some > way to wrap JFR events in scoped objects. That would make the code much > simpler to read, and easier to add JFR events in the future too. Yes, I agree. In the future I will probably add constructors to all JFR events so that they can be directly assigned to a scoped object. The other way of adding commit() on destruction of the JFR events directly was not considered as easy to read when I talked to the JFR people. > However this seems out of scope for this particular change. > > Thanks, > Thomas > Thanks for your review Thomas. /Leo From kim.barrett at oracle.com Tue Sep 11 19:24:29 2018 From: kim.barrett at oracle.com (Kim Barrett) Date: Tue, 11 Sep 2018 15:24:29 -0400 Subject: RFR (S): 8209843: Optimize oop scan closure closures wrt to reference processing in G1 In-Reply-To: References: <74dab00bcc1885f04b04378a9120e854dabe5bfc.camel@oracle.com> <0734c7b2-31bf-c741-0efd-36729fceff6c@oracle.com> Message-ID: <5A73DCFA-17A2-4045-BB87-718E56E2AABA@oracle.com> > On Sep 4, 2018, at 6:35 AM, Thomas Schatzl wrote: > > http://cr.openjdk.java.net/~tschatzl/8209843/webrev.1/ (full only, > sorry, but the change is very small) > > Thanks, > Thomas I feel like there's a bit of a tangled mess in this area, with various potential inconsistencies that make the code harder to understand than it needs to be. That said, I don't think your change is making that any worse, and in some cases is making it better. So okay, looks good. From manc at google.com Wed Sep 12 03:32:57 2018 From: manc at google.com (Man Cao) Date: Tue, 11 Sep 2018 20:32:57 -0700 Subject: RFR (S) 8210562: Change the verbosity threshold of logging for [oopstorage, ref] Message-ID: Hi all, I noticed that excessive logging messages appear in JDK11 with -Xlog:ref*=debug, which is the documented replacement for -XX:+PrintReferenceGC . I have a fix for it. Could someone review it? Webrev: http://cr.openjdk.java.net/~jcbeyler/8210562/webrev.00/ Bug: https://bugs.openjdk.java.net/browse/JDK-8210562 It changed the log tag "ref" to "oops" in oopStorage.cpp, and improved and increased verbosity level. I find that "ref" is mainly used for logging about processing java.lang.ref.Reference, but oopStorage is for JNI handles. Currently "oops" tag seems unused in HotSpot. The change uses debug level for messages about OopStorage::Block and unusual events, and trace level for the common alloc/release of individual oops. Below are some test results in terms of lines of logging messages: With varying levels of verbosity for oopstorage*: $ java -Xlog:oopstorage*= -Xms500m -Xmx500m -jar dacapo-9.12-bach.jar tradesoap info debug trace without patch 107934 112990 112701 with patch 0 5425 112805 For -Xlog:ref*=debug: $ java -Xlog:ref*=debug -Xms500m -Xmx500m -jar dacapo-9.12-bach.jar tradesoap jdk10 1008 tip without patch 108394 tip with patch 1054 Alternative is to advise users to change PrintReferenceGC and -Xlog:ref*=debug to -Xlog:gc+ref*=debug, as Kim suggested in https://bugs.openjdk.java.net/browse/JDK-8210562. However, that possibly involves changing the official flag mapping documentation in JDK9 and JDK10 , and imposes extra work on users who have migrated old print flags to new UL flags from JDK8 to JDK9/JDK10 according to the official doc. Thanks, Man -------------- next part -------------- An HTML attachment was scrubbed... URL: From kim.barrett at oracle.com Wed Sep 12 13:00:05 2018 From: kim.barrett at oracle.com (Kim Barrett) Date: Wed, 12 Sep 2018 09:00:05 -0400 Subject: RFR (S) 8210562: Change the verbosity threshold of logging for [oopstorage,ref] In-Reply-To: References: Message-ID: <91BE44AF-9346-4BEF-809C-2B79756A06B8@oracle.com> > On Sep 11, 2018, at 11:32 PM, Man Cao wrote: > > Hi all, > > I noticed that excessive logging messages appear in JDK11 with -Xlog:ref*=debug, which is the documented replacement for -XX:+PrintReferenceGC. > I have a fix for it. Could someone review it? > Webrev: http://cr.openjdk.java.net/~jcbeyler/8210562/webrev.00/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8210562 > > It changed the log tag "ref" to "oops" in oopStorage.cpp, and improved and increased verbosity level. I find that "ref" is mainly used for logging about processing java.lang.ref.Reference, but oopStorage is for JNI handles. Currently "oops" tag seems unused in HotSpot. > The change uses debug level for messages about OopStorage::Block and unusual events, and trace level for the common alloc/release of individual oops. > > Below are some test results in terms of lines of logging messages: > With varying levels of verbosity for oopstorage*: > $ java -Xlog:oopstorage*= -Xms500m -Xmx500m -jar dacapo-9.12-bach.jar tradesoap > info debug trace > without patch 107934 112990 112701 > with patch 0 5425 112805 > > For -Xlog:ref*=debug: > $ java -Xlog:ref*=debug -Xms500m -Xmx500m -jar dacapo-9.12-bach.jar tradesoap > jdk10 1008 > tip without patch 108394 > tip with patch 1054 > > > Alternative is to advise users to change PrintReferenceGC and -Xlog:ref*=debug to -Xlog:gc+ref*=debug, as Kim suggested in https://bugs.openjdk.java.net/browse/JDK-8210562. However, that possibly involves changing the official flag mapping documentation in JDK9 and JDK10, and imposes extra work on users who have migrated old print flags to new UL flags from JDK8 to JDK9/JDK10 according to the official doc. > > Thanks, > Man As I noted in the bug, I think the problem here is the documented translation from the old -XX:+PrintReferenceGC to using UL. Rather than -Xlog:ref*=debug, it should be -Xlog:gc+ref*=debug. And then the output is entirely reasonable. So I don?t think this change is needed. OopStorage is used for a lot more than JNI handles; StringTable (weak) refs, class holder (weak) handles, JNI (weak and string) handles, possibly more coming. ?ref? seems like a reasonable tag there. Having to choose an otherwise unused tag to avoid problems would, to me, indicate a serious design flaw in UL. Fortunately, I don?t think it?s that bad, just that the documented usage is wrong. Reducing the levels for the logging in OopStorage might be reasonable, but I think that?s a separate issue. From manc at google.com Wed Sep 12 18:27:23 2018 From: manc at google.com (Man Cao) Date: Wed, 12 Sep 2018 11:27:23 -0700 Subject: RFR (S) 8210562: Change the verbosity threshold of logging for [oopstorage, ref] In-Reply-To: <91BE44AF-9346-4BEF-809C-2B79756A06B8@oracle.com> References: <91BE44AF-9346-4BEF-809C-2B79756A06B8@oracle.com> Message-ID: Thanks for the response, Kim! Replies below. > I think the problem here is the documented translation from the old -XX:+PrintReferenceGC to using UL. Rather than -Xlog:ref*=debug, it should be -Xlog:gc+ref*=debug. Thomas also agreed on fixing the documentation instead in the bug. I'm not familiar with the process of changing documentation. Should we also change JDK9 and JDK10's documentation, or only JDK11's documentation? Since this whole issue is about to UL and consistent behavior with old flag, should I also CC serviceability-dev at openjdk.java.net on this thread? > OopStorage is used for a lot more than JNI handles; StringTable (weak) refs, class holder (weak) handles, JNI (weak and string) handles, possibly more coming. Thanks for the correction. I wasn't looking into every detail of OopStorage. I should have put "oopStorage is unrelated to processing java.lang.ref.Reference" instead. > Reducing the levels for the logging in OopStorage might be reasonable, but I think that?s a separate issue. I can revert the uses of "oops" back to "ref" and ask JC to create another webrev. Should we open a new RFE for that? I think JDK-8210562 might be OK for this purpose, as its title is about "Change the verbosity threshold" for logging. -Man On Wed, Sep 12, 2018 at 6:00 AM Kim Barrett wrote: > > On Sep 11, 2018, at 11:32 PM, Man Cao wrote: > > > > Hi all, > > > > I noticed that excessive logging messages appear in JDK11 with > -Xlog:ref*=debug, which is the documented replacement for > -XX:+PrintReferenceGC. > > I have a fix for it. Could someone review it? > > Webrev: http://cr.openjdk.java.net/~jcbeyler/8210562/webrev.00/ > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210562 > > > > It changed the log tag "ref" to "oops" in oopStorage.cpp, and improved > and increased verbosity level. I find that "ref" is mainly used for logging > about processing java.lang.ref.Reference, but oopStorage is for JNI > handles. Currently "oops" tag seems unused in HotSpot. > > The change uses debug level for messages about OopStorage::Block and > unusual events, and trace level for the common alloc/release of individual > oops. > > > > Below are some test results in terms of lines of logging messages: > > With varying levels of verbosity for oopstorage*: > > $ java -Xlog:oopstorage*= -Xms500m -Xmx500m -jar > dacapo-9.12-bach.jar tradesoap > > info debug trace > > without patch 107934 112990 112701 > > with patch 0 5425 112805 > > > > For -Xlog:ref*=debug: > > $ java -Xlog:ref*=debug -Xms500m -Xmx500m -jar dacapo-9.12-bach.jar > tradesoap > > jdk10 1008 > > tip without patch 108394 > > tip with patch 1054 > > > > > > Alternative is to advise users to change PrintReferenceGC and > -Xlog:ref*=debug to -Xlog:gc+ref*=debug, as Kim suggested in > https://bugs.openjdk.java.net/browse/JDK-8210562. However, that possibly > involves changing the official flag mapping documentation in JDK9 and > JDK10, and imposes extra work on users who have migrated old print flags to > new UL flags from JDK8 to JDK9/JDK10 according to the official doc. > > > > Thanks, > > Man > > As I noted in the bug, I think the problem here is the documented > translation from the old > -XX:+PrintReferenceGC to using UL. Rather than -Xlog:ref*=debug, it > should be > -Xlog:gc+ref*=debug. And then the output is entirely reasonable. > > So I don?t think this change is needed. > > OopStorage is used for a lot more than JNI handles; StringTable (weak) > refs, class > holder (weak) handles, JNI (weak and string) handles, possibly more coming. > ?ref? seems like a reasonable tag there. Having to choose an otherwise > unused > tag to avoid problems would, to me, indicate a serious design flaw in UL. > Fortunately, > I don?t think it?s that bad, just that the documented usage is wrong. > > Reducing the levels for the logging in OopStorage might be reasonable, but > I think > that?s a separate issue. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkennke at redhat.com Wed Sep 12 20:11:51 2018 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 12 Sep 2018 22:11:51 +0200 Subject: RFR: JDK-8210656: Object equals abstraction for BarrierSetC2 Message-ID: <07f491e5-fb06-86a3-f6f1-1b9d37544fb0@redhat.com> This introduces an abstraction to deal with object equality in BarrierSetC2. This is needed by GCs that can have different copies of same objects alive like Shenandoah. The approach chosen here is slightly different than we did in e.g. BarrierSetAssembler and the runtime Access API: instead of owning the whole equality, it only provides a resolve-like method to resolve the operands to stable values. The reason for doing this is that it's easier to do this way in intrinsics if those barriers are detached from the actual CmpP. This is because the barriers create new memory states, and we'd have to create memphis around those things, which is considerably more complex. I chose to add a new resolve_for_obj_equals(a, b) method instead of using two calls to resolve(a); resolve(b); because this allows for optimization: if any of a or b is known to be NULL, we can elide barriers for both. This is not possible to do with two independent resolve() calls. Bug: https://bugs.openjdk.java.net/browse/JDK-8210656 Webrev: http://cr.openjdk.java.net/~rkennke/JDK-8210656/webrev.00/ Testing: passes hotspot/jtreg:tier1 What do you think about this? Thanks, Roman -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From jiangli.zhou at oracle.com Thu Sep 13 00:20:24 2018 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Wed, 12 Sep 2018 17:20:24 -0700 Subject: RFR: 8210193: gc/g1/mixedgc/TestOldGenCollectionUsage.java fails intermittently with OutOfMemoryError in CDS mode In-Reply-To: References: Message-ID: Including the hotspot-gc mailing list. Also fixed a typo in the webrev. It was caught in my testing and also spotted by Ioi (thanks!). Thanks, Jiangli On 9/12/18 11:16 AM, Jiangli Zhou wrote: > Please review the test fix for OutOfMemoryError failure with > TestOldGenCollectionUsage.java in CDS mode. The fix uses 14M java heap > and also handles OutOfMemoryError to prevent similar issue in the > future. Please see detailed analysis in the bug report. I also applied > the same OutOfMemoryError handling to TestLogging.java, as suggested > by Paul Hohensee. > > webrev: http://cr.openjdk.java.net/~jiangli/8210193/webrev.00/ > > bug: https://bugs.openjdk.java.net/browse/JDK-8210193 > > Tested with tier1 - tier3 for TestOldGenCollectionUsage.java > (retesting with TestLogging.java change). Also tested tier6 and tier7 > with the default CDS archive patch. > > Thanks, > > Jiangli > From erik.osterlund at oracle.com Thu Sep 13 10:43:21 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 13 Sep 2018 12:43:21 +0200 Subject: RFR: JDK-8210656: Object equals abstraction for BarrierSetC2 In-Reply-To: <07f491e5-fb06-86a3-f6f1-1b9d37544fb0@redhat.com> References: <07f491e5-fb06-86a3-f6f1-1b9d37544fb0@redhat.com> Message-ID: <5B9A3F49.4090905@oracle.com> Hi Roman, Interesting. So semantically, this is cmp(resolve(a), resolve(b)), but you have circumstances in which the barriers are unnecessary and can be elided. Any of them having null in their type is one reason, but I suppose there are surely other reasons as well (such as finding dominating write barriers). I see two different approaches for this barrier elision: 1) Elide it during parsing (as you propose) 2) Elide it during Optimize (which I think conceptually looks like a natural fit) I originally proposed a function on BarrierSetC2 that I think I called optimize_barriers() or something like that. The idea was to use this hook to let GC barrier code shave off pointless (not to be confused with useless) barriers that can be removed. Roland thought that this seemed too specific to ZGC to warrant a general API, and I agreed, because indeed only ZGC used this hook at the time. This is today ZBarrierSetC2::find_dominating_barriers which is called straight from Optimize. I wonder if it would make sense to re-instate that hook. Then you could use the existing resolve() barriers during parsing, and leave barrier elision tricks (null checks included, plus other tricks you might have up your sleeve) to Optimize. For example, you might be able to walk your list of barriers and disconnect these pointless barriers. What do you think? Thanks, /Erik On 2018-09-12 22:11, Roman Kennke wrote: > This introduces an abstraction to deal with object equality in > BarrierSetC2. This is needed by GCs that can have different copies of > same objects alive like Shenandoah. > > The approach chosen here is slightly different than we did in e.g. > BarrierSetAssembler and the runtime Access API: instead of owning the > whole equality, it only provides a resolve-like method to resolve the > operands to stable values. The reason for doing this is that it's easier > to do this way in intrinsics if those barriers are detached from the > actual CmpP. This is because the barriers create new memory states, and > we'd have to create memphis around those things, which is considerably > more complex. > > I chose to add a new resolve_for_obj_equals(a, b) method instead of > using two calls to resolve(a); resolve(b); because this allows for > optimization: if any of a or b is known to be NULL, we can elide > barriers for both. This is not possible to do with two independent > resolve() calls. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8210656 > Webrev: > http://cr.openjdk.java.net/~rkennke/JDK-8210656/webrev.00/ > > Testing: passes hotspot/jtreg:tier1 > > What do you think about this? > > Thanks, > Roman > From rkennke at redhat.com Thu Sep 13 10:51:29 2018 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 13 Sep 2018 12:51:29 +0200 Subject: RFR: JDK-8210656: Object equals abstraction for BarrierSetC2 In-Reply-To: <5B9A3F49.4090905@oracle.com> References: <07f491e5-fb06-86a3-f6f1-1b9d37544fb0@redhat.com> <5B9A3F49.4090905@oracle.com> Message-ID: <1cd9d4d1-5f18-15da-2c30-44effc1b8bf2@redhat.com> Hi Erik, > Interesting. So semantically, this is cmp(resolve(a), resolve(b)), but > you have circumstances in which the barriers are unnecessary and can be > elided. Any of them having null in their type is one reason, but I > suppose there are surely other reasons as well (such as finding > dominating write barriers). Yes. We can already handle reasons that relate to 'stand-alone' barriers (like dominating write-barriers and others). However, this one is different because it relates to the *combination* of the two operands. I.e. a property of operand A or B would affect barriers for both A *and* B. This seems tricky to do after parsing. I guess we could look at CmpP, check their operands for known-null, and elide the write-barriers then, but this also means we need to check if the write-barriers haven't found other uses in the meantime, etc). Overall, this seemed *much* more hassle, whereas during parsing it comes quite naturally. See our impl: https://paste.fedoraproject.org/paste/Hr~nKkm4HnZo3hmcw3Snnw Roland: how hard/feasible would it be to do something like Erik proposed? I.e. use the usual resolve() for obj-eq and elide barriers later? It might have additional advantage (not sure) to catch cases where type of an object becomes known-null later in the optimization process? Roman -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From per.liden at oracle.com Thu Sep 13 11:14:46 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 13 Sep 2018 13:14:46 +0200 Subject: RFR: 8210710: Rename ThreadLocalAllocBuffer::myThread() to thread() Message-ID: I propose that we rename ThreadLocalAllocBuffer::myThread() to ThreadLocalAllocBuffer::thread(), since we typically don't do camelCase naming in hotspot. I got annoyed by this when working on an unrelated TLAB change, so I broke this out into a separate patch. Bug: https://bugs.openjdk.java.net/browse/JDK-8210710 Webrev: http://cr.openjdk.java.net/~pliden/8210710/webrev.0 /Per From per.liden at oracle.com Thu Sep 13 11:14:50 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 13 Sep 2018 13:14:50 +0200 Subject: RFR: 8210711: Remove unused offset getters in ThreadLocalAllocBuffer Message-ID: The following offset getters in ThreadLocalAllocBuffer are unused and can be removed. They are left-overs from when we had FastTLABRefill support. I stumbled over these when working on an unrelated TLAB change, so I broke this out into a separate patch. static ByteSize size_offset(); static ByteSize refill_waste_limit_offset(); static ByteSize number_of_refills_offset(); static ByteSize fast_refill_waste_offset(); static ByteSize slow_allocations_offset(); Bug: https://bugs.openjdk.java.net/browse/JDK-8210711 Webrev: http://cr.openjdk.java.net/~pliden/8210711/webrev.0 /Per From rkennke at redhat.com Thu Sep 13 11:17:10 2018 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 13 Sep 2018 13:17:10 +0200 Subject: RFR: 8210710: Rename ThreadLocalAllocBuffer::myThread() to thread() In-Reply-To: References: Message-ID: <9e778d30-a9fa-8f2f-8e30-be4a8c6247ff@redhat.com> Looks good to me. Thanks! Roman > I propose that we rename ThreadLocalAllocBuffer::myThread() to > ThreadLocalAllocBuffer::thread(), since we typically don't do camelCase > naming in hotspot. I got annoyed by this when working on an unrelated > TLAB change, so I broke this out into a separate patch. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210710 > Webrev: http://cr.openjdk.java.net/~pliden/8210710/webrev.0 > > /Per -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rkennke at redhat.com Thu Sep 13 11:17:49 2018 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 13 Sep 2018 13:17:49 +0200 Subject: RFR: 8210711: Remove unused offset getters in ThreadLocalAllocBuffer In-Reply-To: References: Message-ID: Good by me. Thanks! Roman > The following offset getters in ThreadLocalAllocBuffer are unused and > can be removed. They are left-overs from when we had FastTLABRefill > support. I stumbled over these when working on an unrelated TLAB change, > so I broke this out into a separate patch. > > static ByteSize size_offset(); > static ByteSize refill_waste_limit_offset(); > static ByteSize number_of_refills_offset(); > static ByteSize fast_refill_waste_offset(); > static ByteSize slow_allocations_offset(); > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210711 > Webrev: http://cr.openjdk.java.net/~pliden/8210711/webrev.0 > > /Per -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From per.liden at oracle.com Thu Sep 13 11:23:13 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 13 Sep 2018 13:23:13 +0200 Subject: RFR: 8210711: Remove unused offset getters in ThreadLocalAllocBuffer In-Reply-To: References: Message-ID: <898a286b-5332-9efa-39c1-4f11951cdaad@oracle.com> Thanks for reviewing, Roman! /Per On 09/13/2018 01:17 PM, Roman Kennke wrote: > Good by me. Thanks! > Roman > > >> The following offset getters in ThreadLocalAllocBuffer are unused and >> can be removed. They are left-overs from when we had FastTLABRefill >> support. I stumbled over these when working on an unrelated TLAB change, >> so I broke this out into a separate patch. >> >> static ByteSize size_offset(); >> static ByteSize refill_waste_limit_offset(); >> static ByteSize number_of_refills_offset(); >> static ByteSize fast_refill_waste_offset(); >> static ByteSize slow_allocations_offset(); >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210711 >> Webrev: http://cr.openjdk.java.net/~pliden/8210711/webrev.0 >> >> /Per > > From per.liden at oracle.com Thu Sep 13 11:27:21 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 13 Sep 2018 13:27:21 +0200 Subject: RFR: 8210710: Rename ThreadLocalAllocBuffer::myThread() to thread() In-Reply-To: <9e778d30-a9fa-8f2f-8e30-be4a8c6247ff@redhat.com> References: <9e778d30-a9fa-8f2f-8e30-be4a8c6247ff@redhat.com> Message-ID: Thanks Roman! /Per On 09/13/2018 01:17 PM, Roman Kennke wrote: > Looks good to me. Thanks! > Roman > >> I propose that we rename ThreadLocalAllocBuffer::myThread() to >> ThreadLocalAllocBuffer::thread(), since we typically don't do camelCase >> naming in hotspot. I got annoyed by this when working on an unrelated >> TLAB change, so I broke this out into a separate patch. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210710 >> Webrev: http://cr.openjdk.java.net/~pliden/8210710/webrev.0 >> >> /Per > > From thomas.schatzl at oracle.com Thu Sep 13 11:35:35 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 13 Sep 2018 13:35:35 +0200 Subject: RFR (S) 8210562: Change the verbosity threshold of logging for [oopstorage, ref] In-Reply-To: References: <91BE44AF-9346-4BEF-809C-2B79756A06B8@oracle.com> Message-ID: Hi Man, On Wed, 2018-09-12 at 11:27 -0700, Man Cao wrote: > Thanks for the response, Kim! Replies below. > > > I think the problem here is the documented translation from the old > > -XX:+PrintReferenceGC to using UL. Rather than -Xlog:ref*=debug, > > it should be -Xlog:gc+ref*=debug. > Thomas also agreed on fixing the documentation instead in the bug. > I'm not familiar with the process of changing documentation. Should > we also change JDK9 and JDK10's documentation, or only JDK11's > documentation? I created JDK-8210712 for the doc team. > Since this whole issue is about to UL and consistent behavior with > old flag, should I also CC serviceability-dev at openjdk.java.net on > this thread? I do not think this is needed for just a doc error. [...] > Should we open a new RFE for that? I think JDK-8210562 might be OK > for this purpose, as its title is about "Change the verbosity > threshold" for logging. If you change the old RFE, please put the old description in the comments, otherwise the link to JDK-8210712 will be confusing. Feel free to create a new RFE and close this issue maybe as duplicate as JDK-8210712 though. Thanks, Thomas From thomas.schatzl at oracle.com Thu Sep 13 11:37:15 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 13 Sep 2018 13:37:15 +0200 Subject: RFR: 8210711: Remove unused offset getters in ThreadLocalAllocBuffer In-Reply-To: References: Message-ID: <35225f573356aab16027b448d45b5d9dfd3d3bff.camel@oracle.com> Hi, On Thu, 2018-09-13 at 13:14 +0200, Per Liden wrote: > The following offset getters in ThreadLocalAllocBuffer are unused > and can be removed. They are left-overs from when we had > FastTLABRefill support. I stumbled over these when working on an > unrelated TLAB change, so I broke this out into a separate patch. > > static ByteSize size_offset(); > static ByteSize refill_waste_limit_offset(); > static ByteSize number_of_refills_offset(); > static ByteSize fast_refill_waste_offset(); > static ByteSize slow_allocations_offset(); > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210711 > Webrev: http://cr.openjdk.java.net/~pliden/8210711/webrev.0 > looks good to me. Thomas From thomas.schatzl at oracle.com Thu Sep 13 11:38:27 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 13 Sep 2018 13:38:27 +0200 Subject: RFR: 8210710: Rename ThreadLocalAllocBuffer::myThread() to thread() In-Reply-To: References: Message-ID: Hi Per, On Thu, 2018-09-13 at 13:14 +0200, Per Liden wrote: > I propose that we rename ThreadLocalAllocBuffer::myThread() to > ThreadLocalAllocBuffer::thread(), since we typically don't do > camelCase naming in hotspot. I got annoyed by this when working on an > unrelated TLAB change, so I broke this out into a separate patch. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210710 > Webrev: http://cr.openjdk.java.net/~pliden/8210710/webrev.0 > looks good. Thomas From per.liden at oracle.com Thu Sep 13 11:46:13 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 13 Sep 2018 13:46:13 +0200 Subject: RFR: 8210713: Let CollectedHeap::ensure_parsability() take care of TLAB statistics gathering Message-ID: A call to CollectedHeap::ensure_parsability(retire_tlabs), where retire_tlabs is true must always be preceded by a call to CollectedHeap::accumulate_statistics_all_tlabs(), otherwise we will loose TLAB statistics as it will be reset when the TLABs are retired. Instead of having all collectors do this dance and risk getting it wrong, I propose that we move this responsibility into ensure_parsability(), which will gather the TLAB statistics if it's told to retire all TLABs. Bug: https://bugs.openjdk.java.net/browse/JDK-8210713 Webrev: http://cr.openjdk.java.net/~pliden/8210713/webrev.0 /Per From per.liden at oracle.com Thu Sep 13 11:48:35 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 13 Sep 2018 13:48:35 +0200 Subject: RFR: 8210711: Remove unused offset getters in ThreadLocalAllocBuffer In-Reply-To: <35225f573356aab16027b448d45b5d9dfd3d3bff.camel@oracle.com> References: <35225f573356aab16027b448d45b5d9dfd3d3bff.camel@oracle.com> Message-ID: <74dbddca-ccb9-865e-ace9-b44e044aeac7@oracle.com> Thanks Thomas! /Per On 09/13/2018 01:37 PM, Thomas Schatzl wrote: > Hi, > > On Thu, 2018-09-13 at 13:14 +0200, Per Liden wrote: >> The following offset getters in ThreadLocalAllocBuffer are unused >> and can be removed. They are left-overs from when we had >> FastTLABRefill support. I stumbled over these when working on an >> unrelated TLAB change, so I broke this out into a separate patch. >> >> static ByteSize size_offset(); >> static ByteSize refill_waste_limit_offset(); >> static ByteSize number_of_refills_offset(); >> static ByteSize fast_refill_waste_offset(); >> static ByteSize slow_allocations_offset(); >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210711 >> Webrev: http://cr.openjdk.java.net/~pliden/8210711/webrev.0 >> > > looks good to me. > > Thomas > From per.liden at oracle.com Thu Sep 13 11:48:47 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 13 Sep 2018 13:48:47 +0200 Subject: RFR: 8210710: Rename ThreadLocalAllocBuffer::myThread() to thread() In-Reply-To: References: Message-ID: Thanks Thomas! /Per On 09/13/2018 01:38 PM, Thomas Schatzl wrote: > Hi Per, > > On Thu, 2018-09-13 at 13:14 +0200, Per Liden wrote: >> I propose that we rename ThreadLocalAllocBuffer::myThread() to >> ThreadLocalAllocBuffer::thread(), since we typically don't do >> camelCase naming in hotspot. I got annoyed by this when working on an >> unrelated TLAB change, so I broke this out into a separate patch. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210710 >> Webrev: http://cr.openjdk.java.net/~pliden/8210710/webrev.0 >> > > looks good. > > Thomas > From martin.doerr at sap.com Thu Sep 13 11:55:18 2018 From: martin.doerr at sap.com (Doerr, Martin) Date: Thu, 13 Sep 2018 11:55:18 +0000 Subject: RFR: 8210711: Remove unused offset getters in ThreadLocalAllocBuffer In-Reply-To: <898a286b-5332-9efa-39c1-4f11951cdaad@oracle.com> References: <898a286b-5332-9efa-39c1-4f11951cdaad@oracle.com> Message-ID: <1e6633cd1ffc4af6ba0c37c1550120b7@sap.com> Hi Per, looks good to me, too. Best regards, Martin -----Original Message----- From: hotspot-gc-dev On Behalf Of Per Liden Sent: Donnerstag, 13. September 2018 13:23 To: Roman Kennke ; hotspot-gc-dev Subject: Re: RFR: 8210711: Remove unused offset getters in ThreadLocalAllocBuffer Thanks for reviewing, Roman! /Per On 09/13/2018 01:17 PM, Roman Kennke wrote: > Good by me. Thanks! > Roman > > >> The following offset getters in ThreadLocalAllocBuffer are unused and >> can be removed. They are left-overs from when we had FastTLABRefill >> support. I stumbled over these when working on an unrelated TLAB change, >> so I broke this out into a separate patch. >> >> static ByteSize size_offset(); >> static ByteSize refill_waste_limit_offset(); >> static ByteSize number_of_refills_offset(); >> static ByteSize fast_refill_waste_offset(); >> static ByteSize slow_allocations_offset(); >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210711 >> Webrev: http://cr.openjdk.java.net/~pliden/8210711/webrev.0 >> >> /Per > > From per.liden at oracle.com Thu Sep 13 12:13:36 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 13 Sep 2018 14:13:36 +0200 Subject: RFR: 8210711: Remove unused offset getters in ThreadLocalAllocBuffer In-Reply-To: <1e6633cd1ffc4af6ba0c37c1550120b7@sap.com> References: <898a286b-5332-9efa-39c1-4f11951cdaad@oracle.com> <1e6633cd1ffc4af6ba0c37c1550120b7@sap.com> Message-ID: <2567c49f-f5da-12e1-b2f0-d00895871d51@oracle.com> Thanks Martin! /Per On 09/13/2018 01:55 PM, Doerr, Martin wrote: > Hi Per, > > looks good to me, too. > > Best regards, > Martin > > -----Original Message----- > From: hotspot-gc-dev On Behalf Of Per Liden > Sent: Donnerstag, 13. September 2018 13:23 > To: Roman Kennke ; hotspot-gc-dev > Subject: Re: RFR: 8210711: Remove unused offset getters in ThreadLocalAllocBuffer > > Thanks for reviewing, Roman! > > /Per > > On 09/13/2018 01:17 PM, Roman Kennke wrote: >> Good by me. Thanks! >> Roman >> >> >>> The following offset getters in ThreadLocalAllocBuffer are unused and >>> can be removed. They are left-overs from when we had FastTLABRefill >>> support. I stumbled over these when working on an unrelated TLAB change, >>> so I broke this out into a separate patch. >>> >>> static ByteSize size_offset(); >>> static ByteSize refill_waste_limit_offset(); >>> static ByteSize number_of_refills_offset(); >>> static ByteSize fast_refill_waste_offset(); >>> static ByteSize slow_allocations_offset(); >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8210711 >>> Webrev: http://cr.openjdk.java.net/~pliden/8210711/webrev.0 >>> >>> /Per >> >> From rkennke at redhat.com Thu Sep 13 12:15:21 2018 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 13 Sep 2018 14:15:21 +0200 Subject: RFR: JDK-8210656: Object equals abstraction for BarrierSetC2 In-Reply-To: <5B9A3F49.4090905@oracle.com> References: <07f491e5-fb06-86a3-f6f1-1b9d37544fb0@redhat.com> <5B9A3F49.4090905@oracle.com> Message-ID: <8b48dd82-7162-1d5a-527a-fc068a72c009@redhat.com> Hi Erik, I talked to Roland about this. It turns out that we already have this optimization pass, and could just as well live with cmp(resolve(a), resolve(b)). We need a little (shenandoah-specific-) hook in CmpPNode::Ideal() for that though (but we'd need that anyway I suppose). If you agree with that, I'll withdraw this RFR. Ok? Roman > Hi Roman, > > Interesting. So semantically, this is cmp(resolve(a), resolve(b)), but > you have circumstances in which the barriers are unnecessary and can be > elided. Any of them having null in their type is one reason, but I > suppose there are surely other reasons as well (such as finding > dominating write barriers). > > I see two different approaches for this barrier elision: > 1) Elide it during parsing (as you propose) > 2) Elide it during Optimize (which I think conceptually looks like a > natural fit) > > I originally proposed a function on BarrierSetC2 that I think I called > optimize_barriers() or something like that. The idea was to use this > hook to let GC barrier code shave off pointless (not to be confused with > useless) barriers that can be removed. Roland thought that this seemed > too specific to ZGC to warrant a general API, and I agreed, because > indeed only ZGC used this hook at the time. This is today > ZBarrierSetC2::find_dominating_barriers which is called straight from > Optimize. > > I wonder if it would make sense to re-instate that hook. Then you could > use the existing resolve() barriers during parsing, and leave barrier > elision tricks (null checks included, plus other tricks you might have > up your sleeve) to Optimize. For example, you might be able to walk your > list of barriers and disconnect these pointless barriers. What do you > think? > > Thanks, > /Erik > > On 2018-09-12 22:11, Roman Kennke wrote: >> This introduces an abstraction to deal with object equality in >> BarrierSetC2. This is needed by GCs that can have different copies of >> same objects alive like Shenandoah. >> >> The approach chosen here is slightly different than we did in e.g. >> BarrierSetAssembler and the runtime Access API: instead of owning the >> whole equality, it only provides a resolve-like method to resolve the >> operands to stable values. The reason for doing this is that it's easier >> to do this way in intrinsics if those barriers are detached from the >> actual CmpP. This is because the barriers create new memory states, and >> we'd have to create memphis around those things, which is considerably >> more complex. >> >> I chose to add a new resolve_for_obj_equals(a, b) method instead of >> using two calls to resolve(a); resolve(b); because this allows for >> optimization: if any of a or b is known to be NULL, we can elide >> barriers for both. This is not possible to do with two independent >> resolve() calls. >> >> Bug: >> https://bugs.openjdk.java.net/browse/JDK-8210656 >> Webrev: >> http://cr.openjdk.java.net/~rkennke/JDK-8210656/webrev.00/ >> >> Testing: passes hotspot/jtreg:tier1 >> >> What do you think about this? >> >> Thanks, >> Roman >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From stefan.johansson at oracle.com Thu Sep 13 12:30:37 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Thu, 13 Sep 2018 14:30:37 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: References: <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> <65063e5f31702649e12b485b570ecab9ebd32d01.camel@oracle.com> <66c536399b2f218af0d85bd3bced44f204e3dbf4.camel@oracle.com> <2821cf70-c4d5-016f-b976-5ecc850f377c@oracle.com> Message-ID: <57b10708-b870-cb98-0df0-fab650f431e4@oracle.com> Hi Rodrigo, Sorry for being a bit late into the discussion. We've had some internal discussions and realized that there are some questions that I need to bring up here. I'm trying to better understand under what circumstances this feature is to be used and how a user should use the different flags to tweak it to their use case. To me it feels like GCFrequency would be enough to make sure that the VM returns memory on a timely basis. And if the flag is managed, it can be controlled to not do periodic GCs during high load. With that we get a good way to periodically try to reduce the committed heap. The reason I ask is because I have a hard time seeing how we can implement a generic policy for when the system is idle. A policy that will apply well to most use cases. For some cases having the flags you propose might be good, but for other there might be a different set of options needed. If this is the case then maybe the logic and policy of when to do this can live outside the VM, while the code to periodically do GCs lives within the VM. What do you think about that? I understand the problems you've stated with having the policy outside that VM, but at least we have more information to act on there. We know that many have asked for features similar to this one and it would be nice to get input from others on this to make sure we implement something that benefits the whole user base as much as possible. So anyone with a use case that could benefit from this, please chime in. Regards, Stefan On 2018-09-07 17:37, Rodrigo Bruno wrote: > Hi Per and Thomas, > > thank you for your comments. > > I think it is possible to implement this feature using the service > thread or using a separate thread. > I see some pros and cons of having a separate thread: > > Pros: > - using the service thread exposes something that is G1 specific to the > rest of the JVM. > Thus, using a separate thread, hides this feature from the outsite. > > Cons: > - Having a manageable timeout is a bit more tricky to implement in a > separate/dedicated thread. > We need to be able to handle switch on and off. It might require some > variable pooling. > - It requires some more memory. > > Regardless of the path taken, I can prepare a new version of the patch > whenever we decide on this. > > cheers, > rodrigo > > Per Liden > escreveu > no dia sexta, 7/09/2018 ?(s) 11:58: > > Hi Thomas, > > On 09/07/2018 10:10 AM, Thomas Schatzl wrote: > [...] > >? ? overnight I thought a bit of the implementation, and given the > > problem with heap usage of the new thread, and the requirement of > being > > able to turn on/off that feature by a managed variable, the best > change > > would probably reusing the service thread as you did in the initial > > change. > > I'm not convinced that this should be handled outside of G1. If there's > a need to have the flag manageable at runtime (is that really the > case?), you could just always start the G1DetectIdleThread and have it > check the flag. I wouldn't worry too much about the memory overhead for > the stack. > > cheers, > Per > From erik.osterlund at oracle.com Thu Sep 13 12:51:16 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 13 Sep 2018 14:51:16 +0200 Subject: RFR: JDK-8210656: Object equals abstraction for BarrierSetC2 In-Reply-To: <8b48dd82-7162-1d5a-527a-fc068a72c009@redhat.com> References: <07f491e5-fb06-86a3-f6f1-1b9d37544fb0@redhat.com> <5B9A3F49.4090905@oracle.com> <8b48dd82-7162-1d5a-527a-fc068a72c009@redhat.com> Message-ID: <5B9A5D44.7040409@oracle.com> Hi Roman, I'm glad this idea works well for you. If you need an Ideal hook for CmpPNode anyway for barrier optimizations, then I suppose we should sort something out there. In that API though, it would be great if it was not specific to CmpPNode. I'm thinking something along the lines of Node* BarrierSetC2::ideal_node(Node* n), and then figure out if something should or should not be done for a given node in the backend. That way, if we need ideal hooks for other node types, we could reuse the same API to ask the BarrierSetC2 if it has more ideal nodes. What do you think? Thanks, /Erik On 2018-09-13 14:15, Roman Kennke wrote: > Hi Erik, > > I talked to Roland about this. It turns out that we already have this > optimization pass, and could just as well live with cmp(resolve(a), > resolve(b)). We need a little (shenandoah-specific-) hook in > CmpPNode::Ideal() for that though (but we'd need that anyway I suppose). > If you agree with that, I'll withdraw this RFR. Ok? > > Roman > >> Hi Roman, >> >> Interesting. So semantically, this is cmp(resolve(a), resolve(b)), but >> you have circumstances in which the barriers are unnecessary and can be >> elided. Any of them having null in their type is one reason, but I >> suppose there are surely other reasons as well (such as finding >> dominating write barriers). >> >> I see two different approaches for this barrier elision: >> 1) Elide it during parsing (as you propose) >> 2) Elide it during Optimize (which I think conceptually looks like a >> natural fit) >> >> I originally proposed a function on BarrierSetC2 that I think I called >> optimize_barriers() or something like that. The idea was to use this >> hook to let GC barrier code shave off pointless (not to be confused with >> useless) barriers that can be removed. Roland thought that this seemed >> too specific to ZGC to warrant a general API, and I agreed, because >> indeed only ZGC used this hook at the time. This is today >> ZBarrierSetC2::find_dominating_barriers which is called straight from >> Optimize. >> >> I wonder if it would make sense to re-instate that hook. Then you could >> use the existing resolve() barriers during parsing, and leave barrier >> elision tricks (null checks included, plus other tricks you might have >> up your sleeve) to Optimize. For example, you might be able to walk your >> list of barriers and disconnect these pointless barriers. What do you >> think? >> >> Thanks, >> /Erik >> >> On 2018-09-12 22:11, Roman Kennke wrote: >>> This introduces an abstraction to deal with object equality in >>> BarrierSetC2. This is needed by GCs that can have different copies of >>> same objects alive like Shenandoah. >>> >>> The approach chosen here is slightly different than we did in e.g. >>> BarrierSetAssembler and the runtime Access API: instead of owning the >>> whole equality, it only provides a resolve-like method to resolve the >>> operands to stable values. The reason for doing this is that it's easier >>> to do this way in intrinsics if those barriers are detached from the >>> actual CmpP. This is because the barriers create new memory states, and >>> we'd have to create memphis around those things, which is considerably >>> more complex. >>> >>> I chose to add a new resolve_for_obj_equals(a, b) method instead of >>> using two calls to resolve(a); resolve(b); because this allows for >>> optimization: if any of a or b is known to be NULL, we can elide >>> barriers for both. This is not possible to do with two independent >>> resolve() calls. >>> >>> Bug: >>> https://bugs.openjdk.java.net/browse/JDK-8210656 >>> Webrev: >>> http://cr.openjdk.java.net/~rkennke/JDK-8210656/webrev.00/ >>> >>> Testing: passes hotspot/jtreg:tier1 >>> >>> What do you think about this? >>> >>> Thanks, >>> Roman >>> > From per.liden at oracle.com Thu Sep 13 13:59:47 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 13 Sep 2018 15:59:47 +0200 Subject: RFR: 8210714: ZGC: ZWeakRootsIterator should no longer call reset/finish_dead_counter() Message-ID: <4217e45b-47d7-3ded-f377-3feb97286003@oracle.com> When the ZConcurrentStringTable option was removed (JDK-8210061) and that behavior was made the default and always on, we left calls to StringTable::reset/finish_dead_counter() that are no longer needed. Bug: https://bugs.openjdk.java.net/browse/JDK-8210714 Webrev: http://cr.openjdk.java.net/~pliden/8210714/webrev.0 /Per From erik.osterlund at oracle.com Thu Sep 13 15:33:15 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 13 Sep 2018 17:33:15 +0200 Subject: RFR: 8210714: ZGC: ZWeakRootsIterator should no longer call reset/finish_dead_counter() In-Reply-To: <4217e45b-47d7-3ded-f377-3feb97286003@oracle.com> References: <4217e45b-47d7-3ded-f377-3feb97286003@oracle.com> Message-ID: <5B9A833B.3040600@oracle.com> Hi Per, Looks good! /Erik On 2018-09-13 15:59, Per Liden wrote: > When the ZConcurrentStringTable option was removed (JDK-8210061) and > that behavior was made the default and always on, we left calls to > StringTable::reset/finish_dead_counter() that are no longer needed. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210714 > Webrev: http://cr.openjdk.java.net/~pliden/8210714/webrev.0 > > /Per From rkennke at redhat.com Thu Sep 13 15:37:48 2018 From: rkennke at redhat.com (Roman Kennke) Date: Thu, 13 Sep 2018 17:37:48 +0200 Subject: RFR: JDK-8210656: Object equals abstraction for BarrierSetC2 In-Reply-To: <5B9A5D44.7040409@oracle.com> References: <07f491e5-fb06-86a3-f6f1-1b9d37544fb0@redhat.com> <5B9A3F49.4090905@oracle.com> <8b48dd82-7162-1d5a-527a-fc068a72c009@redhat.com> <5B9A5D44.7040409@oracle.com> Message-ID: <87d66c05-9f5c-8e18-a589-b47d21d72681@redhat.com> Hi Erik, > I'm glad this idea works well for you. If you need an Ideal hook for > CmpPNode anyway for barrier optimizations, then I suppose we should sort > something out there. In that API though, it would be great if it was not > specific to CmpPNode. I'm thinking something along the lines of Node* > BarrierSetC2::ideal_node(Node* n), and then figure out if something > should or should not be done for a given node in the backend. That way, > if we need ideal hooks for other node types, we could reuse the same API > to ask the BarrierSetC2 if it has more ideal nodes. What do you think? Yeah, that would actually be good. We have at least one more place that I know of where we hook into Ideal(). The API would be something like this: Node* ideal_node(PhaseGVN* phase, Node* n, bool can_reshape) const; And a sample usage from CallLeafNode would look like this: Node* CallLeafNode::Ideal(PhaseGVN* phase, bool can_reshape) { Node* ideal = BarrierSet::barrier_set_c2()->barrier_set_c2()->ideal_node(phase, n, can_reshape); if (ideal != NULL) { return ideal; } return CallNode::Ideal(phase, n, can_reshape); } Unfortunately (or maybe fortunately) this can't be inserted generically into Node::Ideal(..) because subclasses can't be expected to always call the super implementation. Thanks for reviewing! I'll withdraw this RFR and push the additional resolve() hooks via another RFE. Cheers, Roman > Thanks, > /Erik > > On 2018-09-13 14:15, Roman Kennke wrote: >> Hi Erik, >> >> I talked to Roland about this. It turns out that we already have this >> optimization pass, and could just as well live with cmp(resolve(a), >> resolve(b)). We need a little (shenandoah-specific-) hook in >> CmpPNode::Ideal() for that though (but we'd need that anyway I suppose). >> If you agree with that, I'll withdraw this RFR. Ok? >> >> Roman >> >>> Hi Roman, >>> >>> Interesting. So semantically, this is cmp(resolve(a), resolve(b)), but >>> you have circumstances in which the barriers are unnecessary and can be >>> elided. Any of them having null in their type is one reason, but I >>> suppose there are surely other reasons as well (such as finding >>> dominating write barriers). >>> >>> I see two different approaches for this barrier elision: >>> 1) Elide it during parsing (as you propose) >>> 2) Elide it during Optimize (which I think conceptually looks like a >>> natural fit) >>> >>> I originally proposed a function on BarrierSetC2 that I think I called >>> optimize_barriers() or something like that. The idea was to use this >>> hook to let GC barrier code shave off pointless (not to be confused with >>> useless) barriers that can be removed. Roland thought that this seemed >>> too specific to ZGC to warrant a general API, and I agreed, because >>> indeed only ZGC used this hook at the time. This is today >>> ZBarrierSetC2::find_dominating_barriers which is called straight from >>> Optimize. >>> >>> I wonder if it would make sense to re-instate that hook. Then you could >>> use the existing resolve() barriers during parsing, and leave barrier >>> elision tricks (null checks included, plus other tricks you might have >>> up your sleeve) to Optimize. For example, you might be able to walk your >>> list of barriers and disconnect these pointless barriers. What do you >>> think? >>> >>> Thanks, >>> /Erik >>> >>> On 2018-09-12 22:11, Roman Kennke wrote: >>>> This introduces an abstraction to deal with object equality in >>>> BarrierSetC2. This is needed by GCs that can have different copies of >>>> same objects alive like Shenandoah. >>>> >>>> The approach chosen here is slightly different than we did in e.g. >>>> BarrierSetAssembler and the runtime Access API: instead of owning the >>>> whole equality, it only provides a resolve-like method to resolve the >>>> operands to stable values. The reason for doing this is that it's >>>> easier >>>> to do this way in intrinsics if those barriers are detached from the >>>> actual CmpP. This is because the barriers create new memory states, and >>>> we'd have to create memphis around those things, which is considerably >>>> more complex. >>>> >>>> I chose to add a new resolve_for_obj_equals(a, b) method instead of >>>> using two calls to resolve(a); resolve(b); because this allows for >>>> optimization: if any of a or b is known to be NULL, we can elide >>>> barriers for both. This is not possible to do with two independent >>>> resolve() calls. >>>> >>>> Bug: >>>> https://bugs.openjdk.java.net/browse/JDK-8210656 >>>> Webrev: >>>> http://cr.openjdk.java.net/~rkennke/JDK-8210656/webrev.00/ >>>> >>>> Testing: passes hotspot/jtreg:tier1 >>>> >>>> What do you think about this? >>>> >>>> Thanks, >>>> Roman >>>> >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From manc at google.com Fri Sep 14 00:01:06 2018 From: manc at google.com (Man Cao) Date: Thu, 13 Sep 2018 17:01:06 -0700 Subject: RFR (S) 8210724: Change the verbosity threshold of logging for [oopstorage, ref] Message-ID: Hi all, Could anyone review this logging change for oopstorage? Webrev: http://cr.openjdk.java.net/~jcbeyler/8210724/webrev.00/ Bug: https://bugs.openjdk.java.net/browse/JDK-8210724 This is similar to the webrev I previously sent for https://bugs.openjdk.java.net/browse/JDK-8210562. Kim mentioned it might be reasonable to improve the logging in oopStorage.cpp. We discussed and decided to create a separate issue for improving the logging. The patch changes the logging level for [oopstorage,ref] to trace, and consistently uses debug level for [oopstorage,blocks] and unusual events, and uses log_info() for printing statistics. I think this would make the logging for oopstorage more user-friendly. Below is comparison of lines of messages: $ java -Xlog:oopstorage*= -Xms500m -Xmx500m -jar dacapo-9.12-bach.jar tradesoap info | debug | trace without patch: 107934 | 112990 | 112701 with patch: 1694 | 5247 | 112241 -Man -------------- next part -------------- An HTML attachment was scrubbed... URL: From manc at google.com Fri Sep 14 00:04:20 2018 From: manc at google.com (Man Cao) Date: Thu, 13 Sep 2018 17:04:20 -0700 Subject: RFR (S) 8210562: Change the verbosity threshold of logging for [oopstorage, ref] In-Reply-To: References: <91BE44AF-9346-4BEF-809C-2B79756A06B8@oracle.com> Message-ID: Thanks! JC created a separate RFE for logging: JDK-8210724. I have sent an RFR for it. -Man On Thu, Sep 13, 2018 at 4:35 AM Thomas Schatzl wrote: > Hi Man, > > On Wed, 2018-09-12 at 11:27 -0700, Man Cao wrote: > > Thanks for the response, Kim! Replies below. > > > > > I think the problem here is the documented translation from the old > > > -XX:+PrintReferenceGC to using UL. Rather than -Xlog:ref*=debug, > > > it should be -Xlog:gc+ref*=debug. > > > Thomas also agreed on fixing the documentation instead in the bug. > > I'm not familiar with the process of changing documentation. Should > > we also change JDK9 and JDK10's documentation, or only JDK11's > > documentation? > > I created JDK-8210712 for the doc team. > > > Since this whole issue is about to UL and consistent behavior with > > old flag, should I also CC serviceability-dev at openjdk.java.net on > > this thread? > > I do not think this is needed for just a doc error. > > [...] > > > Should we open a new RFE for that? I think JDK-8210562 might be OK > > for this purpose, as its title is about "Change the verbosity > > threshold" for logging. > > If you change the old RFE, please put the old description in the > comments, otherwise the link to JDK-8210712 will be confusing. > Feel free to create a new RFE and close this issue maybe as duplicate > as JDK-8210712 though. > > Thanks, > Thomas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kim.barrett at oracle.com Fri Sep 14 00:41:11 2018 From: kim.barrett at oracle.com (Kim Barrett) Date: Thu, 13 Sep 2018 20:41:11 -0400 Subject: RFR (S) 8210724: Change the verbosity threshold of logging for [oopstorage,ref] In-Reply-To: References: Message-ID: <9F76DF3D-D7C4-4187-A48E-B656D8904503@oracle.com> > On Sep 13, 2018, at 8:01 PM, Man Cao wrote: > > Hi all, > > Could anyone review this logging change for oopstorage? > Webrev: http://cr.openjdk.java.net/~jcbeyler/8210724/webrev.00/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8210724 > > This is similar to the webrev I previously sent for https://bugs.openjdk.java.net/browse/JDK-8210562. Kim mentioned it might be reasonable to improve the logging in oopStorage.cpp. We discussed and decided to create a separate issue for improving the logging. > > The patch changes the logging level for [oopstorage,ref] to trace, and consistently uses debug level for [oopstorage,blocks] and unusual events, and uses log_info() for printing statistics. I think this would make the logging for oopstorage more user-friendly. > > Below is comparison of lines of messages: > $ java -Xlog:oopstorage*= -Xms500m -Xmx500m -jar dacapo-9.12-bach.jar tradesoap > info | debug | trace > without patch: 107934 | 112990 | 112701 > with patch: 1694 | 5247 | 112241 > > -Man Looks good. From rkennke at redhat.com Fri Sep 14 07:36:12 2018 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 14 Sep 2018 09:36:12 +0200 Subject: RFC: Piggy-back monitor-deflation and nmethod-marking on GC Message-ID: <1517ac6a-c55c-4f4a-9f4d-b3299f2185c1@redhat.com> Regarding those issues: https://bugs.openjdk.java.net/browse/JDK-8132849 https://bugs.openjdk.java.net/browse/JDK-8132849 I.e. increased pause times caused by monitor-deflation and nmethod-marking, there has been the suggestion by the reporter to piggy-back those activities on GC thread-scanning. We do have code in Shenandoah that does this since quite a while. The idea is to introduce two new methods in VM_Operation: virtual bool deflates_idle_monitors(); virtual bool marks_nmethods(); That basically tell the runtime whether or not a particular VM_Operation (e.g. InitMark) can take over any of the two activties, and if so, skip the corresponding activity in the preceding SP-cleanup phase. This means that the VM_Operation needs to handle this itself. The change also includes some changes in sychronizer, thread etc to make this relatively easy to implement from a GC standpoint. Question: would there be interest in getting this upstreamed? If so, I would put together a patch for reviewing. Alternatively or additionally, at least for monitor deflation, there is the JEP Draft: Concurrent Monitor Deflation: https://bugs.openjdk.java.net/browse/JDK-8183909 But this seems like a very serious amount of work. WDYT? Roman -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From stefan.johansson at oracle.com Fri Sep 14 07:37:20 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Fri, 14 Sep 2018 09:37:20 +0200 Subject: RFR (S) 8210724: Change the verbosity threshold of logging for [oopstorage,ref] In-Reply-To: <9F76DF3D-D7C4-4187-A48E-B656D8904503@oracle.com> References: <9F76DF3D-D7C4-4187-A48E-B656D8904503@oracle.com> Message-ID: <0403dd98-2ee1-a472-ba0d-7ecbbae8be79@oracle.com> On 2018-09-14 02:41, Kim Barrett wrote: >> On Sep 13, 2018, at 8:01 PM, Man Cao wrote: >> >> Hi all, >> >> Could anyone review this logging change for oopstorage? >> Webrev: http://cr.openjdk.java.net/~jcbeyler/8210724/webrev.00/ >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210724 >> >> This is similar to the webrev I previously sent for https://bugs.openjdk.java.net/browse/JDK-8210562. Kim mentioned it might be reasonable to improve the logging in oopStorage.cpp. We discussed and decided to create a separate issue for improving the logging. >> >> The patch changes the logging level for [oopstorage,ref] to trace, and consistently uses debug level for [oopstorage,blocks] and unusual events, and uses log_info() for printing statistics. I think this would make the logging for oopstorage more user-friendly. >> >> Below is comparison of lines of messages: >> $ java -Xlog:oopstorage*= -Xms500m -Xmx500m -jar dacapo-9.12-bach.jar tradesoap >> info | debug | trace >> without patch: 107934 | 112990 | 112701 >> with patch: 1694 | 5247 | 112241 >> >> -Man > > Looks good. > Looks good to me too, Stefan From erik.osterlund at oracle.com Fri Sep 14 08:04:08 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Fri, 14 Sep 2018 10:04:08 +0200 Subject: RFR: 8210713: Let CollectedHeap::ensure_parsability() take care of TLAB statistics gathering In-Reply-To: References: Message-ID: <5B9B6B78.5020102@oracle.com> Hi Per, Looks good. Thanks, /Erik On 2018-09-13 13:46, Per Liden wrote: > A call to CollectedHeap::ensure_parsability(retire_tlabs), where > retire_tlabs is true must always be preceded by a call to > CollectedHeap::accumulate_statistics_all_tlabs(), otherwise we will > loose TLAB statistics as it will be reset when the TLABs are retired. > Instead of having all collectors do this dance and risk getting it > wrong, I propose that we move this responsibility into > ensure_parsability(), which will gather the TLAB statistics if it's > told to retire all TLABs. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210713 > Webrev: http://cr.openjdk.java.net/~pliden/8210713/webrev.0 > > /Per From per.liden at oracle.com Fri Sep 14 08:27:39 2018 From: per.liden at oracle.com (Per Liden) Date: Fri, 14 Sep 2018 10:27:39 +0200 Subject: RFR: 8210713: Let CollectedHeap::ensure_parsability() take care of TLAB statistics gathering In-Reply-To: <5B9B6B78.5020102@oracle.com> References: <5B9B6B78.5020102@oracle.com> Message-ID: <13c9ab24-f1a4-ceb0-1f9a-219a84ded2f1@oracle.com> Thanks Erik! /Per On 09/14/2018 10:04 AM, Erik ?sterlund wrote: > Hi Per, > > Looks good. > > Thanks, > /Erik > > On 2018-09-13 13:46, Per Liden wrote: >> A call to CollectedHeap::ensure_parsability(retire_tlabs), where >> retire_tlabs is true must always be preceded by a call to >> CollectedHeap::accumulate_statistics_all_tlabs(), otherwise we will >> loose TLAB statistics as it will be reset when the TLABs are retired. >> Instead of having all collectors do this dance and risk getting it >> wrong, I propose that we move this responsibility into >> ensure_parsability(), which will gather the TLAB statistics if it's >> told to retire all TLABs. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210713 >> Webrev: http://cr.openjdk.java.net/~pliden/8210713/webrev.0 >> >> /Per > From per.liden at oracle.com Fri Sep 14 08:51:33 2018 From: per.liden at oracle.com (Per Liden) Date: Fri, 14 Sep 2018 10:51:33 +0200 Subject: RFR: 8210714: ZGC: ZWeakRootsIterator should no longer call reset/finish_dead_counter() In-Reply-To: <5B9A833B.3040600@oracle.com> References: <4217e45b-47d7-3ded-f377-3feb97286003@oracle.com> <5B9A833B.3040600@oracle.com> Message-ID: <7e72e07b-08bf-654b-2fa3-b1fa9a8fa29d@oracle.com> Thanks Erik! /Per On 09/13/2018 05:33 PM, Erik ?sterlund wrote: > Hi Per, > > Looks good! > > /Erik > > On 2018-09-13 15:59, Per Liden wrote: >> When the ZConcurrentStringTable option was removed (JDK-8210061) and >> that behavior was made the default and always on, we left calls to >> StringTable::reset/finish_dead_counter() that are no longer needed. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210714 >> Webrev: http://cr.openjdk.java.net/~pliden/8210714/webrev.0 >> >> /Per > From per.liden at oracle.com Fri Sep 14 11:37:34 2018 From: per.liden at oracle.com (Per Liden) Date: Fri, 14 Sep 2018 13:37:34 +0200 Subject: RFR: 8210713: Let CollectedHeap::ensure_parsability() take care of TLAB statistics gathering In-Reply-To: References: Message-ID: <579d6261-a7f8-9c6e-119b-c957b98a40ff@oracle.com> I should probably mention that this patch is just an intermediate step towards the end goal, which is to allow a GC to retire TLABs and collect statistics with a single pass over the thread list and in parallel if so desired. I'm just breaking up the steps to get there in more easy to review pieces, which also have some value on their own. cheers, Per On 09/13/2018 01:46 PM, Per Liden wrote: > A call to CollectedHeap::ensure_parsability(retire_tlabs), where > retire_tlabs is true must always be preceded by a call to > CollectedHeap::accumulate_statistics_all_tlabs(), otherwise we will > loose TLAB statistics as it will be reset when the TLABs are retired. > Instead of having all collectors do this dance and risk getting it > wrong, I propose that we move this responsibility into > ensure_parsability(), which will gather the TLAB statistics if it's told > to retire all TLABs. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210713 > Webrev: http://cr.openjdk.java.net/~pliden/8210713/webrev.0 > > /Per From per.liden at oracle.com Fri Sep 14 11:47:05 2018 From: per.liden at oracle.com (Per Liden) Date: Fri, 14 Sep 2018 13:47:05 +0200 Subject: RFR: 8210753: Make ThreadLocalAllocBuffer::resize() public Message-ID: Make ThreadLocalAllocBuffer::resize() public to allow a GC to resize TLABs in parallel. With resize() public I propose that we remove ThreadLocalAllocBuffer::resize_all_tlabs() and let CollectedHeap::resize_all_tlabs() iterate over the threads, just like CollectedHeap::ensure_parsability() does. Bug: https://bugs.openjdk.java.net/browse/JDK-8210753 Webrev: http://cr.openjdk.java.net/~pliden/8210753/webrev.0 /Per From erik.osterlund at oracle.com Fri Sep 14 11:55:07 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Fri, 14 Sep 2018 13:55:07 +0200 Subject: RFR: 8210753: Make ThreadLocalAllocBuffer::resize() public In-Reply-To: References: Message-ID: <5B9BA19B.4020805@oracle.com> Hi Per, Looks good. Thanks, /Erik On 2018-09-14 13:47, Per Liden wrote: > Make ThreadLocalAllocBuffer::resize() public to allow a GC to resize > TLABs in parallel. With resize() public I propose that we remove > ThreadLocalAllocBuffer::resize_all_tlabs() and let > CollectedHeap::resize_all_tlabs() iterate over the threads, just like > CollectedHeap::ensure_parsability() does. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210753 > Webrev: http://cr.openjdk.java.net/~pliden/8210753/webrev.0 > > /Per From per.liden at oracle.com Fri Sep 14 11:52:53 2018 From: per.liden at oracle.com (Per Liden) Date: Fri, 14 Sep 2018 13:52:53 +0200 Subject: RFR: 8210753: Make ThreadLocalAllocBuffer::resize() public In-Reply-To: References: Message-ID: <30a7a606-ea1d-0bf1-2927-1bd7a2a6df28@oracle.com> Thanks Erik! /Per On 09/14/2018 01:47 PM, Per Liden wrote: > Make ThreadLocalAllocBuffer::resize() public to allow a GC to resize > TLABs in parallel. With resize() public I propose that we remove > ThreadLocalAllocBuffer::resize_all_tlabs() and let > CollectedHeap::resize_all_tlabs() iterate over the threads, just like > CollectedHeap::ensure_parsability() does. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210753 > Webrev: http://cr.openjdk.java.net/~pliden/8210753/webrev.0 > > /Per From rkennke at redhat.com Fri Sep 14 12:56:07 2018 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 14 Sep 2018 14:56:07 +0200 Subject: RFR: JDK-8210752: Remaining explicit barriers for C2 Message-ID: <15be8e2d-dba5-2e8a-c851-b6821b81d4b3@redhat.com> Please review the following change: JDK-8210187 introduced explicit barriers for C2. There've been a few missing: - Unsafe accesses also require explicit barriers when it's unknown if the access is on-heap or off-heap. In this case, C2 turns the access into a raw access, in which case the access_load/store APIs cannot determine what to do. Emitting explicit barriers solves this for Shenandoah: in case of raw access, base will be NULL, which gets handled by a null-check (in this case the barrier is ignored), for on-heap access, the null-check will fail and the barrier triggered correctly. - One arraycopy barrier on dst erroneously emitted for ACCESS_READ where it should be ACCESS_WRITE (my mistake) - Object equality using CmpP requires stable oops, and thus barriers on both operands. - vectorizedMismatch() and copyMemory() also require explicit barriers before building the addresses and feeding them into the calls. Bug: https://bugs.openjdk.java.net/browse/JDK-8210752 Webrev: http://cr.openjdk.java.net/~rkennke/JDK-8210752/webrev.00/ Thanks, Roman -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rkennke at redhat.com Fri Sep 14 14:56:17 2018 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 14 Sep 2018 16:56:17 +0200 Subject: RFC: Piggy-back monitor-deflation and nmethod-marking on GC In-Reply-To: <1517ac6a-c55c-4f4a-9f4d-b3299f2185c1@redhat.com> References: <1517ac6a-c55c-4f4a-9f4d-b3299f2185c1@redhat.com> Message-ID: > Regarding those issues: > > https://bugs.openjdk.java.net/browse/JDK-8132849 > https://bugs.openjdk.java.net/browse/JDK-8132849 Oops. I meant to include: https://bugs.openjdk.java.net/browse/JDK-8153224 Roman > I.e. increased pause times caused by monitor-deflation and > nmethod-marking, there has been the suggestion by the reporter to > piggy-back those activities on GC thread-scanning. > > We do have code in Shenandoah that does this since quite a while. > > The idea is to introduce two new methods in VM_Operation: > > virtual bool deflates_idle_monitors(); > virtual bool marks_nmethods(); > > That basically tell the runtime whether or not a particular VM_Operation > (e.g. InitMark) can take over any of the two activties, and if so, skip > the corresponding activity in the preceding SP-cleanup phase. This means > that the VM_Operation needs to handle this itself. The change also > includes some changes in sychronizer, thread etc to make this relatively > easy to implement from a GC standpoint. > > Question: would there be interest in getting this upstreamed? If so, I > would put together a patch for reviewing. > > Alternatively or additionally, at least for monitor deflation, there is > the JEP Draft: Concurrent Monitor Deflation: > https://bugs.openjdk.java.net/browse/JDK-8183909 > > But this seems like a very serious amount of work. > > WDYT? > > Roman > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From jcbeyler at google.com Fri Sep 14 15:12:48 2018 From: jcbeyler at google.com (JC Beyler) Date: Fri, 14 Sep 2018 08:12:48 -0700 Subject: RFR: 8210753: Make ThreadLocalAllocBuffer::resize() public In-Reply-To: <30a7a606-ea1d-0bf1-2927-1bd7a2a6df28@oracle.com> References: <30a7a606-ea1d-0bf1-2927-1bd7a2a6df28@oracle.com> Message-ID: Hi Per, I noticed when working on this code that I always disliked the resize method naming: really what it is doing is calculating the next size for a given TLAB to be done when the tlab is full. I don't think we'd automatically want to rename the resize method but it is only called once currently so changing its name could be done here easily and any reader would understand that tlab.resize() does not resize; it merely calculates the next size. Were you to consider it, I think calculate_next_size would seem to be more appropriate (and we could remove the comment in the first line of the method since now it is redundant with the name of the method). Anyway, whatever you do decide, it looks good to me (not a reviewer though), Jc On Fri, Sep 14, 2018 at 4:53 AM Per Liden wrote: > Thanks Erik! > > /Per > > On 09/14/2018 01:47 PM, Per Liden wrote: > > Make ThreadLocalAllocBuffer::resize() public to allow a GC to resize > > TLABs in parallel. With resize() public I propose that we remove > > ThreadLocalAllocBuffer::resize_all_tlabs() and let > > CollectedHeap::resize_all_tlabs() iterate over the threads, just like > > CollectedHeap::ensure_parsability() does. > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210753 > > Webrev: http://cr.openjdk.java.net/~pliden/8210753/webrev.0 > > > > /Per > -- Thanks, Jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From per.liden at oracle.com Fri Sep 14 16:08:55 2018 From: per.liden at oracle.com (Per Liden) Date: Fri, 14 Sep 2018 18:08:55 +0200 Subject: RFR: 8210753: Make ThreadLocalAllocBuffer::resize() public In-Reply-To: References: <30a7a606-ea1d-0bf1-2927-1bd7a2a6df28@oracle.com> Message-ID: Hi JC, On 09/14/2018 05:12 PM, JC Beyler wrote: > Hi Per, > > I noticed when working on this code that I always disliked the resize > method naming: really what it is doing is calculating the next size for > a given TLAB to be done when the tlab is full. I don't think we'd > automatically want to rename the resize method but it is only called > once currently so changing its name could be done here easily and any > reader would understand that tlab.resize() does not resize; it merely > calculates the next size. > > Were you to consider it, I think calculate_next_size would seem to be > more appropriate (and we could remove the comment in the first line of > the method since now it is redundant with the name of the method). > > Anyway, whatever you do decide, it looks good to me (not a reviewer though), I see your point. However, there are never any active TLABs (they have all been retired) when resize() is called by the GC, so it can only refer to the size of the next allocated TLAB. That might not be super obvious, but I think I'd prefer to keep the name, since it has a history and I think this well understood among the GC folks. Thanks for reviewing! cheers, Per > Jc > > On Fri, Sep 14, 2018 at 4:53 AM Per Liden > wrote: > > Thanks Erik! > > /Per > > On 09/14/2018 01:47 PM, Per Liden wrote: > > Make ThreadLocalAllocBuffer::resize() public to allow a GC to resize > > TLABs in parallel. With resize() public I propose that we remove > > ThreadLocalAllocBuffer::resize_all_tlabs() and let > > CollectedHeap::resize_all_tlabs() iterate over the threads, just > like > > CollectedHeap::ensure_parsability() does. > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210753 > > Webrev: http://cr.openjdk.java.net/~pliden/8210753/webrev.0 > > > > /Per > > > > -- > > Thanks, > Jc From jiangli.zhou at oracle.com Fri Sep 14 17:12:16 2018 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Fri, 14 Sep 2018 10:12:16 -0700 Subject: RFR: 8210237: gc/stress/TestStressIHOPMultiThread.java fails with 'Unexpected exit from test [exit code: 1]' in CDS mode Message-ID: Please help review the fix for test TestStressIHOPMultiThread.java. This test has the similar issue as the one fixed earlier this week for JDK-8210193. Please see the bug report for detailed analysis. ? webrev: http://cr.openjdk.java.net/~jiangli/8210237/webrev.00/ ? bug: https://bugs.openjdk.java.net/browse/JDK-8210237 Tested with tier1 - tier3. Also tested with tier6 and tier7 with the default CDS archive patch. Thanks, Jiangli From jcbeyler at google.com Fri Sep 14 19:17:09 2018 From: jcbeyler at google.com (JC Beyler) Date: Fri, 14 Sep 2018 12:17:09 -0700 Subject: RFR (S) 8210724: Change the verbosity threshold of logging for [oopstorage, ref] In-Reply-To: <0403dd98-2ee1-a472-ba0d-7ecbbae8be79@oracle.com> References: <9F76DF3D-D7C4-4187-A48E-B656D8904503@oracle.com> <0403dd98-2ee1-a472-ba0d-7ecbbae8be79@oracle.com> Message-ID: Thanks both, I updated the webrev, sent it on the submit repo, it passed, and then pushed it. Have a great weekend, Jc On Fri, Sep 14, 2018 at 12:38 AM Stefan Johansson < stefan.johansson at oracle.com> wrote: > > > On 2018-09-14 02:41, Kim Barrett wrote: > >> On Sep 13, 2018, at 8:01 PM, Man Cao wrote: > >> > >> Hi all, > >> > >> Could anyone review this logging change for oopstorage? > >> Webrev: http://cr.openjdk.java.net/~jcbeyler/8210724/webrev.00/ > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210724 > >> > >> This is similar to the webrev I previously sent for > https://bugs.openjdk.java.net/browse/JDK-8210562. Kim mentioned it might > be reasonable to improve the logging in oopStorage.cpp. We discussed and > decided to create a separate issue for improving the logging. > >> > >> The patch changes the logging level for [oopstorage,ref] to trace, and > consistently uses debug level for [oopstorage,blocks] and unusual events, > and uses log_info() for printing statistics. I think this would make the > logging for oopstorage more user-friendly. > >> > >> Below is comparison of lines of messages: > >> $ java -Xlog:oopstorage*= -Xms500m -Xmx500m -jar > dacapo-9.12-bach.jar tradesoap > >> info | debug | trace > >> without patch: 107934 | 112990 | 112701 > >> with patch: 1694 | 5247 | 112241 > >> > >> -Man > > > > Looks good. > > > > Looks good to me too, > Stefan > > -- Thanks, Jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From jiangli.zhou at oracle.com Fri Sep 14 19:22:02 2018 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Fri, 14 Sep 2018 12:22:02 -0700 Subject: RFR: 8210237: gc/stress/TestStressIHOPMultiThread.java fails with 'Unexpected exit from test [exit code: 1]' in CDS mode In-Reply-To: <08CC7521-C4FF-4910-BE02-89627B8681D7@oracle.com> References: <08CC7521-C4FF-4910-BE02-89627B8681D7@oracle.com> Message-ID: <5f487141-aa57-a4a4-78ca-feca2d8316d8@oracle.com> Hi Gerard, Thanks for the review! The assumption (can allocate up to the specified amount under low memory conditions) made by the test for all different configurations may not always be true and OutOfMemoryError could happen with some scenarios. The fix is to handle possible OutOfMemoryError in the test. The test is intended for IHOP testing with logging output. Adding the OOM handling does not seem to change the original intention. Sorry, I should have included more details in the RFR request. Thanks, Jiangli On 9/14/18 11:55 AM, Gerard Ziemski wrote: > hi Jiangli, > > So the fix here is to provide more diagnostic info, so that it can be later used to determine the real issue? > > Looks good if that?s the intend. > > > cheers > >> On Sep 14, 2018, at 12:12 PM, Jiangli Zhou wrote: >> >> Please help review the fix for test TestStressIHOPMultiThread.java. This test has the similar issue as the one fixed earlier this week for JDK-8210193. Please see the bug report for detailed analysis. >> >> webrev: http://cr.openjdk.java.net/~jiangli/8210237/webrev.00/ >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8210237 >> >> Tested with tier1 - tier3. Also tested with tier6 and tier7 with the default CDS archive patch. >> >> Thanks, >> >> Jiangli >> From calvin.cheung at oracle.com Fri Sep 14 19:52:30 2018 From: calvin.cheung at oracle.com (Calvin Cheung) Date: Fri, 14 Sep 2018 12:52:30 -0700 Subject: RFR: 8210237: gc/stress/TestStressIHOPMultiThread.java fails with 'Unexpected exit from test [exit code: 1]' in CDS mode In-Reply-To: References: Message-ID: <5B9C117D.3020904@oracle.com> Looks good. thanks, Calvin On 9/14/18, 10:12 AM, Jiangli Zhou wrote: > Please help review the fix for test TestStressIHOPMultiThread.java. > This test has the similar issue as the one fixed earlier this week for > JDK-8210193. Please see the bug report for detailed analysis. > > webrev: http://cr.openjdk.java.net/~jiangli/8210237/webrev.00/ > > bug: https://bugs.openjdk.java.net/browse/JDK-8210237 > > Tested with tier1 - tier3. Also tested with tier6 and tier7 with the > default CDS archive patch. > > Thanks, > > Jiangli > From jiangli.zhou at oracle.com Fri Sep 14 19:57:31 2018 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Fri, 14 Sep 2018 12:57:31 -0700 Subject: RFR: 8210237: gc/stress/TestStressIHOPMultiThread.java fails with 'Unexpected exit from test [exit code: 1]' in CDS mode In-Reply-To: <5B9C117D.3020904@oracle.com> References: <5B9C117D.3020904@oracle.com> Message-ID: <30277F01-A3D0-4C41-968E-6CDF92C341E6@oracle.com> Thanks! Jiangli > On Sep 14, 2018, at 12:52 PM, Calvin Cheung wrote: > > Looks good. > > thanks, > Calvin > >> On 9/14/18, 10:12 AM, Jiangli Zhou wrote: >> Please help review the fix for test TestStressIHOPMultiThread.java. This test has the similar issue as the one fixed earlier this week for JDK-8210193. Please see the bug report for detailed analysis. >> >> webrev: http://cr.openjdk.java.net/~jiangli/8210237/webrev.00/ >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8210237 >> >> Tested with tier1 - tier3. Also tested with tier6 and tier7 with the default CDS archive patch. >> >> Thanks, >> >> Jiangli >> From jiangli.zhou at oracle.com Fri Sep 14 22:39:39 2018 From: jiangli.zhou at oracle.com (Jiangli Zhou) Date: Fri, 14 Sep 2018 15:39:39 -0700 Subject: RFR: 8210237: gc/stress/TestStressIHOPMultiThread.java fails with 'Unexpected exit from test [exit code: 1]' in CDS mode In-Reply-To: References: <08CC7521-C4FF-4910-BE02-89627B8681D7@oracle.com> <5f487141-aa57-a4a4-78ca-feca2d8316d8@oracle.com> Message-ID: <5821b048-4260-32f7-d772-f66446a204eb@oracle.com> Hi Gerard, On 9/14/18 2:08 PM, Gerard Ziemski wrote: > hi Jiangli, > > Thank you for the clarification. > > I wonder if we have some agreed upon exit status that tells Mach5 framework (and others) that a test finished trivially, so it can be marked as ?skipped?, ?not run?, or ?trivially run?? Seems we have a few tests that may not get to finish their run due to various reasons. That's a good question. From what I understand, for a test that's not run/skipped under various conditions, a SkippedException can be thrown by the test. The harness marks the execution as skipped instead of passed normally. For TestStressIHOPMultiThread.java (with the fix), if OOM occurs in one thread, other thread continues until specified timeout reaches. It's not skipped, but probably could be considered as 'trivially run' (less load). I did some searches but couldn't find a jtreg status corresponding to? 'trivially run'. It sounds like a good idea to have this kind of fine grained status. Thanks, Jiangli > > > cheers > > >> On Sep 14, 2018, at 2:22 PM, Jiangli Zhou wrote: >> >> Hi Gerard, >> >> Thanks for the review! The assumption (can allocate up to the specified amount under low memory conditions) made by the test for all different configurations may not always be true and OutOfMemoryError could happen with some scenarios. The fix is to handle possible OutOfMemoryError in the test. The test is intended for IHOP testing with logging output. Adding the OOM handling does not seem to change the original intention. Sorry, I should have included more details in the RFR request. >> >> Thanks, >> >> Jiangli >> >> >> On 9/14/18 11:55 AM, Gerard Ziemski wrote: >>> hi Jiangli, >>> >>> So the fix here is to provide more diagnostic info, so that it can be later used to determine the real issue? >>> >>> Looks good if that?s the intend. >>> >>> >>> cheers >>> >>>> On Sep 14, 2018, at 12:12 PM, Jiangli Zhou wrote: >>>> >>>> Please help review the fix for test TestStressIHOPMultiThread.java. This test has the similar issue as the one fixed earlier this week for JDK-8210193. Please see the bug report for detailed analysis. >>>> >>>> webrev: http://cr.openjdk.java.net/~jiangli/8210237/webrev.00/ >>>> >>>> bug: https://bugs.openjdk.java.net/browse/JDK-8210237 >>>> >>>> Tested with tier1 - tier3. Also tested with tier6 and tier7 with the default CDS archive patch. >>>> >>>> Thanks, >>>> >>>> Jiangli >>>> From rbruno at gsd.inesc-id.pt Sat Sep 15 20:14:48 2018 From: rbruno at gsd.inesc-id.pt (Rodrigo Bruno) Date: Sat, 15 Sep 2018 20:14:48 +0000 Subject: hotspot-gc-dev Digest, Vol 135, Issue 15 In-Reply-To: References: Message-ID: Hi Stefan, > ------------------------------ > > Message: 3 > Date: Thu, 13 Sep 2018 14:30:37 +0200 > From: Stefan Johansson > To: hotspot-gc-dev at openjdk.java.net > Subject: Re: RFR: bug: Timely Reducing Unused Committed Memory > Message-ID: <57b10708-b870-cb98-0df0-fab650f431e4 at oracle.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hi Rodrigo, > > Sorry for being a bit late into the discussion. We've had some internal > discussions and realized that there are some questions that I need to > bring up here. > > I'm trying to better understand under what circumstances this feature is > to be used and how a user should use the different flags to tweak it to > their use case. To me it feels like GCFrequency would be enough to make > sure that the VM returns memory on a timely basis. And if the flag is > managed, it can be controlled to not do periodic GCs during high load. > With that we get a good way to periodically try to reduce the committed > heap. > > The reason I ask is because I have a hard time seeing how we can > implement a generic policy for when the system is idle. A policy that > will apply well to most use cases. For some cases having the flags you > propose might be good, but for other there might be a different set of > options needed. If this is the case then maybe the logic and policy of > when to do this can live outside the VM, while the code to periodically > do GCs lives within the VM. What do you think about that? I understand > the problems you've stated with having the policy outside that VM, but > at least we have more information to act on there. > > I understand that it is hard to define what is idle. However, if we require the user to provide one, I guess that most regular users that suffer from the problem that this patch is trying to solve will simply not do it because it requires knowledge and effort. If we provide an idle check that we think will benefit most users, then we are probably helping a lot of users. For those that the default idle check is not good enough, they can always disable this idle check and implement the idle check logic it in an external tool. We can also change the semantics of "idleness". Currently it checks the load. I think that checking the allocation rate might be another good option (instead of load). The only corner case is an application that does not allocate but consumes a lot of CPU. For this case, we might only trigger compaction at most once because, as it does not allocate memory, we will not get over committed memory (i.e., the other checks will prevent it). The opposite is also possible (almost idle application that allocates a lot of memory) but in this scenario I don't think we want to trigger an idle compaction. Having said that, I am open to change this flag or even remove it as it is one of the hardest to get right. cheers, rodrigo > We know that many have asked for features similar to this one and it > would be nice to get input from others on this to make sure we implement > something that benefits the whole user base as much as possible. So > anyone with a use case that could benefit from this, please chime in. > > Regards, > Stefan > > > > On 2018-09-07 17:37, Rodrigo Bruno wrote: > > Hi Per and Thomas, > > > > thank you for your comments. > > > > I think it is possible to implement this feature using the service > > thread or using a separate thread. > > I see some pros and cons of having a separate thread: > > > > Pros: > > - using the service thread exposes something that is G1 specific to the > > rest of the JVM. > > Thus, using a separate thread, hides this feature from the outsite. > > > > Cons: > > - Having a manageable timeout is a bit more tricky to implement in a > > separate/dedicated thread. > > We need to be able to handle switch on and off. It might require some > > variable pooling. > > - It requires some more memory. > > > > Regardless of the path taken, I can prepare a new version of the patch > > whenever we decide on this. > > > > cheers, > > rodrigo > > > > Per Liden > escreveu > > no dia sexta, 7/09/2018 ?(s) 11:58: > > > > Hi Thomas, > > > > On 09/07/2018 10:10 AM, Thomas Schatzl wrote: > > [...] > > >? ? overnight I thought a bit of the implementation, and given the > > > problem with heap usage of the new thread, and the requirement of > > being > > > able to turn on/off that feature by a managed variable, the best > > change > > > would probably reusing the service thread as you did in the > initial > > > change. > > > > I'm not convinced that this should be handled outside of G1. If > there's > > a need to have the flag manageable at runtime (is that really the > > case?), you could just always start the G1DetectIdleThread and have > it > > check the flag. I wouldn't worry too much about the memory overhead > for > > the stack. > > > > cheers, > > Per > > > > > ------------------------------ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Mon Sep 17 03:52:29 2018 From: david.holmes at oracle.com (David Holmes) Date: Mon, 17 Sep 2018 13:52:29 +1000 Subject: RFC: Piggy-back monitor-deflation and nmethod-marking on GC In-Reply-To: <1517ac6a-c55c-4f4a-9f4d-b3299f2185c1@redhat.com> References: <1517ac6a-c55c-4f4a-9f4d-b3299f2185c1@redhat.com> Message-ID: <2a4c820c-fefd-8355-61c1-949662132901@oracle.com> Hi Roman, On 14/09/2018 5:36 PM, Roman Kennke wrote: > Regarding those issues: > > https://bugs.openjdk.java.net/browse/JDK-8132849 > https://bugs.openjdk.java.net/browse/JDK-8132849 > > I.e. increased pause times caused by monitor-deflation and > nmethod-marking, there has been the suggestion by the reporter to > piggy-back those activities on GC thread-scanning. > > We do have code in Shenandoah that does this since quite a while. > > The idea is to introduce two new methods in VM_Operation: > > virtual bool deflates_idle_monitors(); > virtual bool marks_nmethods(); > > That basically tell the runtime whether or not a particular VM_Operation > (e.g. InitMark) can take over any of the two activties, and if so, skip > the corresponding activity in the preceding SP-cleanup phase. This means > that the VM_Operation needs to handle this itself. The change also > includes some changes in sychronizer, thread etc to make this relatively > easy to implement from a GC standpoint. So the VM_op hands off these tasks to GC threads? This seems a bit "klunky" to me. What you're basically achieving is concurrent cleanup whilst still at a safepoint, but this really has nothing to do with GC - you're just "hijacking" the GC threads because it is convenient. I'd rather see a more concerted effort to avoid having a monitor deflation problem in the first place - as per my comment in JDK-8153224. That is a slightly different, but related, goal to the JEP. Cheers, David > Question: would there be interest in getting this upstreamed? If so, I > would put together a patch for reviewing. > > Alternatively or additionally, at least for monitor deflation, there is > the JEP Draft: Concurrent Monitor Deflation: > https://bugs.openjdk.java.net/browse/JDK-8183909 > > But this seems like a very serious amount of work. > > WDYT? > > Roman > From sangheon.kim at oracle.com Mon Sep 17 18:31:58 2018 From: sangheon.kim at oracle.com (sangheon.kim at oracle.com) Date: Mon, 17 Sep 2018 11:31:58 -0700 Subject: RFR(M): 8204908: NVDIMM for POGC and G1GC - ReserveSpace.cpp changes are mostly eliminated/no collector specific code. In-Reply-To: References: <5df6d080894cfad5e6486a00f28b6ccfc5ca633f.camel@oracle.com> <87e5373f-a2e0-083e-6421-0209f519dca5@oracle.com> <177b91b2-c0ed-d7ad-84ab-ce479f07474a@oracle.com> Message-ID: <0b6a7684-3120-ce80-2069-1a0d163b14ba@oracle.com> Hi Kishor, On 9/4/18 10:41 PM, Kharbas, Kishor wrote: > Hi, > I have uploaded implementation for parallel scavenge at http://cr.openjdk.java.net/~kkharbas/8204908/webrev-8204908.00 > Majority of the implementation is handled in two new files - adjoiningGenerationsForHeteroHeap.cpp and psFileBackedVirtualspace.cpp. Would love to hear your feedback and suggestions. Sorry for late review. ---------------- General comments. 1. Looks like this patch is not based on latest repository, as it fails with -Wreorder issue. 2. I see many wrong alignment location of having only 2 spaces after new line that is continued. e.g. a->b(c, __d, xxxx); // 2 spaces this should be a->b(c, _____d, xxx); // 'd' should locate under 'c' as those are in the same context. 3. I see assertion failures with below options combinations. There could be more... Please run jtreg tests before posting webrev. -XX:+UseLargePages -XX:+UseSHM -version -XX:+UseLargePages -XX:+UseNUMA -version -XX:+UseLargePages -XX:AllocateHeapAt=. -version ========================= src/hotspot/share/gc/parallel/adjoiningGenerations.cpp - Copyright update 43?? _virtual_spaces(new AdjoiningVirtualSpaces(old_young_rs, policy->min_old_size(), 44?????????????????? policy->min_young_size(), alignment) ) { - Wrong alignment? ========================= src/hotspot/share/gc/parallel/adjoiningGenerations.hpp - Copyright update 62?? AdjoiningGenerations(); - Why we need this ctor? ========================= /src/hotspot/share/gc/parallel/adjoiningVirtualSpaces.hpp - Copyright update ? 88?? virtual PSVirtualSpace* high() { return _high; } ? 89?? virtual PSVirtualSpace* low()? { return _low; } - Those methods don't need 'virtual' specifier as high()/low() methods are only used at ctor of AdjoiningGenerations. The other calling site is ctor of AdjoiningGenerationsForHeteroHeap but it is used another type of high()/low() which returns _young_vs or _old_vs. ========================= src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp - no comments. ========================= src/hotspot/share/gc/parallel/psOldGen.cpp ========================= src/hotspot/share/gc/parallel/psOldGen.hpp ? 32 #include "gc/parallel/psFileBackedVirtualspace.hpp" - Include this header file at cpp seems better rather than hpp. ========================= src/hotspot/share/gc/parallel/adjoiningGenerationsForHeteroHeap.cpp ?? 1 /* ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. ?? 3 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. - Wrong alignment. The second/below star should locate under first line star. - Similarily there are more wrong alignment locations. ? . Line 49, 50 ? . 52, 53 ? . 54, 55 ? . 64, 65 ? . 67, 68 69 70 ? . 72, 73 74 75 76 ? . 79, 80 ? . 81, 82 ? . 85, 86 ? . 87, 88 ? . 106, 107 108 109 ? . 114, 115 ? . 166, 167 168 ? . 178, 179 180 ? . 186, 187 188 ? . 218, 219 220 ? . 230, 231 232 ? . 239, 240 241 ? 59?? // Initialize the virtual spaces.? Then pass the ? 60?? // a virtual to each generation for initialization of the - Then pass 'the a' virtual to each generation. 'the a'? ? 64?? (static_cast (_virtual_spaces))->initialize(max_old_byte_size, init_old_byte_size, ? 65???? init_young_byte_size); - Just making 'initilize' method as 'virtual' seems better. ? 39 AdjoiningGenerationsForHeteroHeap::AdjoiningGenerationsForHeteroHeap(ReservedSpace old_young_rs, size_t total_size_limit, ? 40?? GenerationSizer* policy, size_t alignment) : _total_size_limit(total_size_limit) { - Wrong alirnment at line 40. 'Generation' should be under 'ReservedSpace' at line 39. ? 49?? _virtual_spaces = new HeteroVirtualSpaces(old_young_rs, min_old_byte_size,? 70???? (static_cast (_virtual_spaces))->max_young_size()); ? 75???? (static_cast (_virtual_spaces))->max_old_size(), - Instead assigning at line 49 and then cast to HeteroVirtualSpaces*, create/initialize HeteroVirtualSpaces, get max_young/old_size() and the assign to _virtual_spaces would be better alternative. ?109?? _min_young_byte_size(min_yg_byte_size), _max_total_size(max_total_size) { ?110?? _max_old_byte_size = _max_total_size - _min_young_byte_size; ?111?? _max_young_byte_size = _max_total_size - _min_old_byte_size; - _max_old_byte_size and _max_young_byte_size can move to initialization list. ?117?? // This the reserved space exclusively for old generation. ?122?? // This the reserved space exclusively for young generation. - Missing 'is'? i.e. // This *is* the reserved space exclusively for old generation. ?131???? vm_exit_during_initialization("Could not reserve enough space for " ?132?????? "object heap"); - 'object heap' can stay same line. Or changing align is necessary. ?137???? vm_exit_during_initialization("Could not reserve enough space for " ?138?????? "object heap"); - 'object heap' can stay same line. Or changing align is necessary. ?152?? if (uncommitted_in_old > 0) { - This condition seems redundant as uncommitted_in_old is type of size_t. ?159?? if (bytes_needed == 0) { ?160???? return true; ?161?? } - This condition, can move to inside of 'if (uncommitted_in_old > 0)' condition because if uncommitted_in_old is zero, there's no way bytes_needed to be zero - bytes_needed is guaranteed not to be zero from caller site, so safe to move between line 154 and 155. The condition to return true is 'uncommitted_size == bytes_needed && success of expand_by()'. ?176???? bool ret = _young_vs->shrink_by(shrink_size); ?177???? assert(ret, "We should be able to shrink young space"); - I would prefer to add more conditions below if we fails to shrink. i.e. just assert seems not enough. ?189?? _old_vs->expand_by(bytes_to_add_in_old); - Check the return value of expand_by(). ?211?? if (bytes_needed == 0) { ?212???? return true; ?213?? } - Same as 'adjust_boundary_down()' ?244?? DEBUG_ONLY(size_t total_size_after = _young_vs->reserved_size() + _old_vs->reserved_size()); ?245?? assert(total_size_after == total_size_before, "should be equal"); - Why adjust_boundary_up() doesn't have this kind of check? ========================= src/hotspot/share/gc/parallel/adjoiningGenerationsForHeteroHeap.hpp ?? 1 /* ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. ?? 3 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. ?? 4 * - Wrong alignment. The second/below star should locate under first line star. ? 37?? size_t total_size_limit() { ? 38???? return _total_size_limit; ? 39?? } - I don't think we need this but if you prefer to have, add 'const'. ? 45???? PSVirtualSpace*??? _young_vs; ? 46???? PSVirtualSpace*??? _old_vs; - Can't we use 'AdjoiningVirtualSpaces::_low' and '_high' instead? If it is not the case, adding high(),low() may result in confusion between AdjoiningVirtualSpaces::high() and low(). So use other name for current HeteroVirtualSpaces::high()/low(). ? 55???? HeteroVirtualSpaces(ReservedSpace rs, ? 56?????? size_t min_old_byte_size, ? 57?????? size_t min_young_byte_size, size_t max_total_size, ? 58?????? size_t alignment); - Wrong alignment. Line 56 ~ 58 should be same as the location of 'ReservedSpace' at line 55. ? 67???? size_t max_young_size() { return _max_young_byte_size; } ? 68???? size_t max_old_size() { return _max_old_byte_size; } - 'const'? ? 70???? void initialize(size_t initial_old_reserved_size, size_t init_low_byte_size, ? 71?????? size_t init_high_byte_size); - Wrong alignment ? 82?? size_t reserved_byte_size(); - 'const'? ========================= ?src/hotspot/share/gc/parallel/psFileBackedVirtualspace.cpp ?? 1 /* ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. - Wrong alignment. The second/below star should locate under first line star. ? 36?? if (_fd == -1) { ? 37???? return; ? 38?? } - I prefer to have 'initialize()' method to handle when fails to create the heap file. Current implementation seems easy to call additional 'initialize()'. ? Ctor of PSVirtualSpace doesn't have any failure path(e.g. allocation), so no further check is needed around line 18:psOldGen.cpp. But yours is different, so something should be checked. ? 33?? _mapping_succeeded = false; ? 34?? _file_path = path; ? 46?? _mapping_succeeded = true; ? 47?? _special = true; - Can move to initialization list with proper initial value. ? 55?? assert(special(), "_special should always be true for PSFileBackedVirtualSpace"); ? 66?? assert(special(), "_special should always be true for PSFileBackedVirtualSpace"); - For these 2, can we have more specific message? e.g. to include meaning of expand or shrink. ========================= ?src/hotspot/share/gc/parallel/psFileBackedVirtualspace.hpp ?? 1 /* ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. - Wrong alignment. The second/below star should locate under first line star. ? 37?? PSFileBackedVirtualSpace(ReservedSpace rs, const char* file_path); ? 38?? bool?? expand_by(size_t bytes); - Just recommendation to add new line between 37 and 38. ? 43 #endif // SHARE_VM_GC_PARALLEL_PSFILEBACKEDVIRTUALSPACE_HPP - Last line is not terminated with a newline. Thanks, Sangheon > > I will post G1 GC implementation in a few days. > > Thanks > Kishor. > >> -----Original Message----- >> From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] >> Sent: Thursday, August 30, 2018 4:06 PM >> To: Kharbas, Kishor ; Thomas St?fe >> ; Thomas Schatzl >> >> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime > runtime-dev at openjdk.java.net>; Viswanathan, Sandhya >> ; Aundhe, Shirish >> >> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - >> ReserveSpace.cpp changes are mostly eliminated/no collector specific code. >> >> Hi Kishor, >> >> On 8/30/18 11:55 AM, Kharbas, Kishor wrote: >>> Hi Sangheon, >>> >>> So far I don?t see a need to do so. I will post my implementation code >> today or tomorrow, if we see a need or any simplification by changing >> classes in VirtualSpace.hpp, we can discuss that. >> Okay. >> >> Thanks, >> Sangheon >> >> >>> Regards, >>> -Kishor >>> >>>> -----Original Message----- >>>> From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] >>>> Sent: Wednesday, August 29, 2018 10:17 AM >>>> To: Kharbas, Kishor ; Thomas St?fe >>>> ; Thomas Schatzl >> >>>> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime >>> runtime-dev at openjdk.java.net>; Viswanathan, Sandhya >>>> ; Aundhe, Shirish >>>> >>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - >>>> ReserveSpace.cpp changes are mostly eliminated/no collector specific >> code. >>>> Hi Kishor, >>>> >>>> On 8/29/18 9:52 AM, Kharbas, Kishor wrote: >>>>> Hi Sangheon, >>>>> >>>>> Thanks for reviewing the design. >>>>> Since the collectors do not use them for heap memory, I did not have >>>>> to override VirtualSpace >>>> Sorry, I meant ReservedSpace and its friends at >>>> share/memory/virtualspace.hpp. >>>> >>>> Thanks, >>>> Sangheon >>>> >>>> >>>>> -Kishor >>>>>> -----Original Message----- >>>>>> From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] >>>>>> Sent: Tuesday, August 28, 2018 2:38 PM >>>>>> To: Kharbas, Kishor ; Thomas St?fe >>>>>> ; Thomas Schatzl >>>> >>>>>> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime >>>>> runtime-dev at openjdk.java.net>; Viswanathan, Sandhya >>>>>> ; Aundhe, Shirish >>>>>> >>>>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - >>>>>> ReserveSpace.cpp changes are mostly eliminated/no collector >>>>>> specific >>>> code. >>>>>> Hi Kishor, >>>>>> >>>>>> On 8/21/18 10:57 AM, Kharbas, Kishor wrote: >>>>>>> Hi All, >>>>>>> >>>>>>> Thank you for your valuable comments and feedback in this thread >>>>>>> so >>>> far. >>>>>>> After taken in all the comments and reading thoroughly through the >>>>>>> code, I >>>>>> feel that the complexity added to virtualSpace.cpp is due to lack >>>>>> of abstraction in VirtualSpace and at GC level. NV-DIMM size and >>>>>> file paths are being passed all the way to OS calls. >>>>>>> So I went back to the drawing board and created a high level >>>>>>> design to remove all the pain points of current implementation. I >>>>>>> have uploaded the design at >>>>>>> >> http://cr.openjdk.java.net/~kkharbas/8202286/Design%20for%20JEP%20JDK >>>>>> - >>>>>>> 8202286.pdf I would love to hear your feedback and suggestions. >>>>>>> Once we get confidence in the design, I will work on the >> implementation. >>>>>> The design looks good to me. >>>>>> If you are planning to change VirtualSpace at >>>>>> share/memory/virtualspace.hpp, including it on the design document >>>>>> would be helpful too. >>>>>> >>>>>> Probably folks involved in the previous discussions would say >>>>>> whether the design well covers their concerns or not. >>>>>> >>>>>> Thanks, >>>>>> Sangheon >>>>>> >>>>>> >>>>>>> PS: >>>>>>> 1. Vinay has transitioned to another team in Intel, so he won't be >>>>>>> able to >>>>>> continue on this jep. >>>>>>> 2. If you feel this should be part of JEP discussion thread, we >>>>>>> can take it >>>>>> there. >>>>>>> Thanks and regards, >>>>>>> Kishor >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Thomas St?fe [mailto:thomas.stuefe at gmail.com] >>>>>>>> Sent: Friday, June 22, 2018 9:25 AM >>>>>>>> To: Thomas Schatzl >>>>>>>> Cc: Awasthi, Vinay K ; Paul Su >>>>>>>> ; hotspot-gc-dev at openjdk.java.net; Hotspot >>>> dev >>>>>>>> runtime ; Viswanathan, >>>>>> Sandhya >>>>>>>> ; Aundhe, Shirish >>>>>>>> ; Kharbas, Kishor >>>>>>>> >>>>>>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - >>>>>>>> ReserveSpace.cpp changes are mostly eliminated/no collector >>>>>>>> specific >>>>>> code. >>>>>>>> Hi Thomas, >>>>>>>> >>>>>>>> On Fri, Jun 22, 2018 at 4:44 PM, Thomas Schatzl >>>>>>>> wrote: >>>>>>>>> Hi Thomas, >>>>>>>>> >>>>>>>>> On Tue, 2018-06-19 at 13:40 +0200, Thomas St?fe wrote: >>>>>>>>>> Hi Vinay, >>>>>>>>>> >>>>>>>>>> On Mon, Jun 18, 2018 at 6:47 PM, Awasthi, Vinay K >>>>>>>>>> wrote: >>>>>>>>>>> Hi Thomas, >>>>>>>>>>> >>>>>>>>>>> Os::commit_memory calls map_memory_to_file which is same >> as >>>>>>>>>>> os::reserve_memory. >>>>>>>>>>> >>>>>>>>>>> I am failing to see why os::reserve_memory can call >>>>>>>>>>> map_memory_to_file (i.e. tie it to mmap) but commit_memory >>>>>> can't... >>>>>>>>>>> Before this patch, commit_memory never dealt with >>>>>>>>>>> incrementally committing pages to device so there has to be a >>>>>>>>>>> way to pass file descriptor and offset. Windows has no such >>>>>>>>>>> capability to manage incremental commits. All other OSes do >>>>>>>>>>> and that is why map_memory_to_file is used (which by the way >>>>>>>>>>> also works on Windows). >>>>>>>>>> AIX uses System V shared memory by default, which follows a >>>>>>>>>> different allocation scheme (semantics more like Windows >>>>>> VirtualAlloc... >>>>>>>>>> calls). >>>>>>>>>> >>>>>>>>>> But my doubts are not limited to that one, see my earlier >>>>>>>>>> replies and those of others. It really makes sense to step back >>>>>>>>>> one step and discuss the JEP first. >>>>>>>>>> >>>>>>>>> There is already a discussion thread as David mentioned >>>>>>>>> (http://mail.op >>>>>>>>> enjdk.java.net/pipermail/hotspot-gc-dev/2018-May/022092.html) >>>> that >>>>>>>>> so far nobody answered to. >>>>>>>>> >>>>>>>> Ah, I thought he wanted to have the JEP discussed in the comments >>>>>>>> section of the JEP itself. >>>>>>>> >>>>>>>>> I believe discussion about contents the JEP and the >>>>>>>>> implementation should be separate. >>>>>>>>> >>>>>>>> makes sense. >>>>>>>> >>>>>>>>> So far what I gathered from the responses to the implementation, >>>>>>>>> the proposed idea itself is not the issue here (allowing the use >>>>>>>>> of NVDIMM memory for parts of the heap for allowing the use of >>>>>>>>> larger heaps to improve overall performance; I am not saying >>>>>>>>> that the text doesn't need a bit more work :) ), but rather how >>>>>>>>> an implementation of this JEP should proceed. >>>>>>>> I have no problem with adding NVDIMM support. I think it is a >>>>>>>> cool >>>>>> feature. >>>>>>>> Also, the awkwardness off the memory management abstraction >> layer >>>>>>>> in hotspot has always been a sore point with me (I originally >>>>>>>> implemented the AIX mm layer in os_aix.cpp, and those are painful >>>>>>>> memories). So, I have a lot of sympathy for Vinays struggles. >>>>>>>> Unfortunately not much time atm, but I will respond to your mail. >>>>>>>> >>>>>>>>> Let's discuss the non-implementation stuff in that thread. Vinay >>>>>>>>> or I can repost the proposal email if you do not have it any >>>>>>>>> more so that answers will be in-thread. >>>>>>>>> >>>>>>>> Okay, sounds good. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Thomas >>>>>>>> >>>>>>>> (one of us should change his name to make this less confusing :-) >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Thomas >>>>>>>>> From rkennke at redhat.com Tue Sep 18 07:58:42 2018 From: rkennke at redhat.com (Roman Kennke) Date: Tue, 18 Sep 2018 09:58:42 +0200 Subject: RFR: JDK-8210829: Modularize allocations in C2 Message-ID: <8fe187e2-b3fb-5473-257f-e4f6b6a94fcb@redhat.com> Similar to what we've done before to runtime, interpreter and C1, allocations should be owned and implemented by GC, and possible to override by specific collectors. For example, Shenandoah lays out objects differently in heap, and needs one extra store to initialize objects. This proposed change factors out the interesting part of object allocation (i.e. the actual allocation) into BarrierSetC2. It's mostly a move-and-rename-job. I had to move some little things around, that is: - for the need-gc-check, I'm passing back the needgc_ctrl to plug into slow-path - for prefetching, instead of passing around the 'length' node, only to determine the number of prefetch lines, I determine this early, and pass through the lines arg. - i_o, needgc_ctrl, fast_oop_ctrl, fast_oop_rawmem are passed as in/out or out-args to stitch together into the regions and phis as appropriate. I see no easy way around that. I tested this using hotspot/jtreg:tier1 and also verified that this fills Shenandoah's needs and run tier3_gc_shenandoah testsuite. http://cr.openjdk.java.net/~rkennke/JDK-8210829/webrev.00/ Can I please get reviews? Thanks, Roman -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From leo.korinth at oracle.com Tue Sep 18 15:10:46 2018 From: leo.korinth at oracle.com (Leo Korinth) Date: Tue, 18 Sep 2018 17:10:46 +0200 Subject: RFR: 8196341: Add JFR events for parallel phases of G1 In-Reply-To: <8c0e4d82-0c87-da99-328e-548b8d7c12dd@oracle.com> References: <8c0e4d82-0c87-da99-328e-548b8d7c12dd@oracle.com> Message-ID: <510a6229-2e9b-5d83-4e8a-740d55758206@oracle.com> Hi, could I get another review on this? Thanks, Leo On 05/09/18 14:27, Leo Korinth wrote: > Hi, > > I have added JFR events for the parallel phases of G1. > > This change adds JFR events for most values of the enumeration > GCParPhases. I have also added a test case (TestG1ParallelPhases.java). > > This is a rebase/rework of a previous post I did 31th of January that > never got pushed. > > Enhancement: > https://bugs.openjdk.java.net/browse/JDK-8196341 > > Webrev: > http://cr.openjdk.java.net/~lkorinth/8196341/00/ > > Testing: > - mach5 hs-tier1,hs-tier2,hs-tier3 > - brief visual inspection of result in JMC > - new test TestG1ParallelPhases.java (100 runs) > > Thanks, > Leo From vladimir.kozlov at oracle.com Tue Sep 18 17:41:51 2018 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Tue, 18 Sep 2018 10:41:51 -0700 Subject: RFR: JDK-8210829: Modularize allocations in C2 In-Reply-To: <8fe187e2-b3fb-5473-257f-e4f6b6a94fcb@redhat.com> References: <8fe187e2-b3fb-5473-257f-e4f6b6a94fcb@redhat.com> Message-ID: <09c53a34-eb68-a614-ee03-468673004b3b@oracle.com> Hi Roman, This looks good. I looked through changes and it generates the same ideal graph as before. It seems you unintentionally changed indent of the comment in barrierSetC2.hpp Thanks, Vladimir On 9/18/18 12:58 AM, Roman Kennke wrote: > Similar to what we've done before to runtime, interpreter and C1, > allocations should be owned and implemented by GC, and possible to > override by specific collectors. For example, Shenandoah lays out > objects differently in heap, and needs one extra store to initialize > objects. > > This proposed change factors out the interesting part of object > allocation (i.e. the actual allocation) into BarrierSetC2. It's mostly a > move-and-rename-job. I had to move some little things around, that is: > - for the need-gc-check, I'm passing back the needgc_ctrl to plug into > slow-path > - for prefetching, instead of passing around the 'length' node, only to > determine the number of prefetch lines, I determine this early, and pass > through the lines arg. > - i_o, needgc_ctrl, fast_oop_ctrl, fast_oop_rawmem are passed as in/out > or out-args to stitch together into the regions and phis as appropriate. > I see no easy way around that. > > I tested this using hotspot/jtreg:tier1 and also verified that this > fills Shenandoah's needs and run tier3_gc_shenandoah testsuite. > > http://cr.openjdk.java.net/~rkennke/JDK-8210829/webrev.00/ > > Can I please get reviews? > Thanks, > Roman > From sangheon.kim at oracle.com Tue Sep 18 19:14:22 2018 From: sangheon.kim at oracle.com (sangheon.kim at oracle.com) Date: Tue, 18 Sep 2018 12:14:22 -0700 Subject: RFR: 8196341: Add JFR events for parallel phases of G1 In-Reply-To: <510a6229-2e9b-5d83-4e8a-740d55758206@oracle.com> References: <8c0e4d82-0c87-da99-328e-548b8d7c12dd@oracle.com> <510a6229-2e9b-5d83-4e8a-740d55758206@oracle.com> Message-ID: <4c5c40d9-97e5-afa4-0503-565dd4a66686@oracle.com> Hi Leo, On 9/18/18 8:10 AM, Leo Korinth wrote: > Hi, could I get another review on this? > > Thanks, Leo > > On 05/09/18 14:27, Leo Korinth wrote: >> Hi, >> >> I have added JFR events for the parallel phases of G1. >> >> This change adds JFR events for most values of the enumeration >> GCParPhases. I have also added a test case (TestG1ParallelPhases.java). >> >> This is a rebase/rework of a previous post I did 31th of January that >> never got pushed. >> >> Enhancement: >> https://bugs.openjdk.java.net/browse/JDK-8196341 >> >> Webrev: >> http://cr.openjdk.java.net/~lkorinth/8196341/00/ This looks good to me. And agree to Thomas about scoped objects suggestion. Thanks, Sangheon >> >> Testing: >> - mach5 hs-tier1,hs-tier2,hs-tier3 >> - brief visual inspection of result in JMC >> - new test TestG1ParallelPhases.java (100 runs) >> >> Thanks, >> Leo From per.liden at oracle.com Tue Sep 18 20:13:56 2018 From: per.liden at oracle.com (Per Liden) Date: Tue, 18 Sep 2018 22:13:56 +0200 Subject: RFR: 8210857: Allow retiring TLABs and collecting statistics in parallel Message-ID: There is currently no good way of retiring TLABs and collecting their statistics in parallel. The current code for doing this also traverses the thread list twice, which is unnecessary. I propose that we adjust the TLAB API to allow this to be done in parallel, in a single pass. A few notes on this patch: * The GlobalTLABStats class is restructured into the ThreadLocalAllocStats class to allow for a more flexible way of collected statistics when retiring TLABs (flexible in the sense that it allows for parallel operations). * The TLAB API is slightly adjusted. The "make_parsable(bool retire_tlab)" is broken into two functions, "make_parsable()" to just make a TLAB parsable, and "retire(ThreadLocalAllocStats* stats)" to retire and optionally collects statistics. * GCs are unaffected by this change, in the sense that they continue to call CollectedHeap::ensure_parsability(). However, all GCs will benefit from now doing a single pass over the thread list instead of two to retire and collect stats. * This is the last patch in my series of TLAB related patches. The remaining patches only touch ZGC to actually retire/resize/remap TLABs in parallel. Bug: https://bugs.openjdk.java.net/browse/JDK-8210857 Webrev: http://cr.openjdk.java.net/~pliden/8210857/webrev.0 /Per From per.liden at oracle.com Tue Sep 18 20:16:27 2018 From: per.liden at oracle.com (Per Liden) Date: Tue, 18 Sep 2018 22:16:27 +0200 Subject: RFR: 8210857: Allow retiring TLABs and collecting statistics in parallel In-Reply-To: References: Message-ID: <33ca946c-0803-a5cb-41f2-e4fa55de34ff@oracle.com> On 09/18/2018 10:13 PM, Per Liden wrote: > There is currently no good way of retiring TLABs and collecting their > statistics in parallel. The current code for doing this also traverses > the thread list twice, which is unnecessary. I propose that we adjust > the TLAB API to allow this to be done in parallel, in a single pass. > > A few notes on this patch: > > * The GlobalTLABStats class is restructured into the > ThreadLocalAllocStats class to allow for a more flexible way of > collected statistics when retiring TLABs (flexible in the sense that it > allows for parallel operations). > > * The TLAB API is slightly adjusted. The "make_parsable(bool > retire_tlab)" is broken into two functions, "make_parsable()" to just > make a TLAB parsable, and "retire(ThreadLocalAllocStats* stats)" to > retire and optionally collects statistics. > > * GCs are unaffected by this change, in the sense that they continue to > call CollectedHeap::ensure_parsability(). However, all GCs will benefit > from now doing a single pass over the thread list instead of two to > retire and collect stats. > > * This is the last patch in my series of TLAB related patches. The > remaining patches only touch ZGC to actually retire/resize/remap TLABs > in parallel. > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210857 > Webrev: http://cr.openjdk.java.net/~pliden/8210857/webrev.0 Testing: Passed tier{1,2,3,4,5,6} on linux-x86_64 (product and debug) /Per From per.liden at oracle.com Tue Sep 18 20:22:04 2018 From: per.liden at oracle.com (Per Liden) Date: Tue, 18 Sep 2018 22:22:04 +0200 Subject: RFR: 8210881: ZGC: Introduce ZRootsIteratorClosure Message-ID: <0a7125af-59a0-34cf-42b5-b44f72d84735@oracle.com> Introduce ZRootsIteratorClosure to allow piggybacking per-thread operations when iterating over thread roots. Use this to update the thread local address bad mask, instead of having that logic buried inside the ZRootIterator, which should not have knowledge of this. This mechanism will later also be used to retire/resize/remap TLABs. Bug: https://bugs.openjdk.java.net/browse/JDK-8210881 Webrev: http://cr.openjdk.java.net/~pliden/8210881/webrev.0 /Per From per.liden at oracle.com Tue Sep 18 20:24:28 2018 From: per.liden at oracle.com (Per Liden) Date: Tue, 18 Sep 2018 22:24:28 +0200 Subject: RFR: 8210883: ZGC: Parallel retire/resize/remap of TLABs Message-ID: <655f86ce-fde9-973d-46e2-a36ff39980fa@oracle.com> Let ZGC retire/resize/remap TLABs in parallel. This change builds on JDK-8210857 - "Allow retiring TLABs and collecting statistics in parallel?", which is currently also out on review. Bug: https://bugs.openjdk.java.net/browse/JDK-8210883 Webrev: http://cr.openjdk.java.net/~pliden/8210883/webrev.0 /Per From per.liden at oracle.com Tue Sep 18 21:01:47 2018 From: per.liden at oracle.com (Per Liden) Date: Tue, 18 Sep 2018 23:01:47 +0200 Subject: RFR: 8210884: ZGC: Remove insertion of filler objects Message-ID: <42d20236-4358-ff0a-deaf-ff3ca6c524a6@oracle.com> Remove insertion of filler objects since we don't depend or use those in any way, since ZGC does not depend on having a parsable heap. Bug: https://bugs.openjdk.java.net/browse/JDK-8210884 Webrev: http://cr.openjdk.java.net/~pliden/8210884/webrev.0 /Per From rwestrel at redhat.com Tue Sep 18 21:32:58 2018 From: rwestrel at redhat.com (Roland Westrelin) Date: Tue, 18 Sep 2018 23:32:58 +0200 Subject: RFR: JDK-8210752: Remaining explicit barriers for C2 In-Reply-To: <15be8e2d-dba5-2e8a-c851-b6821b81d4b3@redhat.com> References: <15be8e2d-dba5-2e8a-c851-b6821b81d4b3@redhat.com> Message-ID: > http://cr.openjdk.java.net/~rkennke/JDK-8210752/webrev.00/ That looks good to me. Roland. From rwestrel at redhat.com Tue Sep 18 21:39:00 2018 From: rwestrel at redhat.com (Roland Westrelin) Date: Tue, 18 Sep 2018 23:39:00 +0200 Subject: RFR: JDK-8210829: Modularize allocations in C2 In-Reply-To: <8fe187e2-b3fb-5473-257f-e4f6b6a94fcb@redhat.com> References: <8fe187e2-b3fb-5473-257f-e4f6b6a94fcb@redhat.com> Message-ID: > http://cr.openjdk.java.net/~rkennke/JDK-8210829/webrev.00/ That looks good to me. Roland. From leo.korinth at oracle.com Wed Sep 19 08:19:58 2018 From: leo.korinth at oracle.com (Leo Korinth) Date: Wed, 19 Sep 2018 10:19:58 +0200 Subject: RFR: 8196341: Add JFR events for parallel phases of G1 In-Reply-To: <4c5c40d9-97e5-afa4-0503-565dd4a66686@oracle.com> References: <8c0e4d82-0c87-da99-328e-548b8d7c12dd@oracle.com> <510a6229-2e9b-5d83-4e8a-740d55758206@oracle.com> <4c5c40d9-97e5-afa4-0503-565dd4a66686@oracle.com> Message-ID: <52f8aa0e-7453-3fb3-2f68-13cdc462724d@oracle.com> Thank you for the review Sangheon. /Leo On 18/09/18 21:14, sangheon.kim at oracle.com wrote: > Hi Leo, > > On 9/18/18 8:10 AM, Leo Korinth wrote: >> Hi, could I get another review on this? >> >> Thanks, Leo >> >> On 05/09/18 14:27, Leo Korinth wrote: >>> Hi, >>> >>> I have added JFR events for the parallel phases of G1. >>> >>> This change adds JFR events for most values of the enumeration >>> GCParPhases. I have also added a test case (TestG1ParallelPhases.java). >>> >>> This is a rebase/rework of a previous post I did 31th of January that >>> never got pushed. >>> >>> Enhancement: >>> https://bugs.openjdk.java.net/browse/JDK-8196341 >>> >>> Webrev: >>> http://cr.openjdk.java.net/~lkorinth/8196341/00/ > This looks good to me. > And agree to Thomas about scoped objects suggestion. > > Thanks, > Sangheon > > >>> >>> Testing: >>> - mach5 hs-tier1,hs-tier2,hs-tier3 >>> - brief visual inspection of result in JMC >>> - new test TestG1ParallelPhases.java (100 runs) >>> >>> Thanks, >>> Leo > From stefan.johansson at oracle.com Wed Sep 19 08:37:31 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Wed, 19 Sep 2018 10:37:31 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: <57b10708-b870-cb98-0df0-fab650f431e4@oracle.com> References: <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> <65063e5f31702649e12b485b570ecab9ebd32d01.camel@oracle.com> <66c536399b2f218af0d85bd3bced44f204e3dbf4.camel@oracle.com> <2821cf70-c4d5-016f-b976-5ecc850f377c@oracle.com> <57b10708-b870-cb98-0df0-fab650f431e4@oracle.com> Message-ID: Hi Rodrigo, I pasted your reply here to keep the discussion in one thread. > I understand that it is hard to define what is idle. However, if we require the > user to provide one, I guess that most regular users that suffer from the problem > that this patch is trying to solve will simply not do it because it requires knowledge > and effort. If we provide an idle check that we think will benefit most users, then > we are probably helping a lot of users. For those that the default idle check is > not good enough, they can always disable this idle check and implement the idle > check logic it in an external tool. > I agree, if we can find a solution that benefits most users, we should do it. And this is why I would like to hear from more users if this would benefit their use cases. Another thing that I don't fully understand is why the flags are manageable if there isn't supposed to be some external logic that sets them? > We can also change the semantics of "idleness". Currently it checks the load. > I think that checking the allocation rate might be another good option (instead of > load). The only corner case is an application that does not allocate but consumes > a lot of CPU. For this case, we might only trigger compaction at most once because, > as it does not allocate memory, we will not get over committed memory (i.e., the other > checks will prevent it). The opposite is also possible (almost idle application that allocates > a lot of memory) but in this scenario I don't think we want to trigger an idle compaction. > This is my main problem when it comes to determine "idleness", for some applications allocation rate will be the correct metric, for others it will be the load and for a third something different. It feels like it is always possible to come up with a case that needs something different. > Having said that, I am open to change this flag or even remove it as it is one of the > hardest to get right. > As I said before, to me it feels like just having a periodic GC interval flag that is manageable would be a good start. Maybe have constraint that the periodic GC only occurs if no other GCs have happened during the interval. Could you explain how your use case would suffer from such limitations? Thanks, Stefan > cheers, > rodrigo On 2018-09-13 14:30, Stefan Johansson wrote: > Hi Rodrigo, > > Sorry for being a bit late into the discussion. We've had some internal > discussions and realized that there are some questions that I need to > bring up here. > > I'm trying to better understand under what circumstances this feature is > to be used and how a user should use the different flags to tweak it to > their use case. To me it feels like GCFrequency would be enough to make > sure that the VM returns memory on a timely basis. And if the flag is > managed, it can be controlled to not do periodic GCs during high load. > With that we get a good way to periodically try to reduce the committed > heap. > > The reason I ask is because I have a hard time seeing how we can > implement a generic policy for when the system is idle. A policy that > will apply well to most use cases. For some cases having the flags you > propose might be good, but for other there might be a different set of > options needed. If this is the case then maybe the logic and policy of > when to do this can live outside the VM, while the code to periodically > do GCs lives within the VM. What do you think about that? I understand > the problems you've stated with having the policy outside that VM, but > at least we have more information to act on there. > > We know that many have asked for features similar to this one and it > would be nice to get input from others on this to make sure we implement > something that benefits the whole user base as much as possible. So > anyone with a use case that could benefit from this, please chime in. > > Regards, > Stefan > > > > On 2018-09-07 17:37, Rodrigo Bruno wrote: >> Hi Per and Thomas, >> >> thank you for your comments. >> >> I think it is possible to implement this feature using the service >> thread or using a separate thread. >> I see some pros and cons of having a separate thread: >> >> Pros: >> - using the service thread exposes something that is G1 specific to >> the rest of the JVM. >> Thus, using a separate thread, hides this feature from the outsite. >> >> Cons: >> - Having a manageable timeout is a bit more tricky to implement in a >> separate/dedicated thread. >> We need to be able to handle switch on and off. It might require some >> variable pooling. >> - It requires some more memory. >> >> Regardless of the path taken, I can prepare a new version of the patch >> whenever we decide on this. >> >> cheers, >> rodrigo >> >> Per Liden > >> escreveu no dia sexta, 7/09/2018 ?(s) 11:58: >> >> ??? Hi Thomas, >> >> ??? On 09/07/2018 10:10 AM, Thomas Schatzl wrote: >> ??? [...] >> ???? >? ? overnight I thought a bit of the implementation, and given the >> ???? > problem with heap usage of the new thread, and the requirement of >> ??? being >> ???? > able to turn on/off that feature by a managed variable, the best >> ??? change >> ???? > would probably reusing the service thread as you did in the >> initial >> ???? > change. >> >> ??? I'm not convinced that this should be handled outside of G1. If >> there's >> ??? a need to have the flag manageable at runtime (is that really the >> ??? case?), you could just always start the G1DetectIdleThread and >> have it >> ??? check the flag. I wouldn't worry too much about the memory >> overhead for >> ??? the stack. >> >> ??? cheers, >> ??? Per >> From rkennke at redhat.com Wed Sep 19 09:40:31 2018 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 19 Sep 2018 11:40:31 +0200 Subject: RFR: JDK-8210829: Modularize allocations in C2 In-Reply-To: <09c53a34-eb68-a614-ee03-468673004b3b@oracle.com> References: <8fe187e2-b3fb-5473-257f-e4f6b6a94fcb@redhat.com> <09c53a34-eb68-a614-ee03-468673004b3b@oracle.com> Message-ID: Thanks, Vladimir! I'll fix the comment and push it through jdk/submit before pushing to jdk/jdk. Roman > Hi Roman, > > This looks good. I looked through changes and it generates the same > ideal graph as before. > It seems you unintentionally changed indent of the comment in > barrierSetC2.hpp > > Thanks, > Vladimir > > On 9/18/18 12:58 AM, Roman Kennke wrote: >> Similar to what we've done before to runtime, interpreter and C1, >> allocations should be owned and implemented by GC, and possible to >> override by specific collectors. For example, Shenandoah lays out >> objects differently in heap, and needs one extra store to initialize >> objects. >> >> This proposed change factors out the interesting part of object >> allocation (i.e. the actual allocation) into BarrierSetC2. It's mostly a >> move-and-rename-job. I had to move some little things around, that is: >> - for the need-gc-check, I'm passing back the needgc_ctrl to plug into >> slow-path >> - for prefetching, instead of passing around the 'length' node, only to >> determine the number of prefetch lines, I determine this early, and pass >> through the lines arg. >> - i_o, needgc_ctrl, fast_oop_ctrl, fast_oop_rawmem are passed as in/out >> or out-args to stitch together into the regions and phis as appropriate. >> I see no easy way around that. >> >> I tested this using hotspot/jtreg:tier1 and also verified that this >> fills Shenandoah's needs and run tier3_gc_shenandoah testsuite. >> >> http://cr.openjdk.java.net/~rkennke/JDK-8210829/webrev.00/ >> >> Can I please get reviews? >> Thanks, >> Roman >> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From stefan.johansson at oracle.com Wed Sep 19 11:37:45 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Wed, 19 Sep 2018 13:37:45 +0200 Subject: RFR: 8210713: Let CollectedHeap::ensure_parsability() take care of TLAB statistics gathering In-Reply-To: References: Message-ID: <4bee595d-eb87-9083-2573-2105d59f0b5a@oracle.com> On 2018-09-13 13:46, Per Liden wrote: > A call to CollectedHeap::ensure_parsability(retire_tlabs), where > retire_tlabs is true must always be preceded by a call to > CollectedHeap::accumulate_statistics_all_tlabs(), otherwise we will > loose TLAB statistics as it will be reset when the TLABs are retired. > Instead of having all collectors do this dance and risk getting it > wrong, I propose that we move this responsibility into > ensure_parsability(), which will gather the TLAB statistics if it's > told to retire all TLABs. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210713 > Webrev: http://cr.openjdk.java.net/~pliden/8210713/webrev.0 > Looks good, Stefan > /Per From per.liden at oracle.com Wed Sep 19 12:07:53 2018 From: per.liden at oracle.com (Per Liden) Date: Wed, 19 Sep 2018 14:07:53 +0200 Subject: RFR: 8210713: Let CollectedHeap::ensure_parsability() take care of TLAB statistics gathering In-Reply-To: <4bee595d-eb87-9083-2573-2105d59f0b5a@oracle.com> References: <4bee595d-eb87-9083-2573-2105d59f0b5a@oracle.com> Message-ID: <74307929-8242-fdae-4aea-2201529dd484@oracle.com> Thanks Stefan! /Per On 09/19/2018 01:37 PM, Stefan Johansson wrote: > > > On 2018-09-13 13:46, Per Liden wrote: >> A call to CollectedHeap::ensure_parsability(retire_tlabs), where >> retire_tlabs is true must always be preceded by a call to >> CollectedHeap::accumulate_statistics_all_tlabs(), otherwise we will >> loose TLAB statistics as it will be reset when the TLABs are retired. >> Instead of having all collectors do this dance and risk getting it >> wrong, I propose that we move this responsibility into >> ensure_parsability(), which will gather the TLAB statistics if it's >> told to retire all TLABs. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210713 >> Webrev: http://cr.openjdk.java.net/~pliden/8210713/webrev.0 >> > Looks good, > Stefan >> /Per > From zgu at redhat.com Wed Sep 19 13:30:57 2018 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 19 Sep 2018 09:30:57 -0400 Subject: RFR(XXS) 8210879: ClassLoaderStatsClosure does raw oop comparison Message-ID: <1729861c-8fcd-4b16-8bed-ce4e44ebe270@redhat.com> Hi, Please review this small change. GCs, such as Shenandoah, need proper barrier for oop comparison. Bug: https://bugs.openjdk.java.net/browse/JDK-8210879 Webrev: http://cr.openjdk.java.net/~zgu/8210879/webrev.00/ Test: hotspot_gc on Linux x64 (fastdebug and release) Thanks, -Zhengyu From shade at redhat.com Wed Sep 19 14:01:52 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 19 Sep 2018 16:01:52 +0200 Subject: RFR(XXS) 8210879: ClassLoaderStatsClosure does raw oop comparison In-Reply-To: <1729861c-8fcd-4b16-8bed-ce4e44ebe270@redhat.com> References: <1729861c-8fcd-4b16-8bed-ce4e44ebe270@redhat.com> Message-ID: <38e31e36-8d22-d629-8da3-23962a8d4d01@redhat.com> On 09/19/2018 03:30 PM, Zhengyu Gu wrote: > Bug: https://bugs.openjdk.java.net/browse/JDK-8210879 > Webrev: http://cr.openjdk.java.net/~zgu/8210879/webrev.00/ This looks trivial and good to me. Thanks, -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From zgu at redhat.com Wed Sep 19 14:05:33 2018 From: zgu at redhat.com (Zhengyu Gu) Date: Wed, 19 Sep 2018 10:05:33 -0400 Subject: RFR(XXS) 8210879: ClassLoaderStatsClosure does raw oop comparison In-Reply-To: <38e31e36-8d22-d629-8da3-23962a8d4d01@redhat.com> References: <1729861c-8fcd-4b16-8bed-ce4e44ebe270@redhat.com> <38e31e36-8d22-d629-8da3-23962a8d4d01@redhat.com> Message-ID: Thanks, Aleksey. -Zhengyu On 09/19/2018 10:01 AM, Aleksey Shipilev wrote: > On 09/19/2018 03:30 PM, Zhengyu Gu wrote: >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210879 >> Webrev: http://cr.openjdk.java.net/~zgu/8210879/webrev.00/ > > This looks trivial and good to me. > > Thanks, > -Aleksey > From erik.osterlund at oracle.com Wed Sep 19 14:09:53 2018 From: erik.osterlund at oracle.com (Erik Osterlund) Date: Wed, 19 Sep 2018 16:09:53 +0200 Subject: RFR: 8210881: ZGC: Introduce ZRootsIteratorClosure In-Reply-To: <0a7125af-59a0-34cf-42b5-b44f72d84735@oracle.com> References: <0a7125af-59a0-34cf-42b5-b44f72d84735@oracle.com> Message-ID: Hi Per, Looks good. An implementation using behaviours would have been even better of course though... ;) Thanks, /Erik > On 18 Sep 2018, at 22:22, Per Liden wrote: > > Introduce ZRootsIteratorClosure to allow piggybacking per-thread operations when iterating over thread roots. Use this to update the thread local address bad mask, instead of having that logic buried inside the ZRootIterator, which should not have knowledge of this. This mechanism will later also be used to retire/resize/remap TLABs. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210881 > Webrev: http://cr.openjdk.java.net/~pliden/8210881/webrev.0 > > /Per From erik.osterlund at oracle.com Wed Sep 19 14:23:51 2018 From: erik.osterlund at oracle.com (Erik Osterlund) Date: Wed, 19 Sep 2018 16:23:51 +0200 Subject: RFR: 8210884: ZGC: Remove insertion of filler objects In-Reply-To: <42d20236-4358-ff0a-deaf-ff3ca6c524a6@oracle.com> References: <42d20236-4358-ff0a-deaf-ff3ca6c524a6@oracle.com> Message-ID: <4DCD51B3-EDBA-4977-9442-3600CFFA6A87@oracle.com> Hi Per, Looks good. Thanks, /Erik > On 18 Sep 2018, at 23:01, Per Liden wrote: > > Remove insertion of filler objects since we don't depend or use those in any way, since ZGC does not depend on having a parsable heap. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210884 > Webrev: http://cr.openjdk.java.net/~pliden/8210884/webrev.0 > > /Per From per.liden at oracle.com Wed Sep 19 15:53:11 2018 From: per.liden at oracle.com (Per Liden) Date: Wed, 19 Sep 2018 17:53:11 +0200 Subject: RFR: 8210884: ZGC: Remove insertion of filler objects In-Reply-To: <4DCD51B3-EDBA-4977-9442-3600CFFA6A87@oracle.com> References: <42d20236-4358-ff0a-deaf-ff3ca6c524a6@oracle.com> <4DCD51B3-EDBA-4977-9442-3600CFFA6A87@oracle.com> Message-ID: Thanks Erik! /Per On 09/19/2018 04:23 PM, Erik Osterlund wrote: > Hi Per, > > Looks good. > > Thanks, > /Erik > >> On 18 Sep 2018, at 23:01, Per Liden wrote: >> >> Remove insertion of filler objects since we don't depend or use those in any way, since ZGC does not depend on having a parsable heap. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210884 >> Webrev: http://cr.openjdk.java.net/~pliden/8210884/webrev.0 >> >> /Per > From per.liden at oracle.com Wed Sep 19 15:55:21 2018 From: per.liden at oracle.com (Per Liden) Date: Wed, 19 Sep 2018 17:55:21 +0200 Subject: RFR: 8210881: ZGC: Introduce ZRootsIteratorClosure In-Reply-To: References: <0a7125af-59a0-34cf-42b5-b44f72d84735@oracle.com> Message-ID: On 09/19/2018 04:09 PM, Erik Osterlund wrote: > Hi Per, > > Looks good. An implementation using behaviours would have been even better of course though... ;) Thanks Erik! Let's see if using behaviors here is better if/when that patch gets in ;) /Per > > Thanks, > /Erik > >> On 18 Sep 2018, at 22:22, Per Liden wrote: >> >> Introduce ZRootsIteratorClosure to allow piggybacking per-thread operations when iterating over thread roots. Use this to update the thread local address bad mask, instead of having that logic buried inside the ZRootIterator, which should not have knowledge of this. This mechanism will later also be used to retire/resize/remap TLABs. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210881 >> Webrev: http://cr.openjdk.java.net/~pliden/8210881/webrev.0 >> >> /Per > From kishor.kharbas at intel.com Wed Sep 19 16:34:32 2018 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Wed, 19 Sep 2018 16:34:32 +0000 Subject: RFR(M): 8204908: NVDIMM for POGC and G1GC - ReserveSpace.cpp changes are mostly eliminated/no collector specific code. In-Reply-To: <0b6a7684-3120-ce80-2069-1a0d163b14ba@oracle.com> References: <5df6d080894cfad5e6486a00f28b6ccfc5ca633f.camel@oracle.com> <87e5373f-a2e0-083e-6421-0209f519dca5@oracle.com> <177b91b2-c0ed-d7ad-84ab-ce479f07474a@oracle.com> <0b6a7684-3120-ce80-2069-1a0d163b14ba@oracle.com> Message-ID: Thanks Sangheon, I have uploaded the updated patch at, http://cr.openjdk.java.net/~kkharbas/8204908/webrev-8204908.01/ http://cr.openjdk.java.net/~kkharbas/8204908/webrev-8204908.01_to_00/ I have fixed all the indentation and format related comments, others I have pasted here below with my comments inline. ============================= > src/hotspot/share/gc/parallel/adjoiningGenerations.hpp > - Copyright update > > 62 AdjoiningGenerations(); > - Why we need this ctor? > >> I need this default ctor for constructing adjoiningGenerationsForHeteroHeap, since I don't want to call the non-default constructor of adjoiningGenerations. > ========================= > /src/hotspot/share/gc/parallel/adjoiningVirtualSpaces.hpp > - Copyright update > > 88 virtual PSVirtualSpace* high() { return _high; } > 89 virtual PSVirtualSpace* low() { return _low; } > - Those methods don't need 'virtual' specifier as high()/low() methods are > only used at ctor of AdjoiningGenerations. The other calling site is ctor of > AdjoiningGenerationsForHeteroHeap but it is used another type of > high()/low() which returns _young_vs or _old_vs. > >> I feel overriding these methods is a good idea from design standpoint; code changes in future would take benefit of this. ======================== > src/hotspot/share/gc/parallel/adjoiningGenerationsForHeteroHeap.hpp > 45 PSVirtualSpace* _young_vs; > 46 PSVirtualSpace* _old_vs; > - Can't we use 'AdjoiningVirtualSpaces::_low' and '_high' instead? If it is not > the case, adding high(),low() may result in confusion between > AdjoiningVirtualSpaces::high() and low(). So use other name for current > HeteroVirtualSpaces::high()/low(). > >> AdjoiningVirtualSpaces contains two adjacent virtual spaces in the same reserved space and separated by a boundary. That?s why the name 'high' and 'low'. >> The class I added - HeteroVirtualSpaces, are not adjacent and do not share same reserved space. So I have names them '_young_vs' and '_old_vs'. But from outside, i.e, users of VirtualSpaces, they call high() and low() to access these spaces. So I have not changed the function names. > -----Original Message----- > From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] > Sent: Monday, September 17, 2018 11:32 AM > To: Kharbas, Kishor ; Thomas St?fe > ; Thomas Schatzl > > Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime runtime-dev at openjdk.java.net>; Viswanathan, Sandhya > ; Aundhe, Shirish > > Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > ReserveSpace.cpp changes are mostly eliminated/no collector specific code. > > Hi Kishor, > > > On 9/4/18 10:41 PM, Kharbas, Kishor wrote: > > Hi, > > I have uploaded implementation for parallel scavenge at > > http://cr.openjdk.java.net/~kkharbas/8204908/webrev-8204908.00 > > Majority of the implementation is handled in two new files - > adjoiningGenerationsForHeteroHeap.cpp and psFileBackedVirtualspace.cpp. > Would love to hear your feedback and suggestions. > Sorry for late review. > > ---------------- > General comments. > > 1. Looks like this patch is not based on latest repository, as it fails with - > Wreorder issue. > > 2. I see many wrong alignment location of having only 2 spaces after new > line that is continued. > e.g. > a->b(c, > __d, xxxx); // 2 spaces > this should be > a->b(c, > _____d, xxx); // 'd' should locate under 'c' as those are in the same context. > > 3. I see assertion failures with below options combinations. There could be > more... Please run jtreg tests before posting webrev. > -XX:+UseLargePages -XX:+UseSHM -version > -XX:+UseLargePages -XX:+UseNUMA -version -XX:+UseLargePages - > XX:AllocateHeapAt=. -version > > ========================= > src/hotspot/share/gc/parallel/adjoiningGenerations.cpp > - Copyright update > > 43?? _virtual_spaces(new AdjoiningVirtualSpaces(old_young_rs, > policy->min_old_size(), > 44?????????????????? policy->min_young_size(), alignment) ) { > - Wrong alignment? > > > ========================= > src/hotspot/share/gc/parallel/adjoiningGenerations.hpp > - Copyright update > > 62?? AdjoiningGenerations(); > - Why we need this ctor? > > > ========================= > /src/hotspot/share/gc/parallel/adjoiningVirtualSpaces.hpp > - Copyright update > > ? 88?? virtual PSVirtualSpace* high() { return _high; } > ? 89?? virtual PSVirtualSpace* low()? { return _low; } > - Those methods don't need 'virtual' specifier as high()/low() methods are > only used at ctor of AdjoiningGenerations. The other calling site is ctor of > AdjoiningGenerationsForHeteroHeap but it is used another type of > high()/low() which returns _young_vs or _old_vs. > > > ========================= > src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp > - no comments. > > > ========================= > src/hotspot/share/gc/parallel/psOldGen.cpp > ========================= > src/hotspot/share/gc/parallel/psOldGen.hpp > ? 32 #include "gc/parallel/psFileBackedVirtualspace.hpp" > - Include this header file at cpp seems better rather than hpp. > > ========================= > src/hotspot/share/gc/parallel/adjoiningGenerationsForHeteroHeap.cpp > ?? 1 /* > ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. > ?? 3 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE > HEADER. > - Wrong alignment. The second/below star should locate under first line star. > - Similarily there are more wrong alignment locations. > ? . Line 49, 50 > ? . 52, 53 > ? . 54, 55 > ? . 64, 65 > ? . 67, 68 69 70 > ? . 72, 73 74 75 76 > ? . 79, 80 > ? . 81, 82 > ? . 85, 86 > ? . 87, 88 > ? . 106, 107 108 109 > ? . 114, 115 > ? . 166, 167 168 > ? . 178, 179 180 > ? . 186, 187 188 > ? . 218, 219 220 > ? . 230, 231 232 > ? . 239, 240 241 > > > ? 59?? // Initialize the virtual spaces.? Then pass the > ? 60?? // a virtual to each generation for initialization of the > - Then pass 'the a' virtual to each generation. 'the a'? > > ? 64?? (static_cast > (_virtual_spaces))->initialize(max_old_byte_size, > init_old_byte_size, > ? 65???? init_young_byte_size); > - Just making 'initilize' method as 'virtual' seems better. > > ? 39 > AdjoiningGenerationsForHeteroHeap::AdjoiningGenerationsForHeteroHeap( > ReservedSpace > old_young_rs, size_t total_size_limit, > ? 40?? GenerationSizer* policy, size_t alignment) : > _total_size_limit(total_size_limit) { > - Wrong alirnment at line 40. 'Generation' should be under 'ReservedSpace' > at line 39. > > ? 49?? _virtual_spaces = new HeteroVirtualSpaces(old_young_rs, > min_old_byte_size,? 70???? (static_cast > (_virtual_spaces))->max_young_size()); > ? 75???? (static_cast > (_virtual_spaces))->max_old_size(), > - Instead assigning at line 49 and then cast to HeteroVirtualSpaces*, > create/initialize HeteroVirtualSpaces, get max_young/old_size() and the > assign to _virtual_spaces would be better alternative. > > ?109?? _min_young_byte_size(min_yg_byte_size), > _max_total_size(max_total_size) { > ?110?? _max_old_byte_size = _max_total_size - _min_young_byte_size; > ?111?? _max_young_byte_size = _max_total_size - _min_old_byte_size; > - _max_old_byte_size and _max_young_byte_size can move to initialization > list. > > ?117?? // This the reserved space exclusively for old generation. > ?122?? // This the reserved space exclusively for young generation. > - Missing 'is'? i.e. // This *is* the reserved space exclusively for old > generation. > > ?131???? vm_exit_during_initialization("Could not reserve enough space for " > ?132?????? "object heap"); > - 'object heap' can stay same line. Or changing align is necessary. > > ?137???? vm_exit_during_initialization("Could not reserve enough space for " > ?138?????? "object heap"); > - 'object heap' can stay same line. Or changing align is necessary. > > ?152?? if (uncommitted_in_old > 0) { > - This condition seems redundant as uncommitted_in_old is type of size_t. > > ?159?? if (bytes_needed == 0) { > ?160???? return true; > ?161?? } > - This condition, can move to inside of 'if (uncommitted_in_old > 0)' > condition because if uncommitted_in_old is zero, there's no way > bytes_needed to be zero - bytes_needed is guaranteed not to be zero from > caller site, so safe to move between line 154 and 155. The condition to > return true is 'uncommitted_size == bytes_needed && success of > expand_by()'. > > ?176???? bool ret = _young_vs->shrink_by(shrink_size); > ?177???? assert(ret, "We should be able to shrink young space"); > - I would prefer to add more conditions below if we fails to shrink. > i.e. just assert seems not enough. > > ?189?? _old_vs->expand_by(bytes_to_add_in_old); > - Check the return value of expand_by(). > > ?211?? if (bytes_needed == 0) { > ?212???? return true; > ?213?? } > - Same as 'adjust_boundary_down()' > > ?244?? DEBUG_ONLY(size_t total_size_after = _young_vs->reserved_size() > + _old_vs->reserved_size()); > ?245?? assert(total_size_after == total_size_before, "should be equal"); > - Why adjust_boundary_up() doesn't have this kind of check? > > > ========================= > src/hotspot/share/gc/parallel/adjoiningGenerationsForHeteroHeap.hpp > ?? 1 /* > ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. > ?? 3 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE > HEADER. > ?? 4 * > - Wrong alignment. The second/below star should locate under first line star. > > ? 37?? size_t total_size_limit() { > ? 38???? return _total_size_limit; > ? 39?? } > - I don't think we need this but if you prefer to have, add 'const'. > > ? 45???? PSVirtualSpace*??? _young_vs; > ? 46???? PSVirtualSpace*??? _old_vs; > - Can't we use 'AdjoiningVirtualSpaces::_low' and '_high' instead? If it is not > the case, adding high(),low() may result in confusion between > AdjoiningVirtualSpaces::high() and low(). So use other name for current > HeteroVirtualSpaces::high()/low(). > > ? 55???? HeteroVirtualSpaces(ReservedSpace rs, > ? 56?????? size_t min_old_byte_size, > ? 57?????? size_t min_young_byte_size, size_t max_total_size, > ? 58?????? size_t alignment); > - Wrong alignment. Line 56 ~ 58 should be same as the location of > 'ReservedSpace' at line 55. > > ? 67???? size_t max_young_size() { return _max_young_byte_size; } > ? 68???? size_t max_old_size() { return _max_old_byte_size; } > - 'const'? > > ? 70???? void initialize(size_t initial_old_reserved_size, size_t > init_low_byte_size, > ? 71?????? size_t init_high_byte_size); > - Wrong alignment > > ? 82?? size_t reserved_byte_size(); > - 'const'? > > ========================= > ?src/hotspot/share/gc/parallel/psFileBackedVirtualspace.cpp > ?? 1 /* > ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. > - Wrong alignment. The second/below star should locate under first line star. > > ? 36?? if (_fd == -1) { > ? 37???? return; > ? 38?? } > - I prefer to have 'initialize()' method to handle when fails to create the heap > file. Current implementation seems easy to call additional 'initialize()'. > ? Ctor of PSVirtualSpace doesn't have any failure path(e.g. > allocation), so no further check is needed around line 18:psOldGen.cpp. > But yours is different, so something should be checked. > > ? 33?? _mapping_succeeded = false; > ? 34?? _file_path = path; > ? 46?? _mapping_succeeded = true; > ? 47?? _special = true; > - Can move to initialization list with proper initial value. > > ? 55?? assert(special(), "_special should always be true for > PSFileBackedVirtualSpace"); > ? 66?? assert(special(), "_special should always be true for > PSFileBackedVirtualSpace"); > - For these 2, can we have more specific message? e.g. to include meaning > of expand or shrink. > > > ========================= > ?src/hotspot/share/gc/parallel/psFileBackedVirtualspace.hpp > ?? 1 /* > ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. > - Wrong alignment. The second/below star should locate under first line star. > > ? 37?? PSFileBackedVirtualSpace(ReservedSpace rs, const char* file_path); > ? 38?? bool?? expand_by(size_t bytes); > - Just recommendation to add new line between 37 and 38. > > ? 43 #endif // SHARE_VM_GC_PARALLEL_PSFILEBACKEDVIRTUALSPACE_HPP > - Last line is not terminated with a newline. > > > Thanks, > Sangheon > > > > > > I will post G1 GC implementation in a few days. > > > > Thanks > > Kishor. > > > >> -----Original Message----- > >> From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] > >> Sent: Thursday, August 30, 2018 4:06 PM > >> To: Kharbas, Kishor ; Thomas St?fe > >> ; Thomas Schatzl > > >> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime >> runtime-dev at openjdk.java.net>; Viswanathan, Sandhya > >> ; Aundhe, Shirish > >> > >> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > >> ReserveSpace.cpp changes are mostly eliminated/no collector specific > code. > >> > >> Hi Kishor, > >> > >> On 8/30/18 11:55 AM, Kharbas, Kishor wrote: > >>> Hi Sangheon, > >>> > >>> So far I don?t see a need to do so. I will post my implementation > >>> code > >> today or tomorrow, if we see a need or any simplification by changing > >> classes in VirtualSpace.hpp, we can discuss that. > >> Okay. > >> > >> Thanks, > >> Sangheon > >> > >> > >>> Regards, > >>> -Kishor > >>> > >>>> -----Original Message----- > >>>> From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] > >>>> Sent: Wednesday, August 29, 2018 10:17 AM > >>>> To: Kharbas, Kishor ; Thomas St?fe > >>>> ; Thomas Schatzl > >> > >>>> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime >>>> runtime-dev at openjdk.java.net>; Viswanathan, Sandhya > >>>> ; Aundhe, Shirish > >>>> > >>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > >>>> ReserveSpace.cpp changes are mostly eliminated/no collector > >>>> specific > >> code. > >>>> Hi Kishor, > >>>> > >>>> On 8/29/18 9:52 AM, Kharbas, Kishor wrote: > >>>>> Hi Sangheon, > >>>>> > >>>>> Thanks for reviewing the design. > >>>>> Since the collectors do not use them for heap memory, I did not > >>>>> have to override VirtualSpace > >>>> Sorry, I meant ReservedSpace and its friends at > >>>> share/memory/virtualspace.hpp. > >>>> > >>>> Thanks, > >>>> Sangheon > >>>> > >>>> > >>>>> -Kishor > >>>>>> -----Original Message----- > >>>>>> From: sangheon.kim at oracle.com > [mailto:sangheon.kim at oracle.com] > >>>>>> Sent: Tuesday, August 28, 2018 2:38 PM > >>>>>> To: Kharbas, Kishor ; Thomas St?fe > >>>>>> ; Thomas Schatzl > >>>> > >>>>>> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime > >>>>>> ; Viswanathan, Sandhya > >>>>>> ; Aundhe, Shirish > >>>>>> > >>>>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > >>>>>> ReserveSpace.cpp changes are mostly eliminated/no collector > >>>>>> specific > >>>> code. > >>>>>> Hi Kishor, > >>>>>> > >>>>>> On 8/21/18 10:57 AM, Kharbas, Kishor wrote: > >>>>>>> Hi All, > >>>>>>> > >>>>>>> Thank you for your valuable comments and feedback in this thread > >>>>>>> so > >>>> far. > >>>>>>> After taken in all the comments and reading thoroughly through > >>>>>>> the code, I > >>>>>> feel that the complexity added to virtualSpace.cpp is due to lack > >>>>>> of abstraction in VirtualSpace and at GC level. NV-DIMM size and > >>>>>> file paths are being passed all the way to OS calls. > >>>>>>> So I went back to the drawing board and created a high level > >>>>>>> design to remove all the pain points of current implementation. > >>>>>>> I have uploaded the design at > >>>>>>> > >> > http://cr.openjdk.java.net/~kkharbas/8202286/Design%20for%20JEP%20JDK > >>>>>> - > >>>>>>> 8202286.pdf I would love to hear your feedback and suggestions. > >>>>>>> Once we get confidence in the design, I will work on the > >> implementation. > >>>>>> The design looks good to me. > >>>>>> If you are planning to change VirtualSpace at > >>>>>> share/memory/virtualspace.hpp, including it on the design > >>>>>> document would be helpful too. > >>>>>> > >>>>>> Probably folks involved in the previous discussions would say > >>>>>> whether the design well covers their concerns or not. > >>>>>> > >>>>>> Thanks, > >>>>>> Sangheon > >>>>>> > >>>>>> > >>>>>>> PS: > >>>>>>> 1. Vinay has transitioned to another team in Intel, so he won't > >>>>>>> be able to > >>>>>> continue on this jep. > >>>>>>> 2. If you feel this should be part of JEP discussion thread, we > >>>>>>> can take it > >>>>>> there. > >>>>>>> Thanks and regards, > >>>>>>> Kishor > >>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: Thomas St?fe [mailto:thomas.stuefe at gmail.com] > >>>>>>>> Sent: Friday, June 22, 2018 9:25 AM > >>>>>>>> To: Thomas Schatzl > >>>>>>>> Cc: Awasthi, Vinay K ; Paul Su > >>>>>>>> ; hotspot-gc-dev at openjdk.java.net; > Hotspot > >>>> dev > >>>>>>>> runtime ; Viswanathan, > >>>>>> Sandhya > >>>>>>>> ; Aundhe, Shirish > >>>>>>>> ; Kharbas, Kishor > >>>>>>>> > >>>>>>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > >>>>>>>> ReserveSpace.cpp changes are mostly eliminated/no collector > >>>>>>>> specific > >>>>>> code. > >>>>>>>> Hi Thomas, > >>>>>>>> > >>>>>>>> On Fri, Jun 22, 2018 at 4:44 PM, Thomas Schatzl > >>>>>>>> wrote: > >>>>>>>>> Hi Thomas, > >>>>>>>>> > >>>>>>>>> On Tue, 2018-06-19 at 13:40 +0200, Thomas St?fe wrote: > >>>>>>>>>> Hi Vinay, > >>>>>>>>>> > >>>>>>>>>> On Mon, Jun 18, 2018 at 6:47 PM, Awasthi, Vinay K > >>>>>>>>>> wrote: > >>>>>>>>>>> Hi Thomas, > >>>>>>>>>>> > >>>>>>>>>>> Os::commit_memory calls map_memory_to_file which is same > >> as > >>>>>>>>>>> os::reserve_memory. > >>>>>>>>>>> > >>>>>>>>>>> I am failing to see why os::reserve_memory can call > >>>>>>>>>>> map_memory_to_file (i.e. tie it to mmap) but > commit_memory > >>>>>> can't... > >>>>>>>>>>> Before this patch, commit_memory never dealt with > >>>>>>>>>>> incrementally committing pages to device so there has to be > >>>>>>>>>>> a way to pass file descriptor and offset. Windows has no > >>>>>>>>>>> such capability to manage incremental commits. All other > >>>>>>>>>>> OSes do and that is why map_memory_to_file is used (which > by > >>>>>>>>>>> the way also works on Windows). > >>>>>>>>>> AIX uses System V shared memory by default, which follows a > >>>>>>>>>> different allocation scheme (semantics more like Windows > >>>>>> VirtualAlloc... > >>>>>>>>>> calls). > >>>>>>>>>> > >>>>>>>>>> But my doubts are not limited to that one, see my earlier > >>>>>>>>>> replies and those of others. It really makes sense to step > >>>>>>>>>> back one step and discuss the JEP first. > >>>>>>>>>> > >>>>>>>>> There is already a discussion thread as David mentioned > >>>>>>>>> (http://mail.op > >>>>>>>>> enjdk.java.net/pipermail/hotspot-gc-dev/2018- > May/022092.html) > >>>> that > >>>>>>>>> so far nobody answered to. > >>>>>>>>> > >>>>>>>> Ah, I thought he wanted to have the JEP discussed in the > >>>>>>>> comments section of the JEP itself. > >>>>>>>> > >>>>>>>>> I believe discussion about contents the JEP and the > >>>>>>>>> implementation should be separate. > >>>>>>>>> > >>>>>>>> makes sense. > >>>>>>>> > >>>>>>>>> So far what I gathered from the responses to the > >>>>>>>>> implementation, the proposed idea itself is not the issue here > >>>>>>>>> (allowing the use of NVDIMM memory for parts of the heap for > >>>>>>>>> allowing the use of larger heaps to improve overall > >>>>>>>>> performance; I am not saying that the text doesn't need a bit > >>>>>>>>> more work :) ), but rather how an implementation of this JEP > should proceed. > >>>>>>>> I have no problem with adding NVDIMM support. I think it is a > >>>>>>>> cool > >>>>>> feature. > >>>>>>>> Also, the awkwardness off the memory management abstraction > >> layer > >>>>>>>> in hotspot has always been a sore point with me (I originally > >>>>>>>> implemented the AIX mm layer in os_aix.cpp, and those are > >>>>>>>> painful memories). So, I have a lot of sympathy for Vinays > struggles. > >>>>>>>> Unfortunately not much time atm, but I will respond to your mail. > >>>>>>>> > >>>>>>>>> Let's discuss the non-implementation stuff in that thread. > >>>>>>>>> Vinay or I can repost the proposal email if you do not have it > >>>>>>>>> any more so that answers will be in-thread. > >>>>>>>>> > >>>>>>>> Okay, sounds good. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Thomas > >>>>>>>> > >>>>>>>> (one of us should change his name to make this less confusing > >>>>>>>> :-) > >>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Thomas > >>>>>>>>> From rkennke at redhat.com Wed Sep 19 17:08:11 2018 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 19 Sep 2018 19:08:11 +0200 Subject: RFR: JDK-8210829: Modularize allocations in C2 In-Reply-To: <09c53a34-eb68-a614-ee03-468673004b3b@oracle.com> References: <8fe187e2-b3fb-5473-257f-e4f6b6a94fcb@redhat.com> <09c53a34-eb68-a614-ee03-468673004b3b@oracle.com> Message-ID: Alright, submit repo came back with UNSTABLE. Can somebody here check it and get back to me? Build Details: 2018-09-19-1536076.roman.source 0 Failed Tests Mach5 Tasks Results Summary KILLED: 0 PASSED: 70 UNABLE_TO_RUN: 3 NA: 0 FAILED: 0 EXECUTED_WITH_FAILURE: 2 Test 2 Not run tier1-solaris-sparc-jdk_open_test_hotspot_jtreg_tier1_common-solaris-sparcv9-57 Dependency task failed: mach5...-1909-solaris-sparcv9-solaris-sparcv9-build-8 tier1-solaris-sparc-jdk_open_test_hotspot_jtreg_tier1_common-solaris-sparcv9-debug-58 Dependency task failed: mach5...solaris-sparcv9-debug-solaris-sparcv9-build-9 > Hi Roman, > > This looks good. I looked through changes and it generates the same > ideal graph as before. > It seems you unintentionally changed indent of the comment in > barrierSetC2.hpp > > Thanks, > Vladimir > > On 9/18/18 12:58 AM, Roman Kennke wrote: >> Similar to what we've done before to runtime, interpreter and C1, >> allocations should be owned and implemented by GC, and possible to >> override by specific collectors. For example, Shenandoah lays out >> objects differently in heap, and needs one extra store to initialize >> objects. >> >> This proposed change factors out the interesting part of object >> allocation (i.e. the actual allocation) into BarrierSetC2. It's mostly a >> move-and-rename-job. I had to move some little things around, that is: >> - for the need-gc-check, I'm passing back the needgc_ctrl to plug into >> slow-path >> - for prefetching, instead of passing around the 'length' node, only to >> determine the number of prefetch lines, I determine this early, and pass >> through the lines arg. >> - i_o, needgc_ctrl, fast_oop_ctrl, fast_oop_rawmem are passed as in/out >> or out-args to stitch together into the regions and phis as appropriate. >> I see no easy way around that. >> >> I tested this using hotspot/jtreg:tier1 and also verified that this >> fills Shenandoah's needs and run tier3_gc_shenandoah testsuite. >> >> http://cr.openjdk.java.net/~rkennke/JDK-8210829/webrev.00/ >> >> Can I please get reviews? >> Thanks, >> Roman >> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From vladimir.kozlov at oracle.com Wed Sep 19 17:58:53 2018 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Wed, 19 Sep 2018 10:58:53 -0700 Subject: RFR: JDK-8210829: Modularize allocations in C2 In-Reply-To: References: <8fe187e2-b3fb-5473-257f-e4f6b6a94fcb@redhat.com> <09c53a34-eb68-a614-ee03-468673004b3b@oracle.com> Message-ID: <81b25b8b-c18f-28b5-1e25-3ecab35d6dc6@oracle.com> Crypto library build failed - 8210912. Mikael just pushed the fix - update your copy: http://hg.openjdk.java.net/jdk/jdk/rev/15094d12a632 Vladimir On 9/19/18 10:08 AM, Roman Kennke wrote: > Alright, submit repo came back with UNSTABLE. Can somebody here check it > and get back to me? > > Build Details: 2018-09-19-1536076.roman.source > 0 Failed Tests > Mach5 Tasks Results Summary > > KILLED: 0 > PASSED: 70 > UNABLE_TO_RUN: 3 > NA: 0 > FAILED: 0 > EXECUTED_WITH_FAILURE: 2 > Test > > 2 Not run > > tier1-solaris-sparc-jdk_open_test_hotspot_jtreg_tier1_common-solaris-sparcv9-57 > Dependency task failed: > mach5...-1909-solaris-sparcv9-solaris-sparcv9-build-8 > > tier1-solaris-sparc-jdk_open_test_hotspot_jtreg_tier1_common-solaris-sparcv9-debug-58 > Dependency task failed: > mach5...solaris-sparcv9-debug-solaris-sparcv9-build-9 > > >> Hi Roman, >> >> This looks good. I looked through changes and it generates the same >> ideal graph as before. >> It seems you unintentionally changed indent of the comment in >> barrierSetC2.hpp >> >> Thanks, >> Vladimir >> >> On 9/18/18 12:58 AM, Roman Kennke wrote: >>> Similar to what we've done before to runtime, interpreter and C1, >>> allocations should be owned and implemented by GC, and possible to >>> override by specific collectors. For example, Shenandoah lays out >>> objects differently in heap, and needs one extra store to initialize >>> objects. >>> >>> This proposed change factors out the interesting part of object >>> allocation (i.e. the actual allocation) into BarrierSetC2. It's mostly a >>> move-and-rename-job. I had to move some little things around, that is: >>> - for the need-gc-check, I'm passing back the needgc_ctrl to plug into >>> slow-path >>> - for prefetching, instead of passing around the 'length' node, only to >>> determine the number of prefetch lines, I determine this early, and pass >>> through the lines arg. >>> - i_o, needgc_ctrl, fast_oop_ctrl, fast_oop_rawmem are passed as in/out >>> or out-args to stitch together into the regions and phis as appropriate. >>> I see no easy way around that. >>> >>> I tested this using hotspot/jtreg:tier1 and also verified that this >>> fills Shenandoah's needs and run tier3_gc_shenandoah testsuite. >>> >>> http://cr.openjdk.java.net/~rkennke/JDK-8210829/webrev.00/ >>> >>> Can I please get reviews? >>> Thanks, >>> Roman >>> > > From rkennke at redhat.com Wed Sep 19 20:05:23 2018 From: rkennke at redhat.com (Roman Kennke) Date: Wed, 19 Sep 2018 22:05:23 +0200 Subject: RFR: JDK-8210829: Modularize allocations in C2 In-Reply-To: <81b25b8b-c18f-28b5-1e25-3ecab35d6dc6@oracle.com> References: <8fe187e2-b3fb-5473-257f-e4f6b6a94fcb@redhat.com> <09c53a34-eb68-a614-ee03-468673004b3b@oracle.com> <81b25b8b-c18f-28b5-1e25-3ecab35d6dc6@oracle.com> Message-ID: Thanks, Vladimir. That fixed it, build came out clean and I pushed the change. Thanks, Roman > Crypto library build failed - 8210912. > Mikael just pushed the fix - update your copy: > > http://hg.openjdk.java.net/jdk/jdk/rev/15094d12a632 > > Vladimir > > On 9/19/18 10:08 AM, Roman Kennke wrote: >> Alright, submit repo came back with UNSTABLE. Can somebody here check it >> and get back to me? >> >> Build Details: 2018-09-19-1536076.roman.source >> 0 Failed Tests >> Mach5 Tasks Results Summary >> >> ???? KILLED: 0 >> ???? PASSED: 70 >> ???? UNABLE_TO_RUN: 3 >> ???? NA: 0 >> ???? FAILED: 0 >> ???? EXECUTED_WITH_FAILURE: 2 >> ???? Test >> >> ???????? 2 Not run >> >> tier1-solaris-sparc-jdk_open_test_hotspot_jtreg_tier1_common-solaris-sparcv9-57 >> >> Dependency task failed: >> mach5...-1909-solaris-sparcv9-solaris-sparcv9-build-8 >> >> tier1-solaris-sparc-jdk_open_test_hotspot_jtreg_tier1_common-solaris-sparcv9-debug-58 >> >> Dependency task failed: >> mach5...solaris-sparcv9-debug-solaris-sparcv9-build-9 >> >> >>> Hi Roman, >>> >>> This looks good. I looked through changes and it generates the same >>> ideal graph as before. >>> It seems you unintentionally changed indent of the comment in >>> barrierSetC2.hpp >>> >>> Thanks, >>> Vladimir >>> >>> On 9/18/18 12:58 AM, Roman Kennke wrote: >>>> Similar to what we've done before to runtime, interpreter and C1, >>>> allocations should be owned and implemented by GC, and possible to >>>> override by specific collectors. For example, Shenandoah lays out >>>> objects differently in heap, and needs one extra store to initialize >>>> objects. >>>> >>>> This proposed change factors out the interesting part of object >>>> allocation (i.e. the actual allocation) into BarrierSetC2. It's >>>> mostly a >>>> move-and-rename-job. I had to move some little things around, that is: >>>> - for the need-gc-check, I'm passing back the needgc_ctrl to plug into >>>> slow-path >>>> - for prefetching, instead of passing around the 'length' node, only to >>>> determine the number of prefetch lines, I determine this early, and >>>> pass >>>> through the lines arg. >>>> - i_o, needgc_ctrl, fast_oop_ctrl, fast_oop_rawmem are passed as in/out >>>> or out-args to stitch together into the regions and phis as >>>> appropriate. >>>> I see no easy way around that. >>>> >>>> I tested this using hotspot/jtreg:tier1 and also verified that this >>>> fills Shenandoah's needs and run tier3_gc_shenandoah testsuite. >>>> >>>> http://cr.openjdk.java.net/~rkennke/JDK-8210829/webrev.00/ >>>> >>>> Can I please get reviews? >>>> Thanks, >>>> Roman >>>> >> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From kishor.kharbas at intel.com Wed Sep 19 22:57:22 2018 From: kishor.kharbas at intel.com (Kharbas, Kishor) Date: Wed, 19 Sep 2018 22:57:22 +0000 Subject: RFR(M): 8204908: NVDIMM for POGC and G1GC - ReserveSpace.cpp changes are mostly eliminated/no collector specific code. References: <5df6d080894cfad5e6486a00f28b6ccfc5ca633f.camel@oracle.com> <87e5373f-a2e0-083e-6421-0209f519dca5@oracle.com> <177b91b2-c0ed-d7ad-84ab-ce479f07474a@oracle.com> <0b6a7684-3120-ce80-2069-1a0d163b14ba@oracle.com> Message-ID: Hi, I have an small update to the patch to disable UseCompressedOops. http://cr.openjdk.java.net/~kkharbas/8204908/webrev-8204908.02/ http://cr.openjdk.java.net/~kkharbas/8204908/webrev-8204908.02_to_01/ Regards, Kishor > -----Original Message----- > From: Kharbas, Kishor > Sent: Wednesday, September 19, 2018 9:35 AM > To: 'sangheon.kim at oracle.com' ; Thomas > St?fe ; Thomas Schatzl > > Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime runtime-dev at openjdk.java.net>; Viswanathan, Sandhya > ; Aundhe, Shirish > ; Kharbas, Kishor > Subject: RE: RFR(M): 8204908: NVDIMM for POGC and G1GC - > ReserveSpace.cpp changes are mostly eliminated/no collector specific code. > > Thanks Sangheon, > > I have uploaded the updated patch at, > http://cr.openjdk.java.net/~kkharbas/8204908/webrev-8204908.01/ > http://cr.openjdk.java.net/~kkharbas/8204908/webrev-8204908.01_to_00/ > > I have fixed all the indentation and format related comments, others I have > pasted here below with my comments inline. > > ============================= > > src/hotspot/share/gc/parallel/adjoiningGenerations.hpp > > - Copyright update > > > > 62 AdjoiningGenerations(); > > - Why we need this ctor? > > > >> I need this default ctor for constructing > adjoiningGenerationsForHeteroHeap, since I don't want to call the non- > default constructor of adjoiningGenerations. > > > ========================= > > /src/hotspot/share/gc/parallel/adjoiningVirtualSpaces.hpp > > - Copyright update > > > > 88 virtual PSVirtualSpace* high() { return _high; } > > 89 virtual PSVirtualSpace* low() { return _low; } > > - Those methods don't need 'virtual' specifier as high()/low() methods > > are only used at ctor of AdjoiningGenerations. The other calling site > > is ctor of AdjoiningGenerationsForHeteroHeap but it is used another > > type of > > high()/low() which returns _young_vs or _old_vs. > > > >> I feel overriding these methods is a good idea from design standpoint; > code changes in future would take benefit of this. > > ======================== > > src/hotspot/share/gc/parallel/adjoiningGenerationsForHeteroHeap.hpp > > 45 PSVirtualSpace* _young_vs; > > 46 PSVirtualSpace* _old_vs; > > - Can't we use 'AdjoiningVirtualSpaces::_low' and '_high' instead? If > > it is not the case, adding high(),low() may result in confusion > > between > > AdjoiningVirtualSpaces::high() and low(). So use other name for > > current HeteroVirtualSpaces::high()/low(). > > > >> AdjoiningVirtualSpaces contains two adjacent virtual spaces in the same > reserved space and separated by a boundary. That?s why the name 'high' and > 'low'. > >> The class I added - HeteroVirtualSpaces, are not adjacent and do not > share same reserved space. So I have names them '_young_vs' and '_old_vs'. > But from outside, i.e, users of VirtualSpaces, they call high() and low() to > access these spaces. So I have not changed the function names. > > > -----Original Message----- > > From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] > > Sent: Monday, September 17, 2018 11:32 AM > > To: Kharbas, Kishor ; Thomas St?fe > > ; Thomas Schatzl > > > Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime > runtime-dev at openjdk.java.net>; Viswanathan, Sandhya > > ; Aundhe, Shirish > > > > Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > > ReserveSpace.cpp changes are mostly eliminated/no collector specific > code. > > > > Hi Kishor, > > > > > > On 9/4/18 10:41 PM, Kharbas, Kishor wrote: > > > Hi, > > > I have uploaded implementation for parallel scavenge at > > > http://cr.openjdk.java.net/~kkharbas/8204908/webrev-8204908.00 > > > Majority of the implementation is handled in two new files - > > adjoiningGenerationsForHeteroHeap.cpp and > psFileBackedVirtualspace.cpp. > > Would love to hear your feedback and suggestions. > > Sorry for late review. > > > > ---------------- > > General comments. > > > > 1. Looks like this patch is not based on latest repository, as it > > fails with - Wreorder issue. > > > > 2. I see many wrong alignment location of having only 2 spaces after > > new line that is continued. > > e.g. > > a->b(c, > > __d, xxxx); // 2 spaces > > this should be > > a->b(c, > > _____d, xxx); // 'd' should locate under 'c' as those are in the same context. > > > > 3. I see assertion failures with below options combinations. There > > could be more... Please run jtreg tests before posting webrev. > > -XX:+UseLargePages -XX:+UseSHM -version -XX:+UseLargePages > > -XX:+UseNUMA -version -XX:+UseLargePages - XX:AllocateHeapAt=. > > -version > > > > ========================= > > src/hotspot/share/gc/parallel/adjoiningGenerations.cpp > > - Copyright update > > > > 43?? _virtual_spaces(new AdjoiningVirtualSpaces(old_young_rs, > > policy->min_old_size(), > > 44?????????????????? policy->min_young_size(), alignment) ) { > > - Wrong alignment? > > > > > > ========================= > > src/hotspot/share/gc/parallel/adjoiningGenerations.hpp > > - Copyright update > > > > 62?? AdjoiningGenerations(); > > - Why we need this ctor? > > > > > > ========================= > > /src/hotspot/share/gc/parallel/adjoiningVirtualSpaces.hpp > > - Copyright update > > > > ? 88?? virtual PSVirtualSpace* high() { return _high; } > > ? 89?? virtual PSVirtualSpace* low()? { return _low; } > > - Those methods don't need 'virtual' specifier as high()/low() methods > > are only used at ctor of AdjoiningGenerations. The other calling site > > is ctor of AdjoiningGenerationsForHeteroHeap but it is used another > > type of > > high()/low() which returns _young_vs or _old_vs. > > > > > > ========================= > > src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp > > - no comments. > > > > > > ========================= > > src/hotspot/share/gc/parallel/psOldGen.cpp > > ========================= > > src/hotspot/share/gc/parallel/psOldGen.hpp > > ? 32 #include "gc/parallel/psFileBackedVirtualspace.hpp" > > - Include this header file at cpp seems better rather than hpp. > > > > ========================= > > src/hotspot/share/gc/parallel/adjoiningGenerationsForHeteroHeap.cpp > > ?? 1 /* > > ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. > > ?? 3 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE > HEADER. > > - Wrong alignment. The second/below star should locate under first line > star. > > - Similarily there are more wrong alignment locations. > > ? . Line 49, 50 > > ? . 52, 53 > > ? . 54, 55 > > ? . 64, 65 > > ? . 67, 68 69 70 > > ? . 72, 73 74 75 76 > > ? . 79, 80 > > ? . 81, 82 > > ? . 85, 86 > > ? . 87, 88 > > ? . 106, 107 108 109 > > ? . 114, 115 > > ? . 166, 167 168 > > ? . 178, 179 180 > > ? . 186, 187 188 > > ? . 218, 219 220 > > ? . 230, 231 232 > > ? . 239, 240 241 > > > > > > ? 59?? // Initialize the virtual spaces.? Then pass the > > ? 60?? // a virtual to each generation for initialization of the > > - Then pass 'the a' virtual to each generation. 'the a'? > > > > ? 64?? (static_cast > > (_virtual_spaces))->initialize(max_old_byte_size > > , > > init_old_byte_size, > > ? 65???? init_young_byte_size); > > - Just making 'initilize' method as 'virtual' seems better. > > > > ? 39 > > > AdjoiningGenerationsForHeteroHeap::AdjoiningGenerationsForHeteroHeap( > > ReservedSpace > > old_young_rs, size_t total_size_limit, > > ? 40?? GenerationSizer* policy, size_t alignment) : > > _total_size_limit(total_size_limit) { > > - Wrong alirnment at line 40. 'Generation' should be under 'ReservedSpace' > > at line 39. > > > > ? 49?? _virtual_spaces = new HeteroVirtualSpaces(old_young_rs, > > min_old_byte_size,? 70???? (static_cast > > (_virtual_spaces))->max_young_size()); > > ? 75???? (static_cast > > (_virtual_spaces))->max_old_size(), > > - Instead assigning at line 49 and then cast to HeteroVirtualSpaces*, > > create/initialize HeteroVirtualSpaces, get max_young/old_size() and > > the assign to _virtual_spaces would be better alternative. > > > > ?109?? _min_young_byte_size(min_yg_byte_size), > > _max_total_size(max_total_size) { > > ?110?? _max_old_byte_size = _max_total_size - _min_young_byte_size; > > ?111?? _max_young_byte_size = _max_total_size - _min_old_byte_size; > > - _max_old_byte_size and _max_young_byte_size can move to > > initialization list. > > > > ?117?? // This the reserved space exclusively for old generation. > > ?122?? // This the reserved space exclusively for young generation. > > - Missing 'is'? i.e. // This *is* the reserved space exclusively for > > old generation. > > > > ?131???? vm_exit_during_initialization("Could not reserve enough space for " > > ?132?????? "object heap"); > > - 'object heap' can stay same line. Or changing align is necessary. > > > > ?137???? vm_exit_during_initialization("Could not reserve enough space for " > > ?138?????? "object heap"); > > - 'object heap' can stay same line. Or changing align is necessary. > > > > ?152?? if (uncommitted_in_old > 0) { > > - This condition seems redundant as uncommitted_in_old is type of size_t. > > > > ?159?? if (bytes_needed == 0) { > > ?160???? return true; > > ?161?? } > > - This condition, can move to inside of 'if (uncommitted_in_old > 0)' > > condition because if uncommitted_in_old is zero, there's no way > > bytes_needed to be zero - bytes_needed is guaranteed not to be zero > > from caller site, so safe to move between line 154 and 155. The > > condition to return true is 'uncommitted_size == bytes_needed && > > success of expand_by()'. > > > > ?176???? bool ret = _young_vs->shrink_by(shrink_size); > > ?177???? assert(ret, "We should be able to shrink young space"); > > - I would prefer to add more conditions below if we fails to shrink. > > i.e. just assert seems not enough. > > > > ?189?? _old_vs->expand_by(bytes_to_add_in_old); > > - Check the return value of expand_by(). > > > > ?211?? if (bytes_needed == 0) { > > ?212???? return true; > > ?213?? } > > - Same as 'adjust_boundary_down()' > > > > ?244?? DEBUG_ONLY(size_t total_size_after = > > _young_vs->reserved_size() > > + _old_vs->reserved_size()); > > ?245?? assert(total_size_after == total_size_before, "should be > > equal"); > > - Why adjust_boundary_up() doesn't have this kind of check? > > > > > > ========================= > > src/hotspot/share/gc/parallel/adjoiningGenerationsForHeteroHeap.hpp > > ?? 1 /* > > ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. > > ?? 3 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE > HEADER. > > ?? 4 * > > - Wrong alignment. The second/below star should locate under first line > star. > > > > ? 37?? size_t total_size_limit() { > > ? 38???? return _total_size_limit; > > ? 39?? } > > - I don't think we need this but if you prefer to have, add 'const'. > > > > ? 45???? PSVirtualSpace*??? _young_vs; > > ? 46???? PSVirtualSpace*??? _old_vs; > > - Can't we use 'AdjoiningVirtualSpaces::_low' and '_high' instead? If > > it is not the case, adding high(),low() may result in confusion > > between > > AdjoiningVirtualSpaces::high() and low(). So use other name for > > current HeteroVirtualSpaces::high()/low(). > > > > ? 55???? HeteroVirtualSpaces(ReservedSpace rs, > > ? 56?????? size_t min_old_byte_size, > > ? 57?????? size_t min_young_byte_size, size_t max_total_size, > > ? 58?????? size_t alignment); > > - Wrong alignment. Line 56 ~ 58 should be same as the location of > > 'ReservedSpace' at line 55. > > > > ? 67???? size_t max_young_size() { return _max_young_byte_size; } > > ? 68???? size_t max_old_size() { return _max_old_byte_size; } > > - 'const'? > > > > ? 70???? void initialize(size_t initial_old_reserved_size, size_t > > init_low_byte_size, > > ? 71?????? size_t init_high_byte_size); > > - Wrong alignment > > > > ? 82?? size_t reserved_byte_size(); > > - 'const'? > > > > ========================= > > ?src/hotspot/share/gc/parallel/psFileBackedVirtualspace.cpp > > ?? 1 /* > > ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. > > - Wrong alignment. The second/below star should locate under first line > star. > > > > ? 36?? if (_fd == -1) { > > ? 37???? return; > > ? 38?? } > > - I prefer to have 'initialize()' method to handle when fails to > > create the heap file. Current implementation seems easy to call additional > 'initialize()'. > > ? Ctor of PSVirtualSpace doesn't have any failure path(e.g. > > allocation), so no further check is needed around line 18:psOldGen.cpp. > > But yours is different, so something should be checked. > > > > ? 33?? _mapping_succeeded = false; > > ? 34?? _file_path = path; > > ? 46?? _mapping_succeeded = true; > > ? 47?? _special = true; > > - Can move to initialization list with proper initial value. > > > > ? 55?? assert(special(), "_special should always be true for > > PSFileBackedVirtualSpace"); > > ? 66?? assert(special(), "_special should always be true for > > PSFileBackedVirtualSpace"); > > - For these 2, can we have more specific message? e.g. to include > > meaning of expand or shrink. > > > > > > ========================= > > ?src/hotspot/share/gc/parallel/psFileBackedVirtualspace.hpp > > ?? 1 /* > > ?? 2 * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. > > - Wrong alignment. The second/below star should locate under first line > star. > > > > ? 37?? PSFileBackedVirtualSpace(ReservedSpace rs, const char* > > file_path); > > ? 38?? bool?? expand_by(size_t bytes); > > - Just recommendation to add new line between 37 and 38. > > > > ? 43 #endif // > SHARE_VM_GC_PARALLEL_PSFILEBACKEDVIRTUALSPACE_HPP > > - Last line is not terminated with a newline. > > > > > > Thanks, > > Sangheon > > > > > > > > > > I will post G1 GC implementation in a few days. > > > > > > Thanks > > > Kishor. > > > > > >> -----Original Message----- > > >> From: sangheon.kim at oracle.com [mailto:sangheon.kim at oracle.com] > > >> Sent: Thursday, August 30, 2018 4:06 PM > > >> To: Kharbas, Kishor ; Thomas St?fe > > >> ; Thomas Schatzl > > > > >> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime > >> runtime-dev at openjdk.java.net>; Viswanathan, Sandhya > > >> ; Aundhe, Shirish > > >> > > >> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > > >> ReserveSpace.cpp changes are mostly eliminated/no collector > > >> specific > > code. > > >> > > >> Hi Kishor, > > >> > > >> On 8/30/18 11:55 AM, Kharbas, Kishor wrote: > > >>> Hi Sangheon, > > >>> > > >>> So far I don?t see a need to do so. I will post my implementation > > >>> code > > >> today or tomorrow, if we see a need or any simplification by > > >> changing classes in VirtualSpace.hpp, we can discuss that. > > >> Okay. > > >> > > >> Thanks, > > >> Sangheon > > >> > > >> > > >>> Regards, > > >>> -Kishor > > >>> > > >>>> -----Original Message----- > > >>>> From: sangheon.kim at oracle.com > [mailto:sangheon.kim at oracle.com] > > >>>> Sent: Wednesday, August 29, 2018 10:17 AM > > >>>> To: Kharbas, Kishor ; Thomas St?fe > > >>>> ; Thomas Schatzl > > >> > > >>>> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime > > >>>> ; Viswanathan, Sandhya > > >>>> ; Aundhe, Shirish > > >>>> > > >>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > > >>>> ReserveSpace.cpp changes are mostly eliminated/no collector > > >>>> specific > > >> code. > > >>>> Hi Kishor, > > >>>> > > >>>> On 8/29/18 9:52 AM, Kharbas, Kishor wrote: > > >>>>> Hi Sangheon, > > >>>>> > > >>>>> Thanks for reviewing the design. > > >>>>> Since the collectors do not use them for heap memory, I did not > > >>>>> have to override VirtualSpace > > >>>> Sorry, I meant ReservedSpace and its friends at > > >>>> share/memory/virtualspace.hpp. > > >>>> > > >>>> Thanks, > > >>>> Sangheon > > >>>> > > >>>> > > >>>>> -Kishor > > >>>>>> -----Original Message----- > > >>>>>> From: sangheon.kim at oracle.com > > [mailto:sangheon.kim at oracle.com] > > >>>>>> Sent: Tuesday, August 28, 2018 2:38 PM > > >>>>>> To: Kharbas, Kishor ; Thomas St?fe > > >>>>>> ; Thomas Schatzl > > >>>> > > >>>>>> Cc: hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime > > >>>>>> ; Viswanathan, Sandhya > > >>>>>> ; Aundhe, Shirish > > >>>>>> > > >>>>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > > >>>>>> ReserveSpace.cpp changes are mostly eliminated/no collector > > >>>>>> specific > > >>>> code. > > >>>>>> Hi Kishor, > > >>>>>> > > >>>>>> On 8/21/18 10:57 AM, Kharbas, Kishor wrote: > > >>>>>>> Hi All, > > >>>>>>> > > >>>>>>> Thank you for your valuable comments and feedback in this > > >>>>>>> thread so > > >>>> far. > > >>>>>>> After taken in all the comments and reading thoroughly through > > >>>>>>> the code, I > > >>>>>> feel that the complexity added to virtualSpace.cpp is due to > > >>>>>> lack of abstraction in VirtualSpace and at GC level. NV-DIMM > > >>>>>> size and file paths are being passed all the way to OS calls. > > >>>>>>> So I went back to the drawing board and created a high level > > >>>>>>> design to remove all the pain points of current implementation. > > >>>>>>> I have uploaded the design at > > >>>>>>> > > >> > > > http://cr.openjdk.java.net/~kkharbas/8202286/Design%20for%20JEP%20JDK > > >>>>>> - > > >>>>>>> 8202286.pdf I would love to hear your feedback and suggestions. > > >>>>>>> Once we get confidence in the design, I will work on the > > >> implementation. > > >>>>>> The design looks good to me. > > >>>>>> If you are planning to change VirtualSpace at > > >>>>>> share/memory/virtualspace.hpp, including it on the design > > >>>>>> document would be helpful too. > > >>>>>> > > >>>>>> Probably folks involved in the previous discussions would say > > >>>>>> whether the design well covers their concerns or not. > > >>>>>> > > >>>>>> Thanks, > > >>>>>> Sangheon > > >>>>>> > > >>>>>> > > >>>>>>> PS: > > >>>>>>> 1. Vinay has transitioned to another team in Intel, so he > > >>>>>>> won't be able to > > >>>>>> continue on this jep. > > >>>>>>> 2. If you feel this should be part of JEP discussion thread, > > >>>>>>> we can take it > > >>>>>> there. > > >>>>>>> Thanks and regards, > > >>>>>>> Kishor > > >>>>>>> > > >>>>>>>> -----Original Message----- > > >>>>>>>> From: Thomas St?fe [mailto:thomas.stuefe at gmail.com] > > >>>>>>>> Sent: Friday, June 22, 2018 9:25 AM > > >>>>>>>> To: Thomas Schatzl > > >>>>>>>> Cc: Awasthi, Vinay K ; Paul Su > > >>>>>>>> ; hotspot-gc-dev at openjdk.java.net; > > Hotspot > > >>>> dev > > >>>>>>>> runtime ; > Viswanathan, > > >>>>>> Sandhya > > >>>>>>>> ; Aundhe, Shirish > > >>>>>>>> ; Kharbas, Kishor > > >>>>>>>> > > >>>>>>>> Subject: Re: RFR(M): 8204908: NVDIMM for POGC and G1GC - > > >>>>>>>> ReserveSpace.cpp changes are mostly eliminated/no collector > > >>>>>>>> specific > > >>>>>> code. > > >>>>>>>> Hi Thomas, > > >>>>>>>> > > >>>>>>>> On Fri, Jun 22, 2018 at 4:44 PM, Thomas Schatzl > > >>>>>>>> wrote: > > >>>>>>>>> Hi Thomas, > > >>>>>>>>> > > >>>>>>>>> On Tue, 2018-06-19 at 13:40 +0200, Thomas St?fe wrote: > > >>>>>>>>>> Hi Vinay, > > >>>>>>>>>> > > >>>>>>>>>> On Mon, Jun 18, 2018 at 6:47 PM, Awasthi, Vinay K > > >>>>>>>>>> wrote: > > >>>>>>>>>>> Hi Thomas, > > >>>>>>>>>>> > > >>>>>>>>>>> Os::commit_memory calls map_memory_to_file which is > same > > >> as > > >>>>>>>>>>> os::reserve_memory. > > >>>>>>>>>>> > > >>>>>>>>>>> I am failing to see why os::reserve_memory can call > > >>>>>>>>>>> map_memory_to_file (i.e. tie it to mmap) but > > commit_memory > > >>>>>> can't... > > >>>>>>>>>>> Before this patch, commit_memory never dealt with > > >>>>>>>>>>> incrementally committing pages to device so there has to > > >>>>>>>>>>> be a way to pass file descriptor and offset. Windows has > > >>>>>>>>>>> no such capability to manage incremental commits. All > > >>>>>>>>>>> other OSes do and that is why map_memory_to_file is used > > >>>>>>>>>>> (which > > by > > >>>>>>>>>>> the way also works on Windows). > > >>>>>>>>>> AIX uses System V shared memory by default, which follows a > > >>>>>>>>>> different allocation scheme (semantics more like Windows > > >>>>>> VirtualAlloc... > > >>>>>>>>>> calls). > > >>>>>>>>>> > > >>>>>>>>>> But my doubts are not limited to that one, see my earlier > > >>>>>>>>>> replies and those of others. It really makes sense to step > > >>>>>>>>>> back one step and discuss the JEP first. > > >>>>>>>>>> > > >>>>>>>>> There is already a discussion thread as David mentioned > > >>>>>>>>> (http://mail.op > > >>>>>>>>> enjdk.java.net/pipermail/hotspot-gc-dev/2018- > > May/022092.html) > > >>>> that > > >>>>>>>>> so far nobody answered to. > > >>>>>>>>> > > >>>>>>>> Ah, I thought he wanted to have the JEP discussed in the > > >>>>>>>> comments section of the JEP itself. > > >>>>>>>> > > >>>>>>>>> I believe discussion about contents the JEP and the > > >>>>>>>>> implementation should be separate. > > >>>>>>>>> > > >>>>>>>> makes sense. > > >>>>>>>> > > >>>>>>>>> So far what I gathered from the responses to the > > >>>>>>>>> implementation, the proposed idea itself is not the issue > > >>>>>>>>> here (allowing the use of NVDIMM memory for parts of the > > >>>>>>>>> heap for allowing the use of larger heaps to improve overall > > >>>>>>>>> performance; I am not saying that the text doesn't need a > > >>>>>>>>> bit more work :) ), but rather how an implementation of this > > >>>>>>>>> JEP > > should proceed. > > >>>>>>>> I have no problem with adding NVDIMM support. I think it is a > > >>>>>>>> cool > > >>>>>> feature. > > >>>>>>>> Also, the awkwardness off the memory management abstraction > > >> layer > > >>>>>>>> in hotspot has always been a sore point with me (I originally > > >>>>>>>> implemented the AIX mm layer in os_aix.cpp, and those are > > >>>>>>>> painful memories). So, I have a lot of sympathy for Vinays > > struggles. > > >>>>>>>> Unfortunately not much time atm, but I will respond to your > mail. > > >>>>>>>> > > >>>>>>>>> Let's discuss the non-implementation stuff in that thread. > > >>>>>>>>> Vinay or I can repost the proposal email if you do not have > > >>>>>>>>> it any more so that answers will be in-thread. > > >>>>>>>>> > > >>>>>>>> Okay, sounds good. > > >>>>>>>> > > >>>>>>>> Thanks, > > >>>>>>>> Thomas > > >>>>>>>> > > >>>>>>>> (one of us should change his name to make this less confusing > > >>>>>>>> :-) > > >>>>>>>> > > >>>>>>>>> Thanks, > > >>>>>>>>> Thomas > > >>>>>>>>> From kim.barrett at oracle.com Wed Sep 19 23:44:59 2018 From: kim.barrett at oracle.com (Kim Barrett) Date: Wed, 19 Sep 2018 19:44:59 -0400 Subject: RFR(XXS) 8210879: ClassLoaderStatsClosure does raw oop comparison In-Reply-To: <1729861c-8fcd-4b16-8bed-ce4e44ebe270@redhat.com> References: <1729861c-8fcd-4b16-8bed-ce4e44ebe270@redhat.com> Message-ID: <14BD507A-CEEE-4F2A-8423-0436E107A2E7@oracle.com> > On Sep 19, 2018, at 9:30 AM, Zhengyu Gu wrote: > > Hi, > > Please review this small change. > > GCs, such as Shenandoah, need proper barrier for oop comparison. > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210879 > Webrev: http://cr.openjdk.java.net/~zgu/8210879/webrev.00/ > > > Test: > > hotspot_gc on Linux x64 (fastdebug and release) > > > Thanks, > > -Zhengyu Looks good. From per.liden at oracle.com Thu Sep 20 10:04:27 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 20 Sep 2018 12:04:27 +0200 Subject: RFR: 8210857: Allow retiring TLABs and collecting statistics in parallel In-Reply-To: <33ca946c-0803-a5cb-41f2-e4fa55de34ff@oracle.com> References: <33ca946c-0803-a5cb-41f2-e4fa55de34ff@oracle.com> Message-ID: On 09/18/2018 10:16 PM, Per Liden wrote: > On 09/18/2018 10:13 PM, Per Liden wrote: >> There is currently no good way of retiring TLABs and collecting their >> statistics in parallel. The current code for doing this also traverses >> the thread list twice, which is unnecessary. I propose that we adjust >> the TLAB API to allow this to be done in parallel, in a single pass. >> >> A few notes on this patch: >> >> * The GlobalTLABStats class is restructured into the >> ThreadLocalAllocStats class to allow for a more flexible way of >> collected statistics when retiring TLABs (flexible in the sense that >> it allows for parallel operations). >> >> * The TLAB API is slightly adjusted. The "make_parsable(bool >> retire_tlab)" is broken into two functions, "make_parsable()" to just >> make a TLAB parsable, and "retire(ThreadLocalAllocStats* stats)" to >> retire and optionally collects statistics. >> >> * GCs are unaffected by this change, in the sense that they continue >> to call CollectedHeap::ensure_parsability(). However, all GCs will >> benefit from now doing a single pass over the thread list instead of >> two to retire and collect stats. >> >> * This is the last patch in my series of TLAB related patches. The >> remaining patches only touch ZGC to actually retire/resize/remap TLABs >> in parallel. >> >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210857 >> Webrev: http://cr.openjdk.java.net/~pliden/8210857/webrev.0 > > Testing: Passed tier{1,2,3,4,5,6} on linux-x86_64 (product and debug) Discussed with Erik and Stefan off-line, which resulted in the following adjustment. This adjustment actually fixes a long standing bug we've had, when make_parsable(false) is called and ZeroTLAB is true, in which case we actually retire the TLAB but failed to call Thread::incr_allocated_bytes() to update the number of allocated bytes. Diff: http://cr.openjdk.java.net/~pliden/8210857/webrev.0vs1 Full: http://cr.openjdk.java.net/~pliden/8210857/webrev.1 /Per From per.liden at oracle.com Thu Sep 20 10:08:53 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 20 Sep 2018 12:08:53 +0200 Subject: RFR: 8210883: ZGC: Parallel retire/resize/remap of TLABs In-Reply-To: <655f86ce-fde9-973d-46e2-a36ff39980fa@oracle.com> References: <655f86ce-fde9-973d-46e2-a36ff39980fa@oracle.com> Message-ID: <5a37954a-a9b1-bcaf-2c58-0f3ec9692b37@oracle.com> On 09/18/2018 10:24 PM, Per Liden wrote: > Let ZGC retire/resize/remap TLABs in parallel. This change builds on > JDK-8210857 - "Allow retiring TLABs and collecting statistics in > parallel?", which is currently also out on review. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210883 > Webrev: http://cr.openjdk.java.net/~pliden/8210883/webrev.0 While off-line reviewing with Erik we noticed that the ZMarkRootsIteratorClosure obviously has to be owned by the ZMarkRootsTask for the stat collection to work out as intended. To be symmetric, ZRelocateRootsIteratorClosure/ZRelocateRootsTask were adjusted in the same way, even if it doesn't really matter in that case. Diff: http://cr.openjdk.java.net/~pliden/8210883/webrev.0vs1 Full: http://cr.openjdk.java.net/~pliden/8210883/webrev.0 /Per From erik.osterlund at oracle.com Thu Sep 20 10:23:26 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 20 Sep 2018 12:23:26 +0200 Subject: RFR: 8210857: Allow retiring TLABs and collecting statistics in parallel In-Reply-To: References: <33ca946c-0803-a5cb-41f2-e4fa55de34ff@oracle.com> Message-ID: <5BA3751E.5070500@oracle.com> Hi Per, Looks good. Thanks, /Erik On 2018-09-20 12:04, Per Liden wrote: > On 09/18/2018 10:16 PM, Per Liden wrote: >> On 09/18/2018 10:13 PM, Per Liden wrote: >>> There is currently no good way of retiring TLABs and collecting >>> their statistics in parallel. The current code for doing this also >>> traverses the thread list twice, which is unnecessary. I propose >>> that we adjust the TLAB API to allow this to be done in parallel, in >>> a single pass. >>> >>> A few notes on this patch: >>> >>> * The GlobalTLABStats class is restructured into the >>> ThreadLocalAllocStats class to allow for a more flexible way of >>> collected statistics when retiring TLABs (flexible in the sense that >>> it allows for parallel operations). >>> >>> * The TLAB API is slightly adjusted. The "make_parsable(bool >>> retire_tlab)" is broken into two functions, "make_parsable()" to >>> just make a TLAB parsable, and "retire(ThreadLocalAllocStats* >>> stats)" to retire and optionally collects statistics. >>> >>> * GCs are unaffected by this change, in the sense that they continue >>> to call CollectedHeap::ensure_parsability(). However, all GCs will >>> benefit from now doing a single pass over the thread list instead of >>> two to retire and collect stats. >>> >>> * This is the last patch in my series of TLAB related patches. The >>> remaining patches only touch ZGC to actually retire/resize/remap >>> TLABs in parallel. >>> >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8210857 >>> Webrev: http://cr.openjdk.java.net/~pliden/8210857/webrev.0 >> >> Testing: Passed tier{1,2,3,4,5,6} on linux-x86_64 (product and debug) > > Discussed with Erik and Stefan off-line, which resulted in the > following adjustment. This adjustment actually fixes a long standing > bug we've had, when make_parsable(false) is called and ZeroTLAB is > true, in which case we actually retire the TLAB but failed to call > Thread::incr_allocated_bytes() to update the number of allocated bytes. > > Diff: http://cr.openjdk.java.net/~pliden/8210857/webrev.0vs1 > Full: http://cr.openjdk.java.net/~pliden/8210857/webrev.1 > > /Per From erik.osterlund at oracle.com Thu Sep 20 10:23:57 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Thu, 20 Sep 2018 12:23:57 +0200 Subject: RFR: 8210883: ZGC: Parallel retire/resize/remap of TLABs In-Reply-To: <5a37954a-a9b1-bcaf-2c58-0f3ec9692b37@oracle.com> References: <655f86ce-fde9-973d-46e2-a36ff39980fa@oracle.com> <5a37954a-a9b1-bcaf-2c58-0f3ec9692b37@oracle.com> Message-ID: <5BA3753D.2090504@oracle.com> Hi Per, Looks good. Thanks, /Erik On 2018-09-20 12:08, Per Liden wrote: > On 09/18/2018 10:24 PM, Per Liden wrote: >> Let ZGC retire/resize/remap TLABs in parallel. This change builds on >> JDK-8210857 - "Allow retiring TLABs and collecting statistics in >> parallel?", which is currently also out on review. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210883 >> Webrev: http://cr.openjdk.java.net/~pliden/8210883/webrev.0 > > While off-line reviewing with Erik we noticed that the > ZMarkRootsIteratorClosure obviously has to be owned by the > ZMarkRootsTask for the stat collection to work out as intended. > To be symmetric, ZRelocateRootsIteratorClosure/ZRelocateRootsTask were > adjusted in the same way, even if it doesn't really matter in that case. > > Diff: http://cr.openjdk.java.net/~pliden/8210883/webrev.0vs1 > Full: http://cr.openjdk.java.net/~pliden/8210883/webrev.0 > > /Per From per.liden at oracle.com Thu Sep 20 10:21:05 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 20 Sep 2018 12:21:05 +0200 Subject: RFR: 8210883: ZGC: Parallel retire/resize/remap of TLABs In-Reply-To: <5BA3753D.2090504@oracle.com> References: <655f86ce-fde9-973d-46e2-a36ff39980fa@oracle.com> <5a37954a-a9b1-bcaf-2c58-0f3ec9692b37@oracle.com> <5BA3753D.2090504@oracle.com> Message-ID: Thanks Erik! /Per On 09/20/2018 12:23 PM, Erik ?sterlund wrote: > Hi Per, > > Looks good. > > Thanks, > /Erik > > On 2018-09-20 12:08, Per Liden wrote: >> On 09/18/2018 10:24 PM, Per Liden wrote: >>> Let ZGC retire/resize/remap TLABs in parallel. This change builds on >>> JDK-8210857 - "Allow retiring TLABs and collecting statistics in >>> parallel?", which is currently also out on review. >>> >>> Bug: https://bugs.openjdk.java.net/browse/JDK-8210883 >>> Webrev: http://cr.openjdk.java.net/~pliden/8210883/webrev.0 >> >> While off-line reviewing with Erik we noticed that the >> ZMarkRootsIteratorClosure obviously has to be owned by the >> ZMarkRootsTask for the stat collection to work out as intended. >> To be symmetric, ZRelocateRootsIteratorClosure/ZRelocateRootsTask were >> adjusted in the same way, even if it doesn't really matter in that case. >> >> Diff: http://cr.openjdk.java.net/~pliden/8210883/webrev.0vs1 >> Full: http://cr.openjdk.java.net/~pliden/8210883/webrev.0 >> >> /Per > From per.liden at oracle.com Thu Sep 20 10:21:22 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 20 Sep 2018 12:21:22 +0200 Subject: RFR: 8210857: Allow retiring TLABs and collecting statistics in parallel In-Reply-To: <5BA3751E.5070500@oracle.com> References: <33ca946c-0803-a5cb-41f2-e4fa55de34ff@oracle.com> <5BA3751E.5070500@oracle.com> Message-ID: Thanks Erik! /Per On 09/20/2018 12:23 PM, Erik ?sterlund wrote: > Hi Per, > > Looks good. > > Thanks, > /Erik > > On 2018-09-20 12:04, Per Liden wrote: >> On 09/18/2018 10:16 PM, Per Liden wrote: >>> On 09/18/2018 10:13 PM, Per Liden wrote: >>>> There is currently no good way of retiring TLABs and collecting >>>> their statistics in parallel. The current code for doing this also >>>> traverses the thread list twice, which is unnecessary. I propose >>>> that we adjust the TLAB API to allow this to be done in parallel, in >>>> a single pass. >>>> >>>> A few notes on this patch: >>>> >>>> * The GlobalTLABStats class is restructured into the >>>> ThreadLocalAllocStats class to allow for a more flexible way of >>>> collected statistics when retiring TLABs (flexible in the sense that >>>> it allows for parallel operations). >>>> >>>> * The TLAB API is slightly adjusted. The "make_parsable(bool >>>> retire_tlab)" is broken into two functions, "make_parsable()" to >>>> just make a TLAB parsable, and "retire(ThreadLocalAllocStats* >>>> stats)" to retire and optionally collects statistics. >>>> >>>> * GCs are unaffected by this change, in the sense that they continue >>>> to call CollectedHeap::ensure_parsability(). However, all GCs will >>>> benefit from now doing a single pass over the thread list instead of >>>> two to retire and collect stats. >>>> >>>> * This is the last patch in my series of TLAB related patches. The >>>> remaining patches only touch ZGC to actually retire/resize/remap >>>> TLABs in parallel. >>>> >>>> >>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8210857 >>>> Webrev: http://cr.openjdk.java.net/~pliden/8210857/webrev.0 >>> >>> Testing: Passed tier{1,2,3,4,5,6} on linux-x86_64 (product and debug) >> >> Discussed with Erik and Stefan off-line, which resulted in the >> following adjustment. This adjustment actually fixes a long standing >> bug we've had, when make_parsable(false) is called and ZeroTLAB is >> true, in which case we actually retire the TLAB but failed to call >> Thread::incr_allocated_bytes() to update the number of allocated bytes. >> >> Diff: http://cr.openjdk.java.net/~pliden/8210857/webrev.0vs1 >> Full: http://cr.openjdk.java.net/~pliden/8210857/webrev.1 >> >> /Per > From stefan.johansson at oracle.com Thu Sep 20 12:08:52 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Thu, 20 Sep 2018 14:08:52 +0200 Subject: RFR: 8210857: Allow retiring TLABs and collecting statistics in parallel In-Reply-To: References: <33ca946c-0803-a5cb-41f2-e4fa55de34ff@oracle.com> <5BA3751E.5070500@oracle.com> Message-ID: <03df05f6-5b30-70e5-2602-7453633989fa@oracle.com> On 2018-09-20 12:21, Per Liden wrote: > Thanks Erik! > > /Per > > On 09/20/2018 12:23 PM, Erik ?sterlund wrote: >> Hi Per, >> >> Looks good. >> >> Thanks, >> /Erik >> >> On 2018-09-20 12:04, Per Liden wrote: >>> On 09/18/2018 10:16 PM, Per Liden wrote: >>>> On 09/18/2018 10:13 PM, Per Liden wrote: >>>>> There is currently no good way of retiring TLABs and collecting >>>>> their statistics in parallel. The current code for doing this also >>>>> traverses the thread list twice, which is unnecessary. I propose >>>>> that we adjust the TLAB API to allow this to be done in parallel, >>>>> in a single pass. >>>>> >>>>> A few notes on this patch: >>>>> >>>>> * The GlobalTLABStats class is restructured into the >>>>> ThreadLocalAllocStats class to allow for a more flexible way of >>>>> collected statistics when retiring TLABs (flexible in the sense >>>>> that it allows for parallel operations). >>>>> >>>>> * The TLAB API is slightly adjusted. The "make_parsable(bool >>>>> retire_tlab)" is broken into two functions, "make_parsable()" to >>>>> just make a TLAB parsable, and "retire(ThreadLocalAllocStats* >>>>> stats)" to retire and optionally collects statistics. >>>>> >>>>> * GCs are unaffected by this change, in the sense that they >>>>> continue to call CollectedHeap::ensure_parsability(). However, all >>>>> GCs will benefit from now doing a single pass over the thread list >>>>> instead of two to retire and collect stats. >>>>> >>>>> * This is the last patch in my series of TLAB related patches. The >>>>> remaining patches only touch ZGC to actually retire/resize/remap >>>>> TLABs in parallel. >>>>> >>>>> >>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8210857 >>>>> Webrev: http://cr.openjdk.java.net/~pliden/8210857/webrev.0 >>>> >>>> Testing: Passed tier{1,2,3,4,5,6} on linux-x86_64 (product and debug) >>> >>> Discussed with Erik and Stefan off-line, which resulted in the >>> following adjustment. This adjustment actually fixes a long standing >>> bug we've had, when make_parsable(false) is called and ZeroTLAB is >>> true, in which case we actually retire the TLAB but failed to call >>> Thread::incr_allocated_bytes() to update the number of allocated bytes. >>> >>> Diff: http://cr.openjdk.java.net/~pliden/8210857/webrev.0vs1 >>> Full: http://cr.openjdk.java.net/~pliden/8210857/webrev.1 Nice cleanup, looks good. Stefan >>> >>> /Per >> From per.liden at oracle.com Thu Sep 20 12:11:36 2018 From: per.liden at oracle.com (Per Liden) Date: Thu, 20 Sep 2018 14:11:36 +0200 Subject: RFR: 8210857: Allow retiring TLABs and collecting statistics in parallel In-Reply-To: <03df05f6-5b30-70e5-2602-7453633989fa@oracle.com> References: <33ca946c-0803-a5cb-41f2-e4fa55de34ff@oracle.com> <5BA3751E.5070500@oracle.com> <03df05f6-5b30-70e5-2602-7453633989fa@oracle.com> Message-ID: On 09/20/2018 02:08 PM, Stefan Johansson wrote: [...] >>>> Discussed with Erik and Stefan off-line, which resulted in the >>>> following adjustment. This adjustment actually fixes a long standing >>>> bug we've had, when make_parsable(false) is called and ZeroTLAB is >>>> true, in which case we actually retire the TLAB but failed to call >>>> Thread::incr_allocated_bytes() to update the number of allocated bytes. >>>> >>>> Diff: http://cr.openjdk.java.net/~pliden/8210857/webrev.0vs1 >>>> Full: http://cr.openjdk.java.net/~pliden/8210857/webrev.1 > Nice cleanup, looks good. > Stefan Thanks for reviewing, Stefan. /Per From zgu at redhat.com Thu Sep 20 12:38:01 2018 From: zgu at redhat.com (Zhengyu Gu) Date: Thu, 20 Sep 2018 08:38:01 -0400 Subject: RFR(XXS) 8210879: ClassLoaderStatsClosure does raw oop comparison In-Reply-To: <14BD507A-CEEE-4F2A-8423-0436E107A2E7@oracle.com> References: <1729861c-8fcd-4b16-8bed-ce4e44ebe270@redhat.com> <14BD507A-CEEE-4F2A-8423-0436E107A2E7@oracle.com> Message-ID: Thanks, Kim! I pushed it based on trivial rule, before seeing this email. Sorry, I can not add your name as a reviewer. -Zhengyu On 09/19/2018 07:44 PM, Kim Barrett wrote: >> On Sep 19, 2018, at 9:30 AM, Zhengyu Gu wrote: >> >> Hi, >> >> Please review this small change. >> >> GCs, such as Shenandoah, need proper barrier for oop comparison. >> >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8210879 >> Webrev: http://cr.openjdk.java.net/~zgu/8210879/webrev.00/ >> >> >> Test: >> >> hotspot_gc on Linux x64 (fastdebug and release) >> >> >> Thanks, >> >> -Zhengyu > > Looks good. > From thomas.schatzl at oracle.com Thu Sep 20 14:31:58 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 20 Sep 2018 16:31:58 +0200 Subject: RFR (S): 8210557: G1 next bitmap verification at the end of concurrent mark sometimes fails Message-ID: <539931ba2368accf579b42861287f105f6257b27.camel@oracle.com> Hi, can I have reviews for this smal change that removes an unnecessary and actually superfluous assert that can sometimes fail? So if G1 concurrent mark gets aborted by a full gc, it intends to check whether the next mark bitmap for the next concurrent cycle is completely cleared. Unfortunately, G1 full gc uses the next bitmap for its own marking, so depending on when this thread gets scheduled, it may errorneously detect that the next mark bitmap contains marks (well, from the first phase of full gc :)). This bitmap-clear check is actually unnecessary: at the end of full gc the next bitmap is already checked for being completely empty (in G1CollectedHeap:1086). So instead of trying to change scheduling of that verification (to e.g. run outside GC), I opted to remove this verification (and associated code) completely. CR: https://bugs.openjdk.java.net/browse/JDK-8210557 Webrev: http://cr.openjdk.java.net/~tschatzl/8210557/webrev/ Testing: passing hs-tier1-5,jdk-tier1-3 with -XX:+G1VerifyBitmaps; however since this is just removal of verification code there is extremely low risk of introducing a bug. :) Thanks, Thomas From kim.barrett at oracle.com Thu Sep 20 20:51:33 2018 From: kim.barrett at oracle.com (Kim Barrett) Date: Thu, 20 Sep 2018 16:51:33 -0400 Subject: RFR (S): 8210557: G1 next bitmap verification at the end of concurrent mark sometimes fails In-Reply-To: <539931ba2368accf579b42861287f105f6257b27.camel@oracle.com> References: <539931ba2368accf579b42861287f105f6257b27.camel@oracle.com> Message-ID: > On Sep 20, 2018, at 10:31 AM, Thomas Schatzl wrote: > > Hi, > > can I have reviews for this smal change that removes an unnecessary > and actually superfluous assert that can sometimes fail? > > So if G1 concurrent mark gets aborted by a full gc, it intends to check > whether the next mark bitmap for the next concurrent cycle is > completely cleared. > > Unfortunately, G1 full gc uses the next bitmap for its own marking, so > depending on when this thread gets scheduled, it may errorneously > detect that the next mark bitmap contains marks (well, from the first > phase of full gc :)). > > This bitmap-clear check is actually unnecessary: at the end of full gc > the next bitmap is already checked for being completely empty (in > G1CollectedHeap:1086). > > So instead of trying to change scheduling of that verification (to e.g. > run outside GC), I opted to remove this verification (and associated > code) completely. > > CR: > https://bugs.openjdk.java.net/browse/JDK-8210557 > Webrev: > http://cr.openjdk.java.net/~tschatzl/8210557/webrev/ > Testing: > passing hs-tier1-5,jdk-tier1-3 with -XX:+G1VerifyBitmaps; however since > this is just removal of verification code there is extremely low risk > of introducing a bug. :) > > Thanks, > Thomas Looks good. From mikael.vidstedt at oracle.com Fri Sep 21 01:15:46 2018 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Thu, 20 Sep 2018 18:15:46 -0700 Subject: RFR(XS): 8210893: Deprecate TLABStats Message-ID: <190905EC-2082-40E9-80B8-98DC7262CC6E@oracle.com> Please review this change which deprecates the -XX:+TLABStats flag. Bug: https://bugs.openjdk.java.net/browse/JDK-8210892 CSR: https://bugs.openjdk.java.net/browse/JDK-8210893 Webrev: http://cr.openjdk.java.net/~mikael/webrevs/8210893/webrev.00/open/webrev/ * Background (from the bug) After the removal of the -XX:+FastTLABRefill flag in JDK-8194084 the -XX:+TLABStats flag no longer has any effect. It should be deprecated and eventually removed. * Testing Ran VMDeprecatedOptions.java locally, will run tier1 before pushing. Cheers, Mikael -------------- next part -------------- An HTML attachment was scrubbed... URL: From per.liden at oracle.com Fri Sep 21 06:45:14 2018 From: per.liden at oracle.com (Per Liden) Date: Fri, 21 Sep 2018 08:45:14 +0200 Subject: RFR(XS): 8210893: Deprecate TLABStats In-Reply-To: <190905EC-2082-40E9-80B8-98DC7262CC6E@oracle.com> References: <190905EC-2082-40E9-80B8-98DC7262CC6E@oracle.com> Message-ID: Looks good. /Per On 09/21/2018 03:15 AM, Mikael Vidstedt wrote: > > Please review this change which deprecates the -XX:+TLABStats flag. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210892 > CSR: https://bugs.openjdk.java.net/browse/JDK-8210893 > Webrev: > http://cr.openjdk.java.net/~mikael/webrevs/8210893/webrev.00/open/webrev/ > > * Background (from the bug) > > After the removal of the -XX:+FastTLABRefill flag in?JDK-8194084?the > -XX:+TLABStats flag no longer has any effect. It should be deprecated > and eventually removed. > > * Testing > > Ran VMDeprecatedOptions.java locally, will run tier1 before pushing. > > Cheers, > Mikael > From stefan.johansson at oracle.com Fri Sep 21 07:33:55 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Fri, 21 Sep 2018 09:33:55 +0200 Subject: RFR (S): 8210557: G1 next bitmap verification at the end of concurrent mark sometimes fails In-Reply-To: References: <539931ba2368accf579b42861287f105f6257b27.camel@oracle.com> Message-ID: <0d7706b8-7841-d08b-c10c-4e2cfd059824@oracle.com> On 2018-09-20 22:51, Kim Barrett wrote: >> On Sep 20, 2018, at 10:31 AM, Thomas Schatzl wrote: >> >> Hi, >> >> can I have reviews for this smal change that removes an unnecessary >> and actually superfluous assert that can sometimes fail? >> >> So if G1 concurrent mark gets aborted by a full gc, it intends to check >> whether the next mark bitmap for the next concurrent cycle is >> completely cleared. >> >> Unfortunately, G1 full gc uses the next bitmap for its own marking, so >> depending on when this thread gets scheduled, it may errorneously >> detect that the next mark bitmap contains marks (well, from the first >> phase of full gc :)). >> >> This bitmap-clear check is actually unnecessary: at the end of full gc >> the next bitmap is already checked for being completely empty (in >> G1CollectedHeap:1086). >> >> So instead of trying to change scheduling of that verification (to e.g. >> run outside GC), I opted to remove this verification (and associated >> code) completely. >> >> CR: >> https://bugs.openjdk.java.net/browse/JDK-8210557 >> Webrev: >> http://cr.openjdk.java.net/~tschatzl/8210557/webrev/ >> Testing: >> passing hs-tier1-5,jdk-tier1-3 with -XX:+G1VerifyBitmaps; however since >> this is just removal of verification code there is extremely low risk >> of introducing a bug. :) >> >> Thanks, >> Thomas > > Looks good. > Looks good to me too. Stefan From thomas.schatzl at oracle.com Fri Sep 21 07:41:18 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 21 Sep 2018 09:41:18 +0200 Subject: RFR (S): 8210557: G1 next bitmap verification at the end of concurrent mark sometimes fails In-Reply-To: <0d7706b8-7841-d08b-c10c-4e2cfd059824@oracle.com> References: <539931ba2368accf579b42861287f105f6257b27.camel@oracle.com> <0d7706b8-7841-d08b-c10c-4e2cfd059824@oracle.com> Message-ID: <2b6bb9ac4e169f5f5957b1228d2aa5ccea3ceedc.camel@oracle.com> Hi Stefan, Kim, On Fri, 2018-09-21 at 09:33 +0200, Stefan Johansson wrote: > > On 2018-09-20 22:51, Kim Barrett wrote: > > > On Sep 20, 2018, at 10:31 AM, Thomas Schatzl > > le.com> wrote: > > > > > > Hi, > > > > > > can I have reviews for this smal change that removes an > > > unnecessary and actually superfluous assert that can sometimes > > > fail? > > > > > > [...] > > > Thanks, > > > Thomas > > > > Looks good. > > > > Looks good to me too. > Stefan thanks for your reviews. Thomas From thomas.schatzl at oracle.com Fri Sep 21 07:43:04 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Fri, 21 Sep 2018 09:43:04 +0200 Subject: RFR(XS): 8210893: Deprecate TLABStats In-Reply-To: <190905EC-2082-40E9-80B8-98DC7262CC6E@oracle.com> References: <190905EC-2082-40E9-80B8-98DC7262CC6E@oracle.com> Message-ID: <6ac1814728de5efef9ec290bb81053e064e111e7.camel@oracle.com> Hi, On Thu, 2018-09-20 at 18:15 -0700, Mikael Vidstedt wrote: > > Please review this change which deprecates the -XX:+TLABStats flag. > > [...] looks good. Thomas From rkennke at redhat.com Fri Sep 21 12:48:39 2018 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 21 Sep 2018 14:48:39 +0200 Subject: RFR: JDK-8210752: Remaining explicit barriers for C2 In-Reply-To: References: <15be8e2d-dba5-2e8a-c851-b6821b81d4b3@redhat.com> Message-ID: Hi Roland, thanks for reviewing! Any other reviews? Can I push that stuff? Roman >> http://cr.openjdk.java.net/~rkennke/JDK-8210752/webrev.00/ > > That looks good to me. > > Roland. > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From shade at redhat.com Fri Sep 21 13:13:07 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 21 Sep 2018 15:13:07 +0200 Subject: RFR: JDK-8210752: Remaining explicit barriers for C2 In-Reply-To: References: <15be8e2d-dba5-2e8a-c851-b6821b81d4b3@redhat.com> Message-ID: <0b832f84-2555-5251-8164-b05732ae8b4a@redhat.com> On 09/21/2018 02:48 PM, Roman Kennke wrote: > Any other reviews? Can I push that stuff? > >>> http://cr.openjdk.java.net/~rkennke/JDK-8210752/webrev.00/ Looks good to me too. Thanks, -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From synytskyy at jelastic.com Fri Sep 21 13:35:16 2018 From: synytskyy at jelastic.com (Ruslan Synytsky) Date: Fri, 21 Sep 2018 16:35:16 +0300 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: References: <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> <65063e5f31702649e12b485b570ecab9ebd32d01.camel@oracle.com> <66c536399b2f218af0d85bd3bced44f204e3dbf4.camel@oracle.com> <2821cf70-c4d5-016f-b976-5ecc850f377c@oracle.com> <57b10708-b870-cb98-0df0-fab650f431e4@oracle.com> Message-ID: Dear Stefan and Rodrigo, thank you for moving this forward. ---------- Forwarded message --------- From: Stefan Johansson Date: quarta, 19/09/2018 ?(s) 10:45 Subject: Re: RFR: bug: Timely Reducing Unused Committed Memory To: , Hi Rodrigo, I pasted your reply here to keep the discussion in one thread. >> I understand that it is hard to define what is idle. However, if we require the >> user to provide one, I guess that most regular users that suffer from the problem >> that this patch is trying to solve will simply not do it because it requires knowledge >> and effort. If we provide an idle check that we think will benefit most users, then >> we are probably helping a lot of users. For those that the default idle check is >> not good enough, they can always disable this idle check and implement the idle >> check logic it in an external tool. >> > I agree, if we can find a solution that benefits most users, we should > do it. And this is why I would like to hear from more users if this > would benefit their use cases. I believe the default idle definition should be based on the major bottlenecks: RAM, CPU and IO loads as well as the network. RAM - we try to improve. IO - I?m not sure if we can measure IO load properly inside JVM. If possible then it's good to add too. If not then we can skip it for now, as it can be measured and triggered by outside logic. Network is not involved in GC process, correct? So no need for that. CPU looks the most obvious and already implemented, seems like a good option to start from. > Another thing that I don't fully > understand is why the flags are manageable if there isn't supposed to be > some external logic that sets them? Some advanced users, for example cloud platform or software vendors, will be able to apply an additional logic based on their custom needs / specifics. Such flexibility enables more use cases and it helps to collect more feedback for the further default improvements. >> We can also change the semantics of "idleness". Currently it checks the load. >> I think that checking the allocation rate might be another good option (instead of >> load). The only corner case is an application that does not allocate but consumes >> a lot of CPU. For this case, we might only trigger compaction at most once because, >> as it does not allocate memory, we will not get over committed memory (i.e., the other >> checks will prevent it). The opposite is also possible (almost idle application that allocates >> a lot of memory) but in this scenario I don't think we want to trigger an idle compaction. >> > This is my main problem when it comes to determine "idleness", for some > applications allocation rate will be the correct metric, for others it > will be the load and for a third something different. It feels like it > is always possible to come up with a case that needs something different. I would prefer to start with the most obvious one - based on CPU, give it to try to more people by promoting the fact that JVM is elastic now, and we will get more feedback that can be converted into an additional logic later. >> Having said that, I am open to change this flag or even remove it as it is one of the >> hardest to get right. >> > As I said before, to me it feels like just having a periodic GC interval > flag that is manageable would be a good start. Maybe have constraint > that the periodic GC only occurs if no other GCs have happened during > the interval. Decision based on the previous GC cycles is very good proposal. I think we need to take it into account somehow, but I'm not so deep on it. Input of others will be helpful here. > Could you explain how your use case would suffer from such > limitations? In my opinion, CPU load spikes is clearly one of the major use cases eligible for defaults. Thank you > Thanks, > Stefan >> cheers, >> rodrigo On 2018-09-13 14:30, Stefan Johansson wrote: > Hi Rodrigo, > > Sorry for being a bit late into the discussion. We've had some internal > discussions and realized that there are some questions that I need to > bring up here. > > I'm trying to better understand under what circumstances this feature is > to be used and how a user should use the different flags to tweak it to > their use case. To me it feels like GCFrequency would be enough to make > sure that the VM returns memory on a timely basis. And if the flag is > managed, it can be controlled to not do periodic GCs during high load. > With that we get a good way to periodically try to reduce the committed > heap. > > The reason I ask is because I have a hard time seeing how we can > implement a generic policy for when the system is idle. A policy that > will apply well to most use cases. For some cases having the flags you > propose might be good, but for other there might be a different set of > options needed. If this is the case then maybe the logic and policy of > when to do this can live outside the VM, while the code to periodically > do GCs lives within the VM. What do you think about that? I understand > the problems you've stated with having the policy outside that VM, but > at least we have more information to act on there. > > We know that many have asked for features similar to this one and it > would be nice to get input from others on this to make sure we implement > something that benefits the whole user base as much as possible. So > anyone with a use case that could benefit from this, please chime in. > > Regards, > Stefan > > > > On 2018-09-07 17:37, Rodrigo Bruno wrote: >> Hi Per and Thomas, >> >> thank you for your comments. >> >> I think it is possible to implement this feature using the service >> thread or using a separate thread. >> I see some pros and cons of having a separate thread: >> >> Pros: >> - using the service thread exposes something that is G1 specific to >> the rest of the JVM. >> Thus, using a separate thread, hides this feature from the outsite. >> >> Cons: >> - Having a manageable timeout is a bit more tricky to implement in a >> separate/dedicated thread. >> We need to be able to handle switch on and off. It might require some >> variable pooling. >> - It requires some more memory. >> >> Regardless of the path taken, I can prepare a new version of the patch >> whenever we decide on this. >> >> cheers, >> rodrigo >> >> Per Liden > >> escreveu no dia sexta, 7/09/2018 ?(s) 11:58: >> >> Hi Thomas, >> >> On 09/07/2018 10:10 AM, Thomas Schatzl wrote: >> [...] >> > overnight I thought a bit of the implementation, and given the >> > problem with heap usage of the new thread, and the requirement of >> being >> > able to turn on/off that feature by a managed variable, the best >> change >> > would probably reusing the service thread as you did in the >> initial >> > change. >> >> I'm not convinced that this should be handled outside of G1. If >> there's >> a need to have the flag manageable at runtime (is that really the >> case?), you could just always start the G1DetectIdleThread and >> have it >> check the flag. I wouldn't worry too much about the memory >> overhead for >> the stack. >> >> cheers, >> Per >> -- Ruslan CEO @ Jelastic -------------- next part -------------- An HTML attachment was scrubbed... URL: From nijiaben at perfma.com Sat Sep 22 06:55:40 2018 From: nijiaben at perfma.com (=?utf-8?B?bmlqaWFiZW4=?=) Date: Sat, 22 Sep 2018 14:55:40 +0800 Subject: A Bug Of Metaspace After Full GC Message-ID: Hi, all We use metaspace instead of perm since JDK8, I found that after the Full GC, the size of the metaspace has always been the same from the GC log. I have looked at the source code of hotspot roughly. It has not bean fixed in the latest version. The problem which may have some impact on us to troubleshoot some metaspace problems. I wrote a test case, and simply analyzed the hotspot code, and showed the output before and after the modification. I hope this problem can be fixed as soon as possible. Demo? import java.io.File; import java.net.URL; import java.net.URLClassLoader; /** * * @author nijiaben * @version v1.0 * @createTime 2018?09?22? 12:32:32 PM * */ public class ClassLoaderTest { private static URL[] urls = new URL[1]; static { try { File jarFile = new File("/home/admin/.m2/repository/org/slf4j/slf4j-api/1.7.21/slf4j-api-1.7.21.jar"); urls[0] = jarFile.toURI().toURL(); } catch (Exception e) { e.printStackTrace(); } } public static void main(String args[]) { for(int i=0;i<1000;i++) { loadClass(); } System.gc(); } public static void loadClass() { try { URLClassLoader ucl = new URLClassLoader(urls); Class.forName("org.slf4j.Logger", false, ucl); } catch (Exception e) { e.printStackTrace(); } } } My demo is very simple, just constantly create a new class loader to load a specific class, after loading, execute a System GC, so that we can see the relevant GC log output, the JVM parameters we run are -Xmx1000M -Xms1000M -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails ClassLoaderTest The GC log is [Full GC (System.gc()) [CMS: 0K->9921K(683264K), 0.1003988 secs] 49083K->9921K(989952K), [Metaspace: 12916K->12916K(1064960K)], 0.1060065 secs] [Times: user=0.06 sys=0.05, real=0.11 secs] Let's look at the source code of hotspot, GenCollectedHeap::do_collection: if (PrintGCDetails) { print_heap_change(gch_prev_used); // Print metaspace info for full GC with PrintGCDetails flag. if (complete) { MetaspaceAux::print_metaspace_change(metadata_prev_used); } } for (int j = max_level_collected; j >= 0; j -= 1) { // Adjust generation sizes. _gens[j]->compute_new_size(); } if (complete) { // Delete metaspaces for unloaded class loaders and clean up loader_data graph ClassLoaderDataGraph::purge(); MetaspaceAux::verify_metrics(); // Resize the metaspace capacity after full collections MetaspaceGC::compute_new_size(); update_full_collections_completed(); } We see that metaspace's gc log output is executed before ClassLoaderDataGraph::purge(); , that means the size of the metaspace is printed without reclaiming the memory of the metaspace, so the GC log seems no change before and after the GC metaspace. After knowing the specific problem, we can make an adjustment to the code to move the logic of the output of metaspace size after to ClassLoaderDataGraph::purge()?just like this: if (PrintGCDetails) { print_heap_change(gch_prev_used); } for (int j = max_level_collected; j >= 0; j -= 1) { // Adjust generation sizes. _gens[j]->compute_new_size(); } if (complete) { // Delete metaspaces for unloaded class loaders and clean up loader_data graph ClassLoaderDataGraph::purge(); MetaspaceAux::verify_metrics(); // Resize the metaspace capacity after full collections MetaspaceGC::compute_new_size(); update_full_collections_completed(); if (PrintGCDetails) { // Print metaspace info for full GC with PrintGCDetails flag. MetaspaceAux::print_metaspace_change(metadata_prev_used); } } At this point we recompile hotspot, execute the above demo again, you can see the correct output as follows [Full GC (System.gc()) [CMS: 0K->9921K(683264K), 0.0852627 secs] 49083K->9921K(989952K), [Metaspace: 12913K->3280K(1058816K)], 0.0891207 secs] [Times: user=0.05 sys=0.04, real=0.09 secs] I also probably thought about why this problem is caused. This is the version before JDK8, such as JDK7, you can see the same code location: if (PrintGCDetails) { print_heap_change(gch_prev_used); // Print perm gen info for full GC with PrintGCDetails flag. if (complete) { print_perm_heap_change(perm_prev_used); } } for (int j = max_level_collected; j >= 0; j -= 1) { // Adjust generation sizes. _gens[j]->compute_new_size(); } if (complete) { // Ask the permanent generation to adjust size for full collections perm()->compute_new_size(); update_full_collections_completed(); } It can be seen that the GC engineer at that time simply replaced print_perm_heap_change(perm_prev_used);with MetaspaceAux::print_metaspace_change(metadata_prev_used);but forgot the difference between Perm and Metaspace. Thanks, nijiaben -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.buck at oracle.com Tue Sep 25 01:37:29 2018 From: david.buck at oracle.com (David Buck) Date: Tue, 25 Sep 2018 10:37:29 +0900 Subject: RFR (S) [8u] backport 8141421: Various test fail with OOME on win x86 Message-ID: <54c04c8e-7677-3723-5335-7738917fa575@oracle.com> Hi! May I please get a review of this simple backport to 8u-dev? The only reason the JDK 9 fix does not apply as-is to JDK 8 is because of changes to assert in JDK 9 [0]. bug report: https://bugs.openjdk.java.net/browse/JDK-8141421 JDK 9 review thread: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2016-January/016251.html JDK 9 changeset: http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/2de6311c5afc JDK 8 webrev (for review): http://cr.openjdk.java.net/~dbuck/8141421_hotspot_jdk8_ver01/ Cheers, -Buck [0] https://bugs.openjdk.java.net/browse/JDK-8080775 From stefan.johansson at oracle.com Tue Sep 25 14:48:55 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Tue, 25 Sep 2018 16:48:55 +0200 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: References: <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> <65063e5f31702649e12b485b570ecab9ebd32d01.camel@oracle.com> <66c536399b2f218af0d85bd3bced44f204e3dbf4.camel@oracle.com> <2821cf70-c4d5-016f-b976-5ecc850f377c@oracle.com> <57b10708-b870-cb98-0df0-fab650f431e4@oracle.com> Message-ID: <3927596f-8847-ba95-d4c6-3c5f3656330c@oracle.com> Thanks Ruslan for your input, On 2018-09-21 15:35, Ruslan Synytsky wrote: > Dear Stefan and Rodrigo, thank you for moving this forward. > > ---------- Forwarded message --------- >> From: *Stefan Johansson* > > >> Date: quarta, 19/09/2018 ?(s) 10:45 >> Subject: Re: RFR: bug: Timely Reducing Unused Committed Memory >> To: > >, > > >> >> >> Hi Rodrigo, >> >> I pasted your reply here to keep the discussion in one thread. >> >> >> I understand that it is hard to define what is idle. However, if we >> require the >> >> user to provide one, I guess that most regular users that suffer >> from the problem >> >> that this patch is trying to solve will simply not do it because it >> requires knowledge >> >> and effort. If we provide an idle check that we think will benefit >> most users, then >> >> we are probably helping a lot of users. For those that the default >> idle check is >> >> not good enough, they can always disable this idle check and >> implement the idle >> >> check logic it in an external tool. >> >> >> > I agree, if we can find a solution that benefits most users, we should >> > do it. And this is why I would like to hear from more users if this >> > would benefit their use cases. > I believe the default idle definition should be based on the major > bottlenecks: RAM, CPU and IO loads as well as the network. RAM - we try > to improve. IO - I?m not sure if we can measure IO load properly inside > JVM. If possible then it's good to add too. If not then we can skip it > for now, as it can be measured and triggered by outside logic. Network > is not involved in GC process, correct? So no need for that. CPU looks > the most obvious and already implemented, seems like a good option to > start from. I agree that CPU can look obvious, but making decisions in the VM based on the system load might be hard. For example the avg load might be low while the current process is fairly active. Another question, when running in the cloud, what load is the user expecting us to compare against, the overall system or the local container. I'm actually not entirely sure what the getloadavg() call return in case of running in a container. > >> > Another thing that I don't fully >> > understand is why the flags are manageable if there isn't supposed >> to be >> > some external logic that sets them? > Some advanced users, for example cloud platform or software vendors, > will be able to apply an additional logic based on their custom needs / > specifics. Such flexibility enables more use cases and it helps to > collect more feedback for the further default improvements. That's how I would expect it to be used as well, thanks for clarifying your viewpoint. >> >> >> We can also change the semantics of "idleness".? Currently it >> checks the load. >> >> I think that checking the allocation rate might be another good >> option (instead of >> >> load). The only corner case is? an application that does not >> allocate but consumes >> >> a lot of CPU. For this case, we might only trigger compaction at >> most once because, >> >> as it does not allocate memory, we will not get over committed >> memory (i.e., the other >> >> checks will prevent it). The opposite is also possible (almost idle >> application that allocates >> >> a lot of memory) but in this scenario I don't think we want to >> trigger an idle compaction. >> >> >> >> > This is my main problem when it comes to determine "idleness", for some >> > applications allocation rate will be the correct metric, for others it >> > will be the load and for a third something different. It feels like it >> > is always possible to come up with a case that needs something >> different. > I would prefer to start with the most obvious one - based on CPU, give > it to try to more people by promoting the fact that JVM is elastic now, > and we will get more feedback that can be converted into an additional > logic later. > So basically, the first version would have two flags, one to turn on periodic GCs (currently named GCFrequency) and one to control at which average load (MaxLoadGC) these GCs will kick in? >> >> Having said that, I am open to change this flag or even remove it >> as it is one of the >> >> hardest to get right. >> >> >> >> > As I said before, to me it feels like just having a periodic GC >> interval >> > flag that is manageable would be a good start. Maybe have constraint >> > that the periodic GC only occurs if no other GCs have happened during >> > the interval. >> > Decision based on the previous GC cycles is very good proposal. I think > we need to take it into account somehow, but I'm not so deep on it. > Input of others will be helpful here. I guess there are corner cases in this area as well, but I guess the simple constraint I described might be a good start. But as you say, input from others would be very helpful. >> > Could you explain how your use case would suffer from such >> > limitations? > In my opinion, CPU load spikes is clearly one of the major use cases > eligible?for defaults. This is clear and good use case where I guess having a load threshold should really help. Thanks, Stefan > > Thank you > >> >> > Thanks, >> > Stefan >> >> >> cheers, >> >> rodrigo >> >> >> On 2018-09-13 14:30, Stefan Johansson wrote: >> > Hi Rodrigo, >> > >> > Sorry for being a bit late into the discussion. We've had some internal >> > discussions and realized that there are some questions that I need to >> > bring up here. >> > >> > I'm trying to better understand under what circumstances this >> feature is >> > to be used and how a user should use the different flags to tweak it to >> > their use case. To me it feels like GCFrequency would be enough to make >> > sure that the VM returns memory on a timely basis. And if the flag is >> > managed, it can be controlled to not do periodic GCs during high load. >> > With that we get a good way to periodically try to reduce the committed >> > heap. >> > >> > The reason I ask is because I have a hard time seeing how we can >> > implement a generic policy for when the system is idle. A policy that >> > will apply well to most use cases. For some cases having the flags you >> > propose might be good, but for other there might be a different set of >> > options needed. If this is the case then maybe the logic and policy of >> > when to do this can live outside the VM, while the code to periodically >> > do GCs lives within the VM. What do you think about that? I understand >> > the problems you've stated with having the policy outside that VM, but >> > at least we have more information to act on there. >> > >> > We know that many have asked for features similar to this one and it >> > would be nice to get input from others on this to make sure we >> implement >> > something that benefits the whole user base as much as possible. So >> > anyone with a use case that could benefit from this, please chime in. >> > >> > Regards, >> > Stefan >> > >> > >> > >> > On 2018-09-07 17:37, Rodrigo Bruno wrote: >> >> Hi Per and Thomas, >> >> >> >> thank you for your comments. >> >> >> >> I think it is possible to implement this feature using the service >> >> thread or using a separate thread. >> >> I see some pros and cons of having a separate thread: >> >> >> >> Pros: >> >> - using the service thread exposes something that is G1 specific to >> >> the rest of the JVM. >> >> Thus, using a separate thread, hides this feature from the outsite. >> >> >> >> Cons: >> >> - Having a manageable timeout is a bit more tricky to implement in a >> >> separate/dedicated thread. >> >> We need to be able to handle switch on and off. It might require some >> >> variable pooling. >> >> - It requires some more memory. >> >> >> >> Regardless of the path taken, I can prepare a new version of the patch >> >> whenever we decide on this. >> >> >> >> cheers, >> >> rodrigo >> >> >> >> Per Liden >> >> >> >> escreveu no dia sexta, 7/09/2018 ?(s) 11:58: >> >> >> >> ??? Hi Thomas, >> >> >> >> ??? On 09/07/2018 10:10 AM, Thomas Schatzl wrote: >> >> ??? [...] >> >> ???? >? ? overnight I thought a bit of the implementation, and >> given the >> >> ???? > problem with heap usage of the new thread, and the >> requirement of >> >> ??? being >> >> ???? > able to turn on/off that feature by a managed variable, the best >> >> ??? change >> >> ???? > would probably reusing the service thread as you did in the >> >> initial >> >> ???? > change. >> >> >> >> ??? I'm not convinced that this should be handled outside of G1. If >> >> there's >> >> ??? a need to have the flag manageable at runtime (is that really the >> >> ??? case?), you could just always start the G1DetectIdleThread and >> >> have it >> >> ??? check the flag. I wouldn't worry too much about the memory >> >> overhead for >> >> ??? the stack. >> >> >> >> ??? cheers, >> >> ??? Per >> >> > > > > -- > Ruslan > CEO @ Jelastic From daniel.daugherty at oracle.com Tue Sep 25 15:11:18 2018 From: daniel.daugherty at oracle.com (Daniel D. Daugherty) Date: Tue, 25 Sep 2018 11:11:18 -0400 Subject: RFR: 8208686: [AOT] JVMTI ResourceExhausted event repeated for same allocation In-Reply-To: <910A4A3C-3EEF-4167-84D5-9819C83D6FC1@oracle.com> References: <910A4A3C-3EEF-4167-84D5-9819C83D6FC1@oracle.com> Message-ID: <64cbe730-5c9a-a04d-9eee-a56abfbb8e07@oracle.com> Adding serviceability-dev at ... since this is JVM/TI... Dan On 9/25/18 10:48 AM, Doug Simon wrote: > A major design point of Graal is to treat allocations as non-side effecting to give more freedom to the optimizer by reducing the number of distinct FrameStates that need to be managed. When failing an allocation, Graal will deoptimize to the last side effecting instruction before the allocation. This mean the VM code for heap allocation will potentially be executed twice, once from Graal compiled code and then again in the interpreter. While this is perfectly fine according to the JVM specification, it can cause confusing behavior for JVMTI based tools. They will receive 2 ResourceExhausted events for a single allocation. Furthermore, the first ResourceExhausted event (on the Graal allocation slow path) might denote a bytecode instruction that performs no allocation, making it hard to debug the memory failure. > > The proposed solution is to add an extra set of JVMCI VM runtime calls for allocation. These entry points will attempt the allocation and upon failure, > skip side-effects such as posting JVMTI events or handling -XX:OnOutOfMemoryError. The compiled code using these entry points is expected deoptmize on null. > > The path from these new entry points to where allocation can fail goes through quite a bit of VM code. One could modify all these paths by: > * Returning null instead of throwing an exception on failure. > * Adding a `bool null_on_fail` argument to all relevant methods. > * Adding extra null checking where necessary after each call to these methods when `null_on_fail == true`. > This represents a significant number of changes. > > Instead, the proposed solution introduces a new _in_retryable_allocation thread-local. This way, only the entry points and allocation routines that raise an exception need to be modified. Failure is communicated back to the new entry points by throwing a special pre-allocated OOME object (i.e., Universe::out_of_memory_error_retry()) which must not propagate back to Java code. Use of this object is not strictly necessary; it is introduced to highlight/document the special allocation mode. > > The proposed solution is at http://cr.openjdk.java.net/~dnsimon/8208686. > THE JBS bug is: https://bugs.openjdk.java.net/browse/JDK-8208686 > > -Doug From jcbeyler at google.com Tue Sep 25 19:57:35 2018 From: jcbeyler at google.com (JC Beyler) Date: Tue, 25 Sep 2018 12:57:35 -0700 Subject: A Bug Of Metaspace After Full GC In-Reply-To: References: Message-ID: Hi Nijiaben, I'm not a GC engineer but thought I would look at this by curiosity (for note, I also created https://bugs.openjdk.java.net/browse/JDK-8211123 for tracking purposes). It seems you are using an older JDK because the code has changed somewhat there. I looked at this and used your reproducer on JDK12 tip: Using CMS (which is being deprecated from what I know and see in the log message): $ ~/mercurial/jdk12-gc-print/build/linux-x86_64-server-release/images/jdk/bin/java -XX:+UseConcMarkSweepGC -Xmx1000M -Xms1000M -Xlog:gc* ClassLoaderTest OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. ... [0.559s][info][gc,metaspace ] GC(0) Metaspace: 15267K->15267K(1067008K) (Note: I am using -Xlog:gc* since -XX:+PrintGCDetails is being deprecated as well from what the log message is saying) So the bug is there for CMS, but for G1, it seems to work: $ ~/mercurial/jdk12-gc-print/build/linux-x86_64-server-release/images/jdk/bin/java -Xmx1000M -Xms1000M -Xlog:gc* ClassLoaderTest ... [0.653s][info][gc,metaspace ] GC(0) Metaspace: 15266K->5266K(1062912K) ... So I looked at the other ones as well and as a summary: - The bug seems to show up for Serial, CMS - The bug does not seem to show up for Parallel, G1 I then updated the code like what you were showing though things have changed in tip so my change became a simple one-line move: diff -r 8f66a57054b7 src/hotspot/share/gc/shared/genCollectedHeap.cpp --- a/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 10:30:32 2018 -0700 +++ b/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 12:50:54 2018 -0700 @@ -649,7 +649,6 @@ complete = complete || collected_old; print_heap_change(young_prev_used, old_prev_used); - MetaspaceUtils::print_metaspace_change(metadata_prev_used); // Adjust generation sizes. if (collected_old) { @@ -665,6 +664,7 @@ MetaspaceGC::compute_new_size(); update_full_collections_completed(); } + MetaspaceUtils::print_metaspace_change(metadata_prev_used); // Track memory usage and detect low memory after GC finishes MemoryService::track_memory_usage(); With this, I looked at the various GC results and this fixes: - Serial, CMS; while keeping Parallel and G1 working (seemingly) Finally, note that all GC logging now are logging the summary at the exit and this line shows up: [0.560s][info][gc,heap,exit ] Metaspace used 5277K, capacity 5310K, committed 15232K, reserved 1062912K For all GCs, so though the line "Metaspace before -> after" is false, you can get the the after with the final line. If the GC experts on this list want me to make a webrev out of this and get reviews for it, I can do that. Hoping that helps, Jc On Fri, Sep 21, 2018 at 11:56 PM nijiaben wrote: > Hi, all > > We use metaspace instead of perm since JDK8, I found that after the Full > GC, the size of the metaspace has always been the same from the GC log. I > have looked at the source code of hotspot roughly. It has not bean fixed in > the latest version. The problem which may have some impact on us to > troubleshoot some metaspace problems. > > I wrote a test case, and simply analyzed the hotspot code, and showed the > output before and after the modification. I hope this problem can be fixed > as soon as possible. > > Demo? > > import java.io.File; > import java.net.URL; > import java.net.URLClassLoader; > > /** > * > * @author nijiaben > * @version v1.0 > * @createTime 2018?09?22? 12:32:32 PM > * > */ > public class ClassLoaderTest { > private static URL[] urls = new URL[1]; > > static { > try { > File jarFile = new File("/home/admin/.m2/repository/org/slf4j/slf4j-api/1.7.21/slf4j-api-1.7.21.jar"); > urls[0] = jarFile.toURI().toURL(); > } catch (Exception e) { > e.printStackTrace(); > } > } > > public static void main(String args[]) { > for(int i=0;i<1000;i++) { > loadClass(); > } > System.gc(); > } > > public static void loadClass() { > try { > URLClassLoader ucl = new URLClassLoader(urls); > Class.forName("org.slf4j.Logger", false, ucl); > } catch (Exception e) { > e.printStackTrace(); > } > } > } > > My demo is very simple, just constantly create a new class loader to load > a specific class, after loading, execute a System GC, so that we can see > the relevant GC log output, the JVM parameters we run are > > -Xmx1000M -Xms1000M -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails ClassLoaderTest > > The GC log is > > [Full GC (System.gc()) [CMS: 0K->9921K(683264K), 0.1003988 secs] 49083K->9921K(989952K), [Metaspace: 12916K->12916K(1064960K)], 0.1060065 secs] [Times: user=0.06 sys=0.05, real=0.11 secs] > > Let's look at the source code of hotspot, > > GenCollectedHeap::do_collection: > > if (PrintGCDetails) { > print_heap_change(gch_prev_used); > > // Print metaspace info for full GC with PrintGCDetails flag. > if (complete) { > MetaspaceAux::print_metaspace_change(metadata_prev_used); > } > } > > for (int j = max_level_collected; j >= 0; j -= 1) { > // Adjust generation sizes. > _gens[j]->compute_new_size(); > } > > if (complete) { > // Delete metaspaces for unloaded class loaders and clean up loader_data graph > ClassLoaderDataGraph::purge(); > MetaspaceAux::verify_metrics(); > // Resize the metaspace capacity after full collections > MetaspaceGC::compute_new_size(); > update_full_collections_completed(); > } > > We see that metaspace's gc log output is executed before > ClassLoaderDataGraph::purge(); , that means the size of the metaspace is > printed without reclaiming the memory of the metaspace, so the GC log seems > no change before and after the GC metaspace. > > After knowing the specific problem, we can make an adjustment to the code > to move the logic of the output of metaspace size after to > ClassLoaderDataGraph::purge()?just like this: > > if (PrintGCDetails) { > print_heap_change(gch_prev_used); > } > > for (int j = max_level_collected; j >= 0; j -= 1) { > // Adjust generation sizes. > _gens[j]->compute_new_size(); > } > > if (complete) { > // Delete metaspaces for unloaded class loaders and clean up loader_data graph > ClassLoaderDataGraph::purge(); > MetaspaceAux::verify_metrics(); > // Resize the metaspace capacity after full collections > MetaspaceGC::compute_new_size(); > update_full_collections_completed(); > if (PrintGCDetails) { > // Print metaspace info for full GC with PrintGCDetails flag. > MetaspaceAux::print_metaspace_change(metadata_prev_used); > } > } > > At this point we recompile hotspot, execute the above demo again, you can > see the correct output as follows > > [Full GC (System.gc()) [CMS: 0K->9921K(683264K), 0.0852627 secs] 49083K->9921K(989952K), [Metaspace: 12913K->3280K(1058816K)], 0.0891207 secs] [Times: user=0.05 sys=0.04, real=0.09 secs] > > I also probably thought about why this problem is caused. This is the > version before JDK8, such as JDK7, you can see the same code location: > > if (PrintGCDetails) { > print_heap_change(gch_prev_used); > > // Print perm gen info for full GC with PrintGCDetails flag. > if (complete) { > print_perm_heap_change(perm_prev_used); > } > } > > for (int j = max_level_collected; j >= 0; j -= 1) { > // Adjust generation sizes. > _gens[j]->compute_new_size(); > } > > if (complete) { > // Ask the permanent generation to adjust size for full collections > perm()->compute_new_size(); > update_full_collections_completed(); > } > > It can be seen that the GC engineer at that time simply replaced > print_perm_heap_change(perm_prev_used);with > MetaspaceAux::print_metaspace_change(metadata_prev_used);but forgot the > difference between Perm and Metaspace. > > Thanks, > > nijiaben > -- Thanks, Jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From rednaxelafx at gmail.com Tue Sep 25 20:08:02 2018 From: rednaxelafx at gmail.com (Krystal Mok) Date: Tue, 25 Sep 2018 13:08:02 -0700 Subject: A Bug Of Metaspace After Full GC In-Reply-To: References: Message-ID: Hi JC, Just a quick comment on: So I looked at the other ones as well and as a summary: - The bug seems to show up for Serial, CMS - The bug does not seem to show up for Parallel, G1 That's because Serial (DefNew) / Serial Old (Tenured / GenMarkSweep) / ParNew / CMS use the "generational GC framework" that's been there for years. That's the "GenCollectedHeap" touched by Nijiaben's propsed patch. It was there with the idea that different GC implementations can plug into the same framework and supposedly they could mix-and-match. But later on only a few possible configurations were really tested well and supported. Parallel (ParallelScavenge) / Parallel Old and G1 use their own implementation of the CollectedHeap interface and doesn't participate in that "generational GC framework". So they don't have the same bug at the same place. - Kris On Tue, Sep 25, 2018 at 12:57 PM, JC Beyler wrote: > Hi Nijiaben, > > I'm not a GC engineer but thought I would look at this by curiosity (for > note, I also created https://bugs.openjdk.java.net/browse/JDK-8211123 for > tracking purposes). It seems you are using an older JDK because the code > has changed somewhat there. > > I looked at this and used your reproducer on JDK12 tip: > > Using CMS (which is being deprecated from what I know and see in the log > message): > $ ~/mercurial/jdk12-gc-print/build/linux-x86_64-server-release/images/jdk/bin/java > -XX:+UseConcMarkSweepGC -Xmx1000M -Xms1000M -Xlog:gc* ClassLoaderTest > OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated > in version 9.0 and will likely be removed in a future release. > ... > [0.559s][info][gc,metaspace ] GC(0) Metaspace: 15267K->15267K(1067008K) > > (Note: I am using -Xlog:gc* since -XX:+PrintGCDetails is being deprecated > as well from what the log message is saying) > > So the bug is there for CMS, but for G1, it seems to work: > > $ ~/mercurial/jdk12-gc-print/build/linux-x86_64-server-release/images/jdk/bin/java > -Xmx1000M -Xms1000M -Xlog:gc* ClassLoaderTest > ... > [0.653s][info][gc,metaspace ] GC(0) Metaspace: 15266K->5266K(1062912K) > ... > > So I looked at the other ones as well and as a summary: > - The bug seems to show up for Serial, CMS > - The bug does not seem to show up for Parallel, G1 > > > I then updated the code like what you were showing though things have > changed in tip so my change became a simple one-line move: > > diff -r 8f66a57054b7 src/hotspot/share/gc/shared/genCollectedHeap.cpp > --- a/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 > 10:30:32 2018 -0700 > +++ b/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 > 12:50:54 2018 -0700 > @@ -649,7 +649,6 @@ > complete = complete || collected_old; > > print_heap_change(young_prev_used, old_prev_used); > - MetaspaceUtils::print_metaspace_change(metadata_prev_used); > > // Adjust generation sizes. > if (collected_old) { > @@ -665,6 +664,7 @@ > MetaspaceGC::compute_new_size(); > update_full_collections_completed(); > } > + MetaspaceUtils::print_metaspace_change(metadata_prev_used); > > // Track memory usage and detect low memory after GC finishes > MemoryService::track_memory_usage(); > > With this, I looked at the various GC results and this fixes: > - Serial, CMS; while keeping Parallel and G1 working (seemingly) > > Finally, note that all GC logging now are logging the summary at the exit > and this line shows up: > [0.560s][info][gc,heap,exit ] Metaspace used 5277K, capacity > 5310K, committed 15232K, reserved 1062912K > > For all GCs, so though the line "Metaspace before -> after" is false, you > can get the the after with the final line. > > If the GC experts on this list want me to make a webrev out of this and > get reviews for it, I can do that. > > Hoping that helps, > Jc > > > > On Fri, Sep 21, 2018 at 11:56 PM nijiaben wrote: > >> Hi, all >> >> We use metaspace instead of perm since JDK8, I found that after the Full >> GC, the size of the metaspace has always been the same from the GC log. I >> have looked at the source code of hotspot roughly. It has not bean fixed in >> the latest version. The problem which may have some impact on us to >> troubleshoot some metaspace problems. >> >> I wrote a test case, and simply analyzed the hotspot code, and showed the >> output before and after the modification. I hope this problem can be fixed >> as soon as possible. >> >> Demo? >> >> import java.io.File; >> import java.net.URL; >> import java.net.URLClassLoader; >> >> /** >> * >> * @author nijiaben >> * @version v1.0 >> * @createTime 2018?09?22? 12:32:32 PM >> * >> */ >> public class ClassLoaderTest { >> private static URL[] urls = new URL[1]; >> >> static { >> try { >> File jarFile = new File("/home/admin/.m2/repository/org/slf4j/slf4j-api/1.7.21/slf4j-api-1.7.21.jar"); >> urls[0] = jarFile.toURI().toURL(); >> } catch (Exception e) { >> e.printStackTrace(); >> } >> } >> >> public static void main(String args[]) { >> for(int i=0;i<1000;i++) { >> loadClass(); >> } >> System.gc(); >> } >> >> public static void loadClass() { >> try { >> URLClassLoader ucl = new URLClassLoader(urls); >> Class.forName("org.slf4j.Logger", false, ucl); >> } catch (Exception e) { >> e.printStackTrace(); >> } >> } >> } >> >> My demo is very simple, just constantly create a new class loader to load >> a specific class, after loading, execute a System GC, so that we can see >> the relevant GC log output, the JVM parameters we run are >> >> -Xmx1000M -Xms1000M -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails ClassLoaderTest >> >> The GC log is >> >> [Full GC (System.gc()) [CMS: 0K->9921K(683264K), 0.1003988 secs] 49083K->9921K(989952K), [Metaspace: 12916K->12916K(1064960K)], 0.1060065 secs] [Times: user=0.06 sys=0.05, real=0.11 secs] >> >> Let's look at the source code of hotspot, >> >> GenCollectedHeap::do_collection: >> >> if (PrintGCDetails) { >> print_heap_change(gch_prev_used); >> >> // Print metaspace info for full GC with PrintGCDetails flag. >> if (complete) { >> MetaspaceAux::print_metaspace_change(metadata_prev_used); >> } >> } >> >> for (int j = max_level_collected; j >= 0; j -= 1) { >> // Adjust generation sizes. >> _gens[j]->compute_new_size(); >> } >> >> if (complete) { >> // Delete metaspaces for unloaded class loaders and clean up loader_data graph >> ClassLoaderDataGraph::purge(); >> MetaspaceAux::verify_metrics(); >> // Resize the metaspace capacity after full collections >> MetaspaceGC::compute_new_size(); >> update_full_collections_completed(); >> } >> >> We see that metaspace's gc log output is executed before >> ClassLoaderDataGraph::purge(); , that means the size of the metaspace is >> printed without reclaiming the memory of the metaspace, so the GC log seems >> no change before and after the GC metaspace. >> >> After knowing the specific problem, we can make an adjustment to the code >> to move the logic of the output of metaspace size after to >> ClassLoaderDataGraph::purge()?just like this: >> >> if (PrintGCDetails) { >> print_heap_change(gch_prev_used); >> } >> >> for (int j = max_level_collected; j >= 0; j -= 1) { >> // Adjust generation sizes. >> _gens[j]->compute_new_size(); >> } >> >> if (complete) { >> // Delete metaspaces for unloaded class loaders and clean up loader_data graph >> ClassLoaderDataGraph::purge(); >> MetaspaceAux::verify_metrics(); >> // Resize the metaspace capacity after full collections >> MetaspaceGC::compute_new_size(); >> update_full_collections_completed(); >> if (PrintGCDetails) { >> // Print metaspace info for full GC with PrintGCDetails flag. >> MetaspaceAux::print_metaspace_change(metadata_prev_used); >> } >> } >> >> At this point we recompile hotspot, execute the above demo again, you can >> see the correct output as follows >> >> [Full GC (System.gc()) [CMS: 0K->9921K(683264K), 0.0852627 secs] 49083K->9921K(989952K), [Metaspace: 12913K->3280K(1058816K)], 0.0891207 secs] [Times: user=0.05 sys=0.04, real=0.09 secs] >> >> I also probably thought about why this problem is caused. This is the >> version before JDK8, such as JDK7, you can see the same code location: >> >> if (PrintGCDetails) { >> print_heap_change(gch_prev_used); >> >> // Print perm gen info for full GC with PrintGCDetails flag. >> if (complete) { >> print_perm_heap_change(perm_prev_used); >> } >> } >> >> for (int j = max_level_collected; j >= 0; j -= 1) { >> // Adjust generation sizes. >> _gens[j]->compute_new_size(); >> } >> >> if (complete) { >> // Ask the permanent generation to adjust size for full collections >> perm()->compute_new_size(); >> update_full_collections_completed(); >> } >> >> It can be seen that the GC engineer at that time simply replaced >> print_perm_heap_change(perm_prev_used);with Me >> taspaceAux::print_metaspace_change(metadata_prev_used);but forgot the >> difference between Perm and Metaspace. >> >> Thanks, >> >> nijiaben >> > > > -- > > Thanks, > Jc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcbeyler at google.com Tue Sep 25 20:22:04 2018 From: jcbeyler at google.com (JC Beyler) Date: Tue, 25 Sep 2018 13:22:04 -0700 Subject: A Bug Of Metaspace After Full GC In-Reply-To: References: Message-ID: I did say I was not a GC expert :-) So the question would be: is the patch I showed based on Nijiaben's patch worth fixing and is it right? It seems like the right thing to do for at least fixing SerialGC printouts, no? Jc Ps: here is the patch for simplifying things for everyone then: diff -r 8f66a57054b7 src/hotspot/share/gc/shared/genCollectedHeap.cpp --- a/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 10:30:32 2018 -0700 +++ b/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 12:50:54 2018 -0700 @@ -649,7 +649,6 @@ complete = complete || collected_old; print_heap_change(young_prev_used, old_prev_used); - MetaspaceUtils::print_metaspace_change(metadata_prev_used); // Adjust generation sizes. if (collected_old) { @@ -665,6 +664,7 @@ MetaspaceGC::compute_new_size(); update_full_collections_completed(); } + MetaspaceUtils::print_metaspace_change(metadata_prev_used); // Track memory usage and detect low memory after GC finishes MemoryService::track_memory_usage(); On Tue, Sep 25, 2018 at 1:08 PM Krystal Mok wrote: > Hi JC, > > Just a quick comment on: > > So I looked at the other ones as well and as a summary: > - The bug seems to show up for Serial, CMS > - The bug does not seem to show up for Parallel, G1 > > That's because Serial (DefNew) / Serial Old (Tenured / GenMarkSweep) / > ParNew / CMS use the "generational GC framework" that's been there for > years. That's the "GenCollectedHeap" touched by Nijiaben's propsed patch. > It was there with the idea that different GC implementations can plug into > the same framework and supposedly they could mix-and-match. But later on > only a few possible configurations were really tested well and supported. > > Parallel (ParallelScavenge) / Parallel Old and G1 use their own > implementation of the CollectedHeap interface and doesn't participate in > that "generational GC framework". So they don't have the same bug at the > same place. > > - Kris > > On Tue, Sep 25, 2018 at 12:57 PM, JC Beyler wrote: > >> Hi Nijiaben, >> >> I'm not a GC engineer but thought I would look at this by curiosity (for >> note, I also created https://bugs.openjdk.java.net/browse/JDK-8211123 >> for tracking purposes). It seems you are using an older JDK because the >> code has changed somewhat there. >> >> I looked at this and used your reproducer on JDK12 tip: >> >> Using CMS (which is being deprecated from what I know and see in the log >> message): >> $ >> ~/mercurial/jdk12-gc-print/build/linux-x86_64-server-release/images/jdk/bin/java >> -XX:+UseConcMarkSweepGC -Xmx1000M -Xms1000M -Xlog:gc* ClassLoaderTest >> OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was >> deprecated in version 9.0 and will likely be removed in a future release. >> ... >> [0.559s][info][gc,metaspace ] GC(0) Metaspace: 15267K->15267K(1067008K) >> >> (Note: I am using -Xlog:gc* since -XX:+PrintGCDetails is being deprecated >> as well from what the log message is saying) >> >> So the bug is there for CMS, but for G1, it seems to work: >> >> $ >> ~/mercurial/jdk12-gc-print/build/linux-x86_64-server-release/images/jdk/bin/java >> -Xmx1000M -Xms1000M -Xlog:gc* ClassLoaderTest >> ... >> [0.653s][info][gc,metaspace ] GC(0) Metaspace: 15266K->5266K(1062912K) >> ... >> >> So I looked at the other ones as well and as a summary: >> - The bug seems to show up for Serial, CMS >> - The bug does not seem to show up for Parallel, G1 >> >> >> I then updated the code like what you were showing though things have >> changed in tip so my change became a simple one-line move: >> >> diff -r 8f66a57054b7 src/hotspot/share/gc/shared/genCollectedHeap.cpp >> --- a/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 >> 10:30:32 2018 -0700 >> +++ b/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 >> 12:50:54 2018 -0700 >> @@ -649,7 +649,6 @@ >> complete = complete || collected_old; >> >> print_heap_change(young_prev_used, old_prev_used); >> - MetaspaceUtils::print_metaspace_change(metadata_prev_used); >> >> // Adjust generation sizes. >> if (collected_old) { >> @@ -665,6 +664,7 @@ >> MetaspaceGC::compute_new_size(); >> update_full_collections_completed(); >> } >> + MetaspaceUtils::print_metaspace_change(metadata_prev_used); >> >> // Track memory usage and detect low memory after GC finishes >> MemoryService::track_memory_usage(); >> >> With this, I looked at the various GC results and this fixes: >> - Serial, CMS; while keeping Parallel and G1 working (seemingly) >> >> Finally, note that all GC logging now are logging the summary at the exit >> and this line shows up: >> [0.560s][info][gc,heap,exit ] Metaspace used 5277K, capacity >> 5310K, committed 15232K, reserved 1062912K >> >> For all GCs, so though the line "Metaspace before -> after" is false, you >> can get the the after with the final line. >> >> If the GC experts on this list want me to make a webrev out of this and >> get reviews for it, I can do that. >> >> Hoping that helps, >> Jc >> >> >> >> On Fri, Sep 21, 2018 at 11:56 PM nijiaben wrote: >> >>> Hi, all >>> >>> We use metaspace instead of perm since JDK8, I found that after the Full >>> GC, the size of the metaspace has always been the same from the GC log. I >>> have looked at the source code of hotspot roughly. It has not bean fixed in >>> the latest version. The problem which may have some impact on us to >>> troubleshoot some metaspace problems. >>> >>> I wrote a test case, and simply analyzed the hotspot code, and showed >>> the output before and after the modification. I hope this problem can be >>> fixed as soon as possible. >>> >>> Demo? >>> >>> import java.io.File; >>> import java.net.URL; >>> import java.net.URLClassLoader; >>> >>> /** >>> * >>> * @author nijiaben >>> * @version v1.0 >>> * @createTime 2018?09?22? 12:32:32 PM >>> * >>> */ >>> public class ClassLoaderTest { >>> private static URL[] urls = new URL[1]; >>> >>> static { >>> try { >>> File jarFile = new File("/home/admin/.m2/repository/org/slf4j/slf4j-api/1.7.21/slf4j-api-1.7.21.jar"); >>> urls[0] = jarFile.toURI().toURL(); >>> } catch (Exception e) { >>> e.printStackTrace(); >>> } >>> } >>> >>> public static void main(String args[]) { >>> for(int i=0;i<1000;i++) { >>> loadClass(); >>> } >>> System.gc(); >>> } >>> >>> public static void loadClass() { >>> try { >>> URLClassLoader ucl = new URLClassLoader(urls); >>> Class.forName("org.slf4j.Logger", false, ucl); >>> } catch (Exception e) { >>> e.printStackTrace(); >>> } >>> } >>> } >>> >>> My demo is very simple, just constantly create a new class loader to >>> load a specific class, after loading, execute a System GC, so that we can >>> see the relevant GC log output, the JVM parameters we run are >>> >>> -Xmx1000M -Xms1000M -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails ClassLoaderTest >>> >>> The GC log is >>> >>> [Full GC (System.gc()) [CMS: 0K->9921K(683264K), 0.1003988 secs] 49083K->9921K(989952K), [Metaspace: 12916K->12916K(1064960K)], 0.1060065 secs] [Times: user=0.06 sys=0.05, real=0.11 secs] >>> >>> Let's look at the source code of hotspot, >>> >>> GenCollectedHeap::do_collection: >>> >>> if (PrintGCDetails) { >>> print_heap_change(gch_prev_used); >>> >>> // Print metaspace info for full GC with PrintGCDetails flag. >>> if (complete) { >>> MetaspaceAux::print_metaspace_change(metadata_prev_used); >>> } >>> } >>> >>> for (int j = max_level_collected; j >= 0; j -= 1) { >>> // Adjust generation sizes. >>> _gens[j]->compute_new_size(); >>> } >>> >>> if (complete) { >>> // Delete metaspaces for unloaded class loaders and clean up loader_data graph >>> ClassLoaderDataGraph::purge(); >>> MetaspaceAux::verify_metrics(); >>> // Resize the metaspace capacity after full collections >>> MetaspaceGC::compute_new_size(); >>> update_full_collections_completed(); >>> } >>> >>> We see that metaspace's gc log output is executed before >>> ClassLoaderDataGraph::purge(); , that means the size of the metaspace >>> is printed without reclaiming the memory of the metaspace, so the GC log >>> seems no change before and after the GC metaspace. >>> >>> After knowing the specific problem, we can make an adjustment to the >>> code to move the logic of the output of metaspace size after to >>> ClassLoaderDataGraph::purge()?just like this: >>> >>> if (PrintGCDetails) { >>> print_heap_change(gch_prev_used); >>> } >>> >>> for (int j = max_level_collected; j >= 0; j -= 1) { >>> // Adjust generation sizes. >>> _gens[j]->compute_new_size(); >>> } >>> >>> if (complete) { >>> // Delete metaspaces for unloaded class loaders and clean up loader_data graph >>> ClassLoaderDataGraph::purge(); >>> MetaspaceAux::verify_metrics(); >>> // Resize the metaspace capacity after full collections >>> MetaspaceGC::compute_new_size(); >>> update_full_collections_completed(); >>> if (PrintGCDetails) { >>> // Print metaspace info for full GC with PrintGCDetails flag. >>> MetaspaceAux::print_metaspace_change(metadata_prev_used); >>> } >>> } >>> >>> At this point we recompile hotspot, execute the above demo again, you >>> can see the correct output as follows >>> >>> [Full GC (System.gc()) [CMS: 0K->9921K(683264K), 0.0852627 secs] 49083K->9921K(989952K), [Metaspace: 12913K->3280K(1058816K)], 0.0891207 secs] [Times: user=0.05 sys=0.04, real=0.09 secs] >>> >>> I also probably thought about why this problem is caused. This is the >>> version before JDK8, such as JDK7, you can see the same code location: >>> >>> if (PrintGCDetails) { >>> print_heap_change(gch_prev_used); >>> >>> // Print perm gen info for full GC with PrintGCDetails flag. >>> if (complete) { >>> print_perm_heap_change(perm_prev_used); >>> } >>> } >>> >>> for (int j = max_level_collected; j >= 0; j -= 1) { >>> // Adjust generation sizes. >>> _gens[j]->compute_new_size(); >>> } >>> >>> if (complete) { >>> // Ask the permanent generation to adjust size for full collections >>> perm()->compute_new_size(); >>> update_full_collections_completed(); >>> } >>> >>> It can be seen that the GC engineer at that time simply replaced >>> print_perm_heap_change(perm_prev_used);with >>> MetaspaceAux::print_metaspace_change(metadata_prev_used);but forgot the >>> difference between Perm and Metaspace. >>> >>> Thanks, >>> >>> nijiaben >>> >> >> >> -- >> >> Thanks, >> Jc >> > > -- Thanks, Jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From doug.simon at oracle.com Tue Sep 25 14:48:12 2018 From: doug.simon at oracle.com (Doug Simon) Date: Tue, 25 Sep 2018 16:48:12 +0200 Subject: RFR: 8208686: [AOT] JVMTI ResourceExhausted event repeated for same allocation Message-ID: <910A4A3C-3EEF-4167-84D5-9819C83D6FC1@oracle.com> A major design point of Graal is to treat allocations as non-side effecting to give more freedom to the optimizer by reducing the number of distinct FrameStates that need to be managed. When failing an allocation, Graal will deoptimize to the last side effecting instruction before the allocation. This mean the VM code for heap allocation will potentially be executed twice, once from Graal compiled code and then again in the interpreter. While this is perfectly fine according to the JVM specification, it can cause confusing behavior for JVMTI based tools. They will receive 2 ResourceExhausted events for a single allocation. Furthermore, the first ResourceExhausted event (on the Graal allocation slow path) might denote a bytecode instruction that performs no allocation, making it hard to debug the memory failure. The proposed solution is to add an extra set of JVMCI VM runtime calls for allocation. These entry points will attempt the allocation and upon failure, skip side-effects such as posting JVMTI events or handling -XX:OnOutOfMemoryError. The compiled code using these entry points is expected deoptmize on null. The path from these new entry points to where allocation can fail goes through quite a bit of VM code. One could modify all these paths by: * Returning null instead of throwing an exception on failure. * Adding a `bool null_on_fail` argument to all relevant methods. * Adding extra null checking where necessary after each call to these methods when `null_on_fail == true`. This represents a significant number of changes. Instead, the proposed solution introduces a new _in_retryable_allocation thread-local. This way, only the entry points and allocation routines that raise an exception need to be modified. Failure is communicated back to the new entry points by throwing a special pre-allocated OOME object (i.e., Universe::out_of_memory_error_retry()) which must not propagate back to Java code. Use of this object is not strictly necessary; it is introduced to highlight/document the special allocation mode. The proposed solution is at http://cr.openjdk.java.net/~dnsimon/8208686. THE JBS bug is: https://bugs.openjdk.java.net/browse/JDK-8208686 -Doug From thomas.schatzl at oracle.com Wed Sep 26 07:45:49 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 26 Sep 2018 09:45:49 +0200 Subject: RFR (S) [8u] backport 8141421: Various test fail with OOME on win x86 In-Reply-To: <54c04c8e-7677-3723-5335-7738917fa575@oracle.com> References: <54c04c8e-7677-3723-5335-7738917fa575@oracle.com> Message-ID: Hi David, On Tue, 2018-09-25 at 10:37 +0900, David Buck wrote: > Hi! > > May I please get a review of this simple backport to 8u-dev? The > only reason the JDK 9 fix does not apply as-is to JDK 8 is because of > changes to assert in JDK 9 [0]. > > bug report: > https://bugs.openjdk.java.net/browse/JDK-8141421 > > JDK 9 review thread: > http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2016-January/01 > 6251.html > > JDK 9 changeset: > http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/2de6311c5afc > > JDK 8 webrev (for review): > http://cr.openjdk.java.net/~dbuck/8141421_hotspot_jdk8_ver01/ > > Cheers, > -Buck > > [0] https://bugs.openjdk.java.net/browse/JDK-8080775 > looks good. Could you fix indentation of the assert texts so that the part within the err_msg() call is aligned correctly? So instead of: assert(someCondition, err_msg("....", arg1, arg2, ...)); use assert(someCondition, err_msg("....", arg1, arg2, ....)); to make it more readable (in heapRegionRemSet.cpp, twice). I do not need to see a re-review, but if jdk8u review guidelines tell so, I will certainly look at these minor updates again. Thanks, Thomas From david.buck at oracle.com Wed Sep 26 08:06:25 2018 From: david.buck at oracle.com (David Buck) Date: Wed, 26 Sep 2018 17:06:25 +0900 Subject: RFR (S) [8u] backport 8141421: Various test fail with OOME on win x86 In-Reply-To: References: <54c04c8e-7677-3723-5335-7738917fa575@oracle.com> Message-ID: <0bd6e181-bfd3-49fc-9c66-d4ede9991585@oracle.com> Hi Thomas! Thank you for the review. Out of an (almost pathological) abundance of caution, here an updated webrev: http://cr.openjdk.java.net/~dbuck/8141421_hotspot_jdk8_ver02/ Would you mind taking one last look at the indentation? Cheers, -Buck On 2018/09/26 16:45, Thomas Schatzl wrote: > Hi David, > > On Tue, 2018-09-25 at 10:37 +0900, David Buck wrote: >> Hi! >> >> May I please get a review of this simple backport to 8u-dev? The >> only reason the JDK 9 fix does not apply as-is to JDK 8 is because of >> changes to assert in JDK 9 [0]. >> >> bug report: >> https://bugs.openjdk.java.net/browse/JDK-8141421 >> >> JDK 9 review thread: >> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2016-January/01 >> 6251.html >> >> JDK 9 changeset: >> http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/2de6311c5afc >> >> JDK 8 webrev (for review): >> http://cr.openjdk.java.net/~dbuck/8141421_hotspot_jdk8_ver01/ >> >> Cheers, >> -Buck >> >> [0] https://bugs.openjdk.java.net/browse/JDK-8080775 >> > > looks good. > > Could you fix indentation of the assert texts so that the part within > the err_msg() call is aligned correctly? So instead of: > > assert(someCondition, > err_msg("....", > arg1, arg2, ...)); > > use > > assert(someCondition, > err_msg("....", > arg1, arg2, ....)); > > > to make it more readable (in heapRegionRemSet.cpp, twice). > > I do not need to see a re-review, but if jdk8u review guidelines tell > so, I will certainly look at these minor updates again. > > Thanks, > Thomas > From stefan.johansson at oracle.com Wed Sep 26 08:06:43 2018 From: stefan.johansson at oracle.com (Stefan Johansson) Date: Wed, 26 Sep 2018 10:06:43 +0200 Subject: A Bug Of Metaspace After Full GC In-Reply-To: References: Message-ID: Thanks for taking care of this JC, I think it's worth fixing, so please create a webrev and send out an RFR. Looking a bit more at the code, maybe we should move both the heap printing and metaspace printing to after the update call, just to keep the printing together. Cheers, Stefan On 2018-09-25 22:22, JC Beyler wrote: > I did say I was not a GC expert :-) > > So the question would be: is the patch I showed based on Nijiaben's > patch worth fixing and is it right? It seems like the right thing to do > for at least fixing SerialGC printouts, no? > Jc > > Ps: here is the patch for simplifying things for everyone then: > diff -r 8f66a57054b7 src/hotspot/share/gc/shared/genCollectedHeap.cpp > --- a/src/hotspot/share/gc/shared/genCollectedHeap.cpp? Tue Sep 25 > 10:30:32 2018 -0700 > +++ b/src/hotspot/share/gc/shared/genCollectedHeap.cpp? Tue Sep 25 > 12:50:54 2018 -0700 > @@ -649,7 +649,6 @@ > ? ? ?complete = complete || collected_old; > ? ? ?print_heap_change(young_prev_used, old_prev_used); > -? ? MetaspaceUtils::print_metaspace_change(metadata_prev_used); > ? ? ?// Adjust generation sizes. > ? ? ?if (collected_old) { > @@ -665,6 +664,7 @@ > ? ? ? ?MetaspaceGC::compute_new_size(); > ? ? ? ?update_full_collections_completed(); > ? ? ?} > +? ? MetaspaceUtils::print_metaspace_change(metadata_prev_used); > ? ? ?// Track memory usage and detect low memory after GC finishes > ? ? ?MemoryService::track_memory_usage(); > > On Tue, Sep 25, 2018 at 1:08 PM Krystal Mok > wrote: > > Hi JC, > > Just a quick comment on: > > So I looked at the other ones as well and as a summary: > ? - The bug seems to show up for Serial, CMS > ? - The bug does not seem to show up for Parallel, G1 > > That's because Serial (DefNew) / Serial Old (Tenured / GenMarkSweep) > / ParNew / CMS use the "generational GC framework" that's been there > for years. That's the "GenCollectedHeap" touched by Nijiaben's > propsed patch. > It was there with the idea that different GC implementations can > plug into the same framework and supposedly they could > mix-and-match. But later on only a few possible configurations were > really tested well and supported. > > Parallel (ParallelScavenge) / Parallel Old and G1 use their own > implementation of the CollectedHeap interface and doesn't > participate in that "generational GC framework". So they don't have > the same bug at the same place. > > - Kris > > On Tue, Sep 25, 2018 at 12:57 PM, JC Beyler > wrote: > > Hi Nijiaben, > > I'm not a GC engineer but thought I would look at this by > curiosity (for note, I also created > https://bugs.openjdk.java.net/browse/JDK-8211123 for tracking > purposes). It seems you are using an older JDK because the code > has changed somewhat there. > > I looked at this and used your reproducer on JDK12 tip: > > Using CMS (which is being deprecated from what I know and see in > the log message): > $ > ~/mercurial/jdk12-gc-print/build/linux-x86_64-server-release/images/jdk/bin/java > -XX:+UseConcMarkSweepGC -Xmx1000M -Xms1000M -Xlog:gc* > ClassLoaderTest > OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was > deprecated in version 9.0 and will likely be removed in a future > release. > ... > [0.559s][info][gc,metaspace? ?] GC(0) Metaspace: > 15267K->15267K(1067008K) > > (Note: I am using -Xlog:gc* since -XX:+PrintGCDetails is being > deprecated as well from what the log message is saying) > > So the bug is there for CMS, but for G1, it seems to work: > > $ > ~/mercurial/jdk12-gc-print/build/linux-x86_64-server-release/images/jdk/bin/java > -Xmx1000M -Xms1000M -Xlog:gc* ClassLoaderTest > ... > [0.653s][info][gc,metaspace? ?] GC(0) Metaspace: > 15266K->5266K(1062912K) > ... > > So I looked at the other ones as well and as a summary: > ? - The bug seems to show up for Serial, CMS > ? - The bug does not seem to show up for Parallel, G1 > > > I then updated the code like what you were showing though things > have changed in tip so my change became a simple one-line move: > > diff -r 8f66a57054b7 > src/hotspot/share/gc/shared/genCollectedHeap.cpp > --- a/src/hotspot/share/gc/shared/genCollectedHeap.cpp? Tue Sep > 25 10:30:32 2018 -0700 > +++ b/src/hotspot/share/gc/shared/genCollectedHeap.cpp? Tue Sep > 25 12:50:54 2018 -0700 > @@ -649,7 +649,6 @@ > ? ? ?complete = complete || collected_old; > ? ? ?print_heap_change(young_prev_used, old_prev_used); > -? ? MetaspaceUtils::print_metaspace_change(metadata_prev_used); > ? ? ?// Adjust generation sizes. > ? ? ?if (collected_old) { > @@ -665,6 +664,7 @@ > ? ? ? ?MetaspaceGC::compute_new_size(); > ? ? ? ?update_full_collections_completed(); > ? ? ?} > +? ? MetaspaceUtils::print_metaspace_change(metadata_prev_used); > ? ? ?// Track memory usage and detect low memory after GC finishes > ? ? ?MemoryService::track_memory_usage(); > > With this, I looked at the various GC results and this fixes: > ? - Serial, CMS; while keeping Parallel and G1 working (seemingly) > > Finally, note that all GC logging now are logging the summary at > the exit and this line shows up: > [0.560s][info][gc,heap,exit? ?]? Metaspace? ? ? ?used 5277K, > capacity 5310K, committed 15232K, reserved 1062912K > > For all GCs, so though the line "Metaspace before -> after" is > false, you can get the the after with the final line. > > If the GC experts on this list want me to make a webrev out of > this and get reviews for it, I can do that. > > Hoping that helps, > Jc > > > > On Fri, Sep 21, 2018 at 11:56 PM nijiaben > wrote: > > Hi, all > > We use metaspace instead of perm since JDK8, I found that > after the Full GC, the size of the metaspace has always been > the same from the GC log. I have looked at the source code > of hotspot roughly. It has not bean fixed in the latest > version. The problem which may have some impact on us to > troubleshoot some metaspace problems. > > I wrote a test case, and simply analyzed the hotspot code, > and showed the output before and after the modification. I > hope this problem can be fixed as soon as possible. > > Demo? > > |import java.io.File; import java.net.URL; import > java.net.URLClassLoader; /** * * @author nijiaben * @version > v1.0 * @createTime 2018?09?22? 12:32:32 PM * */ public > class ClassLoaderTest { private static URL[] urls = new > URL[1]; static { try { File jarFile = new > File("/home/admin/.m2/repository/org/slf4j/slf4j-api/1.7.21/slf4j-api-1.7.21.jar"); > urls[0] = jarFile.toURI().toURL(); } catch (Exception e) { > e.printStackTrace(); } } public static void main(String > args[]) { for(int i=0;i<1000;i++) { loadClass(); } > System.gc(); } public static void loadClass() { try { > URLClassLoader ucl = new URLClassLoader(urls); > Class.forName("org.slf4j.Logger", false, ucl); } catch > (Exception e) { e.printStackTrace(); } } }| > > My demo is very simple, just constantly create a new class > loader to load a specific class, after loading, execute a > System GC, so that we can see the relevant GC log output, > the JVM parameters we run are > > |-Xmx1000M -Xms1000M -XX:+UseConcMarkSweepGC > -XX:+PrintGCDetails ClassLoaderTest| > > The GC log is > > |[Full GC (System.gc()) [CMS: 0K->9921K(683264K), 0.1003988 > secs] 49083K->9921K(989952K), [Metaspace: > 12916K->12916K(1064960K)], 0.1060065 secs] [Times: user=0.06 > sys=0.05, real=0.11 secs]| > > Let's look at the source code of hotspot, > > |GenCollectedHeap::do_collection|: > > |if (PrintGCDetails) { print_heap_change(gch_prev_used); // > Print metaspace info for full GC with PrintGCDetails flag. > if (complete) { > MetaspaceAux::print_metaspace_change(metadata_prev_used); } > } for (int j = max_level_collected; j >= 0; j -= 1) { // > Adjust generation sizes. _gens[j]->compute_new_size(); } if > (complete) { // Delete metaspaces for unloaded class loaders > and clean up loader_data graph > ClassLoaderDataGraph::purge(); > MetaspaceAux::verify_metrics(); // Resize the metaspace > capacity after full collections > MetaspaceGC::compute_new_size(); > update_full_collections_completed(); }| > > We see that metaspace's gc log output is executed > before|ClassLoaderDataGraph::purge();|, that means the size > of the metaspace is printed without reclaiming the memory of > the metaspace, so the GC log seems no change before and > after the GC metaspace. > > After knowing the specific problem, we can make an > adjustment to the code to move the logic of the output of > metaspace size after to|ClassLoaderDataGraph::purge()|?just > like this: > > |if (PrintGCDetails) { print_heap_change(gch_prev_used); } > for (int j = max_level_collected; j >= 0; j -= 1) { // > Adjust generation sizes. _gens[j]->compute_new_size(); } if > (complete) { // Delete metaspaces for unloaded class loaders > and clean up loader_data graph > ClassLoaderDataGraph::purge(); > MetaspaceAux::verify_metrics(); // Resize the metaspace > capacity after full collections > MetaspaceGC::compute_new_size(); > update_full_collections_completed(); if (PrintGCDetails) { > // Print metaspace info for full GC with PrintGCDetails > flag. > MetaspaceAux::print_metaspace_change(metadata_prev_used); } }| > > At this point we recompile hotspot, execute the above demo > again, you can see the correct output as follows > > |[Full GC (System.gc()) [CMS: 0K->9921K(683264K), 0.0852627 > secs] 49083K->9921K(989952K), [Metaspace: > 12913K->3280K(1058816K)], 0.0891207 secs] [Times: user=0.05 > sys=0.04, real=0.09 secs]| > > I also probably thought about why this problem is caused. > This is the version before JDK8, such as JDK7, you can see > the same code location: > > |if (PrintGCDetails) { print_heap_change(gch_prev_used); // > Print perm gen info for full GC with PrintGCDetails flag. if > (complete) { print_perm_heap_change(perm_prev_used); } } for > (int j = max_level_collected; j >= 0; j -= 1) { // Adjust > generation sizes. _gens[j]->compute_new_size(); } if > (complete) { // Ask the permanent generation to adjust size > for full collections perm()->compute_new_size(); > update_full_collections_completed(); }| > > It can be seen that the GC engineer at that time simply > replaced|print_perm_heap_change(perm_prev_used);|with|MetaspaceAux::print_metaspace_change(metadata_prev_used);|but > forgot the difference between Perm and Metaspace. > > Thanks, > > nijiaben > > ____ > > > > -- > > Thanks, > Jc > > > > > -- > > Thanks, > Jc From thomas.schatzl at oracle.com Wed Sep 26 08:11:00 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 26 Sep 2018 10:11:00 +0200 Subject: RFR (XS): 8211123: GC Metaspace printing after full gc does not reflect recent changes (Was: Re: A Bug Of Metaspace After Full GC) In-Reply-To: References: Message-ID: Hi, On Tue, 2018-09-25 at 13:22 -0700, JC Beyler wrote: > I did say I was not a GC expert :-) > > So the question would be: is the patch I showed based on Nijiaben's > patch worth fixing and is it right? It seems like the right thing to > do for at least fixing SerialGC printouts, no? Yes. Serial GC is fully supported. I am not sure what release this is reported for, probably 8. Not sure if it is worth backporting (then again, on 8u both CMS and Serial are supported). > Ps: here is the patch for simplifying things for everyone then: > diff -r 8f66a57054b7 src/hotspot/share/gc/shared/genCollectedHeap.cpp > --- a/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 > 10:30:32 2018 -0700 > +++ b/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 > 12:50:54 2018 -0700 > @@ -649,7 +649,6 @@ > complete = complete || collected_old; > > print_heap_change(young_prev_used, old_prev_used); > - MetaspaceUtils::print_metaspace_change(metadata_prev_used); > > // Adjust generation sizes. > if (collected_old) { > @@ -665,6 +664,7 @@ > MetaspaceGC::compute_new_size(); > update_full_collections_completed(); > } > + MetaspaceUtils::print_metaspace_change(metadata_prev_used); > > // Track memory usage and detect low memory after GC finishes > MemoryService::track_memory_usage(); > Looks good. Would it be possible to write a small junit test that checks that metaspace usage decreased after full gc? That should be true for all collectors assuming they do not have class unloading disabled. Thanks, Thomas From thomas.schatzl at oracle.com Wed Sep 26 08:15:19 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 26 Sep 2018 10:15:19 +0200 Subject: RFR (S) [8u] backport 8141421: Various test fail with OOME on win x86 In-Reply-To: <0bd6e181-bfd3-49fc-9c66-d4ede9991585@oracle.com> References: <54c04c8e-7677-3723-5335-7738917fa575@oracle.com> <0bd6e181-bfd3-49fc-9c66-d4ede9991585@oracle.com> Message-ID: <18448aa1498d93fdda7b91d9870d1b2133160919.camel@oracle.com> Hi David, On Wed, 2018-09-26 at 17:06 +0900, David Buck wrote: > Hi Thomas! > > Thank you for the review. > > Out of an (almost pathological) abundance of caution, here an > updated > webrev: > > http://cr.openjdk.java.net/~dbuck/8141421_hotspot_jdk8_ver02/ > > Would you mind taking one last look at the indentation? looks good. :) Thomas From mikael.vidstedt at oracle.com Wed Sep 26 16:44:27 2018 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Wed, 26 Sep 2018 09:44:27 -0700 Subject: RFR(XS): 8210893: Deprecate TLABStats In-Reply-To: <6ac1814728de5efef9ec290bb81053e064e111e7.camel@oracle.com> References: <190905EC-2082-40E9-80B8-98DC7262CC6E@oracle.com> <6ac1814728de5efef9ec290bb81053e064e111e7.camel@oracle.com> Message-ID: For completeness I?ll mention that the subject of this thread is incorrectly using 8210893, which is the CSR. The actually issue is 8210892. Cheers, Mikael > On Sep 21, 2018, at 12:43 AM, Thomas Schatzl wrote: > > Hi, > > On Thu, 2018-09-20 at 18:15 -0700, Mikael Vidstedt wrote: >> >> Please review this change which deprecates the -XX:+TLABStats flag. >> >> > [...] > > looks good. > > Thomas > From jcbeyler at google.com Thu Sep 27 02:21:16 2018 From: jcbeyler at google.com (JC Beyler) Date: Wed, 26 Sep 2018 19:21:16 -0700 Subject: RFR (XS): 8211123: GC Metaspace printing after full gc does not reflect recent changes (Was: Re: A Bug Of Metaspace After Full GC) In-Reply-To: References: Message-ID: Hi Thomas et al, Thanks for moving this thread to a RFR. Here is the webrev and bug: Webrev: http://cr.openjdk.java.net/~jcbeyler/8211123/webrev.00/ Bug: https://bugs.openjdk.java.net/browse/JDK-8211123 For the test, I inspired myself of a few different tests to get this up and running. I didn't find a means to use only the WhiteBox to get the metaspace filled. I did not want to use the .jar that the original reproducer was using since it was using a library. So I used the testcases.jar from the runtime/7116786 test. I also use a separate process to test the various GCs and ensure it works for all. Btw, I've put nijiaben as the contributor for the webrev. Let me know what you think, I imagine I've made the test a bit more complicated than need be, Jc On Wed, Sep 26, 2018 at 1:11 AM Thomas Schatzl wrote: > Hi, > > On Tue, 2018-09-25 at 13:22 -0700, JC Beyler wrote: > > I did say I was not a GC expert :-) > > > > So the question would be: is the patch I showed based on Nijiaben's > > patch worth fixing and is it right? It seems like the right thing to > > do for at least fixing SerialGC printouts, no? > > Yes. Serial GC is fully supported. > > I am not sure what release this is reported for, probably 8. Not sure > if it is worth backporting (then again, on 8u both CMS and Serial are > supported). > > > Ps: here is the patch for simplifying things for everyone then: > > diff -r 8f66a57054b7 src/hotspot/share/gc/shared/genCollectedHeap.cpp > > --- a/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 > > 10:30:32 2018 -0700 > > +++ b/src/hotspot/share/gc/shared/genCollectedHeap.cpp Tue Sep 25 > > 12:50:54 2018 -0700 > > @@ -649,7 +649,6 @@ > > complete = complete || collected_old; > > > > print_heap_change(young_prev_used, old_prev_used); > > - MetaspaceUtils::print_metaspace_change(metadata_prev_used); > > > > // Adjust generation sizes. > > if (collected_old) { > > @@ -665,6 +664,7 @@ > > MetaspaceGC::compute_new_size(); > > update_full_collections_completed(); > > } > > + MetaspaceUtils::print_metaspace_change(metadata_prev_used); > > > > // Track memory usage and detect low memory after GC finishes > > MemoryService::track_memory_usage(); > > > > Looks good. Would it be possible to write a small junit test that > checks that metaspace usage decreased after full gc? > > That should be true for all collectors assuming they do not have class > unloading disabled. > > Thanks, > Thomas > > -- Thanks, Jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Thu Sep 27 12:09:13 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 27 Sep 2018 14:09:13 +0200 Subject: RFR (XS): 8211123: GC Metaspace printing after full gc does not reflect recent changes (Was: Re: A Bug Of Metaspace After Full GC) In-Reply-To: References: Message-ID: <5fd8f58e6d6ae6b1e6f60d4cdc663afda9474894.camel@oracle.com> Hi JC, On Wed, 2018-09-26 at 19:21 -0700, JC Beyler wrote: > Hi Thomas et al, > > Thanks for moving this thread to a RFR. Here is the webrev and bug: > Webrev: http://cr.openjdk.java.net/~jcbeyler/8211123/webrev.00/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8211123 > > For the test, I inspired myself of a few different tests to get this > up and running. I didn't find a means to use only the WhiteBox to get > the metaspace filled. I did not want to use the .jar that the > original reproducer was using since it was using a library. So I used > the testcases.jar from the runtime/7116786 test. > > I also use a separate process to test the various GCs and ensure it > works for all. > > Btw, I've put nijiaben as the contributor for the webrev. > > Let me know what you think, I imagine I've made the test a bit more > complicated than need be, > Jc I do not know a better way than having an external jar to do class loading into a separate class loader in a test. It's safest to duplicate the jar file. The test could be simplified a bit by using the Metaspace MemoryMXBean to get metaspace memory readings (there is an MXBean called "Metaspace"), removing the need for invoking a secondary JVM in the test and parsing the log :) You can then run the test with the different collectors using "main/othervm -XX:+ MyTest" lines. Otherwise the change itself looks good. I would appreciate if the test could be improved as mentioned above, but if you do not want to I could live with it as is. Btw, you can have multiple contributors in that line (or probably multiple Contributed-by lines). Thanks, Thomas From shade at redhat.com Thu Sep 27 13:55:49 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 27 Sep 2018 15:55:49 +0200 Subject: RFR (S) 8211228: Zero build is broken after JDK-8196341 (Add JFR events for parallel phases of G1) Message-ID: Bug: https://bugs.openjdk.java.net/browse/JDK-8211228 Fix: http://cr.openjdk.java.net/~shade/8211228/webrev.01/ Recent addition of JFR events to G1 broke Zero build. That is because Zero builds with JFR disabled, but G1 enabled (it is a separate question why!). It can be fixed by putting JFR_ONLY all around JFR-specific code, or by configuring Zero to avoid building g1gc. But, since it looks like AIX is also affected by this: https://bugs.openjdk.java.net/browse/JDK-8211213 ...I'd prefer to protect the JFR paths. Testing: x86_64 zero build, x86_64 server build Thanks, -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From thomas.stuefe at gmail.com Thu Sep 27 14:17:05 2018 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 27 Sep 2018 16:17:05 +0200 Subject: RFR (XS): 8211123: GC Metaspace printing after full gc does not reflect recent changes (Was: Re: A Bug Of Metaspace After Full GC) In-Reply-To: <5fd8f58e6d6ae6b1e6f60d4cdc663afda9474894.camel@oracle.com> References: <5fd8f58e6d6ae6b1e6f60d4cdc663afda9474894.camel@oracle.com> Message-ID: Hi, On Thu, Sep 27, 2018 at 2:09 PM, Thomas Schatzl wrote: > Hi JC, > > On Wed, 2018-09-26 at 19:21 -0700, JC Beyler wrote: >> Hi Thomas et al, >> >> Thanks for moving this thread to a RFR. Here is the webrev and bug: >> Webrev: http://cr.openjdk.java.net/~jcbeyler/8211123/webrev.00/ >> Bug: https://bugs.openjdk.java.net/browse/JDK-8211123 >> >> For the test, I inspired myself of a few different tests to get this >> up and running. I didn't find a means to use only the WhiteBox to get >> the metaspace filled. I did not want to use the .jar that the >> original reproducer was using since it was using a library. So I used >> the testcases.jar from the runtime/7116786 test. >> >> I also use a separate process to test the various GCs and ensure it >> works for all. >> >> Btw, I've put nijiaben as the contributor for the webrev. >> >> Let me know what you think, I imagine I've made the test a bit more >> complicated than need be, >> Jc > > I do not know a better way than having an external jar to do class > loading into a separate class loader in a test. It's safest to > duplicate the jar file. Just as a suggestion, I did tests - but never really got to jtreg-ify them - for JDK-8198423. They use javax.tools.JavaCompiler to compile classes on the fly and load them into separate class loaders. See e.g. https://github.com/tstuefe/ojdk-repros/blob/master/src/test3/Example2.java (Not the prettiest program ever, but enough to demonstrate the point). Best, Thomas > > The test could be simplified a bit by using the Metaspace MemoryMXBean > to get metaspace memory readings (there is an MXBean called > "Metaspace"), removing the need for invoking a secondary JVM in the > test and parsing the log :) > > You can then run the test with the different collectors using > "main/othervm -XX:+ MyTest" lines. > > Otherwise the change itself looks good. I would appreciate if the test > could be improved as mentioned above, but if you do not want to I could > live with it as is. > > Btw, you can have multiple contributors in that line (or probably > multiple Contributed-by lines). > > Thanks, > Thomas > > From shade at redhat.com Thu Sep 27 18:16:04 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 27 Sep 2018 20:16:04 +0200 Subject: RFR (S) 8211228: Zero build is broken after JDK-8196341 (Add JFR events for parallel phases of G1) In-Reply-To: References: Message-ID: <1b399696-0edc-14ae-13f3-af5d3ab73a6a@redhat.com> On 09/27/2018 03:55 PM, Aleksey Shipilev wrote: > Bug: > https://bugs.openjdk.java.net/browse/JDK-8211228 > > Fix: > http://cr.openjdk.java.net/~shade/8211228/webrev.01/ > > Recent addition of JFR events to G1 broke Zero build. That is because Zero builds with JFR disabled, > but G1 enabled (it is a separate question why!). It can be fixed by putting JFR_ONLY all around > JFR-specific code, or by configuring Zero to avoid building g1gc. But, since it looks like AIX is > also affected by this: > https://bugs.openjdk.java.net/browse/JDK-8211213 I retract this RFR in favor of proper fix on JFR side: https://bugs.openjdk.java.net/browse/JDK-8211239 -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From jcbeyler at google.com Thu Sep 27 18:29:40 2018 From: jcbeyler at google.com (JC Beyler) Date: Thu, 27 Sep 2018 11:29:40 -0700 Subject: RFR (XS): 8211123: GC Metaspace printing after full gc does not reflect recent changes (Was: Re: A Bug Of Metaspace After Full GC) In-Reply-To: References: <5fd8f58e6d6ae6b1e6f60d4cdc663afda9474894.camel@oracle.com> Message-ID: Hi Thomas and Thomas, Thanks for the reviews! @Thomas Schatzl: I looked for using a MXBean but I'm confused about using this. What this fix+test is trying to ascertain is that the moment of the logging of the metaspace data, the new size has been calculated; how do I do that with a MXBean? Would I not always get the current size? So if I did before FullGC, I'd get the previous size, after FullGC, I'd always get the new size, no? I am most likely missing something and I apologize! @Thomas Stufe: I think that is exactly what we could (should?) use; however there are a few questions here: 1) There is no license to that code, could you add one so that we can see if it is compatible for the OpenJDK project? 2) Will bringing this code in for this test be useful for future/other tests? 3) If so, I would actually recommending making a RFE for the test to bring in similar code like that and remove my testcase.jar. That way this bug focuses solely on just getting the print out right with a test and we can refactor it in a second step with a useful common library that can come it slowly with reviews that are concentrating on that library instead, what do you think? Thanks, Jc On Thu, Sep 27, 2018 at 7:17 AM Thomas St?fe wrote: > Hi, > > On Thu, Sep 27, 2018 at 2:09 PM, Thomas Schatzl > wrote: > > Hi JC, > > > > On Wed, 2018-09-26 at 19:21 -0700, JC Beyler wrote: > >> Hi Thomas et al, > >> > >> Thanks for moving this thread to a RFR. Here is the webrev and bug: > >> Webrev: http://cr.openjdk.java.net/~jcbeyler/8211123/webrev.00/ > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8211123 > >> > >> For the test, I inspired myself of a few different tests to get this > >> up and running. I didn't find a means to use only the WhiteBox to get > >> the metaspace filled. I did not want to use the .jar that the > >> original reproducer was using since it was using a library. So I used > >> the testcases.jar from the runtime/7116786 test. > >> > >> I also use a separate process to test the various GCs and ensure it > >> works for all. > >> > >> Btw, I've put nijiaben as the contributor for the webrev. > >> > >> Let me know what you think, I imagine I've made the test a bit more > >> complicated than need be, > >> Jc > > > > I do not know a better way than having an external jar to do class > > loading into a separate class loader in a test. It's safest to > > duplicate the jar file. > > Just as a suggestion, I did tests - but never really got to jtreg-ify > them - for JDK-8198423. > > They use javax.tools.JavaCompiler to compile classes on the fly and > load them into separate class loaders. > > See e.g. > https://github.com/tstuefe/ojdk-repros/blob/master/src/test3/Example2.java > > (Not the prettiest program ever, but enough to demonstrate the point). > > Best, Thomas > > > > > The test could be simplified a bit by using the Metaspace MemoryMXBean > > to get metaspace memory readings (there is an MXBean called > > "Metaspace"), removing the need for invoking a secondary JVM in the > > test and parsing the log :) > > > > You can then run the test with the different collectors using > > "main/othervm -XX:+ MyTest" lines. > > > > Otherwise the change itself looks good. I would appreciate if the test > > could be improved as mentioned above, but if you do not want to I could > > live with it as is. > > > > Btw, you can have multiple contributors in that line (or probably > > multiple Contributed-by lines). > > > > Thanks, > > Thomas > > > > > -- Thanks, Jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Thu Sep 27 19:05:30 2018 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 27 Sep 2018 21:05:30 +0200 Subject: RFR (XS): 8211123: GC Metaspace printing after full gc does not reflect recent changes (Was: Re: A Bug Of Metaspace After Full GC) In-Reply-To: References: <5fd8f58e6d6ae6b1e6f60d4cdc663afda9474894.camel@oracle.com> Message-ID: Hi JC, On Thu, Sep 27, 2018 at 8:29 PM, JC Beyler wrote: > Hi Thomas and Thomas, > > Thanks for the reviews! > > @Thomas Schatzl: I looked for using a MXBean but I'm confused about using > this. What this fix+test is trying to ascertain is that the moment of the > logging of the metaspace data, the new size has been calculated; how do I do > that with a MXBean? Would I not always get the current size? So if I did > before FullGC, I'd get the previous size, after FullGC, I'd always get the > new size, no? I am most likely missing something and I apologize! > > @Thomas Stufe: I think that is exactly what we could (should?) use; however > there are a few questions here: > 1) There is no license to that code, could you add one so that we can see > if it is compatible for the OpenJDK project? There was none? Oops. Sure, I added a license. > 2) Will bringing this code in for this test be useful for future/other > tests? I think so (but not in its current form, these were just quick and dirty tests to trigger a particular pathological OOM scenario - they do what they do but they could be prettier). When I did JDK-8198423, I added tests at C++ level as gtests, see test_metaspace_allocation.cpp. I felt I had more control at that level. However, I still feel I owe the project some nice java level tests, plus, it would be nice to have some metaspace stress scenarios to play around. > 3) If so, I would actually recommending making a RFE for the test to > bring in similar code like that and remove my testcase.jar. That way this > bug focuses solely on just getting the print out right with a test and we > can refactor it in a second step with a useful common library that can come > it slowly with reviews that are concentrating on that library instead, what > do you think? Sounds good. Thanks, Thomas > > Thanks, > Jc > > On Thu, Sep 27, 2018 at 7:17 AM Thomas St?fe > wrote: >> >> Hi, >> >> On Thu, Sep 27, 2018 at 2:09 PM, Thomas Schatzl >> wrote: >> > Hi JC, >> > >> > On Wed, 2018-09-26 at 19:21 -0700, JC Beyler wrote: >> >> Hi Thomas et al, >> >> >> >> Thanks for moving this thread to a RFR. Here is the webrev and bug: >> >> Webrev: http://cr.openjdk.java.net/~jcbeyler/8211123/webrev.00/ >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8211123 >> >> >> >> For the test, I inspired myself of a few different tests to get this >> >> up and running. I didn't find a means to use only the WhiteBox to get >> >> the metaspace filled. I did not want to use the .jar that the >> >> original reproducer was using since it was using a library. So I used >> >> the testcases.jar from the runtime/7116786 test. >> >> >> >> I also use a separate process to test the various GCs and ensure it >> >> works for all. >> >> >> >> Btw, I've put nijiaben as the contributor for the webrev. >> >> >> >> Let me know what you think, I imagine I've made the test a bit more >> >> complicated than need be, >> >> Jc >> > >> > I do not know a better way than having an external jar to do class >> > loading into a separate class loader in a test. It's safest to >> > duplicate the jar file. >> >> Just as a suggestion, I did tests - but never really got to jtreg-ify >> them - for JDK-8198423. >> >> They use javax.tools.JavaCompiler to compile classes on the fly and >> load them into separate class loaders. >> >> See e.g. >> https://github.com/tstuefe/ojdk-repros/blob/master/src/test3/Example2.java >> >> (Not the prettiest program ever, but enough to demonstrate the point). >> >> Best, Thomas >> >> > >> > The test could be simplified a bit by using the Metaspace MemoryMXBean >> > to get metaspace memory readings (there is an MXBean called >> > "Metaspace"), removing the need for invoking a secondary JVM in the >> > test and parsing the log :) >> > >> > You can then run the test with the different collectors using >> > "main/othervm -XX:+ MyTest" lines. >> > >> > Otherwise the change itself looks good. I would appreciate if the test >> > could be improved as mentioned above, but if you do not want to I could >> > live with it as is. >> > >> > Btw, you can have multiple contributors in that line (or probably >> > multiple Contributed-by lines). >> > >> > Thanks, >> > Thomas >> > >> > > > > > -- > > Thanks, > Jc From thomas.schatzl at oracle.com Thu Sep 27 19:19:08 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 27 Sep 2018 21:19:08 +0200 Subject: RFR (XS): 8211123: GC Metaspace printing after full gc does not reflect recent changes (Was: Re: A Bug Of Metaspace After Full GC) In-Reply-To: References: <5fd8f58e6d6ae6b1e6f60d4cdc663afda9474894.camel@oracle.com> Message-ID: <39789d87fd1e9fd3071835187401e50313c24949.camel@oracle.com> Hi JC, On Thu, 2018-09-27 at 21:05 +0200, Thomas St?fe wrote: > Hi JC, > > On Thu, Sep 27, 2018 at 8:29 PM, JC Beyler > wrote: > > Hi Thomas and Thomas, > > > > Thanks for the reviews! > > > > @Thomas Schatzl: I looked for using a MXBean but I'm confused about > > using this. What this fix+test is trying to ascertain is that the > > moment of the logging of the metaspace data, the new size has been > > calculated; how do I do that with a MXBean? Would I not always get > > the current size? So if I did before FullGC, I'd get the previous > > size, after FullGC, I'd always get the new size, no? I am most > > likely missing something and I apologize! No, you are right, the MXBeans will be correct after GC of course, it's the logs that are wrong. So forget what I said. Thomas From jcbeyler at google.com Thu Sep 27 22:55:59 2018 From: jcbeyler at google.com (JC Beyler) Date: Thu, 27 Sep 2018 15:55:59 -0700 Subject: RFR (XS): 8211123: GC Metaspace printing after full gc does not reflect recent changes (Was: Re: A Bug Of Metaspace After Full GC) In-Reply-To: <39789d87fd1e9fd3071835187401e50313c24949.camel@oracle.com> References: <5fd8f58e6d6ae6b1e6f60d4cdc663afda9474894.camel@oracle.com> <39789d87fd1e9fd3071835187401e50313c24949.camel@oracle.com> Message-ID: Thanks both Thomas. I created https://bugs.openjdk.java.net/browse/JDK-8211255 for getting the code in to do the compilation on the fly and being to deprecate the testcase.jar that this would add. I have not put myself as assignee for that one due to me still knee deep in the vmTestbase rebasing. If I get some free cycles, I'll take a stab at it :-) I believe I have a LGTM from Thomas Schatzl for this webrev. Would you Thomas St?fe provide one as well or would someone else? Thanks! Jc On Thu, Sep 27, 2018 at 12:19 PM Thomas Schatzl wrote: > Hi JC, > > On Thu, 2018-09-27 at 21:05 +0200, Thomas St?fe wrote: > > Hi JC, > > > > On Thu, Sep 27, 2018 at 8:29 PM, JC Beyler > > wrote: > > > Hi Thomas and Thomas, > > > > > > Thanks for the reviews! > > > > > > @Thomas Schatzl: I looked for using a MXBean but I'm confused about > > > using this. What this fix+test is trying to ascertain is that the > > > moment of the logging of the metaspace data, the new size has been > > > calculated; how do I do that with a MXBean? Would I not always get > > > the current size? So if I did before FullGC, I'd get the previous > > > size, after FullGC, I'd always get the new size, no? I am most > > > likely missing something and I apologize! > > No, you are right, the MXBeans will be correct after GC of course, it's > the logs that are wrong. > > So forget what I said. > > Thomas > > -- Thanks, Jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Fri Sep 28 05:57:01 2018 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 28 Sep 2018 07:57:01 +0200 Subject: RFR (XS): 8211123: GC Metaspace printing after full gc does not reflect recent changes (Was: Re: A Bug Of Metaspace After Full GC) In-Reply-To: References: <5fd8f58e6d6ae6b1e6f60d4cdc663afda9474894.camel@oracle.com> <39789d87fd1e9fd3071835187401e50313c24949.camel@oracle.com> Message-ID: Hi JC, On Fri, Sep 28, 2018 at 12:55 AM, JC Beyler wrote: > Thanks both Thomas. > > I created https://bugs.openjdk.java.net/browse/JDK-8211255 for getting the > code in to do the compilation on the fly and being to deprecate the > testcase.jar that this would add. I have not put myself as assignee for that > one due to me still knee deep in the vmTestbase rebasing. If I get some free > cycles, I'll take a stab at it :-) > > I believe I have a LGTM from Thomas Schatzl for this webrev. Would you > Thomas St?fe provide one as well or would someone else? > Looks fine to me as well. Not sure if we really need 1G of heapspace in the child process, but I have no strong emotions there. Thanks, Thomas > Thanks! > Jc > > On Thu, Sep 27, 2018 at 12:19 PM Thomas Schatzl > wrote: >> >> Hi JC, >> >> On Thu, 2018-09-27 at 21:05 +0200, Thomas St?fe wrote: >> > Hi JC, >> > >> > On Thu, Sep 27, 2018 at 8:29 PM, JC Beyler >> > wrote: >> > > Hi Thomas and Thomas, >> > > >> > > Thanks for the reviews! >> > > >> > > @Thomas Schatzl: I looked for using a MXBean but I'm confused about >> > > using this. What this fix+test is trying to ascertain is that the >> > > moment of the logging of the metaspace data, the new size has been >> > > calculated; how do I do that with a MXBean? Would I not always get >> > > the current size? So if I did before FullGC, I'd get the previous >> > > size, after FullGC, I'd always get the new size, no? I am most >> > > likely missing something and I apologize! >> >> No, you are right, the MXBeans will be correct after GC of course, it's >> the logs that are wrong. >> >> So forget what I said. >> >> Thomas >> > > > -- > > Thanks, > Jc From nijiaben at perfma.com Fri Sep 28 06:24:12 2018 From: nijiaben at perfma.com (=?utf-8?B?bmlqaWFiZW4=?=) Date: Fri, 28 Sep 2018 14:24:12 +0800 Subject: A little more advice on the metaspace gc log Message-ID: hi all, Thank you so much for the follow-up about the patch that I mentioned about the metaspace last week(A Bug Of Metaspace After Full GC).However,I still have the other suggestion about the metaspace. Please allow me to describe it briefly. Still in the Full GC scenario, we will print the size of the metaspace in the GC log. Suppose we set -XX:MetaspaceSize=100M -XX:MaxMetaspaceSize=100M, which means a metaspace full GC should be triggered once the metaspace usage reaches to 100M, but actually we find the problem from the gc log and only 70899KB is used in the metaspace. [Full GC (Metadata GC Threshold) [PSYoungGen: 5092K->0K(132096K)] [ParOldGen: 85822K->60451K(181760K)] 90914K->60451K(313856K), [Metaspace: 70899K->70899K(1136640K)], 0.1482072 secs] [Times: user=0.52 sys=0.00, real=0.15 secs] First let?s take a look at the code that prints this line of gc log which really depends on the size used by metaspace. void MetaspaceAux::print_metaspace_change(size_t prev_metadata_used) { gclog_or_tty->print(", [Metaspace:"); if (PrintGCDetails && Verbose) { gclog_or_tty->print(" " SIZE_FORMAT "->" SIZE_FORMAT "(" SIZE_FORMAT ")", prev_metadata_used, used_bytes(), reserved_bytes()); } else { gclog_or_tty->print(" " SIZE_FORMAT "K" "->" SIZE_FORMAT "K" "(" SIZE_FORMAT "K)", prev_metadata_used/K, used_bytes()/K, reserved_bytes()/K); } gclog_or_tty->print("]"); } However, the basis for triggering metaspace Full GC is not based on the real usage of metaspace, but the memory size of metaspace committed. size_t MetaspaceGC::allowed_expansion() { size_t committed_bytes = MetaspaceAux::committed_bytes(); size_t capacity_until_gc = capacity_until_GC(); assert(capacity_until_gc >= committed_bytes, err_msg("capacity_until_gc: " SIZE_FORMAT " < committed_bytes: " SIZE_FORMAT, capacity_until_gc, committed_bytes)); size_t left_until_max = MaxMetaspaceSize - committed_bytes; size_t left_until_GC = capacity_until_gc - committed_bytes; size_t left_to_commit = MIN2(left_until_GC, left_until_max); return left_to_commit / BytesPerWord; } Obviously, if we create a lot of classloaders, each of them will correspond to a metaspace data structure. The usage of our metaspace refers to the real in Metanchunk. The size of the memory is allocated, but Metanchunk is not all used up. This problem is simplified. Just like the fragmentation of the CMS GC, we have a certain gap between the memory committed and the memory used, and trigger the metaspace Full GC is based on the size of the committed, so the user feels like the metaspace doesn?t reach our threshold triggers the Full GC, which seems doubtful. Therefore, I would like to make a small suggestion that for the scenario of Full GC, the committed memory size instead of the used memory size will be printed more directly when we print the GC log of metaspace, which will make the user to see it more clearly and turn out that the threshold of metaspace has already reached the trigger of Full GC. The modified log is more conspicuous and indeed triggered when reaching 100M, but it will bring another drawback that we don?t know how much metaspace really have been used. [Full GC (Metadata GC Threshold) [PSYoungGen: 5092K->0K(132096K)] [ParOldGen: 81395K->60429K(181248K)] 86488K->60429K(313344K), [Metaspace: 102400K->70599K(1136640K)], 0.1330421 secs] [Times: user=0.46 sys=0.00, real=0.13 secs] This issue will have some impact on various GC strategies. What do you think about this? I am looking forward to your reply. Thanks, nijiaben -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirk.pepperdine at gmail.com Fri Sep 28 06:37:10 2018 From: kirk.pepperdine at gmail.com (Kirk Pepperdine) Date: Thu, 27 Sep 2018 23:37:10 -0700 Subject: A little more advice on the metaspace gc log In-Reply-To: References: Message-ID: > On Sep 27, 2018, at 11:24 PM, nijiaben wrote: > > hi all, > > Thank you so much for the follow-up about the patch that I mentioned about the metaspace last week(A Bug Of Metaspace After Full GC ).However,I still have the other suggestion about the metaspace. Please allow me to describe it briefly. > > Still in the Full GC scenario, we will print the size of the metaspace in the GC log. Suppose we set -XX:MetaspaceSize=100M -XX:MaxMetaspaceSize=100M, which means a metaspace full GC should be triggered once the metaspace usage reaches to 100M, but actually we find the problem from the gc log and only 70899KB is used in the metaspace. > > [Full GC (Metadata GC Threshold) [PSYoungGen: 5092K->0K(132096K)] [ParOldGen: 85822K->60451K(181760K)] 90914K->60451K(313856K), [Metaspace: 70899K->70899K(1136640K)], 0.1482072 secs] [Times: user=0.52 sys=0.00, real=0.15 secs] > First let?s take a look at the code that prints this line of gc log which really depends on the size used by metaspace. > > void MetaspaceAux::print_metaspace_change(size_t prev_metadata_used) { > gclog_or_tty->print(", [Metaspace:"); > if (PrintGCDetails && Verbose) { > gclog_or_tty->print(" " SIZE_FORMAT > "->" SIZE_FORMAT > "(" SIZE_FORMAT ")", > prev_metadata_used, > used_bytes(), > reserved_bytes()); > } else { > gclog_or_tty->print(" " SIZE_FORMAT "K" > "->" SIZE_FORMAT "K" > "(" SIZE_FORMAT "K)", > prev_metadata_used/K, > used_bytes()/K, > reserved_bytes()/K); > } > > gclog_or_tty->print("]"); > } > However, the basis for triggering metaspace Full GC is not based on the real usage of metaspace, but the memory size of metaspace committed. > > size_t MetaspaceGC::allowed_expansion() { > size_t committed_bytes = MetaspaceAux::committed_bytes(); > size_t capacity_until_gc = capacity_until_GC(); > > assert(capacity_until_gc >= committed_bytes, > err_msg("capacity_until_gc: " SIZE_FORMAT " < committed_bytes: " SIZE_FORMAT, > capacity_until_gc, committed_bytes)); > > size_t left_until_max = MaxMetaspaceSize - committed_bytes; > size_t left_until_GC = capacity_until_gc - committed_bytes; > size_t left_to_commit = MIN2(left_until_GC, left_until_max); > > return left_to_commit / BytesPerWord; > } > Obviously, if we create a lot of classloaders, each of them will correspond to a metaspace data structure. The usage of our metaspace refers to the real in Metanchunk. The size of the memory is allocated, but Metanchunk is not all used up. This problem is simplified. Just like the fragmentation of the CMS GC, we have a certain gap between the memory committed and the memory used, and trigger the metaspace Full GC is based on the size of the committed, so the user feels like the metaspace doesn?t reach our threshold triggers the Full GC, which seems doubtful. > > Therefore, I would like to make a small suggestion that for the scenario of Full GC, the committed memory size instead of the used memory size will be printed more directly when we print the GC log of metaspace, which will make the user to see it more clearly and turn out that the threshold of metaspace has already reached the trigger of Full GC. > > The modified log is more conspicuous and indeed triggered when reaching 100M, but it will bring another drawback that we don?t know how much metaspace really have been used. > > [Full GC (Metadata GC Threshold) [PSYoungGen: 5092K->0K(132096K)] [ParOldGen: 81395K->60429K(181248K)] 86488K->60429K(313344K), [Metaspace: 102400K->70599K(1136640K)], 0.1330421 secs] [Times: user=0.46 sys=0.00, real=0.13 secs] > This issue will have some impact on various GC strategies. > > What do you think about this? I am looking forward to your reply. > Great detective work.. Adding committed to the log seems like a reasonable thing to do. That said there is a lot of tooling that is used to used before->used after(size of pool). Changing the def is a compilation for tools that work across many JVM versions. If you are to add new information in the log could I kindly suggest that you add it in a new column so that it?s distinctly committed or distinctly acclimated. Kind regards, Kirk -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Fri Sep 28 08:31:54 2018 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 28 Sep 2018 10:31:54 +0200 Subject: A little more advice on the metaspace gc log In-Reply-To: References: Message-ID: On Fri, Sep 28, 2018 at 8:24 AM, nijiaben wrote: > hi all, > > Thank you so much for the follow-up about the patch that I mentioned about > the metaspace last week(A Bug Of Metaspace After Full GC).However,I still > have the other suggestion about the metaspace. Please allow me to describe > it briefly. > > Still in the Full GC scenario, we will print the size of the metaspace in > the GC log. Suppose we set -XX:MetaspaceSize=100M -XX:MaxMetaspaceSize=100M, > which means a metaspace full GC should be triggered once the metaspace usage > reaches to 100M, but actually we find the problem from the gc log and only > 70899KB is used in the metaspace. > > [Full GC (Metadata GC Threshold) [PSYoungGen: 5092K->0K(132096K)] > [ParOldGen: 85822K->60451K(181760K)] 90914K->60451K(313856K), [Metaspace: > 70899K->70899K(1136640K)], 0.1482072 secs] [Times: user=0.52 sys=0.00, > real=0.15 secs] > > First let?s take a look at the code that prints this line of gc log which > really depends on the size used by metaspace. > > void MetaspaceAux::print_metaspace_change(size_t prev_metadata_used) { > gclog_or_tty->print(", [Metaspace:"); > if (PrintGCDetails && Verbose) { > gclog_or_tty->print(" " SIZE_FORMAT > "->" SIZE_FORMAT > "(" SIZE_FORMAT ")", > prev_metadata_used, > used_bytes(), > reserved_bytes()); > } else { > gclog_or_tty->print(" " SIZE_FORMAT "K" > "->" SIZE_FORMAT "K" > "(" SIZE_FORMAT "K)", > prev_metadata_used/K, > used_bytes()/K, > reserved_bytes()/K); > } > > gclog_or_tty->print("]"); > } > > However, the basis for triggering metaspace Full GC is not based on the real > usage of metaspace, but the memory size of metaspace committed. > > size_t MetaspaceGC::allowed_expansion() { > size_t committed_bytes = MetaspaceAux::committed_bytes(); > size_t capacity_until_gc = capacity_until_GC(); > > assert(capacity_until_gc >= committed_bytes, > err_msg("capacity_until_gc: " SIZE_FORMAT " < committed_bytes: " > SIZE_FORMAT, > capacity_until_gc, committed_bytes)); > > size_t left_until_max = MaxMetaspaceSize - committed_bytes; > size_t left_until_GC = capacity_until_gc - committed_bytes; > size_t left_to_commit = MIN2(left_until_GC, left_until_max); > > return left_to_commit / BytesPerWord; > } > > Obviously, if we create a lot of classloaders, each of them will correspond > to a metaspace data structure. The usage of our metaspace refers to the real > in Metanchunk. The size of the memory is allocated, but Metanchunk is not > all used up. This problem is simplified. Just like the fragmentation of the > CMS GC, we have a certain gap between the memory committed and the memory > used, and trigger the metaspace Full GC is based on the size of the > committed, so the user feels like the metaspace doesn?t reach our threshold > triggers the Full GC, which seems doubtful. > > Therefore, I would like to make a small suggestion that for the scenario of > Full GC, the committed memory size instead of the used memory size will be > printed more directly when we print the GC log of metaspace, which will make > the user to see it more clearly and turn out that the threshold of metaspace > has already reached the trigger of Full GC. > > The modified log is more conspicuous and indeed triggered when reaching > 100M, but it will bring another drawback that we don?t know how much > metaspace really have been used. > > [Full GC (Metadata GC Threshold) [PSYoungGen: 5092K->0K(132096K)] > [ParOldGen: 81395K->60429K(181248K)] 86488K->60429K(313344K), [Metaspace: > 102400K->70599K(1136640K)], 0.1330421 secs] [Times: user=0.46 sys=0.00, > real=0.13 secs] > > This issue will have some impact on various GC strategies. > > What do you think about this? I am looking forward to your reply. > FWIW, I agree this is confusing, and personally would prefer the committed size be printed as you suggests, but I guess this may break a lot of tooling. Note that the gap between committed and used can be quite large in the following pathological scenarios: a) lots of class loaders which do not load many classes and stop loading classes at an inconvenient time: when having been handed a larger memory chunk after having used up four small chunks - which the JVM takes as proof that this classloader is here to stay and therefore may just as well get a large chunk to gnaw on. This leads to fragmentation and there is no easy way to avoid it. We can play around with chunk sizes and allocation strategies but there will always be pathological scenarios since the JVM needs to guess and may guess wrongly. b) when class loaders died and their memory chunks are returned to the internal metaspace freelists. They are not always uncommitted immediately; in fact they never may be uncommitted. They will be reused when other classloaders load classes, which may never happen. Note: to analyze these situations better we (SAP) added a new command to jcmd, "VM.metaspace". Among other things, the command outputs a "Waste" section, which details both (a) and (b) and some minor other waste scenarios. That command is now upstream in stock OpenJDK, so it should be available in jdk11. Thanks, Thomas > > Thanks, > > nijiaben From rkennke at redhat.com Fri Sep 28 08:45:07 2018 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 28 Sep 2018 10:45:07 +0200 Subject: RFR (trivial): JDK-8211269: Make declaration of Allocation protected in MemAllocator Message-ID: I want to override mem_allocate(Allocation&) but can't because I cannot access the private class Allocation. Let's make it protected. Bug: https://bugs.openjdk.java.net/browse/JDK-8211269 Webrev: http://cr.openjdk.java.net/~rkennke/JDK-8211269/webrev.00/ Ok? Also, counts as trivial I suppose? Thanks, Roman -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From shade at redhat.com Fri Sep 28 08:55:28 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 28 Sep 2018 10:55:28 +0200 Subject: RFR (trivial): JDK-8211269: Make declaration of Allocation protected in MemAllocator In-Reply-To: References: Message-ID: <142f04a8-e392-2f72-784c-18155659cc31@redhat.com> On 09/28/2018 10:45 AM, Roman Kennke wrote: > Webrev: > http://cr.openjdk.java.net/~rkennke/JDK-8211269/webrev.00/ Looks good to me. -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rkennke at redhat.com Fri Sep 28 10:11:54 2018 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 28 Sep 2018 12:11:54 +0200 Subject: RFR: JDK-8211270: GC abstraction to get real object and headers size Message-ID: <1b75ccd7-ba1a-a183-f338-18d3c651827e@redhat.com> Shenandoah GC uses one extra word per object. Some places need to know the real object or header size (fwd ptr plus object). The affected code paths are WB_GetObjectSize(), JvmtiEnv::GetObjectSize() and the PLAB constructor. The patch also adds an assert later in PLAB that uses the header_size(). Bug: https://bugs.openjdk.java.net/browse/JDK-8211270 Webrev: file:///home/rkennke/src/openjdk/webrev/JDK-8211270/webrev.00/ Can I please get reviews? Thanks, Roman -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rkennke at redhat.com Fri Sep 28 10:21:41 2018 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 28 Sep 2018 12:21:41 +0200 Subject: RFR: JDK-8211270: GC abstraction to get real object and headers size In-Reply-To: <1b75ccd7-ba1a-a183-f338-18d3c651827e@redhat.com> References: <1b75ccd7-ba1a-a183-f338-18d3c651827e@redhat.com> Message-ID: Duh. Webrev: http://cr.openjdk.java.net/~rkennke/JDK-8211270/webrev.00/ Thanks, Roman Am 28.09.18 um 12:11 schrieb Roman Kennke: > Shenandoah GC uses one extra word per object. Some places need to know > the real object or header size (fwd ptr plus object). > > The affected code paths are WB_GetObjectSize(), > JvmtiEnv::GetObjectSize() and the PLAB constructor. The patch also adds > an assert later in PLAB that uses the header_size(). > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8211270 > Webrev: > file:///home/rkennke/src/openjdk/webrev/JDK-8211270/webrev.00/ > > Can I please get reviews? > > Thanks, > Roman > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From erik.osterlund at oracle.com Fri Sep 28 13:48:58 2018 From: erik.osterlund at oracle.com (=?UTF-8?Q?Erik_=c3=96sterlund?=) Date: Fri, 28 Sep 2018 15:48:58 +0200 Subject: RFR (trivial): JDK-8211269: Make declaration of Allocation protected in MemAllocator In-Reply-To: References: Message-ID: <5BAE314A.50708@oracle.com> Hi Roman, Looks good. Thanks, /Erik On 2018-09-28 10:45, Roman Kennke wrote: > I want to override mem_allocate(Allocation&) but can't because I cannot > access the private class Allocation. Let's make it protected. > > Bug: > https://bugs.openjdk.java.net/browse/JDK-8211269 > Webrev: > http://cr.openjdk.java.net/~rkennke/JDK-8211269/webrev.00/ > > Ok? Also, counts as trivial I suppose? > > Thanks, > Roman > From rkennke at redhat.com Fri Sep 28 14:59:54 2018 From: rkennke at redhat.com (Roman Kennke) Date: Fri, 28 Sep 2018 16:59:54 +0200 Subject: RFR: JDK-8211279: Verify missing object equals barriers Message-ID: <935559b9-6df2-e337-97b5-5807cf57a154@redhat.com> GCs like Shenandoah require an extra barrier for comparing objects (oop==oop). It is easy to forget or overlook this. It would be very useful to have a flag to turn on extra verification that catches missing object equality barriers. This change inserts an assert into == and != operators for the oop class in oopsHierarchy.hpp. This only gets compiled in fastdebug builds (when CHECK_UNHANDLED_OOPS is on). It also adds a develop flag VerifyObjectEquals that is used to turn on this verification. It also adds a method oopDesc::unsafe_equals(..) to be used in cases where you know what what you are doing, and really want to use direct == comparison without using barriers. This is used in e.g. ReferenceProcessor or all over the place in ShenandoahGC. The change also fixes a couple of places where oops are compared to non-oops like Universe::non_oop_word() to use the oop==void* operator instead, so those don't falsely trip the verification. It doesn't make sense to turn this check on if you're not running a GC that needs it, unless you want to go fix all the oop==oop in the GC itself. Bug: https://bugs.openjdk.java.net/browse/JDK-8211279 Webrev: http://cr.openjdk.java.net/~rkennke/JDK-8211279/webrev.00/ What do you think? Roman -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From vladimir.kozlov at oracle.com Fri Sep 28 17:51:35 2018 From: vladimir.kozlov at oracle.com (Vladimir Kozlov) Date: Fri, 28 Sep 2018 10:51:35 -0700 Subject: RFR: 8208686: [AOT] JVMTI ResourceExhausted event repeated for same allocation In-Reply-To: <64cbe730-5c9a-a04d-9eee-a56abfbb8e07@oracle.com> References: <910A4A3C-3EEF-4167-84D5-9819C83D6FC1@oracle.com> <64cbe730-5c9a-a04d-9eee-a56abfbb8e07@oracle.com> Message-ID: <015e8416-a948-fdc5-46db-8b5d80ba52e8@oracle.com> To let you know, me and Tom R. did review these changes and agreed that it is the least intrusive changes for Hotspot shared code. Thanks, Vladimir On 9/25/18 8:11 AM, Daniel D. Daugherty wrote: > Adding serviceability-dev at ... since this is JVM/TI... > > Dan > > > On 9/25/18 10:48 AM, Doug Simon wrote: >> A major design point of Graal is to treat allocations as non-side effecting to give more freedom >> to the optimizer by reducing the number of distinct FrameStates that need to be managed. When >> failing an allocation, Graal will deoptimize to the last side effecting instruction before the >> allocation. This mean the VM code for heap allocation will potentially be executed twice, once >> from Graal compiled code and then again in the interpreter. While this is perfectly fine according >> to the JVM specification, it can cause confusing behavior for JVMTI based tools. They will receive >> 2 ResourceExhausted events for a single allocation. Furthermore, the first ResourceExhausted event >> (on the Graal allocation slow path) might denote a bytecode instruction that performs no >> allocation, making it hard to debug the memory failure. >> >> The proposed solution is to add an extra set of JVMCI VM runtime calls for allocation. These entry >> points will attempt the allocation and upon failure, >> skip side-effects such as posting JVMTI events or handling -XX:OnOutOfMemoryError. The compiled >> code using these entry points is expected deoptmize on null. >> >> The path from these new entry points to where allocation can fail goes through quite a bit of VM >> code. One could modify all these paths by: >> * Returning null instead of throwing an exception on failure. >> * Adding a `bool null_on_fail` argument to all relevant methods. >> * Adding extra null checking where necessary after each call to these methods when `null_on_fail >> == true`. >> This represents a significant number of changes. >> >> Instead, the proposed solution introduces a new _in_retryable_allocation thread-local. This way, >> only the entry points and allocation routines that raise an exception need to be modified. Failure >> is communicated back to the new entry points by throwing a special pre-allocated OOME object >> (i.e., Universe::out_of_memory_error_retry()) which must not propagate back to Java code. Use of >> this object is not strictly necessary; it is introduced to highlight/document the special >> allocation mode. >> >> The proposed solution is at http://cr.openjdk.java.net/~dnsimon/8208686. >> THE JBS bug is: https://bugs.openjdk.java.net/browse/JDK-8208686 >> >> -Doug > From markus.gronlund at oracle.com Fri Sep 28 20:03:03 2018 From: markus.gronlund at oracle.com (Markus Gronlund) Date: Fri, 28 Sep 2018 13:03:03 -0700 (PDT) Subject: RFR (S) 8211228: Zero build is broken after JDK-8196341 (Add JFR events for parallel phases of G1) In-Reply-To: <1b399696-0edc-14ae-13f3-af5d3ab73a6a@redhat.com> References: <1b399696-0edc-14ae-13f3-af5d3ab73a6a@redhat.com> Message-ID: <326db34b-a6df-48d2-98e7-3858c9b464af@default> Hi Aleksey, I am currently travelling and I have not been following this in detail. As I understand it, JDK-8196341, extended the signatures for the commit() methods, but the associated !INCLUDE_JFR section was not updated, causing compilation errors. I have not checked everything in detail, but it seems the fix does add those missing signature(s). I am ok with this from a JFR perspective. Thanks for addressing Markus -----Original Message----- From: Aleksey Shipilev Sent: den 27 september 2018 14:16 To: hotspot-gc-dev at openjdk.java.net Subject: Re: RFR (S) 8211228: Zero build is broken after JDK-8196341 (Add JFR events for parallel phases of G1) On 09/27/2018 03:55 PM, Aleksey Shipilev wrote: > Bug: > https://bugs.openjdk.java.net/browse/JDK-8211228 > > Fix: > http://cr.openjdk.java.net/~shade/8211228/webrev.01/ > > Recent addition of JFR events to G1 broke Zero build. That is because > Zero builds with JFR disabled, but G1 enabled (it is a separate > question why!). It can be fixed by putting JFR_ONLY all around > JFR-specific code, or by configuring Zero to avoid building g1gc. But, since it looks like AIX is also affected by this: > https://bugs.openjdk.java.net/browse/JDK-8211213 I retract this RFR in favor of proper fix on JFR side: https://bugs.openjdk.java.net/browse/JDK-8211239 -Aleksey From manc at google.com Sat Sep 29 01:46:53 2018 From: manc at google.com (Man Cao) Date: Fri, 28 Sep 2018 18:46:53 -0700 Subject: RFR(XS): 8210716: Detailed GC logging request misses some Message-ID: Hi, Could someone review this minor logging tag change for ParallelGC? It avoids missing log messages with -Xlog:gc*=debug and -Xlog:gc*=trace. Bug: https://bugs.openjdk.java.net/browse/JDK-8210716 Webrev: http://cr.openjdk.java.net/~manc/8210716/webrev.00/ PS: I'd need someone help me push it as I just became an Author. Thanks, Man -------------- next part -------------- An HTML attachment was scrubbed... URL: From synytskyy at jelastic.com Sat Sep 29 07:26:01 2018 From: synytskyy at jelastic.com (Ruslan Synytsky) Date: Sat, 29 Sep 2018 10:26:01 +0300 Subject: RFR: bug: Timely Reducing Unused Committed Memory In-Reply-To: <3927596f-8847-ba95-d4c6-3c5f3656330c@oracle.com> References: <8af5c2468f5133bc1ba2517bf52c3078af516977.camel@oracle.com> <65063e5f31702649e12b485b570ecab9ebd32d01.camel@oracle.com> <66c536399b2f218af0d85bd3bced44f204e3dbf4.camel@oracle.com> <2821cf70-c4d5-016f-b976-5ecc850f377c@oracle.com> <57b10708-b870-cb98-0df0-fab650f431e4@oracle.com> <3927596f-8847-ba95-d4c6-3c5f3656330c@oracle.com> Message-ID: On Tue, Sep 25, 2018 at 17:49 Stefan Johansson wrote: > Thanks Ruslan for your input, > > On 2018-09-21 15:35, Ruslan Synytsky wrote: > > Dear Stefan and Rodrigo, thank you for moving this forward. > > > > ---------- Forwarded message --------- > >> From: *Stefan Johansson* >> > > >> Date: quarta, 19/09/2018 ?(s) 10:45 > >> Subject: Re: RFR: bug: Timely Reducing Unused Committed Memory > >> To: >> >, >> > > >> > >> > >> Hi Rodrigo, > >> > >> I pasted your reply here to keep the discussion in one thread. > >> > >> >> I understand that it is hard to define what is idle. However, if we > >> require the > >> >> user to provide one, I guess that most regular users that suffer > >> from the problem > >> >> that this patch is trying to solve will simply not do it because it > >> requires knowledge > >> >> and effort. If we provide an idle check that we think will benefit > >> most users, then > >> >> we are probably helping a lot of users. For those that the default > >> idle check is > >> >> not good enough, they can always disable this idle check and > >> implement the idle > >> >> check logic it in an external tool. > >> >> > >> > I agree, if we can find a solution that benefits most users, we should > >> > do it. And this is why I would like to hear from more users if this > >> > would benefit their use cases. > > I believe the default idle definition should be based on the major > > bottlenecks: RAM, CPU and IO loads as well as the network. RAM - we try > > to improve. IO - I?m not sure if we can measure IO load properly inside > > JVM. If possible then it's good to add too. If not then we can skip it > > for now, as it can be measured and triggered by outside logic. Network > > is not involved in GC process, correct? So no need for that. CPU looks > > the most obvious and already implemented, seems like a good option to > > start from. > > I agree that CPU can look obvious, but making decisions in the VM based > on the system load might be hard. For example the avg load might be low > while the current process is fairly active. Hi Stefan, you are right, it might be like this. Another question, when > running in the cloud, what load is the user expecting us to compare > against, the overall system or the local container. I'm actually not > entirely sure what the getloadavg() call return in case of running in a > container. Good question! It depends on the used container technology. In short, if it?s a system container then it shows the load of the container, if it?s an application container then the load of the host machine. There is an article on a related topic https://jelastic.com/blog/java-and-memory-limits-in-containers-lxc-docker-and-openvz/ Can we measure CPU usage of JVM process itself and use it for decisions? > > > > >> > Another thing that I don't fully > >> > understand is why the flags are manageable if there isn't supposed > >> to be > >> > some external logic that sets them? > > Some advanced users, for example cloud platform or software vendors, > > will be able to apply an additional logic based on their custom needs / > > specifics. Such flexibility enables more use cases and it helps to > > collect more feedback for the further default improvements. > > That's how I would expect it to be used as well, thanks for clarifying > your viewpoint. > > >> > >> >> We can also change the semantics of "idleness". Currently it > >> checks the load. > >> >> I think that checking the allocation rate might be another good > >> option (instead of > >> >> load). The only corner case is an application that does not > >> allocate but consumes > >> >> a lot of CPU. For this case, we might only trigger compaction at > >> most once because, > >> >> as it does not allocate memory, we will not get over committed > >> memory (i.e., the other > >> >> checks will prevent it). The opposite is also possible (almost idle > >> application that allocates > >> >> a lot of memory) but in this scenario I don't think we want to > >> trigger an idle compaction. > >> >> > >> > >> > This is my main problem when it comes to determine "idleness", for > some > >> > applications allocation rate will be the correct metric, for others it > >> > will be the load and for a third something different. It feels like it > >> > is always possible to come up with a case that needs something > >> different. > > I would prefer to start with the most obvious one - based on CPU, give > > it to try to more people by promoting the fact that JVM is elastic now, > > and we will get more feedback that can be converted into an additional > > logic later. > > > So basically, the first version would have two flags, one to turn on > periodic GCs (currently named GCFrequency) and one to control at which > average load (MaxLoadGC) these GCs will kick in? I think it?s a good starting point. > > >> >> Having said that, I am open to change this flag or even remove it > >> as it is one of the > >> >> hardest to get right. > >> >> > >> > >> > As I said before, to me it feels like just having a periodic GC > >> interval > >> > flag that is manageable would be a good start. Maybe have constraint > >> > that the periodic GC only occurs if no other GCs have happened during > >> > the interval. > >> > > Decision based on the previous GC cycles is very good proposal. I think > > we need to take it into account somehow, but I'm not so deep on it. > > Input of others will be helpful here. > > I guess there are corner cases in this area as well, but I guess the > simple constraint I described might be a good start. But as you say, > input from others would be very helpful. > > >> > Could you explain how your use case would suffer from such > >> > limitations? > > In my opinion, CPU load spikes is clearly one of the major use cases > > eligible for defaults. > > This is clear and good use case where I guess having a load threshold > should really help. Thanks > > Thanks, > Stefan > > > > > Thank you > > > >> > >> > Thanks, > >> > Stefan > >> > >> >> cheers, > >> >> rodrigo > >> > >> > >> On 2018-09-13 14:30, Stefan Johansson wrote: > >> > Hi Rodrigo, > >> > > >> > Sorry for being a bit late into the discussion. We've had some > internal > >> > discussions and realized that there are some questions that I need to > >> > bring up here. > >> > > >> > I'm trying to better understand under what circumstances this > >> feature is > >> > to be used and how a user should use the different flags to tweak it > to > >> > their use case. To me it feels like GCFrequency would be enough to > make > >> > sure that the VM returns memory on a timely basis. And if the flag is > >> > managed, it can be controlled to not do periodic GCs during high load. > >> > With that we get a good way to periodically try to reduce the > committed > >> > heap. > >> > > >> > The reason I ask is because I have a hard time seeing how we can > >> > implement a generic policy for when the system is idle. A policy that > >> > will apply well to most use cases. For some cases having the flags you > >> > propose might be good, but for other there might be a different set of > >> > options needed. If this is the case then maybe the logic and policy of > >> > when to do this can live outside the VM, while the code to > periodically > >> > do GCs lives within the VM. What do you think about that? I understand > >> > the problems you've stated with having the policy outside that VM, but > >> > at least we have more information to act on there. > >> > > >> > We know that many have asked for features similar to this one and it > >> > would be nice to get input from others on this to make sure we > >> implement > >> > something that benefits the whole user base as much as possible. So > >> > anyone with a use case that could benefit from this, please chime in. > >> > > >> > Regards, > >> > Stefan > >> > > >> > > >> > > >> > On 2018-09-07 17:37, Rodrigo Bruno wrote: > >> >> Hi Per and Thomas, > >> >> > >> >> thank you for your comments. > >> >> > >> >> I think it is possible to implement this feature using the service > >> >> thread or using a separate thread. > >> >> I see some pros and cons of having a separate thread: > >> >> > >> >> Pros: > >> >> - using the service thread exposes something that is G1 specific to > >> >> the rest of the JVM. > >> >> Thus, using a separate thread, hides this feature from the outsite. > >> >> > >> >> Cons: > >> >> - Having a manageable timeout is a bit more tricky to implement in a > >> >> separate/dedicated thread. > >> >> We need to be able to handle switch on and off. It might require some > >> >> variable pooling. > >> >> - It requires some more memory. > >> >> > >> >> Regardless of the path taken, I can prepare a new version of the > patch > >> >> whenever we decide on this. > >> >> > >> >> cheers, > >> >> rodrigo > >> >> > >> >> Per Liden > >> >> > >> >> escreveu no dia sexta, 7/09/2018 ?(s) 11:58: > >> >> > >> >> Hi Thomas, > >> >> > >> >> On 09/07/2018 10:10 AM, Thomas Schatzl wrote: > >> >> [...] > >> >> > overnight I thought a bit of the implementation, and > >> given the > >> >> > problem with heap usage of the new thread, and the > >> requirement of > >> >> being > >> >> > able to turn on/off that feature by a managed variable, the > best > >> >> change > >> >> > would probably reusing the service thread as you did in the > >> >> initial > >> >> > change. > >> >> > >> >> I'm not convinced that this should be handled outside of G1. If > >> >> there's > >> >> a need to have the flag manageable at runtime (is that really the > >> >> case?), you could just always start the G1DetectIdleThread and > >> >> have it > >> >> check the flag. I wouldn't worry too much about the memory > >> >> overhead for > >> >> the stack. > >> >> > >> >> cheers, > >> >> Per > >> >> > > > > > > > > -- > > Ruslan > > CEO @ Jelastic > -- Ruslan CEO @ Jelastic -------------- next part -------------- An HTML attachment was scrubbed... URL: From kim.barrett at oracle.com Sat Sep 29 19:19:05 2018 From: kim.barrett at oracle.com (Kim Barrett) Date: Sat, 29 Sep 2018 15:19:05 -0400 Subject: RFR(XS): 8210716: Detailed GC logging request misses some In-Reply-To: References: Message-ID: <0515E1CE-8665-48D4-826A-DFB4AC9A8118@oracle.com> > On Sep 28, 2018, at 9:46 PM, Man Cao wrote: > > Hi, > > Could someone review this minor logging tag change for ParallelGC? It avoids missing log messages with -Xlog:gc*=debug and -Xlog:gc*=trace. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8210716 > Webrev: http://cr.openjdk.java.net/~manc/8210716/webrev.00/ > > PS: I'd need someone help me push it as I just became an Author. > > Thanks, > Man All other uses of the "ergo" tag in gc logging place it as the second tag, e.g. "(gc, ergo, ...)" or "(gc, ergo)". Not that the order matters for logging control, but consistency can help with searching and pattern matching while reading. adjoiningGenerations.cpp needs copyright update. I can sponsor this change for you. From martinrb at google.com Sat Sep 29 23:02:18 2018 From: martinrb at google.com (Martin Buchholz) Date: Sat, 29 Sep 2018 16:02:18 -0700 Subject: RFR(XS): 8210716: Detailed GC logging request misses some In-Reply-To: <0515E1CE-8665-48D4-826A-DFB4AC9A8118@oracle.com> References: <0515E1CE-8665-48D4-826A-DFB4AC9A8118@oracle.com> Message-ID: On Sat, Sep 29, 2018 at 12:19 PM, Kim Barrett wrote: > > adjoiningGenerations.cpp needs copyright update. > FYI there is no need for Google contributions to update copyright years. -------------- next part -------------- An HTML attachment was scrubbed... URL: From manc at google.com Sun Sep 30 00:37:22 2018 From: manc at google.com (Man Cao) Date: Sat, 29 Sep 2018 17:37:22 -0700 Subject: RFR(XS): 8210716: Detailed GC logging request misses some In-Reply-To: References: <0515E1CE-8665-48D4-826A-DFB4AC9A8118@oracle.com> Message-ID: Hi, I updated the patch as Kim suggested: http://cr.openjdk.java.net/~manc/8210716/webrev.01/ I updated the copyright years anyway. -Man On Sat, Sep 29, 2018 at 4:02 PM Martin Buchholz wrote: > > > On Sat, Sep 29, 2018 at 12:19 PM, Kim Barrett > wrote: > >> >> adjoiningGenerations.cpp needs copyright update. >> > > FYI there is no need for Google contributions to update copyright years. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kim.barrett at oracle.com Sun Sep 30 01:29:14 2018 From: kim.barrett at oracle.com (Kim Barrett) Date: Sat, 29 Sep 2018 21:29:14 -0400 Subject: RFR(XS): 8210716: Detailed GC logging request misses some In-Reply-To: References: <0515E1CE-8665-48D4-826A-DFB4AC9A8118@oracle.com> Message-ID: > On Sep 29, 2018, at 8:37 PM, Man Cao wrote: > > Hi, > > I updated the patch as Kim suggested: > http://cr.openjdk.java.net/~manc/8210716/webrev.01/ > > I updated the copyright years anyway. > > -Man Looks good. From thomas.schatzl at oracle.com Sun Sep 30 19:33:28 2018 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Sun, 30 Sep 2018 21:33:28 +0200 Subject: RFR(XS): 8210716: Detailed GC logging request misses some In-Reply-To: References: <0515E1CE-8665-48D4-826A-DFB4AC9A8118@oracle.com> Message-ID: <303f76f5163e6966613c2a37a47c489b1b083ab0.camel@oracle.com> Hi, On Sat, 2018-09-29 at 17:37 -0700, Man Cao wrote: > Hi, > > I updated the patch as Kim suggested: > http://cr.openjdk.java.net/~manc/8210716/webrev.01/ > > I updated the copyright years anyway. looks good. Thomas