From stefan.reich.maker.of.eye at googlemail.com Thu May 5 11:59:39 2022 From: stefan.reich.maker.of.eye at googlemail.com (Stefan Reich) Date: Thu, 5 May 2022 13:59:39 +0200 Subject: Fwd: Native memory report In-Reply-To: References:

Message-ID: ---------- Forwarded message --------- From: Stefan Reich Date: Thu, 5 May 2022 at 13:58 Subject: Re: Native memory report To: Jeremy Whiting Hi Thomas and Jeremy, thank you for your hints. The process is running smoothly right now (resident size <2GB) so I assume there is nothing to debug... I'll keep your suggestions filed if a problem arises again. JVM's name is cleared for now :) Thanks, Stefan On Thu, 5 May 2022 at 13:23, Jeremy Whiting wrote: > Hi Stefan, > Did you try native memory allocation diagnosis ? > This is a blog post showing how to use jemalloc and jeprof to identify > the cause in your JVM. > > > https://technology.blog.gov.uk/2015/12/11/using-jemalloc-to-get-to-the-bottom-of-a-memory-leak/ > > Regards, > Jeremy > > -- > -- > Jeremy Whiting > Senior Software Engineer, Middleware Performance Team > Red Hat > > ------------------------------------------------------------ > Registered Address: Red Hat UK Ltd, Peninsular House, 30 Monument Street, > London. United Kingdom. > Registered in England and Wales under Company Registration No. 03798903. > Directors: Directors:Michael Cunningham (US), Michael O'Neill(Ireland), > Eric Shander (US) > > On Thu, 21 Apr 2022 at 01:55, Stefan Reich < > stefan.reich.maker.of.eye at googlemail.com> wrote: > >> Hi, >> >> the problem occurred again (huge resident size despite small heap). This >> time I could gather this report. What to make of it? >> >> Greetings >> Stefan >> >> Native Memory Tracking: >> >> Total: reserved=13076774KB, committed=10792378KB >> - Java Heap (reserved=2097152KB, committed=1162240KB) >> (mmap: reserved=2097152KB, >> committed=1162240KB) >> >> - Class (reserved=1052755KB, committed=19667KB) >> (classes #23245) >> ( instance classes #22394, array classes >> #851) >> (malloc=4179KB #104371) >> (mmap: reserved=1048576KB, committed=15488KB) >> ( Metadata: ) >> ( reserved=180224KB, committed=177856KB) >> ( used=176408KB) >> ( free=1448KB) >> ( waste=0KB =0.00%) >> ( Class space:) >> ( reserved=1048576KB, committed=15488KB) >> ( used=14496KB) >> ( free=992KB) >> ( waste=0KB =0.00%) >> >> - Thread (reserved=194875KB, committed=18847KB) >> (thread #191) >> (stack: reserved=194340KB, committed=18312KB) >> (malloc=314KB #1142) >> (arena=221KB #378) >> >> - Code (reserved=260127KB, committed=156987KB) >> (malloc=12439KB #33280) >> (mmap: reserved=247688KB, committed=144548KB) >> >> - GC (reserved=9014115KB, committed=8979499KB) >> (malloc=8902991KB #14257646) >> (mmap: reserved=111124KB, committed=76508KB) >> >> - Compiler (reserved=1934KB, committed=1934KB) >> (malloc=1769KB #2309) >> (arena=165KB #5) >> >> - Internal (reserved=14536KB, committed=14536KB) >> (malloc=14500KB #72221) >> (mmap: reserved=36KB, committed=36KB) >> >> - Other (reserved=510KB, committed=510KB) >> (malloc=510KB #320) >> >> - Symbol (reserved=10841KB, committed=10841KB) >> (malloc=8244KB #307589) >> (arena=2597KB #1) >> >> - Native Memory Tracking (reserved=231067KB, committed=231067KB) >> (malloc=38KB #500) >> (tracking overhead=231028KB) >> >> - Shared class space (reserved=12288KB, committed=12044KB) >> (mmap: reserved=12288KB, committed=12044KB) >> >> - Arena Chunk (reserved=4714KB, committed=4714KB) >> (malloc=4714KB) >> >> - Logging (reserved=5KB, committed=5KB) >> (malloc=5KB #211) >> >> - Arguments (reserved=2KB, committed=2KB) >> (malloc=2KB #71) >> >> - Module (reserved=631KB, committed=631KB) >> (malloc=631KB #3213) >> >> - Safepoint (reserved=8KB, committed=8KB) >> (mmap: reserved=8KB, committed=8KB) >> >> - Synchronization (reserved=237KB, committed=237KB) >> (malloc=237KB #1697) >> >> - Metaspace (reserved=755KB, committed=755KB) >> (malloc=755KB #609) >> >> - Unknown (reserved=180224KB, committed=177856KB) >> (mmap: reserved=180224KB, committed=177856KB) >> >> >> -- >> == Gaz.AI == >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > > -- == Gaz.AI == -- == Gaz.AI == -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.reich.maker.of.eye at googlemail.com Fri May 6 19:13:41 2022 From: stefan.reich.maker.of.eye at googlemail.com (Stefan Reich) Date: Fri, 6 May 2022 21:13:41 +0200 Subject: Those memory footprint improvements in JDK 18 Message-ID: I'm right now just trying to get over how amazing this graph is: https://tschatzl.github.io/2022/03/14/jdk18-g1-parallel-gc-changes.html 35-40% savings in memory use just by using JDK 18??? Who would have expected such a major improvement after 17 iterations of the language! Just so I'm sure I'm reading it correctly... the graph basically shows the Java heap's memory footprint in terms of committed native memory. Right? So it would include Eden, tenured generations, humongous objects, but not metadata and code. Is that correct? Basically tell me how much I should celebrate this. lol I did switch another small, pretty crammed (8 GB RAM) server over to JDK 18 and it does feel like there's a lot more memory available on it now and everything is a lot faster too. Cheers Stefan -- == Gaz.AI == -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlie.hunt at oracle.com Fri May 6 21:28:13 2022 From: charlie.hunt at oracle.com (charlie hunt) Date: Fri, 6 May 2022 16:28:13 -0500 Subject: Those memory footprint improvements in JDK 18 In-Reply-To: References: Message-ID: <5b9a10ae-0304-52e1-1fac-c0a8270ae09a@oracle.com> Hi Stefan, The graph Thomas shows in his blog is the GC part of the NMT output which does not include metadata. The GC part of NMT output includes native memory allocated on behalf of the GC itself such as a card table or remembered set .. those things that GC needs to do its work. The native memory allocated for the Java heap are in the "Java Heap" section of the NMT output. Class metadata is in the "Class" section of the NMT output. There is a "Native Memory Tracking Memory Categories" table that lists the sections / categories reported by NMT and a description of each at: https://docs.oracle.com/en/java/javase/17/troubleshoot/diagnostic-tools.html#GUID-5EF7BB07-C903-4EBD-A9C2-EC0E44048D37 hths, Charlie On 5/6/22 2:13 PM, Stefan Reich wrote: > I'm right now just trying to get over how amazing this graph is: > https://tschatzl.github.io/2022/03/14/jdk18-g1-parallel-gc-changes.html > > 35-40% savings in memory use just by using JDK 18??? Who would have > expected such a major improvement after 17 iterations of the language! > > Just so I'm sure I'm reading it correctly... the graph basically shows > the Java heap's memory footprint in terms of committed native memory. > Right? > > So it would include Eden, tenured generations, humongous objects, but > not metadata and code. Is that?correct? > > Basically tell me how much I should celebrate this. lol > > I did switch another small, pretty crammed?(8 GB RAM) server over to > JDK 18 and it does feel like there's a lot more memory available on it > now and everything is a lot faster too. > > Cheers > Stefan > > -- > == Gaz.AI == > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.reich.maker.of.eye at googlemail.com Fri May 6 21:36:06 2022 From: stefan.reich.maker.of.eye at googlemail.com (Stefan Reich) Date: Fri, 6 May 2022 23:36:06 +0200 Subject: Those memory footprint improvements in JDK 18 In-Reply-To: <5b9a10ae-0304-52e1-1fac-c0a8270ae09a@oracle.com> References: <5b9a10ae-0304-52e1-1fac-c0a8270ae09a@oracle.com> Message-ID: Ah, so the memory sections that are now smaller are basically the GC's internal data structures, rather than the general heap? That kind of puts things in perspective. Still a great improvement. Has it been tested by how much the overall memory footprint of the JVM decreases in larger benchmarks? On Fri, 6 May 2022 at 23:28, charlie hunt wrote: > Hi Stefan, > > The graph Thomas shows in his blog is the GC part of the NMT output which > does not include metadata. > > The GC part of NMT output includes native memory allocated on behalf of > the GC itself such as a card table or remembered set .. those things that > GC needs to do its work. The native memory allocated for the Java heap are > in the "Java Heap" section of the NMT output. Class metadata is in the > "Class" section of the NMT output. > > There is a "Native Memory Tracking Memory Categories" table that lists the > sections / categories reported by NMT and a description of each at: > > https://docs.oracle.com/en/java/javase/17/troubleshoot/diagnostic-tools.html#GUID-5EF7BB07-C903-4EBD-A9C2-EC0E44048D37 > > hths, > > Charlie > On 5/6/22 2:13 PM, Stefan Reich wrote: > > I'm right now just trying to get over how amazing this graph is: > https://tschatzl.github.io/2022/03/14/jdk18-g1-parallel-gc-changes.html > > 35-40% savings in memory use just by using JDK 18??? Who would have > expected such a major improvement after 17 iterations of the language! > > Just so I'm sure I'm reading it correctly... the graph basically shows the > Java heap's memory footprint in terms of committed native memory. Right? > > So it would include Eden, tenured generations, humongous objects, but not > metadata and code. Is that correct? > > Basically tell me how much I should celebrate this. lol > > I did switch another small, pretty crammed (8 GB RAM) server over to JDK > 18 and it does feel like there's a lot more memory available on it now and > everything is a lot faster too. > > Cheers > Stefan > > -- > == Gaz.AI == > > _______________________________________________ > hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttps://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -- == Gaz.AI == -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlie.hunt at oracle.com Fri May 6 21:45:19 2022 From: charlie.hunt at oracle.com (charlie hunt) Date: Fri, 6 May 2022 16:45:19 -0500 Subject: [External] : Re: Those memory footprint improvements in JDK 18 In-Reply-To: References: <5b9a10ae-0304-52e1-1fac-c0a8270ae09a@oracle.com> Message-ID: <2381ceb5-2a12-ab8d-6595-898cbc46d151@oracle.com> Yes, GC's internal data structures. How much reduction there will likely vary depending on the application (and G1 region size). The type of application that will experience larger reductions are those applications that have a large number of cross G1 region references. A cache like type of application usually has a large number of these since updates to the cache introduce new references to an older or longer lived object which is likely held in a different G1 region. On 5/6/22 4:36 PM, Stefan Reich wrote: > Ah, so the memory sections that are now smaller are basically the GC's > internal data structures, rather than the general heap? > > That kind of puts things in perspective. Still a great improvement. > Has it been tested by how much the overall memory footprint of the JVM > decreases in larger benchmarks? > > On Fri, 6 May 2022 at 23:28, charlie hunt wrote: > > Hi Stefan, > > The graph Thomas shows in his blog is the GC part of the NMT > output which does not include metadata. > > The GC part of NMT output includes native memory allocated on > behalf of the GC itself such as a card table or remembered set .. > those things that GC needs to do its work. The native memory > allocated for the Java heap are in the "Java Heap" section of the > NMT output. Class metadata is in the "Class" section of the NMT > output. > > There is a "Native Memory Tracking Memory Categories" table that > lists the sections / categories reported by NMT and a description > of each at: > https://docs.oracle.com/en/java/javase/17/troubleshoot/diagnostic-tools.html#GUID-5EF7BB07-C903-4EBD-A9C2-EC0E44048D37 > > hths, > > Charlie > > On 5/6/22 2:13 PM, Stefan Reich wrote: >> I'm right now just trying to get over how amazing this graph is: >> https://tschatzl.github.io/2022/03/14/jdk18-g1-parallel-gc-changes.html >> >> >> >> 35-40% savings in memory use just by using JDK 18??? Who would >> have expected such a major improvement after 17 iterations of the >> language! >> >> Just so I'm sure I'm reading it correctly... the graph basically >> shows the Java heap's memory footprint in terms of committed >> native memory. Right? >> >> So it would include Eden, tenured generations, humongous objects, >> but not metadata and code. Is that?correct? >> >> Basically tell me how much I should celebrate this. lol >> >> I did switch another small, pretty crammed?(8 GB RAM) server over >> to JDK 18 and it does feel like there's a lot more memory >> available on it now and everything is a lot faster too. >> >> Cheers >> Stefan >> >> -- >> == Gaz.AI >> >> == >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > -- > == Gaz.AI > > == -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Mon May 9 08:28:10 2022 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 9 May 2022 10:28:10 +0200 Subject: [External] : Re: Those memory footprint improvements in JDK 18 In-Reply-To: <2381ceb5-2a12-ab8d-6595-898cbc46d151@oracle.com> References: <5b9a10ae-0304-52e1-1fac-c0a8270ae09a@oracle.com> <2381ceb5-2a12-ab8d-6595-898cbc46d151@oracle.com> Message-ID: Hi, On 06.05.22 23:45, charlie hunt wrote: > Yes, GC's internal data structures. as Charlie said. > How much reduction there will likely vary depending on the application > (and G1 region size). The type of application that will experience > larger reductions are those applications that have a large number of > cross G1 region references. A cache like type of application usually has > a large number of these since updates to the cache introduce new > references to an older or longer lived object which is likely held in a > different G1 region. > > On 5/6/22 4:36 PM, Stefan Reich wrote: >> Ah, so the memory sections that are now smaller are basically the GC's >> internal data structures, rather than the general heap? Yes. Significantly so. There has been a presentation for Oracle Developer Live in March that shows the progress in that area for such a cache-like application at https://inside.java/2022/05/02/odl-jdk8-to-jdk18-gc/ ; around 17:49 it talks about memory footprint reductions for the G1 collector over time. Just to make it clear: next to the application, this is highly dependent on the garbage collector; e.g. Parallel GC needs memory roughly constant equal to the "Floor" line on that blog, Serial GC roughly half of that. With all their other tradeoffs of course wrt to latency/throughput. As mentioned in that blog post at the very end, we are working on something that might make G1 GC data structure overhead comparable to Serial GC plus remembered sets (you can see the new "floor" for that change by looking at the "Prototype (calculated)" line, the level in the first ~50s). That change basically removes a constant amount that is exactly 1.5% of Java heap memory size from gc data structure overhead. Maybe that change makes JDK 19, as usual no guarantees. >> That kind of puts things in perspective. Still a great improvement. >> Has it been tested by how much the overall memory footprint of the JVM >> decreases in larger benchmarks? In JDK8 the rule-of-thumb was like that you probably need to consider around 20% of Java heap, JDK 11 around 10% and with JDK 18 (probably) around 5% for G1 GC data structures to be fairly safe for all but the largest outliers with default ergonomics (i.e. that application we use for demonstration purposes is on the upper end). You *can* tune GC remembered set memory footprint quite a bit in earlier releases, but there are also other changes in later JDKs that can't be reproduced by some options. That blog contains a few posts about these. Many (typically throughput-oriented) applications need (much) less remembered set data structure memory though. >> >> On Fri, 6 May 2022 at 23:28, charlie hunt wrote: >> >> Hi Stefan, >> >> The graph Thomas shows in his blog is the GC part of the NMT >> output which does not include metadata. >> >> The GC part of NMT output includes native memory allocated on >> behalf of the GC itself such as a card table or remembered set .. >> those things that GC needs to do its work. The native memory >> allocated for the Java heap are in the "Java Heap" section of the >> NMT output. Class metadata is in the "Class" section of the NMT >> output. >> >> There is a "Native Memory Tracking Memory Categories" table that >> lists the sections / categories reported by NMT and a description >> of each at: >> https://docs.oracle.com/en/java/javase/17/troubleshoot/diagnostic-tools.html#GUID-5EF7BB07-C903-4EBD-A9C2-EC0E44048D37 That documentation isn't current for JDK 18 - there is a new category for GC called "GCCardSet" that measures the memory overhead of the so-called (G1) remembered set only as it is a significant part of it. TL;DR: the previous "GC" category has been split into "GC" and "GCCardSet" and need to be added together to be comparable with "GC" of earlier versions. I'll try to get the documentation updated for JDK 19 at least, possibly JDK 18. >> >> hths, >> >> Charlie >> >> On 5/6/22 2:13 PM, Stefan Reich wrote: >>> I'm right now just trying to get over how amazing this graph is: >>> https://tschatzl.github.io/2022/03/14/jdk18-g1-parallel-gc-changes.html >>> >>> >>> >>> 35-40% savings in memory use just by using JDK 18??? Who would >>> have expected such a major improvement after 17 iterations of the >>> language! >>> >>> Just so I'm sure I'm reading it correctly... the graph basically >>> shows the Java heap's memory footprint in terms of committed >>> native memory. Right? >>> >>> So it would include Eden, tenured generations, humongous objects, >>> but not metadata and code. Is that?correct? >>> >>> Basically tell me how much I should celebrate this. lol >>> >>> I did switch another small, pretty crammed?(8 GB RAM) server over >>> to JDK 18 and it does feel like there's a lot more memory >>> available on it now and everything is a lot faster too. >>> >>> Cheers >>> Stefan >>> Hth, Thomas From stefan.reich.maker.of.eye at googlemail.com Tue May 10 20:09:23 2022 From: stefan.reich.maker.of.eye at googlemail.com (Stefan Reich) Date: Tue, 10 May 2022 22:09:23 +0200 Subject: [External] : Re: Those memory footprint improvements in JDK 18 In-Reply-To: References: <5b9a10ae-0304-52e1-1fac-c0a8270ae09a@oracle.com> <2381ceb5-2a12-ab8d-6595-898cbc46d151@oracle.com> Message-ID: Yes, helps a lot, thanks On Mon, 9 May 2022 at 10:28, Thomas Schatzl wrote: > Hi, > > On 06.05.22 23:45, charlie hunt wrote: > > Yes, GC's internal data structures. > > as Charlie said. > > > How much reduction there will likely vary depending on the application > > (and G1 region size). The type of application that will experience > > larger reductions are those applications that have a large number of > > cross G1 region references. A cache like type of application usually has > > a large number of these since updates to the cache introduce new > > references to an older or longer lived object which is likely held in a > > different G1 region. > > > > On 5/6/22 4:36 PM, Stefan Reich wrote: > >> Ah, so the memory sections that are now smaller are basically the GC's > >> internal data structures, rather than the general heap? > > Yes. Significantly so. There has been a presentation for Oracle > Developer Live in March that shows the progress in that area for such a > cache-like application at > https://inside.java/2022/05/02/odl-jdk8-to-jdk18-gc/ ; around 17:49 it > talks about memory footprint reductions for the G1 collector over time. > > Just to make it clear: next to the application, this is highly dependent > on the garbage collector; e.g. Parallel GC needs memory roughly constant > equal to the "Floor" line on that blog, Serial GC roughly half of that. > With all their other tradeoffs of course wrt to latency/throughput. > > As mentioned in that blog post at the very end, we are working on > something that might make G1 GC data structure overhead comparable to > Serial GC plus remembered sets (you can see the new "floor" for that > change by looking at the "Prototype (calculated)" line, the level in the > first ~50s). That change basically removes a constant amount that is > exactly 1.5% of Java heap memory size from gc data structure overhead. > > Maybe that change makes JDK 19, as usual no guarantees. > > >> That kind of puts things in perspective. Still a great improvement. > >> Has it been tested by how much the overall memory footprint of the JVM > >> decreases in larger benchmarks? > > In JDK8 the rule-of-thumb was like that you probably need to consider > around 20% of Java heap, JDK 11 around 10% and with JDK 18 (probably) > around 5% for G1 GC data structures to be fairly safe for all but the > largest outliers with default ergonomics (i.e. that application we use > for demonstration purposes is on the upper end). > You *can* tune GC remembered set memory footprint quite a bit in earlier > releases, but there are also other changes in later JDKs that can't be > reproduced by some options. That blog contains a few posts about these. > > Many (typically throughput-oriented) applications need (much) less > remembered set data structure memory though. > > >> > >> On Fri, 6 May 2022 at 23:28, charlie hunt > wrote: > >> > >> Hi Stefan, > >> > >> The graph Thomas shows in his blog is the GC part of the NMT > >> output which does not include metadata. > >> > >> The GC part of NMT output includes native memory allocated on > >> behalf of the GC itself such as a card table or remembered set .. > >> those things that GC needs to do its work. The native memory > >> allocated for the Java heap are in the "Java Heap" section of the > >> NMT output. Class metadata is in the "Class" section of the NMT > >> output. > >> > >> There is a "Native Memory Tracking Memory Categories" table that > >> lists the sections / categories reported by NMT and a description > >> of each at: > >> > https://docs.oracle.com/en/java/javase/17/troubleshoot/diagnostic-tools.html#GUID-5EF7BB07-C903-4EBD-A9C2-EC0E44048D37 > > That documentation isn't current for JDK 18 - there is a new category > for GC called "GCCardSet" that measures the memory overhead of the > so-called (G1) remembered set only as it is a significant part of it. > > TL;DR: the previous "GC" category has been split into "GC" and > "GCCardSet" and need to be added together to be comparable with "GC" of > earlier versions. > > I'll try to get the documentation updated for JDK 19 at least, possibly > JDK 18. > > >> > >> hths, > >> > >> Charlie > >> > >> On 5/6/22 2:13 PM, Stefan Reich wrote: > >>> I'm right now just trying to get over how amazing this graph is: > >>> > https://tschatzl.github.io/2022/03/14/jdk18-g1-parallel-gc-changes.html > >>> < > https://urldefense.com/v3/__https://tschatzl.github.io/2022/03/14/jdk18-g1-parallel-gc-changes.html__;!!ACWV5N9M2RV99hQ!OKRokskrYA5lROK6mWSdvn5puKnqs34LsTUVTkWX2Rp65OYC5CRMO5pthP_bsdg-msr3P2Q6ag1T55XNqMseOt88jWtWJhKP$ > > > >>> > >>> > >>> 35-40% savings in memory use just by using JDK 18??? Who would > >>> have expected such a major improvement after 17 iterations of the > >>> language! > >>> > >>> Just so I'm sure I'm reading it correctly... the graph basically > >>> shows the Java heap's memory footprint in terms of committed > >>> native memory. Right? > >>> > >>> So it would include Eden, tenured generations, humongous objects, > >>> but not metadata and code. Is that correct? > >>> > >>> Basically tell me how much I should celebrate this. lol > >>> > >>> I did switch another small, pretty crammed (8 GB RAM) server over > >>> to JDK 18 and it does feel like there's a lot more memory > >>> available on it now and everything is a lot faster too. > >>> > >>> Cheers > >>> Stefan > >>> > > Hth, > Thomas > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -- == Gaz.AI == -------------- next part -------------- An HTML attachment was scrubbed... URL: