From stefan.reich.maker.of.eye at googlemail.com Mon Apr 4 15:49:34 2022
From: stefan.reich.maker.of.eye at googlemail.com (Stefan Reich)
Date: Mon, 4 Apr 2022 17:49:34 +0200
Subject: Huge resident size despite small heap
In-Reply-To: <CAC2-jLFTJs=Ems-JMzMGvfCa2O6Hxa1a2Hwz1kN2MrMiiSJg6w@mail.gmail.com>
References: <CAC2-jLFUP3ukea13dF9BLq4=K=gZRZxDfiyT6Zc5-OO5J_yekg@mail.gmail.com>
<9b70f461-82af-d746-7cc9-583ed1522426@elyograg.org>
<CAC2-jLHvU9J9rxeB+aYuNpemnLrDOhRYm_qDRTy2S=nNWU-NZg@mail.gmail.com>
<8aab9a41-bf56-0137-9a84-e4c8ce20a9b8@elyograg.org>
<CAC2-jLEAgC5PJTCgRU=x8170Yu0K+MKrVkfFPkpsA8bbrC+kqw@mail.gmail.com>
<CAC2-jLFTJs=Ems-JMzMGvfCa2O6Hxa1a2Hwz1kN2MrMiiSJg6w@mail.gmail.com>
Message-ID: <CAC2-jLFe2Che=TABpOH0FVd=SurJ56SH5DkgHMLkTwpbYcU1ew@mail.gmail.com>
Thanks everyone for your help. It seems the process is running stable now
using JDK 18. (Resident size 1.6 GB with 400 MB live Java objects.)
Bit of an anticlimactic ending to the mystery (was there a bug that got
fixed since JDK 16?), but as things are fine, I won't complain.
On Wed, 30 Mar 2022 at 18:59, Stefan Reich <
stefan.reich.maker.of.eye at googlemail.com> wrote:
> Process got to 9GB, yet various diagnostics still unremarkable:
>
> MemoryMXBean.getHeapMemoryUsage() showing 1.5 GB committed
> MemoryMXBean.getNonHeapMemoryUsage() showing 350 MB committed
> Memory pools also looking healthy: https://botcompany.de/images/1103117
>
> (Big Eden space though. Is that normal?)
>
> I did find the courage to upgrade to JDK 18. So far the ship is afloat.
> Have enabled NMT too. Patient will be monitored.
>
> Thanks
>
>
> On Mon, 28 Mar 2022 at 18:23, Stefan Reich <
> stefan.reich.maker.of.eye at googlemail.com> wrote:
>
>> Yeah I gotcha... here it is:
>>
>> root at c8:~# ps -aef|grep 6344
>> root 6344 1 24 Mar22 ? 1-10:46:31 java
>> --illegal-access=permit -Xmx2g --add-opens
>> java.base/jdk.internal.loader=ALL-UNNAMED --add-opens
>> java.base/jdk.internal.module=ALL-UNNAMED --add-opens
>> java.base/jdk.internal.ref=ALL-UNNAMED --add-opens
>> java.base/java.lang.module=ALL-UNNAMED --add-opens
>> java.base/jdk.internal.reflect=ALL-UNNAMED -jar /root/.javax/x30.jar 1013896
>>
>> On Mon, 28 Mar 2022 at 17:52, Shawn Heisey <java at elyograg.org> wrote:
>>
>>> On 3/28/22 09:31, Stefan Reich wrote:
>>> > Hi Shawn - interesting, but that's not it...
>>> > https://botcompany.de/images/1103113
>>>
>>> That makes me suspect that Java is not being given the memory options
>>> you think it is. Take a look at the output from this command, and share
>>> it if you can, substituting the correct PID number if that has changed:
>>>
>>> ps axww | grep 6344
>>>
>>> If the ps on that system doesn't like those options, replace "axww" with
>>> "-aef".
>>>
>>> Thanks,
>>> Shawn
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>
>>
>>
>> --
>> == Gaz.AI ==
>>
>
>
> --
> == Gaz.AI ==
>
--
== Gaz.AI ==
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20220404/4956b687/attachment.htm>
From stefan.reich.maker.of.eye at googlemail.com Thu Apr 21 00:53:15 2022
From: stefan.reich.maker.of.eye at googlemail.com (Stefan Reich)
Date: Thu, 21 Apr 2022 02:53:15 +0200
Subject: Native memory report
Message-ID: <CAC2-jLGnnzh7T-hQmt_otX99o5KTcVudzVPL==Srj_zZ5bApMQ@mail.gmail.com>
Hi,
the problem occurred again (huge resident size despite small heap). This
time I could gather this report. What to make of it?
Greetings
Stefan
Native Memory Tracking:
Total: reserved=13076774KB, committed=10792378KB
- Java Heap (reserved=2097152KB, committed=1162240KB)
(mmap: reserved=2097152KB, committed=1162240KB)
- Class (reserved=1052755KB, committed=19667KB)
(classes #23245)
( instance classes #22394, array classes #851)
(malloc=4179KB #104371)
(mmap: reserved=1048576KB, committed=15488KB)
( Metadata: )
( reserved=180224KB, committed=177856KB)
( used=176408KB)
( free=1448KB)
( waste=0KB =0.00%)
( Class space:)
( reserved=1048576KB, committed=15488KB)
( used=14496KB)
( free=992KB)
( waste=0KB =0.00%)
- Thread (reserved=194875KB, committed=18847KB)
(thread #191)
(stack: reserved=194340KB, committed=18312KB)
(malloc=314KB #1142)
(arena=221KB #378)
- Code (reserved=260127KB, committed=156987KB)
(malloc=12439KB #33280)
(mmap: reserved=247688KB, committed=144548KB)
- GC (reserved=9014115KB, committed=8979499KB)
(malloc=8902991KB #14257646)
(mmap: reserved=111124KB, committed=76508KB)
- Compiler (reserved=1934KB, committed=1934KB)
(malloc=1769KB #2309)
(arena=165KB #5)
- Internal (reserved=14536KB, committed=14536KB)
(malloc=14500KB #72221)
(mmap: reserved=36KB, committed=36KB)
- Other (reserved=510KB, committed=510KB)
(malloc=510KB #320)
- Symbol (reserved=10841KB, committed=10841KB)
(malloc=8244KB #307589)
(arena=2597KB #1)
- Native Memory Tracking (reserved=231067KB, committed=231067KB)
(malloc=38KB #500)
(tracking overhead=231028KB)
- Shared class space (reserved=12288KB, committed=12044KB)
(mmap: reserved=12288KB, committed=12044KB)
- Arena Chunk (reserved=4714KB, committed=4714KB)
(malloc=4714KB)
- Logging (reserved=5KB, committed=5KB)
(malloc=5KB #211)
- Arguments (reserved=2KB, committed=2KB)
(malloc=2KB #71)
- Module (reserved=631KB, committed=631KB)
(malloc=631KB #3213)
- Safepoint (reserved=8KB, committed=8KB)
(mmap: reserved=8KB, committed=8KB)
- Synchronization (reserved=237KB, committed=237KB)
(malloc=237KB #1697)
- Metaspace (reserved=755KB, committed=755KB)
(malloc=755KB #609)
- Unknown (reserved=180224KB, committed=177856KB)
(mmap: reserved=180224KB, committed=177856KB)
--
== Gaz.AI ==
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20220421/ec39a8d6/attachment.htm>
From txiadev at gmail.com Thu Apr 21 02:05:06 2022
From: txiadev at gmail.com (Tianqi Xia)
Date: Thu, 21 Apr 2022 10:05:06 +0800
Subject: Lightweight G1 remembered set logging
Message-ID: <CAMK6dgU4U9U8_j=VZkShNrVXTSBqEEiD_vMsZ8jQn8WofMPa_w@mail.gmail.com>
Hi,
It seems like currently the only way to log rememebered set usage during GC
is setting the G1SummarizeRSetStatsPeriod flag, which can be quite
expensive. Testing on my local box, running BigRamTester (by Thomas Schatzl)
with JDK11 + 10GB heap + 1GB RSet setup, it took hundreds of milliseconds
to process and print those remembered set usage information, and this
happened when the world is stopped.
After a little bit hacking and digging I found most of the time is spent on
the following 2 operations:
1. Calculating the size of PerRegionTable freelist, with 1GB RSet usage,
this freelist can contain more than half a million entries.
2. Calculating the occupancy of OtherRegionTable, which iterates over all
PerRegionTables being used.
After omitting these 2 operations, we can cut down the overall logging time
of remembered set usage to ~1ms, under the same setup.
Can we introduce a "lightweight" mode of remembered set logging that trims
the above mentioned 2 operations? It makes it possible for turning on
remebered set logging in prodcution envrionment, which really helps
diagnose certain performance issues.
Regards,
Tianqi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20220421/867448a3/attachment.htm>
From thomas.schatzl at oracle.com Thu Apr 21 10:20:04 2022
From: thomas.schatzl at oracle.com (Thomas Schatzl)
Date: Thu, 21 Apr 2022 12:20:04 +0200
Subject: Lightweight G1 remembered set logging
In-Reply-To: <CAMK6dgU4U9U8_j=VZkShNrVXTSBqEEiD_vMsZ8jQn8WofMPa_w@mail.gmail.com>
References: <CAMK6dgU4U9U8_j=VZkShNrVXTSBqEEiD_vMsZ8jQn8WofMPa_w@mail.gmail.com>
Message-ID: <2208a52e-7627-9284-48df-7b70e1d33eb0@oracle.com>
Hi,
On 21.04.22 04:05, Tianqi Xia wrote:
> Hi,
>
> It seems like currently the only way to log rememebered set usage during
> GC is setting the G1SummarizeRSetStatsPeriod flag, which can be quite
> expensive. Testing on my local box, running BigRamTester (byThomas
> Schatzl) with JDK11 + 10GB heap + 1GB RSet setup, it took hundreds of
> milliseconds to process and print those remembered set usage
> information, and this happened when the world is stopped.
>
> After a little bit hacking and digging I found most of the time is spent
> on the following 2 operations:
> 1. Calculating the size of PerRegionTable freelist, with 1GB RSet usage,
> this freelist can contain more than half a million entries.
> 2. Calculating the occupancy of OtherRegionTable, which iterates over
> all PerRegionTables being used.
> After omitting these 2 operations, we can cut down the overall logging
> time of remembered set usage to ~1ms, under the same setup.
>
> Can we introduce a "lightweight" mode of remembered set logging that
> trims the above mentioned 2 operations? It makes it possible for turning
> on remebered set logging in prodcution envrionment, which really helps
> diagnose certain performance issues.
a fix for this particular problem has been implemented in JDK-8233919
for JDK 14. Since then that accounting should be very fast (one add per
region, no more iterating through the actual remembered sets), and so
the logging much lighter weight, approximately in the range you state.
Since typically problems with the remembered sets means problems with
remembered set footprint, one option that often helps is to do what
JDK-8223162 (in JDK 13) did, increasing the so-called the sparse entry
sizes via -XX:G1RSetSparseRegionEntries, manually. See the release note
at https://bugs.openjdk.java.net/browse/JDK-8225343 for a comparison of
the old and new default values.
Thanks,
Thomas
From thomas.schatzl at oracle.com Thu Apr 21 11:07:29 2022
From: thomas.schatzl at oracle.com (Thomas Schatzl)
Date: Thu, 21 Apr 2022 13:07:29 +0200
Subject: Native memory report
In-Reply-To: <CAC2-jLGnnzh7T-hQmt_otX99o5KTcVudzVPL==Srj_zZ5bApMQ@mail.gmail.com>
References: <CAC2-jLGnnzh7T-hQmt_otX99o5KTcVudzVPL==Srj_zZ5bApMQ@mail.gmail.com>
Message-ID: <83e9f260-a4da-6c08-4893-42ca6f22fe50@oracle.com>
Hi,
On 21.04.22 02:53, Stefan Reich wrote:
> Hi,
>
> the problem occurred again (huge resident size despite small heap). This
> time I could gather this report. What to make of it?
>
The GC component takes a lot of memory that is not given back. I can
only guess these are the remembered sets.
> [...]
> - ? ? ? ? ? ? ? ? ? ? ? ?GC (reserved=9014115KB, committed=8979499KB)
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? (malloc=8902991KB #14257646)
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? (mmap: reserved=111124KB, committed=76508KB)
> [...]
In the recent email you mentioned that you moved to JDK 18 and "things
are afloat"; if that were the case, and you did not use a different
collector that the default one (G1) there should be a "GCCardSet"
component. See e.g.
https://tschatzl.github.io/2022/03/14/jdk18-g1-parallel-gc-changes.html
for an example.
Are you sure you are running JDK 18? (Or are you running a different
collector than G1 - I maybe overlooked a reference to your GC
configuration in the recent emails? - it is generally a good idea to
restate your setup after coming back with a new question in a new thread
after a few weeks)
Can you track memory consumption via the "summary.diff" jcmd option over
time to see if we are leaking memory somewhere? We are not aware of such
an issue.
Thanks,
Thomas