From lfengnan at uber.com  Fri Feb  8 21:49:39 2019
From: lfengnan at uber.com (Fengnan Li)
Date: Fri, 8 Feb 2019 13:49:39 -0800
Subject: HDFS Namenode with large heap size
Message-ID: <9BC4A85C-E020-464C-85D2-63C6E15B078F@uber.com>

Hi All,

We are trying to use G1 for our HDFS Namenode to see whether it will deliver better GC overall than currently used CMS. However, with the 200G heap size JVM option, the G1 wouldn?t even start our namenode with the production image and will be killed out of memory after running for 1 hours (loading initial data). For the same heap size, CMS can work properly with around 98% throughput and averagely 120ms pause.

We use pretty much the basic options, and tried to tune a little but not much progress. Is there a way to lower down the overall memory footprint for G1?

We managed to start the application with 300G heap size option, but overall G1 will consume about 450G memory, which is problematic.

Thanks,
Fengnan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2710 bytes
Desc: not available
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190208/0b86182e/smime.p7s>

From lfengnan at uber.com  Sat Feb  9 00:37:27 2019
From: lfengnan at uber.com (Fengnan Li)
Date: Fri, 8 Feb 2019 16:37:27 -0800
Subject: HDFS Namenode with large heap size
In-Reply-To: <CA+cQ+tQOObiQ60nrN1qGSEnYqBGefJDc1Ek+9S1Svwhaie3=NQ@mail.gmail.com>
References: <9BC4A85C-E020-464C-85D2-63C6E15B078F@uber.com>
 <CA+cQ+tQOObiQ60nrN1qGSEnYqBGefJDc1Ek+9S1Svwhaie3=NQ@mail.gmail.com>
Message-ID: <28995253-7606-4750-8A89-DE65FB8CBED7@uber.com>

Hi Kris,

Thanks very much for the response!

I didn?t expect the RSS of the Java process to be 300GB when I set the heap size to 300GB since from a lot of other use cases I know that G1 takes more memory overhead compared to CMS. 
I got the 450G number from our internal metrics, which essentially reads the /proc/meminfo file for the memory footprint. On the machine there is no other process taking a lot of memory (more than 1% of total memory or single digit GB).

I turned on the NMT option, and print out the JVM memory stack:

Native Memory Tracking:

Total: reserved=477GB, committed=476GB
-                 Java Heap (reserved=300GB, committed=300GB)
                            (mmap: reserved=300GB, committed=300GB)

-                    Thread (reserved=1GB, committed=1GB)
                            (thread #723)
                            (stack: reserved=1GB, committed=1GB)

-                        GC (reserved=23GB, committed=23GB)
                            (malloc=12GB #20497380)
                            (mmap: reserved=11GB, committed=11GB)

-                  Internal (reserved=152GB, committed=152GB)
                            (malloc=152GB #19364496)

-    Native Memory Tracking (reserved=1GB, committed=1GB)
                            (tracking overhead=1GB)

-                   Unknown (reserved=1GB, committed=0GB)
                            (mmap: reserved=1GB, committed=0GB)

Interal (direct byte buffer) takes a lot of the space. GC overhead looks OK in this case.

This is pretty weird, I run the same app with CMS on the machine, but there is no Internal part from the stack. Do you know why this happens?

Thanks,
Fengnan

> On Feb 8, 2019, at 2:07 PM, Krystal Mok <rednaxelafx at gmail.com> wrote:
> 
> Hi Fengnan,
> 
> This is Kris Mok currently working at Databricks. I used to work on HotSpot and Zing JVMs.
> Just curious how you got to the conclusion of G1 taking 450GB memory. Did you start the VM with -Xmx300g expecting that the RSS of that Java process be close to 300GB? If that's the case, that's a unreasonable expectation to begin with.
> 
> It's a very common case for G1 itself to take a high memory overhead due to its design of the Remembered Set (RSet), and that can be tuned to use less memory by making it more coarse-grained, with the tradeoff that root scanning pause can take longer because more portion of the heap is going to have to be scanned.
> 
> But I doubt that what you're actually seeing. To confirm where the memory went within the HotSpot JVM, please turn on NMT (Native Memory Tracking) and see how much memory each component within the JVM is using.
> 
> https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html <https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html>
> https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html <https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html>
> 
> - Kris
> 
> On Fri, Feb 8, 2019 at 1:51 PM Fengnan Li <lfengnan at uber.com <mailto:lfengnan at uber.com>> wrote:
> Hi All,
> 
> We are trying to use G1 for our HDFS Namenode to see whether it will deliver better GC overall than currently used CMS. However, with the 200G heap size JVM option, the G1 wouldn?t even start our namenode with the production image and will be killed out of memory after running for 1 hours (loading initial data). For the same heap size, CMS can work properly with around 98% throughput and averagely 120ms pause.
> 
> We use pretty much the basic options, and tried to tune a little but not much progress. Is there a way to lower down the overall memory footprint for G1?
> 
> We managed to start the application with 300G heap size option, but overall G1 will consume about 450G memory, which is problematic.
> 
> Thanks,
> Fengnan_______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net <mailto:hotspot-gc-use at openjdk.java.net>
> https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use <https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190208/a571be34/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2710 bytes
Desc: not available
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190208/a571be34/smime.p7s>

From rednaxelafx at gmail.com  Sat Feb  9 08:17:23 2019
From: rednaxelafx at gmail.com (Krystal Mok)
Date: Sat, 9 Feb 2019 00:17:23 -0800
Subject: HDFS Namenode with large heap size
In-Reply-To: <28995253-7606-4750-8A89-DE65FB8CBED7@uber.com>
References: <9BC4A85C-E020-464C-85D2-63C6E15B078F@uber.com>
 <CA+cQ+tQOObiQ60nrN1qGSEnYqBGefJDc1Ek+9S1Svwhaie3=NQ@mail.gmail.com>
 <28995253-7606-4750-8A89-DE65FB8CBED7@uber.com>
Message-ID: <CA+cQ+tSM929q6Ve9xdzZ30P9uQeCjbZy-2v5tZcP++Pn2bV7eQ@mail.gmail.com>

Interesting. Was G1 working so well for your application that it hardly
ever performed Full GCs? Mostly just Young GCs and Mixed GCs? You probably
also turned on "-XX:+ExplicitGCInvokesConcurrent", right?

The "Internal" component can be from a lot of things, but the most
significant source would be DirectByteBuffers as you pointed out.

The reason why I asked about (the lack of) full GCs is because I was
curious whether or not you're seeing a lot of garbage DBBs in the Java heap
that haven't been processed by G1 in the mixed GCs, potentially because
they're in regions that didn't make it into the CSet of mixed GCs.

If you have a running G1 version of that Java application (is it just
NameNode?), could you please try to get it into the state when it consumes
450GB+ memory, and then do two consecutive "jmap -histo:live" on that
process to force two full GCs and see what happens? Yes it'll pause for a
while but if your environment is okay with experiments like this, it'd be
helpful.

- Kris

On Fri, Feb 8, 2019 at 4:37 PM Fengnan Li <lfengnan at uber.com> wrote:

> Hi Kris,
>
> Thanks very much for the response!
>
> I didn?t expect the RSS of the Java process to be 300GB when I set the
> heap size to 300GB since from a lot of other use cases I know that G1 takes
> more memory overhead compared to CMS.
> I got the 450G number from our internal metrics, which essentially reads
> the /proc/meminfo file for the memory footprint. On the machine there is no
> other process taking a lot of memory (more than 1% of total memory or
> single digit GB).
>
> I turned on the NMT option, and print out the JVM memory stack:
>
> Native Memory Tracking:
>
> Total: reserved=477GB, committed=476GB
> -                 Java Heap (reserved=300GB, committed=300GB)
>                             (mmap: reserved=300GB, committed=300GB)
>
> -                    Thread (reserved=1GB, committed=1GB)
>                             (thread #723)
>                             (stack: reserved=1GB, committed=1GB)
>
> -                        GC (reserved=23GB, committed=23GB)
>                             (malloc=12GB #20497380)
>                             (mmap: reserved=11GB, committed=11GB)
>
> -                  Internal (reserved=152GB, committed=152GB)
>                             (malloc=152GB #19364496)
>
> -    Native Memory Tracking (reserved=1GB, committed=1GB)
>                             (tracking overhead=1GB)
>
> -                   Unknown (reserved=1GB, committed=0GB)
>                             (mmap: reserved=1GB, committed=0GB)
>
> Interal (direct byte buffer) takes a lot of the space. GC overhead looks
> OK in this case.
>
> This is pretty weird, I run the same app with CMS on the machine, but
> there is no Internal part from the stack. Do you know why this happens?
>
> Thanks,
> Fengnan
>
> On Feb 8, 2019, at 2:07 PM, Krystal Mok <rednaxelafx at gmail.com> wrote:
>
> Hi Fengnan,
>
> This is Kris Mok currently working at Databricks. I used to work on
> HotSpot and Zing JVMs.
> Just curious how you got to the conclusion of G1 taking 450GB memory. Did
> you start the VM with -Xmx300g expecting that the RSS of that Java process
> be close to 300GB? If that's the case, that's a unreasonable expectation to
> begin with.
>
> It's a very common case for G1 itself to take a high memory overhead due
> to its design of the Remembered Set (RSet), and that can be tuned to use
> less memory by making it more coarse-grained, with the tradeoff that root
> scanning pause can take longer because more portion of the heap is going to
> have to be scanned.
>
> But I doubt that what you're actually seeing. To confirm where the memory
> went within the HotSpot JVM, please turn on NMT (Native Memory Tracking)
> and see how much memory each component within the JVM is using.
>
>
> https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html
> https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html
>
> - Kris
>
> On Fri, Feb 8, 2019 at 1:51 PM Fengnan Li <lfengnan at uber.com> wrote:
>
>> Hi All,
>>
>> We are trying to use G1 for our HDFS Namenode to see whether it will
>> deliver better GC overall than currently used CMS. However, with the 200G
>> heap size JVM option, the G1 wouldn?t even start our namenode with the
>> production image and will be killed out of memory after running for 1 hours
>> (loading initial data). For the same heap size, CMS can work properly with
>> around 98% throughput and averagely 120ms pause.
>>
>> We use pretty much the basic options, and tried to tune a little but not
>> much progress. Is there a way to lower down the overall memory footprint
>> for G1?
>>
>> We managed to start the application with 300G heap size option, but
>> overall G1 will consume about 450G memory, which is problematic.
>>
>> Thanks,
>> Fengnan_______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190209/1e1d55de/attachment.html>

From lfengnan at uber.com  Sat Feb  9 09:02:33 2019
From: lfengnan at uber.com (Fengnan Li)
Date: Sat, 9 Feb 2019 01:02:33 -0800
Subject: HDFS Namenode with large heap size
In-Reply-To: <CA+cQ+tSM929q6Ve9xdzZ30P9uQeCjbZy-2v5tZcP++Pn2bV7eQ@mail.gmail.com>
References: <9BC4A85C-E020-464C-85D2-63C6E15B078F@uber.com>
 <CA+cQ+tQOObiQ60nrN1qGSEnYqBGefJDc1Ek+9S1Svwhaie3=NQ@mail.gmail.com>
 <28995253-7606-4750-8A89-DE65FB8CBED7@uber.com>
 <CA+cQ+tSM929q6Ve9xdzZ30P9uQeCjbZy-2v5tZcP++Pn2bV7eQ@mail.gmail.com>
Message-ID: <183114BA-CCE5-456F-95A4-C2F3A02F5035@uber.com>

I didn?t use `-XX:+ExplicitGCInvokesConcurrent`. If by default it is not turned on then it is not used in our GC.
Indeed I seldomly see full GC, but I wouldn?t call it running well due to the large memory overhead. (Not sure whether it is tunable at this point)

I tried two `jmap -histo:live` and triggered to full GC, some of the gc logs are:

[Full GC (Heap Inspection Initiated GC)  151G->143G(300G), 756.5877774 secs]
   [Eden: 8800.0M(119.7G)->0.0B(119.7G) Survivors: 0.0B->0.0B Heap: 151.7G(300.0G)->143.1G(300.0G)], [Metaspace: 44549K->44549K(45056K)]
 [Times: user=2895.66 sys=2.13, real=756.58 secs]

During the period I didn?t see the overall memory consumption changed? And it is pretty horrible that one GC takes almost 13 minutes, which is not ok in prod.

I tried to search a lot about analyzing JVM non heap memory but couldn?t find any?

Let me know how I can figure out the Internal size.

Thanks,
Fengnan

> On Feb 9, 2019, at 12:17 AM, Krystal Mok <rednaxelafx at gmail.com> wrote:
> 
> Interesting. Was G1 working so well for your application that it hardly ever performed Full GCs? Mostly just Young GCs and Mixed GCs? You probably also turned on "-XX:+ExplicitGCInvokesConcurrent", right?
> 
> The "Internal" component can be from a lot of things, but the most significant source would be DirectByteBuffers as you pointed out.
> 
> The reason why I asked about (the lack of) full GCs is because I was curious whether or not you're seeing a lot of garbage DBBs in the Java heap that haven't been processed by G1 in the mixed GCs, potentially because they're in regions that didn't make it into the CSet of mixed GCs.
> 
> If you have a running G1 version of that Java application (is it just NameNode?), could you please try to get it into the state when it consumes 450GB+ memory, and then do two consecutive "jmap -histo:live" on that process to force two full GCs and see what happens? Yes it'll pause for a while but if your environment is okay with experiments like this, it'd be helpful.
> 
> - Kris
> 
> On Fri, Feb 8, 2019 at 4:37 PM Fengnan Li <lfengnan at uber.com <mailto:lfengnan at uber.com>> wrote:
> Hi Kris,
> 
> Thanks very much for the response!
> 
> I didn?t expect the RSS of the Java process to be 300GB when I set the heap size to 300GB since from a lot of other use cases I know that G1 takes more memory overhead compared to CMS. 
> I got the 450G number from our internal metrics, which essentially reads the /proc/meminfo file for the memory footprint. On the machine there is no other process taking a lot of memory (more than 1% of total memory or single digit GB).
> 
> I turned on the NMT option, and print out the JVM memory stack:
> 
> Native Memory Tracking:
> 
> Total: reserved=477GB, committed=476GB
> -                 Java Heap (reserved=300GB, committed=300GB)
>                             (mmap: reserved=300GB, committed=300GB)
> 
> -                    Thread (reserved=1GB, committed=1GB)
>                             (thread #723)
>                             (stack: reserved=1GB, committed=1GB)
> 
> -                        GC (reserved=23GB, committed=23GB)
>                             (malloc=12GB #20497380)
>                             (mmap: reserved=11GB, committed=11GB)
> 
> -                  Internal (reserved=152GB, committed=152GB)
>                             (malloc=152GB #19364496)
> 
> -    Native Memory Tracking (reserved=1GB, committed=1GB)
>                             (tracking overhead=1GB)
> 
> -                   Unknown (reserved=1GB, committed=0GB)
>                             (mmap: reserved=1GB, committed=0GB)
> 
> Interal (direct byte buffer) takes a lot of the space. GC overhead looks OK in this case.
> 
> This is pretty weird, I run the same app with CMS on the machine, but there is no Internal part from the stack. Do you know why this happens?
> 
> Thanks,
> Fengnan
> 
>> On Feb 8, 2019, at 2:07 PM, Krystal Mok <rednaxelafx at gmail.com <mailto:rednaxelafx at gmail.com>> wrote:
>> 
>> Hi Fengnan,
>> 
>> This is Kris Mok currently working at Databricks. I used to work on HotSpot and Zing JVMs.
>> Just curious how you got to the conclusion of G1 taking 450GB memory. Did you start the VM with -Xmx300g expecting that the RSS of that Java process be close to 300GB? If that's the case, that's a unreasonable expectation to begin with.
>> 
>> It's a very common case for G1 itself to take a high memory overhead due to its design of the Remembered Set (RSet), and that can be tuned to use less memory by making it more coarse-grained, with the tradeoff that root scanning pause can take longer because more portion of the heap is going to have to be scanned.
>> 
>> But I doubt that what you're actually seeing. To confirm where the memory went within the HotSpot JVM, please turn on NMT (Native Memory Tracking) and see how much memory each component within the JVM is using.
>> 
>> https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html <https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html>
>> https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html <https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html>
>> 
>> - Kris
>> 
>> On Fri, Feb 8, 2019 at 1:51 PM Fengnan Li <lfengnan at uber.com <mailto:lfengnan at uber.com>> wrote:
>> Hi All,
>> 
>> We are trying to use G1 for our HDFS Namenode to see whether it will deliver better GC overall than currently used CMS. However, with the 200G heap size JVM option, the G1 wouldn?t even start our namenode with the production image and will be killed out of memory after running for 1 hours (loading initial data). For the same heap size, CMS can work properly with around 98% throughput and averagely 120ms pause.
>> 
>> We use pretty much the basic options, and tried to tune a little but not much progress. Is there a way to lower down the overall memory footprint for G1?
>> 
>> We managed to start the application with 300G heap size option, but overall G1 will consume about 450G memory, which is problematic.
>> 
>> Thanks,
>> Fengnan_______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net <mailto:hotspot-gc-use at openjdk.java.net>
>> https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use <https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190209/d9291454/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2710 bytes
Desc: not available
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190209/d9291454/smime-0001.p7s>

From thomas.schatzl at oracle.com  Sat Feb  9 12:47:54 2019
From: thomas.schatzl at oracle.com (Thomas Schatzl)
Date: Sat, 09 Feb 2019 13:47:54 +0100
Subject: HDFS Namenode with large heap size
In-Reply-To: <9BC4A85C-E020-464C-85D2-63C6E15B078F@uber.com>
References: <9BC4A85C-E020-464C-85D2-63C6E15B078F@uber.com>
Message-ID: <ad6741e1a0eabb3194a3c596e4dd8079c49082df.camel@oracle.com>

Hi Fengnan,

  while I am responding to this email, I will also give explanations
for other questions that came up in this thread already.

Btw, if you ask for help it would be nice to at least mention the JDK
version you are using :) - this matters a lot as you will see.

On Fri, 2019-02-08 at 13:49 -0800, Fengnan Li wrote:
> Hi All,
> 
> We are trying to use G1 for our HDFS Namenode to see whether it will
> deliver better GC overall than currently used CMS. However, with the
> 200G heap size JVM option, the G1 wouldn?t even start our namenode
> with the production image and will be killed out of memory after
> running for 1 hours (loading initial data). For the same heap size,
> CMS can work properly with around 98% throughput and averagely 120ms
> pause.
> 
> We use pretty much the basic options, and tried to tune a little but
> not much progress. Is there a way to lower down the overall memory
> footprint for G1?
> 
> We managed to start the application with 300G heap size option, but
> overall G1 will consume about 450G memory, which is problematic.

Given the huge memory consumption and the long full gc pause times I am
sure you are running some 8u variant.

The problem in particular with large long running problems with jdk8 
are the remembered sets, i.e. some collector internal data structure
that allows g1 to pick particular regions to collect.

The reason why they show up as mtInternal is because of JDK-8176571
[0]. It has never been backported to 8u although it has a "8 backport
candidate" label.

The problem in 8u with the remembered sets can be explained by how it
is used: to pick separate regions for evacuation, for every region you
need to know who is referencing that particular region, basically
making it an O(n^2) data structure in the first place; further the data
structure for a particular region (internally called "Per Region
Table", PRT) is not just one kind of set, but three different data
structures which represent a different tradeoff in memory
usage/retrieval performance, called "sparse", "fine" and "coarse" (in
that order).

Sparse is a bunch of cards (a card spans 512 bytes; consider it
locations in the java heap) in an array, every entry takes 4 bytes (in
8u, 2 bytes later [1]), with very few entries. "Fine" is a bitmap
(spanning 4k of memory per byte, but it must always span a whole
region, i.e. with region size 32MB this table is roughly 8kb in size),
and "coarse" which is just a bit per region (which means: in that whole
region somewhere there is a location that is interesting).

So G1 starts with filling up a sparse array for a given region where
there is a reference into the region's remembered set, and if it fills
up (it's iirc like 20 entries on 32M regions), it expands that region's
remembered set to "fine", i.e. taking a whole 8kb for that region even
if it would only need 21 entries). At some point, if too many PRTs are
used for a single region, it drops that "fine" table and just uses a
bit.

G1 takes some effort to clean out stale remembered set entries (i.e.
cards that do not contain any reference any more) in the "Cleanup"
pause, but the problem is that G1 never moves entries from "fine"
(taking 8kb) to "sparse" (taking ~80 bytes) any more.

So over time these "fine" tables accumulate, taking very large amounts
of memory.

This has been fixed in JDK11 - JDK-8180415 [2] to be exact - G1 now
only keeps remembered sets when they are required, which also means
that no such cruft can accumulate. :) There has been a presentation
about this feature at FOSDEM 2018 [3].

In some applications this can yield enormous savings in remembered set
usage, like by a factor of 100 (I remember some internal application
run at 100+GB Java heap going from 40GB remembered sets to 0.4 GB with
no other tuning) in addition to making gc pauses a lot faster. In your
setup, the Cleanup pauses probably kill you already anyway - with that
change they will be down to a few ms...

So I would seriously consider moving to JDK11 - this will also fix the
long full gc pauses due to the parallel full gc implementation
introduced with JDK10 [4] unless you are stuck with JDK8.

The last significant performance improvements for G1 in the JDK8 line
have been added in August 2015 with 8u60!

Now, with JDK8 there is one option that can seriously reduce memory
usage, in particular by significantly increasing the size of the
"sparse" arrays so that they never get promoted to "fine", and not take
up that much space.

The option to use here is -XX:G1RSetSparseRegionEntries. Bump it to
something like 128 or 256. This will take 512 bytes/1k bytes for a PRT
instead of 8k for the "fine" one, but that is still 16 or 8 times
better... There should be no significant other performance regression.

If you are interested in even more gory details, there has been a
write-up about an estimation to this two years ago [5].

As for the future, we are still working on optimizing remembered set
memory usage. There may be some significant improvements coming in the
next few releases :)

JDK-8017163 [6] is a good starting point that collects references to a
few of those efforts (and how much has been done already).

Hope this helps.

Thanks,
  Thomas

[0] https://bugs.openjdk.java.net/browse/JDK-8176571
[1] https://bugs.openjdk.java.net/browse/JDK-8155721
[2] https://bugs.openjdk.java.net/browse/JDK-8180415
[3] https://archive.fosdem.org/2018/schedule/event/g1/
[4] https://bugs.openjdk.java.net/browse/JDK-8172890
[5] 
http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2017-August/002693.html
[6] https://bugs.openjdk.java.net/browse/JDK-8017163



From thomas.schatzl at oracle.com  Sat Feb  9 13:08:23 2019
From: thomas.schatzl at oracle.com (Thomas Schatzl)
Date: Sat, 09 Feb 2019 14:08:23 +0100
Subject: HDFS Namenode with large heap size
In-Reply-To: <ad6741e1a0eabb3194a3c596e4dd8079c49082df.camel@oracle.com>
References: <9BC4A85C-E020-464C-85D2-63C6E15B078F@uber.com>
 <ad6741e1a0eabb3194a3c596e4dd8079c49082df.camel@oracle.com>
Message-ID: <6973c0c1f1fc02f7023efda0eea438312b7e10e4.camel@oracle.com>

Hi,

  some minor additions:

On Sat, 2019-02-09 at 13:47 +0100, Thomas Schatzl wrote:
> Hi Fengnan,
> 
>   while I am responding to this email, I will also give explanations
> for other questions that came up in this thread already.
> 
> Btw, if you ask for help it would be nice to at least mention the JDK
> version you are using :) - this matters a lot as you will see.
> 
> On Fri, 2019-02-08 at 13:49 -0800, Fengnan Li wrote:
> > Hi All,
> > 
> > We are trying to use G1 for our HDFS Namenode to see whether it
> > will deliver better GC overall than currently used CMS. However,
> > with the 200G heap size JVM option, the G1 wouldn?t even start our
> > namenode with the production image and will be killed out of memory
> > after running for 1 hours (loading initial data). For the same heap
> > size, CMS can work properly with around 98% throughput and
> > averagely 120ms pause.

What does "killed out of memory mean"? An uncaught OOM exception in the
Java program or the OS killing it? What is the Java heap usage for CMS
in this case? Can you provide GC logs?

Also, for generic tuning questions and answer, please have a look at
the official documentation and tuning guide for G1 [0]. That one is for
11, but most advice still applies on 8u except for particular logging
flags/output.

Thanks,
  Thomas

[0] 
https://docs.oracle.com/en/java/javase/11/gctuning/garbage-first-garbage-collector.html#GUID-ED3AB6D3-FD9B-4447-9EDF-983ED2F7A573



From lfengnan at uber.com  Mon Feb 11 21:18:11 2019
From: lfengnan at uber.com (Fengnan Li)
Date: Mon, 11 Feb 2019 13:18:11 -0800
Subject: HDFS Namenode with large heap size
In-Reply-To: <6973c0c1f1fc02f7023efda0eea438312b7e10e4.camel@oracle.com>
References: <9BC4A85C-E020-464C-85D2-63C6E15B078F@uber.com>
 <ad6741e1a0eabb3194a3c596e4dd8079c49082df.camel@oracle.com>
 <6973c0c1f1fc02f7023efda0eea438312b7e10e4.camel@oracle.com>
Message-ID: <20C9F359-A764-444C-8DFE-CF27A0902CCB@uber.com>

Hi Thomas,

Thanks very much for the detailed explanation. For more information, please see below.

Our JDK version:
	openjdk version "1.8.0_171"
	OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
	OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

By `Killed OOM` I meant the process is killed by the system, and it is reflected in /var/log/kern.log

For CMS, there is OOM at all and heap size is set with 200G both min and max (this machine has physically 256G memory). The CMS runs pretty well without Full GC even. 

Hadoop is still testing with Java 11 now so I am not sure I can upgrade to JDK 11 very quickly.

So for our app, I guess I have to use the machine with larger memory to test it. 

Thanks,
Fengnan

> On Feb 9, 2019, at 5:08 AM, Thomas Schatzl <thomas.schatzl at oracle.com> wrote:
> 
> Hi,
> 
>  some minor additions:
> 
> On Sat, 2019-02-09 at 13:47 +0100, Thomas Schatzl wrote:
>> Hi Fengnan,
>> 
>>  while I am responding to this email, I will also give explanations
>> for other questions that came up in this thread already.
>> 
>> Btw, if you ask for help it would be nice to at least mention the JDK
>> version you are using :) - this matters a lot as you will see.
>> 
>> On Fri, 2019-02-08 at 13:49 -0800, Fengnan Li wrote:
>>> Hi All,
>>> 
>>> We are trying to use G1 for our HDFS Namenode to see whether it
>>> will deliver better GC overall than currently used CMS. However,
>>> with the 200G heap size JVM option, the G1 wouldn?t even start our
>>> namenode with the production image and will be killed out of memory
>>> after running for 1 hours (loading initial data). For the same heap
>>> size, CMS can work properly with around 98% throughput and
>>> averagely 120ms pause.
> 
> What does "killed out of memory mean"? An uncaught OOM exception in the
> Java program or the OS killing it? What is the Java heap usage for CMS
> in this case? Can you provide GC logs?
> 
> Also, for generic tuning questions and answer, please have a look at
> the official documentation and tuning guide for G1 [0]. That one is for
> 11, but most advice still applies on 8u except for particular logging
> flags/output.
> 
> Thanks,
>  Thomas
> 
> [0] 
> https://docs.oracle.com/en/java/javase/11/gctuning/garbage-first-garbage-collector.html#GUID-ED3AB6D3-FD9B-4447-9EDF-983ED2F7A573 <https://docs.oracle.com/en/java/javase/11/gctuning/garbage-first-garbage-collector.html#GUID-ED3AB6D3-FD9B-4447-9EDF-983ED2F7A573>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190211/9f902bc2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2710 bytes
Desc: not available
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190211/9f902bc2/smime.p7s>