HDFS Namenode with large heap size

[Fengnan Li]

Hi Thomas,

Thanks very much for the detailed explanation. For more information, please see below.

Our JDK version:
	openjdk version "1.8.0_171"
	OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
	OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

By `Killed OOM` I meant the process is killed by the system, and it is reflected in /var/log/kern.log

For CMS, there is OOM at all and heap size is set with 200G both min and max (this machine has physically 256G memory). The CMS runs pretty well without Full GC even. 

Hadoop is still testing with Java 11 now so I am not sure I can upgrade to JDK 11 very quickly.

So for our app, I guess I have to use the machine with larger memory to test it. 

Thanks,
Fengnan

> On Feb 9, 2019, at 5:08 AM, Thomas Schatzl <thomas.schatzl at oracle.com> wrote:
> 
> Hi,
> 
>  some minor additions:
> 
> On Sat, 2019-02-09 at 13:47 +0100, Thomas Schatzl wrote:
>> Hi Fengnan,
>> 
>>  while I am responding to this email, I will also give explanations
>> for other questions that came up in this thread already.
>> 
>> Btw, if you ask for help it would be nice to at least mention the JDK
>> version you are using :) - this matters a lot as you will see.
>> 
>> On Fri, 2019-02-08 at 13:49 -0800, Fengnan Li wrote:
>>> Hi All,
>>> 
>>> We are trying to use G1 for our HDFS Namenode to see whether it
>>> will deliver better GC overall than currently used CMS. However,
>>> with the 200G heap size JVM option, the G1 wouldn’t even start our
>>> namenode with the production image and will be killed out of memory
>>> after running for 1 hours (loading initial data). For the same heap
>>> size, CMS can work properly with around 98% throughput and
>>> averagely 120ms pause.
> 
> What does "killed out of memory mean"? An uncaught OOM exception in the
> Java program or the OS killing it? What is the Java heap usage for CMS
> in this case? Can you provide GC logs?
> 
> Also, for generic tuning questions and answer, please have a look at
> the official documentation and tuning guide for G1 [0]. That one is for
> 11, but most advice still applies on 8u except for particular logging
> flags/output.
> 
> Thanks,
>  Thomas
> 
> [0] 
> https://docs.oracle.com/en/java/javase/11/gctuning/garbage-first-garbage-collector.html#GUID-ED3AB6D3-FD9B-4447-9EDF-983ED2F7A573 <https://docs.oracle.com/en/java/javase/11/gctuning/garbage-first-garbage-collector.html#GUID-ED3AB6D3-FD9B-4447-9EDF-983ED2F7A573>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190211/9f902bc2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2710 bytes
Desc: not available
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20190211/9f902bc2/smime.p7s>