Garbage Collection tuning advice ?

Bernardo Sanchez bernardo.s at pointclickcare.com
Mon Apr 20 18:54:54 UTC 2015 

CMS is still the best for high performance apps; but truly depends on how your app was created and what it does.  There is no hard rule for what collector to use (i.e. reporting servers need diff settings than app servers…)

The issue you’re having is probably related to running out or memory, or a memory leak.  I don’t believe this to be a ‘Collector’ issue as when Java allocates memory, it requires contiguous blocks, so if it cannot allocate said memory, it will perform a full GC to effectively de-fragment the memory (this could cause stop the world pauses, but not OOM).  As this does not appear to be your issue, you might have a memory leak on your hands (holding on to dead objects that is filling the heap).

Here is a page for JVM options:
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html#G1Options

here is a GC benchmarking doc:
http://blog.mgm-tp.com/2013/12/benchmarking-g1-and-other-java-7-garbage-collectors/

You really need to have GC logging enabled, then use a tool like GCViewer to see how your heap is acting (gcnormal.log file – see below).
Here are some GC Logging Settings:
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/srv/aem/tmp/
-Xloggc:/srv/aem/tmp/gcnormal.log
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintHeapAtGC

I would also suggest taking a heap dump and inspect it with the Eclipse Memory Analyzer (leak suspects) to view what is truly occurring at the slow time.  This can take quite a bit of time to perform, so you’ll probably need at minimum a second server to take over the load.

You can also connect to your JVM using jconsole to see the live heap in action (requires this setting; you can change the port: -agentlib:jdwp=transport=dt_socket,address=8082,server=y,suspend=n)

hope this helps,
b

From: hotspot-gc-use [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Rohit Mohta
Sent: April-20-15 2:05 PM
To: Nicola Abello
Cc: hotspot-gc-use at openjdk.java.net
Subject: Re: Garbage Collection tuning advice ?


Hi,

  We are facing similar issue for our high performance application. We noticed, JVM ergonomic took over and adjusted the values to meet its goal (adaptive policy). This worked in most scenario, but caused problems in some case.

My recommendations will be

a) Turn on GC logging (if not yet done) either for default setting or the one you have mentioned

b) Check if you have a regular premature movement to old generation. If yes, you might want to set a specific new size and/or survivor size.

c) You have 16GB ram but have set Max heap as 8GB. If you are getting OOM, increase Xmx.
On 20 Apr 2015 18:50, "Nicola Abello" <nicola at infielddesign.com<mailto:nicola at infielddesign.com>> wrote:
Hello everyone, I am currently using the Adobe Experience Manager for a Client's site (Java language). It uses openJDK:

#java -version

java version "1.7.0_65"
OpenJDK Runtime Environment (rhel-2.5.1.2.el6_5-x86_64 u65-b17)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

It is running on Rackspace with the following:

vCPU: 4
Memory: 16GB
Guest OS: Red Hat Enterprise Linux 6 (64-bit)

Since it has been in production I have been experiencing very slow performance on the part of the application. It goes like this I launch the app, everything is smooth then 3 to 4 days later the CPU usage spikes to 400% (~4000 users/day hit the site). I got a few OOM exceptions (1 or 2) but mostly the site was exceptionally slow and never becomes an OOM exception. Since I am a novice at Java Memory management I started reading about how it works and found tools like jstat. When the system was overwhelmed the second time around, I ran:

#top

Got the PID of the java process and then pressed shift+H and noted the PIDs of the threads with high CPU percentage. Then I ran

#sudo -uaem jstat <PID>

Got a thread dump and converted the thread PIDs I wrote down previously and searched for their hex value in the dump. After all that, I finally found that it was not surprisingly the Garbage Collector that is flipping out for some reason.

I started reading a lot about Java GC tuning and came up with the following java options.
So restarted the application with the following options:

java -Dcom.day.crx.persistence.tar.IndexMergeDelay=0
-Djackrabbit.maxQueuedEvents=1000000
-Djava.io.tmpdir=/srv/aem/tmp/

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/srv/aem/tmp/
-Xms8192m -Xmx8192m
-XX:PermSize=256m
-XX:MaxPermSize=1024m
-XX:+UseParallelGC
-XX:+UseParallelOldGC
-XX:ParallelGCThreads=4
-XX:NewRatio=1

-Djava.awt.headless=true
-server
-Dsling.run.modes=publish
-jar crx-quickstart/app/cq-quickstart-6.0.0-standalone.jar start
-c crx-quickstart -i launchpad -p 4503
-Dsling.properties=conf/sling.properties

And it looks like it is performing much better but I think that it probably needs more GC tuning.

When I run:

#sudo -uaem jstat <PID> -gcutils

I get this:

S0     S1     E       O       P      YGC      YGCT     FGC     FGCT      GCT
0.00   0.00  55.97 100.00  45.09    4725   521.233   505  4179.584  4700.817

after 4 days that I restarted it.

When I run:

#sudo -uaem jstat <PID> -gccapacity

I get this:

 NGCMN     NGCMX      NGC     S0C   S1C       EC
4194304.0  4194304.0  4194304.0 272896.0 279040.0 3636224.0

 OGCMN   OGCMX  OGC                OC       PGCMN           PGCMX
4194304.0  4194304.0   4194304.0   4194304.0 262144.0  1048576.0

PGC       PC     YGC     FGC
262144.0 262144.0   4725    509

after 4 days that I restarted it.

These result are much better than when I started but I think it can get even better. I'm not really sure what to do next as I'm no GC pro so I was wondering if you guys would have any tips or advice for me on how I could get better app/GC performance and if anything is obvious like ratio's and sizes of youngGen and oldGen ?

How should I set the survivors and eden sizes/ratios ?
Should I change GC type like use CMS GC or G1 ?
How should I proceed ?

Any advice would be helpful.

Best,
Nicola

_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net<mailto:hotspot-gc-use at openjdk.java.net>
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20150420/2d798531/attachment-0001.html>

More information about the hotspot-gc-use mailing list