Garbage Collection tuning advice ?

[Nicola Abello]

Hello everyone, I am currently using the Adobe Experience Manager for a
Client's site (Java language). It uses openJDK:

#*java -version*

*java version "1.7.0_65"*
*OpenJDK Runtime Environment (rhel-2.5.1.2.el6_5-x86_64 u65-b17)*
*OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)*

It is running on Rackspace with the following:

*vCPU:* *4*
*Memory:* *16GB*
*Guest OS:* *Red Hat Enterprise Linux 6 (64-bit)*

Since it has been in production I have been experiencing very slow
performance on the part of the application. It goes like this I launch the
app, everything is smooth then 3 to 4 days later the CPU usage spikes to
400% (~4000 users/day hit the site). I got a few OOM exceptions (1 or 2)
but mostly the site was exceptionally slow and never becomes an OOM
exception. Since I am a novice at Java Memory management I started reading
about how it works and found tools like jstat. When the system was
overwhelmed the second time around, I ran:

#*top*

Got the PID of the java process and then pressed shift+H and noted the PIDs
of the threads with high CPU percentage. Then I ran

#*sudo -uaem jstat <PID>*

Got a thread dump and converted the thread PIDs I wrote down previously and
searched for their hex value in the dump. After all that, I finally found
that it was not surprisingly the Garbage Collector that is flipping out for
some reason.

I started reading a lot about Java GC tuning and came up with the following
java options.
So restarted the application with the following options:

*java* -Dcom.day.crx.persistence.tar.IndexMergeDelay=0
 -Djackrabbit.maxQueuedEvents=1000000
 -Djava.io.tmpdir=/srv/aem/tmp/

* -XX:+HeapDumpOnOutOfMemoryError *
* -XX:HeapDumpPath=/srv/aem/tmp/ *
* -Xms8192m -Xmx8192m *
* -XX:PermSize=256m *
* -XX:MaxPermSize=1024m *
* -XX:+UseParallelGC *
* -XX:+UseParallelOldGC *
* -XX:ParallelGCThreads=4 *
* -XX:NewRatio=1*

 -Djava.awt.headless=true
 -server
 -Dsling.run.modes=publish
 -jar crx-quickstart/app/cq-quickstart-6.0.0-standalone.jar start
-c crx-quickstart -i launchpad -p 4503
 -Dsling.properties=conf/sling.properties

And it looks like it is performing much better but I think that it probably
needs more GC tuning.

When I run:

#*sudo -uaem jstat <PID> -gcutils*

I get this:

S0     S1     E       O       P      YGC      YGCT     FGC     FGCT      GCT

0.00   0.00  55.97 100.00  45.09    4725   521.233   505  4179.584  4700.817

after 4 days that I restarted it.

When I run:

#*sudo -uaem jstat <PID> -gccapacity*

I get this:

 NGCMN     NGCMX      NGC      S0C   S1C        EC
4194304.0  4194304.0  4194304.0 272896.0 279040.0 3636224.0

 OGCMN   OGCMX  OGC                OC       PGCMN           PGCMX
4194304.0  4194304.0   4194304.0   4194304.0 262144.0  1048576.0

PGC        PC     YGC     FGC
262144.0 262144.0   4725    509

after 4 days that I restarted it.

These result are much better than when I started but I think it can get
even better. I'm not really sure what to do next as I'm no GC pro so I was
wondering if you guys would have any tips or advice for me on how I could
get better app/GC performance and if anything is obvious like ratio's and
sizes of youngGen and oldGen ?

How should I set the survivors and eden sizes/ratios ?
Should I change GC type like use CMS GC or G1 ?
How should I proceed ?

Any advice would be helpful.

Best,
Nicola
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20150420/bb0aa7a3/attachment.html>