hotspot-gc-use Digest, Vol 85, Issue 6

[Wang, Yanping]

Hi, Rohit

Which version of JDK are you using? 
It will be nice if you can attach GC logs to help us diagnostic what was going on during runtime.

Thanks
-yanping

-----Original Message-----
From: hotspot-gc-use [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of hotspot-gc-use-request at openjdk.java.net
Sent: Tuesday, April 07, 2015 10:44 AM
To: hotspot-gc-use at openjdk.java.net
Subject: hotspot-gc-use Digest, Vol 85, Issue 6

Send hotspot-gc-use mailing list submissions to
	hotspot-gc-use at openjdk.java.net

To subscribe or unsubscribe via the World Wide Web, visit
	http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
or, via email, send a message with subject or body 'help' to
	hotspot-gc-use-request at openjdk.java.net

You can reach the person managing the list at
	hotspot-gc-use-owner at openjdk.java.net

When replying, please edit your Subject line so it is more specific
than "Re: Contents of hotspot-gc-use digest..."


Today's Topics:

   1. G1 GC tuning (Chaubey, Rohit)


----------------------------------------------------------------------

Message: 1
Date: Tue, 7 Apr 2015 13:10:16 -0400
From: "Chaubey, Rohit" <Rohit.Chaubey at broadridge.com>
To: "hotspot-gc-use at openjdk.java.net"
	<hotspot-gc-use at openjdk.java.net>
Subject: G1 GC tuning
Message-ID:
	<DDCA4F01C163C544B9F7C0DC175B618C06CA0E20BA at JSCPCWEXMAA1.bsg.ad.adp.com>
	
Content-Type: text/plain; charset="us-ascii"

Hello

We are trying to fine tune our high impact application and would appreciate a little help in doing so using G1GC. Following are the application requirements

Application Requirements
* Expected load: 40 concurrent users with  1 Txns/user per second.
* 99th percentile - 250 ms .Desired all response times to be less than 1 second
* 15 GB RAM per JVM for one set(set1) and 12 GB RAM for another set(set 2). Both jvms contain different kinds of data and the application interacts with both to fetch desired output. Both JVMS have the same GC setting. But the kind of data in set 2 is more operation intensive and thus gets more activity that set 1 jvms.
* 2x load capacity. Hopefully we can get to 2X.

We implemented G1GC replacing the CMS as


1)      We have large heaps. Ranging from 8 Gb to 25 GB. The usual problem comes in with the set2 JVMS. Set1 generally does not crash.

2)      CMS was crashing the JVM frequently. A full GC cycle would crash the JVM. It used to happen during the nightly batch run for the application.

3)      We had Xmn upto 50% of the heap.

Once we switched from CMS to G1GC, the set2 jvms stopped crashing frequently. However we did have 2 incidents where the crash did happen. The G1GC parametrs that are being used are as follows:

JAVA_ARGS="$JAVA_ARGS  --J=-Xss${STACK_SIZE} \
                        --J=-XX:+UseG1GC \
                        --J=-XX:MaxGCPauseMillis=200 \
                        --J=-XX:ParallelGCThreads=20 \
                        --J=-XX:InitiatingHeapOccupancyPercent=60 \
                        --J=-XX:SurvivorRatio=2 \
                        --J=-XX:ConcGCThreads=5 \
                        --J=-Xmx$SERVER_HEAP \
                        --J=-Xms$SERVER_HEAP \
                        --J=-DDistributionManager.DISCONNECT_WAIT=$DISCONNECT_WAIT_TIME \
                        --J=-XX:+HeapDumpOnOutOfMemoryError \
                        --J=-XX:HeapDumpPath=${GF_LOG}/jvmdumps \
                        --J=-DgemfireSecurityPropertyFile=$DIR/$cluster_name/runtime/servers/$SERVER_NAME/gfsecurity.properties
                        --J=-verbose:gc \
                        --J=-Xloggc:${GF_LOG}/logs/$SERVER_NAME/gc.log \
                        --J=-XX:+UseGCLogFileRotation --J=-XX:NumberOfGCLogFiles=10 --J=-XX:GCLogFileSize=1m \
                        --J=-XX:+PrintGCDateStamps --J=-XX:+PrintGCDetails --J=-XX:+PrintTenuringDistribution --J=-XX:+PrintAdaptiveSizePolicy"

We tried starting from scratch and worked our way up to the above config with a series of load tests and other activities. Following are the observations that we have seen so far:


*         When the SurvivorRatio element was removed, then we observed a degraded performance. Thus we kept it at 2 instead of default 8.

*         There are a handful of humongous allocations(maybe 3 -4 over a day) that are requested during the batch run.The number of those is not that high as you can see from the attached logs. Should I be changing the region size for them?

Please let me know what can be changed and tested so that we do not encounter the jvm crash during batch times and also maintain the current sla. I have attached all the gc log files for the cluster.


Thanks and Regards,
Rohit Chaubey
Email: rohit.chaubey at broadridge.com<mailto:rohit.chaubey at broadridge.com>
Work: 201-714-3379, BB: 201-618-9230


This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20150407/edc8bf03/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log files.zip
Type: application/x-zip-compressed
Size: 2660835 bytes
Desc: log files.zip
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20150407/edc8bf03/logfiles.zip>

------------------------------

Subject: Digest Footer

_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


------------------------------

End of hotspot-gc-use Digest, Vol 85, Issue 6
*********************************************