YGC time increasing suddenly

[Aaron Daubman]

Luciano,

Until today I wasn't using huge pages, but today I ran a test locking my
> app in only one NUMA node (Wolfgang's tip) and using
> huge pages. As the app has less resources in this case, I decreased the
> request rate to 7k/s. The app ran properly for
> about ~35min and then the same problem appeared.
>

Interesting that I ran the exact same tests last night - enabling hugepages
and shared memory on a NUMA system and then binding the JVM to a single
socket. Apologies if all this is already common knowledge (i've had to do a
lot of reading on this recently at least ;-))

Note that hugepage allocation must be done post-boot in order to actually
allocate the desired number of hugepages on the socket you will be working
with.
I decided to try out the hugeadm utility, and found this to work for me:

/usr/bin/numactl --cpunodebind=0 --membind=0 hugeadm --pool-pages-min
2M:12G --obey-mempolicy --pool-pages-max 2M:12G --create-group-mounts
<groupname> --set-recommended-shmmax --set-shm-group echonest
--set-recommended-min_free_kbytes

This would allocate 12G of 2M hugepages on socket0.
Note that if you use /etc/grub.conf or even sysctl to allocate hugepages,
they will be evenly split among sockets, and so you will have less usable
than desired if locking a process to a single socket.

I found I was actually able to handle slightly higher load at similar
response times using half the systems resources, so I am essentially
wasting money on this dual-socket system =(

Some reading I found useful for this:
https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
"Interaction of Task Memory Policy with Huge Page Allocation/Freeing"

https://www.kernel.org/doc/Documentation/sysctl/vm.txt

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/main-cpu.html#s-cpu-tuning
"4.1.2.2. Controlling NUMA Policy with numactl"

http://users.sdsc.edu/~glockwood/comp/affinity.php#define:numactl

on hugeadm:
https://lwn.net/Articles/376606/

I verified this was working as-desired by:
# cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 HugePages_Total:  6144
Node 0 HugePages_Free:   1541
Node 0 HugePages_Surp:      0
Node 1 HugePages_Total:     0
Node 1 HugePages_Free:      0
Node 1 HugePages_Surp:      0



> What's the best way to setup
> the JVM considering NUMA?Disable NUMA?Run JVM with -XX:+UseNUMA?I'll run a
> test with this parameter.
>

+UseNUMA should definitely help if you are going to run on a multi-socket
NUMA system and if you (IIRC default enabled for your config) enable
+UseParallelGC:
http://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html#numa

As a side question, I am trying to stick with G1, which still lacks NUMA
compatibility, does anybody know why this still is or if it will be coming
anytime soon?
https://bugs.openjdk.java.net/browse/JDK-7005859
http://openjdk.java.net/jeps/157

Another side question that may potentially be useful:
I have found that I need to add -XX:+UseSHM in addition to
-XX:+UseLargePages in order to actually use LargePages - does this make
sense / is this expected? I could find very little documentation /
reference to UseSHM.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20131220/041938f1/attachment.html