jvm swap issue
James Nichols
jamesnichols3 at gmail.com
Fri Mar 2 14:31:39 PST 2012
What is the OS?
If it is Linux, the vm.swappiness = 0 option may help.
I've seen this situation before where the Linux OS will proactively swap
out areas of memory that haven't been used in a while. With a large heap
and infrequent full garbage collections, the Linux virtual memory subsystem
will sometimes do this to pages underneath the JVM's OS-level virtual
memory. If you then have a full garbage collection it will usually cause
major problems if any part of the heap is on swap space.
Typically what I do is set swappiness to 0, but not disable swap, and have
an alert go off on the OS level if it uses even a single block of swap
space. That way there is at least a fail safe, since disabling swap can
make the system crash if it does run out of physical memory.
There are options to use Top that will show you how much of a process's
memory space is on swap, as well as page faults. You don't want either.
Hope that helps.
Jim
On Fri, Mar 2, 2012 at 9:47 AM, the.6th.month at gmail.com <
the.6th.month at gmail.com> wrote:
> hi,hi:
> I've just come across a weird situation where JVM ate swap space
> occasionally even when there's free memory available. Has anyone got any
> idea about how to diagnose such problem?
> The output of top is as follows, and it is sorted by memory usage
> (shift+M):
>
> top - 22:36:16 up 102 days, 10:49, 2 users, load average: 1.68, 1.39,
> 1.34
> Tasks: 98 total, 2 running, 96 sleeping, 0 stopped, 0 zombie
> Cpu(s): 3.9%us, 0.6%sy, 0.0%ni, 94.4%id, 0.0%wa, 0.0%hi, 0.3%si,
> 0.8%st
> Mem: 6291456k total, 6276292k used, 15164k free, 16528k buffers
> Swap: 4192924k total, 39264k used, 4153660k free, 836288k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
>
> 677 root 18 0 5259m 4.8g 10m S 18.9 79.5 398:10.95
> java
>
> 1721 root 34 19 253m 5356 2196 S 0.0 0.1 0:07.81
> yum-updatesd
>
> 1521 ntp 15 0 23412 5044 3916 S 0.0 0.1 0:06.03
> ntpd
>
> 1482 root 15 0 154m 4512 3000 S 0.0 0.1 1:31.02
> snmpd
>
> 14006 root 17 0 88080 3264 2552 S 0.0 0.1 0:00.01
> sshd
>
> 14053 root 18 0 88080 3264 2552 S 0.0 0.1 0:00.00
> sshd
>
> 13088 postfix 15 0 54244 2300 1796 S 0.0 0.0 0:00.00
> pickup
>
> 1592 postfix 15 0 54420 1936 1816 S 0.0 0.0 0:00.09
> qmgr
>
> 1580 root 15 0 54180 1828 1736 S 0.0 0.0 0:00.50
> master
>
> 14008 yue.liu 15 0 88080 1716 976 R 0.0 0.0 0:00.04
> sshd
>
> 14055 yue.liu 15 0 88080 1696 972 S 0.0 0.0 0:00.01
> sshd
>
> 14056 yue.liu 15 0 66096 1596 1204 S 0.0 0.0 0:00.01
> bash
>
> 14159 root 15 0 66096 1580 1192 S 0.0 0.0 0:00.00
> bash
>
> 14101 root 15 0 66096 1576 1196 S 0.0 0.0 0:00.01
> bash
>
> 14009 yue.liu 15 0 66096 1572 1184 S 0.0 0.0 0:00.01
> bash
>
> 1411 haldaemo 15 0 30660 1252 1124 S 0.0 0.0 0:00.14 hald
>
> and the swap usage seems to come from the address space of:
> 2aaad4000000-2aaad7ffc000 rwxp 2aaad4000000 00:00 0
> Size: 65520 kB
> Rss: 50476 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 50476 kB
> Swap: 10192 kB
> Pss: 50476 kB
> according to the output of /proc/pid/smaps.
>
> the output of jmap is as below:
> [root at l-tw14 ~]# jmap 677
> Attaching to process ID 677, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.1-b02
> 0x0000000040000000 49K /home/q/java/jdk1.6.0_26/bin/java
> 0x0000003bf5600000 136K /lib64/ld-2.5.so
> 0x0000003bf5a00000 1681K /lib64/libc-2.5.so
> 0x0000003bf5e00000 22K /lib64/libdl-2.5.so
> 0x0000003bf6200000 142K /lib64/libpthread-2.5.so
> 0x0000003bf6600000 600K /lib64/libm-2.5.so
> 0x0000003bf7200000 52K /lib64/librt-2.5.so
> 0x0000003bf8600000 111K /lib64/libnsl-2.5.so
> 0x0000003bfbe00000 90K /lib64/libresolv-2.5.so
> 0x00002aaaaaab6000 65K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libverify.so
> 0x00002aaaaabc5000 229K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libjava.so
> 0x00002aaaaad00000 52K /lib64/libnss_files-2.5.so
> 0x00002aaaaaf0b000 90K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libzip.so
> 0x00002aaab2dcc000 38K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libmanagement.so
> 0x00002aaab2ed3000 110K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libnet.so
> 0x00002aaab3504000 23K /lib64/libnss_dns-2.5.so
> 0x00002aaab3709000 43K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libnio.so
> 0x00002aaab3818000 744K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libawt.so
> 0x00002aaab39e7000 33K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/headless/libmawt.so
> 0x00002aaab3aed000 655K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libfontmanager.so
> 0x00002aaabc2fc000 718K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libmlib_image.so
> 0x00002aaabc4ab000 221K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libjpeg.so
> 0x00002ac6d0e04000 47K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/jli/libjli.so
> 0x00002ac6d0f11000 13027K
> /home/q/java/jdk1.6.0_26/jre/lib/amd64/server/libjvm.so
>
> [root at l-tw14 ~]# jmap -heap 677
> Attaching to process ID 677, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.1-b02
>
> using thread-local object allocation.
> Parallel GC with 4 thread(s)
>
> Heap Configuration:
> MinHeapFreeRatio = 40
> MaxHeapFreeRatio = 70
> MaxHeapSize = 4194304000 (4000.0MB)
> NewSize = 268435456 (256.0MB)
> MaxNewSize = 268435456 (256.0MB)
> OldSize = 5439488 (5.1875MB)
> NewRatio = 2
> SurvivorRatio = 8
> PermSize = 268435456 (256.0MB)
> MaxPermSize = 268435456 (256.0MB)
>
> Heap Usage:
> PS Young Generation
> Eden Space:
> capacity = 240779264 (229.625MB)
> used = 42971864 (40.981163024902344MB)
> free = 197807400 (188.64383697509766MB)
> 17.84699532929879% used
> From Space:
> capacity = 1769472 (1.6875MB)
> used = 1736704 (1.65625MB)
> free = 32768 (0.03125MB)
> 98.14814814814815% used
> To Space:
> capacity = 13893632 (13.25MB)
> used = 0 (0.0MB)
> free = 13893632 (13.25MB)
> 0.0% used
> PS Old Generation
> capacity = 3925868544 (3744.0MB)
> used = 3600153552 (3433.373977661133MB)
> free = 325714992 (310.6260223388672MB)
> 91.70336478795761% used
> PS Perm Generation
> capacity = 268435456 (256.0MB)
> used = 191475048 (182.6048355102539MB)
> free = 76960408 (73.3951644897461MB)
> 71.33001387119293% used
>
> it seems that the heap usage is not that high enough to occupy swap space.
> weird. And I also checked the nio usage, it was quite trivial:
> [root at l-tw14 ~]#java -classpath .:$JAVA_HOME/lib/sa-jdi.jar
> DirectMemorySize `pgrep java`
> Attaching to process ID 677, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.1-b02
> NIO direct memory:
> reserved size = 0.357348MB (374707 bytes)
> max size = 3968.000000MB (4160749568 bytes)
>
> This problem bothered me quite a few days, and I appreciate all your help.
> Thanks in advance
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120302/08e29137/attachment.html
More information about the hotspot-gc-use
mailing list