jvm swap issue
the.6th.month at gmail.com
the.6th.month at gmail.com
Fri Mar 2 06:47:26 PST 2012
hi,hi:
I've just come across a weird situation where JVM ate swap space
occasionally even when there's free memory available. Has anyone got any
idea about how to diagnose such problem?
The output of top is as follows, and it is sorted by memory usage (shift+M):
top - 22:36:16 up 102 days, 10:49, 2 users, load average: 1.68, 1.39, 1.34
Tasks: 98 total, 2 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.9%us, 0.6%sy, 0.0%ni, 94.4%id, 0.0%wa, 0.0%hi, 0.3%si,
0.8%st
Mem: 6291456k total, 6276292k used, 15164k free, 16528k buffers
Swap: 4192924k total, 39264k used, 4153660k free, 836288k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
677 root 18 0 5259m 4.8g 10m S 18.9 79.5 398:10.95
java
1721 root 34 19 253m 5356 2196 S 0.0 0.1 0:07.81
yum-updatesd
1521 ntp 15 0 23412 5044 3916 S 0.0 0.1 0:06.03
ntpd
1482 root 15 0 154m 4512 3000 S 0.0 0.1 1:31.02
snmpd
14006 root 17 0 88080 3264 2552 S 0.0 0.1 0:00.01
sshd
14053 root 18 0 88080 3264 2552 S 0.0 0.1 0:00.00
sshd
13088 postfix 15 0 54244 2300 1796 S 0.0 0.0 0:00.00
pickup
1592 postfix 15 0 54420 1936 1816 S 0.0 0.0 0:00.09
qmgr
1580 root 15 0 54180 1828 1736 S 0.0 0.0 0:00.50
master
14008 yue.liu 15 0 88080 1716 976 R 0.0 0.0 0:00.04
sshd
14055 yue.liu 15 0 88080 1696 972 S 0.0 0.0 0:00.01
sshd
14056 yue.liu 15 0 66096 1596 1204 S 0.0 0.0 0:00.01
bash
14159 root 15 0 66096 1580 1192 S 0.0 0.0 0:00.00
bash
14101 root 15 0 66096 1576 1196 S 0.0 0.0 0:00.01
bash
14009 yue.liu 15 0 66096 1572 1184 S 0.0 0.0 0:00.01
bash
1411 haldaemo 15 0 30660 1252 1124 S 0.0 0.0 0:00.14 hald
and the swap usage seems to come from the address space of:
2aaad4000000-2aaad7ffc000 rwxp 2aaad4000000 00:00 0
Size: 65520 kB
Rss: 50476 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 50476 kB
Swap: 10192 kB
Pss: 50476 kB
according to the output of /proc/pid/smaps.
the output of jmap is as below:
[root at l-tw14 ~]# jmap 677
Attaching to process ID 677, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.1-b02
0x0000000040000000 49K /home/q/java/jdk1.6.0_26/bin/java
0x0000003bf5600000 136K /lib64/ld-2.5.so
0x0000003bf5a00000 1681K /lib64/libc-2.5.so
0x0000003bf5e00000 22K /lib64/libdl-2.5.so
0x0000003bf6200000 142K /lib64/libpthread-2.5.so
0x0000003bf6600000 600K /lib64/libm-2.5.so
0x0000003bf7200000 52K /lib64/librt-2.5.so
0x0000003bf8600000 111K /lib64/libnsl-2.5.so
0x0000003bfbe00000 90K /lib64/libresolv-2.5.so
0x00002aaaaaab6000 65K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/libverify.so
0x00002aaaaabc5000 229K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/libjava.so
0x00002aaaaad00000 52K /lib64/libnss_files-2.5.so
0x00002aaaaaf0b000 90K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/libzip.so
0x00002aaab2dcc000 38K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/libmanagement.so
0x00002aaab2ed3000 110K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/libnet.so
0x00002aaab3504000 23K /lib64/libnss_dns-2.5.so
0x00002aaab3709000 43K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/libnio.so
0x00002aaab3818000 744K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/libawt.so
0x00002aaab39e7000 33K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/headless/libmawt.so
0x00002aaab3aed000 655K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/libfontmanager.so
0x00002aaabc2fc000 718K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/libmlib_image.so
0x00002aaabc4ab000 221K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/libjpeg.so
0x00002ac6d0e04000 47K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/jli/libjli.so
0x00002ac6d0f11000 13027K
/home/q/java/jdk1.6.0_26/jre/lib/amd64/server/libjvm.so
[root at l-tw14 ~]# jmap -heap 677
Attaching to process ID 677, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.1-b02
using thread-local object allocation.
Parallel GC with 4 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 4194304000 (4000.0MB)
NewSize = 268435456 (256.0MB)
MaxNewSize = 268435456 (256.0MB)
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 268435456 (256.0MB)
MaxPermSize = 268435456 (256.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 240779264 (229.625MB)
used = 42971864 (40.981163024902344MB)
free = 197807400 (188.64383697509766MB)
17.84699532929879% used
>From Space:
capacity = 1769472 (1.6875MB)
used = 1736704 (1.65625MB)
free = 32768 (0.03125MB)
98.14814814814815% used
To Space:
capacity = 13893632 (13.25MB)
used = 0 (0.0MB)
free = 13893632 (13.25MB)
0.0% used
PS Old Generation
capacity = 3925868544 (3744.0MB)
used = 3600153552 (3433.373977661133MB)
free = 325714992 (310.6260223388672MB)
91.70336478795761% used
PS Perm Generation
capacity = 268435456 (256.0MB)
used = 191475048 (182.6048355102539MB)
free = 76960408 (73.3951644897461MB)
71.33001387119293% used
it seems that the heap usage is not that high enough to occupy swap space.
weird. And I also checked the nio usage, it was quite trivial:
[root at l-tw14 ~]#java -classpath .:$JAVA_HOME/lib/sa-jdi.jar
DirectMemorySize `pgrep java`
Attaching to process ID 677, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.1-b02
NIO direct memory:
reserved size = 0.357348MB (374707 bytes)
max size = 3968.000000MB (4160749568 bytes)
This problem bothered me quite a few days, and I appreciate all your help.
Thanks in advance
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120302/6ac7c9e2/attachment-0001.html
More information about the hotspot-gc-use
mailing list