jvm swap issue
the.6th.month at gmail.com
the.6th.month at gmail.com
Mon Mar 5 01:44:06 PST 2012
Hi, James:
I've set swappiness to zero. We'll keep an eye on it for a while to see
whether it is the solution to the problem. So far so good
Best Regards,
Leon
On 3 March 2012 06:31, James Nichols <jamesnichols3 at gmail.com> wrote:
> What is the OS?
>
> If it is Linux, the vm.swappiness = 0 option may help.
>
> I've seen this situation before where the Linux OS will proactively swap
> out areas of memory that haven't been used in a while. With a large heap
> and infrequent full garbage collections, the Linux virtual memory subsystem
> will sometimes do this to pages underneath the JVM's OS-level virtual
> memory. If you then have a full garbage collection it will usually cause
> major problems if any part of the heap is on swap space.
>
> Typically what I do is set swappiness to 0, but not disable swap, and have
> an alert go off on the OS level if it uses even a single block of swap
> space. That way there is at least a fail safe, since disabling swap can
> make the system crash if it does run out of physical memory.
>
> There are options to use Top that will show you how much of a process's
> memory space is on swap, as well as page faults. You don't want either.
>
> Hope that helps.
>
> Jim
>
>
>
>
> On Fri, Mar 2, 2012 at 9:47 AM, the.6th.month at gmail.com <
> the.6th.month at gmail.com> wrote:
>
>> hi,hi:
>> I've just come across a weird situation where JVM ate swap space
>> occasionally even when there's free memory available. Has anyone got any
>> idea about how to diagnose such problem?
>> The output of top is as follows, and it is sorted by memory usage
>> (shift+M):
>>
>> top - 22:36:16 up 102 days, 10:49, 2 users, load average: 1.68, 1.39,
>> 1.34
>> Tasks: 98 total, 2 running, 96 sleeping, 0 stopped, 0 zombie
>> Cpu(s): 3.9%us, 0.6%sy, 0.0%ni, 94.4%id, 0.0%wa, 0.0%hi, 0.3%si,
>> 0.8%st
>> Mem: 6291456k total, 6276292k used, 15164k free, 16528k buffers
>> Swap: 4192924k total, 39264k used, 4153660k free, 836288k cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>>
>> 677 root 18 0 5259m 4.8g 10m S 18.9 79.5 398:10.95
>> java
>>
>> 1721 root 34 19 253m 5356 2196 S 0.0 0.1 0:07.81
>> yum-updatesd
>>
>> 1521 ntp 15 0 23412 5044 3916 S 0.0 0.1 0:06.03
>> ntpd
>>
>> 1482 root 15 0 154m 4512 3000 S 0.0 0.1 1:31.02
>> snmpd
>>
>> 14006 root 17 0 88080 3264 2552 S 0.0 0.1 0:00.01
>> sshd
>>
>> 14053 root 18 0 88080 3264 2552 S 0.0 0.1 0:00.00
>> sshd
>>
>> 13088 postfix 15 0 54244 2300 1796 S 0.0 0.0 0:00.00
>> pickup
>>
>> 1592 postfix 15 0 54420 1936 1816 S 0.0 0.0 0:00.09
>> qmgr
>>
>> 1580 root 15 0 54180 1828 1736 S 0.0 0.0 0:00.50
>> master
>>
>> 14008 yue.liu 15 0 88080 1716 976 R 0.0 0.0 0:00.04
>> sshd
>>
>> 14055 yue.liu 15 0 88080 1696 972 S 0.0 0.0 0:00.01
>> sshd
>>
>> 14056 yue.liu 15 0 66096 1596 1204 S 0.0 0.0 0:00.01
>> bash
>>
>> 14159 root 15 0 66096 1580 1192 S 0.0 0.0 0:00.00
>> bash
>>
>> 14101 root 15 0 66096 1576 1196 S 0.0 0.0 0:00.01
>> bash
>>
>> 14009 yue.liu 15 0 66096 1572 1184 S 0.0 0.0 0:00.01
>> bash
>>
>> 1411 haldaemo 15 0 30660 1252 1124 S 0.0 0.0 0:00.14 hald
>>
>> and the swap usage seems to come from the address space of:
>> 2aaad4000000-2aaad7ffc000 rwxp 2aaad4000000 00:00 0
>> Size: 65520 kB
>> Rss: 50476 kB
>> Shared_Clean: 0 kB
>> Shared_Dirty: 0 kB
>> Private_Clean: 0 kB
>> Private_Dirty: 50476 kB
>> Swap: 10192 kB
>> Pss: 50476 kB
>> according to the output of /proc/pid/smaps.
>>
>> the output of jmap is as below:
>> [root at l-tw14 ~]# jmap 677
>> Attaching to process ID 677, please wait...
>> Debugger attached successfully.
>> Server compiler detected.
>> JVM version is 20.1-b02
>> 0x0000000040000000 49K /home/q/java/jdk1.6.0_26/bin/java
>> 0x0000003bf5600000 136K /lib64/ld-2.5.so
>> 0x0000003bf5a00000 1681K /lib64/libc-2.5.so
>> 0x0000003bf5e00000 22K /lib64/libdl-2.5.so
>> 0x0000003bf6200000 142K /lib64/libpthread-2.5.so
>> 0x0000003bf6600000 600K /lib64/libm-2.5.so
>> 0x0000003bf7200000 52K /lib64/librt-2.5.so
>> 0x0000003bf8600000 111K /lib64/libnsl-2.5.so
>> 0x0000003bfbe00000 90K /lib64/libresolv-2.5.so
>> 0x00002aaaaaab6000 65K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libverify.so
>> 0x00002aaaaabc5000 229K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libjava.so
>> 0x00002aaaaad00000 52K /lib64/libnss_files-2.5.so
>> 0x00002aaaaaf0b000 90K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libzip.so
>> 0x00002aaab2dcc000 38K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libmanagement.so
>> 0x00002aaab2ed3000 110K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libnet.so
>> 0x00002aaab3504000 23K /lib64/libnss_dns-2.5.so
>> 0x00002aaab3709000 43K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libnio.so
>> 0x00002aaab3818000 744K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libawt.so
>> 0x00002aaab39e7000 33K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/headless/libmawt.so
>> 0x00002aaab3aed000 655K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libfontmanager.so
>> 0x00002aaabc2fc000 718K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libmlib_image.so
>> 0x00002aaabc4ab000 221K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/libjpeg.so
>> 0x00002ac6d0e04000 47K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/jli/libjli.so
>> 0x00002ac6d0f11000 13027K
>> /home/q/java/jdk1.6.0_26/jre/lib/amd64/server/libjvm.so
>>
>> [root at l-tw14 ~]# jmap -heap 677
>> Attaching to process ID 677, please wait...
>> Debugger attached successfully.
>> Server compiler detected.
>> JVM version is 20.1-b02
>>
>> using thread-local object allocation.
>> Parallel GC with 4 thread(s)
>>
>> Heap Configuration:
>> MinHeapFreeRatio = 40
>> MaxHeapFreeRatio = 70
>> MaxHeapSize = 4194304000 (4000.0MB)
>> NewSize = 268435456 (256.0MB)
>> MaxNewSize = 268435456 (256.0MB)
>> OldSize = 5439488 (5.1875MB)
>> NewRatio = 2
>> SurvivorRatio = 8
>> PermSize = 268435456 (256.0MB)
>> MaxPermSize = 268435456 (256.0MB)
>>
>> Heap Usage:
>> PS Young Generation
>> Eden Space:
>> capacity = 240779264 (229.625MB)
>> used = 42971864 (40.981163024902344MB)
>> free = 197807400 (188.64383697509766MB)
>> 17.84699532929879% used
>> From Space:
>> capacity = 1769472 (1.6875MB)
>> used = 1736704 (1.65625MB)
>> free = 32768 (0.03125MB)
>> 98.14814814814815% used
>> To Space:
>> capacity = 13893632 (13.25MB)
>> used = 0 (0.0MB)
>> free = 13893632 (13.25MB)
>> 0.0% used
>> PS Old Generation
>> capacity = 3925868544 (3744.0MB)
>> used = 3600153552 (3433.373977661133MB)
>> free = 325714992 (310.6260223388672MB)
>> 91.70336478795761% used
>> PS Perm Generation
>> capacity = 268435456 (256.0MB)
>> used = 191475048 (182.6048355102539MB)
>> free = 76960408 (73.3951644897461MB)
>> 71.33001387119293% used
>>
>> it seems that the heap usage is not that high enough to occupy swap
>> space. weird. And I also checked the nio usage, it was quite trivial:
>> [root at l-tw14 ~]#java -classpath .:$JAVA_HOME/lib/sa-jdi.jar
>> DirectMemorySize `pgrep java`
>> Attaching to process ID 677, please wait...
>> Debugger attached successfully.
>> Server compiler detected.
>> JVM version is 20.1-b02
>> NIO direct memory:
>> reserved size = 0.357348MB (374707 bytes)
>> max size = 3968.000000MB (4160749568 bytes)
>>
>> This problem bothered me quite a few days, and I appreciate all your
>> help. Thanks in advance
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120305/0cefee5e/attachment.html
More information about the hotspot-gc-use
mailing list