<p>Apparently pgpgin/pgpgout may not be that accurate to determine swap file usage: <a href="http://help.lockergnome.com/linux/pgpgin-pgpgout-measure--ftopict506279.html">http://help.lockergnome.com/linux/pgpgin-pgpgout-measure--ftopict506279.html</a></p>
<p>May need to use vmstat and look at si/so instead.</p>
<div class="gmail_quote">On Jan 10, 2012 12:24 AM, "Chi Ho Kwok" <<a href="mailto:chkwok@digibites.nl" target="_blank">chkwok@digibites.nl</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Florian,<div><br></div><div>Uh, you might want to try sar -r as well, that reports memory usage (and man sar for other reporting options, and -f /var/log/sysstat/saXX where xx is the day for older data is useful as well). Page in / out means reading or writing to the swap file, usual cause here is one or more huge background task / cron jobs taking up too much memory forcing other things to swap out. You can try reducing the size of the heap and see if it helps if you're just a little bit short, but otherwise I don't think you can solve this with just VM options.</div>
<div><br></div><div><br></div><div>Here's the relevant section from the manual:</div><div><br></div><div><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
-B Report paging statistics. Some of the metrics below are available only with post 2.5 kernels. The following values are displayed:<br> pgpgin/s<br> Total number of kilobytes the system paged in from disk per second. Note: With old kernels (2.2.x) this value is a number of blocks<br>
per second (and not kilobytes).<br> pgpgout/s<br> Total number of kilobytes the system paged out to disk per second. Note: With old kernels (2.2.x) this value is a number of blocks<br>
per second (and not kilobytes).<br> fault/s<br> Number of page faults (major + minor) made by the system per second. This is not a count of page faults that generate I/O, because<br>
some page faults can be resolved without I/O.<br> majflt/s<br> Number of major faults the system has made per second, those which have required loading a memory page from disk.</blockquote>
</div><div><br></div><div>I'm not sure what kernel you're on, but pgpgin / out being high is a bad thing. Sar seems to report that all faults are minor, but that conflicts with the first two columns.</div><div><br>
</div><div><br></div><div>Chi Ho Kwok</div><div><br><div class="gmail_quote">On Mon, Jan 9, 2012 at 8:47 PM, Florian Binder <span dir="ltr"><<a href="mailto:java@java4.info" target="_blank">java@java4.info</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Yes!<br>
You are right!<br>
I have a lot of page faults when gc is taking so much time.<br>
<br>
For example (sar -B):<br>
00:00:01 pgpgin/s pgpgout/s fault/s majflt/s<br>
00:50:01 0,01 45,18 162,29 0,00<br>
01:00:01 0,02 46,58 170,45 0,00<br>
01:10:02 25313,71 27030,39 27464,37 0,02<br>
01:20:02 23456,85 25371,28 13621,92 0,01<br>
01:30:01 22778,76 22918,60 10136,71 0,03<br>
01:40:11 19020,44 22723,65 8617,42 0,15<br>
01:50:01 5,52 44,22 147,26 0,05<br>
<br>
What is this meaning and how can I avoid it?<br>
<br>
<br>
Flo<br>
<br>
<br>
<br>
Am 09.01.2012 20:33, schrieb Chi Ho Kwok:
<div><div><blockquote type="cite">Just making sure the obvious case is covered: is it
just me or is 6s real > 3.5s user+sys with 13 threads just
plain weird? That means there was 0.5 thread actually running on
the average during that collection.
<div><br>
</div>
<div>Do a sar -B (requires package sysstat) and see if there were
any major pagefaults (or indirectly via cacti and other
monitoring tools via memory usage, load average etc, or even via
cat /proc/vmstat and pgmajfault), I've seen those cause these
kind of times during GC.</div>
<div><br>
</div>
<div><br>
</div>
<div>Chi Ho Kwok<br>
<div><br>
</div>
<div>
<div class="gmail_quote">On Mon, Jan 9, 2012 at 12:08 PM,
Florian Binder <span dir="ltr"><<a href="mailto:java@java4.info" target="_blank">java@java4.info</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi
everybody,<br>
<br>
I am using CMS (with ParNew) gc and have very long (> 6
seconds) young<br>
gc pauses.<br>
As you can see in the log below the old-gen-heap consists
of one large<br>
block, the new Size has 256m, it uses 13 worker threads
and it has to<br>
copy 27505761 words (~210mb) directly from eden to old
gen.<br>
I have seen that this problem occurs only after about one
week of<br>
uptime. Even thought we make a full (compacting) gc every
night.<br>
Since real-time > user-time I assume it might be a
synchronization<br>
problem. Can this be true?<br>
<br>
Do you have any Ideas how I can speed up this gcs?<br>
<br>
Please let me know, if you need more informations.<br>
<br>
Thank you,<br>
Flo<br>
<br>
<br>
##### java -version #####<br>
java version "1.6.0_29"<br>
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)<br>
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed
mode)<br>
<br>
##### The startup parameters: #####<br>
-Xms28G -Xmx28G<br>
-XX:+UseConcMarkSweepGC \<br>
-XX:CMSMaxAbortablePrecleanTime=10000 \<br>
-XX:SurvivorRatio=8 \<br>
-XX:TargetSurvivorRatio=90 \<br>
-XX:MaxTenuringThreshold=31 \<br>
-XX:CMSInitiatingOccupancyFraction=80 \<br>
-XX:NewSize=256M \<br>
<br>
-verbose:gc \<br>
-XX:+PrintFlagsFinal \<br>
-XX:PrintFLSStatistics=1 \<br>
-XX:+PrintGCDetails \<br>
-XX:+PrintGCDateStamps \<br>
-XX:-TraceClassUnloading \<br>
-XX:+PrintGCApplicationConcurrentTime \<br>
-XX:+PrintGCApplicationStoppedTime \<br>
-XX:+PrintTenuringDistribution \<br>
-XX:+CMSClassUnloadingEnabled \<br>
-Dsun.rmi.dgc.server.gcInterval=9223372036854775807 \<br>
-Dsun.rmi.dgc.client.gcInterval=9223372036854775807 \<br>
<br>
-Djava.awt.headless=true<br>
<br>
##### From the out-file (as of +PrintFlagsFinal): #####<br>
ParallelGCThreads = 13<br>
<br>
##### The gc.log-excerpt: #####<br>
Application time: 20,0617700 seconds<br>
2011-12-22T12:02:03.289+0100: [GC Before GC:<br>
Statistics for BinaryTreeDictionary:<br>
------------------------------------<br>
Total Free Space: 1183290963<br>
Max Chunk Size: 1183290963<br>
Number of Blocks: 1<br>
Av. Block Size: 1183290963<br>
Tree Height: 1<br>
Before GC:<br>
Statistics for BinaryTreeDictionary:<br>
------------------------------------<br>
Total Free Space: 0<br>
Max Chunk Size: 0<br>
Number of Blocks: 0<br>
Tree Height: 0<br>
[ParNew<br>
Desired survivor size 25480392 bytes, new threshold 1 (max
31)<br>
- age 1: 28260160 bytes, 28260160 total<br>
: 249216K->27648K(249216K), 6,1808130 secs]<br>
20061765K->20056210K(29332480K)After GC:<br>
Statistics for BinaryTreeDictionary:<br>
------------------------------------<br>
Total Free Space: 1155785202<br>
Max Chunk Size: 1155785202<br>
Number of Blocks: 1<br>
Av. Block Size: 1155785202<br>
Tree Height: 1<br>
After GC:<br>
Statistics for BinaryTreeDictionary:<br>
------------------------------------<br>
Total Free Space: 0<br>
Max Chunk Size: 0<br>
Number of Blocks: 0<br>
Tree Height: 0<br>
, 6,1809440 secs] [Times: user=3,08 sys=0,51, real=6,18
secs]<br>
Total time for which application threads were stopped:
6,1818730 seconds<br>
_______________________________________________<br>
hotspot-gc-use mailing list<br>
<a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br>
<a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>
<br>_______________________________________________<br>
hotspot-gc-use mailing list<br>
<a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br>
<a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a><br>
<br></blockquote></div>