Java application getting paused when pfiles or pstack command run for it
Poonam Bajaj Parhar
poonam.bajaj at oracle.com
Wed Sep 7 16:58:25 UTC 2016
Hello Amit,
If you suspect that the JVM is hung, then you could try jstack -F to
collect the stack traces. And if that too does not work then try
collecting core file, and use jstack and/or native debuggers to collect
the stack traces including native frames to see what is going on.
Thanks,
Poonam
On 9/7/2016 5:59 AM, Charlie Hunt wrote:
> Your description sounds like it may be distributed GC running at its
> default, every 60 minutes. A look at GC logs would help confirm
> that's the case.
>
> GC logs are usually the best place to start looking when looking for
> sources of application pauses.
>
> If it turns out to be distributed GC, you can change the frequency at
> which it runs by adding the following system properties:
> -Dsun.rmi.dgc.client.gcInterval=<milliseconds>
> -Dsun.rmi.dgc.server.gcInterval=<milliseconds>
>
> As an extreme example, you could effectively tell distributed GC to
> not run by setting:
> -Dsun.rmi.dgc.client.gcInterval=Long.MAX_VALUE
> -Dsun.rmi.dgc.server.gcInterval=Long.MAX_VALUE
>
> Or, you could add the following command line option:
> -XX:+DisableExplicitGC
>
> hths,
>
> charlie
>
> ----- Original Message -----
> From: amit.mishra at redknee.com
> To: charlie.hunt at oracle.com
> Cc: hotspot-gc-use at openjdk.java.net
> Sent: Wednesday, September 7, 2016 7:08:23 AM GMT -06:00 US/Canada Central
> Subject: RE: Java application getting paused when pfiles or pstack
> command run for it
>
> Thank you very much Charles, but initially Customer reported a
> Production issue where java Application process hangs periodically for
> a few seconds at fixed point of time of every hour.(say
> 17:41,18:41,19:41).
>
> We tried to investigate it by taking TD’s to see where application
> threads are blocked but TD didn’t came up due to which we conclude
> that it is JVM freeze.
>
> Further to analyze that we were supposed to analyze process threads
> using pstack but as pstack itself is causing application pause so in
> your opinion what are the best commands/tools to analyze JVM freeze
> when Thread dumps stopped coming.
>
> Regards,
>
> Amit
>
> *From:*charlie hunt [mailto:charlie.hunt at oracle.com]
> *Sent:* Wednesday, September 7, 2016 17:29
> *To:* Amit Mishra <amit.mishra at redknee.com>
> *Cc:* hotspot-gc-use at openjdk.java.net
> *Subject:* Re: Java application getting paused when pfiles or pstack
> command run for it
>
> Hi Amit,
>
> pfiles, pstack and pldd stops the process while they do their work.
>
> The following is directly from the p-tools (pfiles,etc.) man page, and
> from Oracle docs on p-tools:
>
> These proc tools (p-tools) stop their target processes while
> inspecting them and reporting the results: files, pled, mmap, and
> stack. A process can do nothing while it is stopped.
>
> As for why it may take up to 15 seconds? It may be that there are a
> large number file descriptors in use, etc. or a (large) portion of
> that time is spent getting all threads in the JVM and app to come to
> stopped state, or a combination of both.
>
> hths,
>
> charlie
>
> On Sep 7, 2016, at 5:33 AM, Amit Mishra <amit.mishra at redknee.com
> <mailto:amit.mishra at redknee.com>> wrote:
>
> Hello Charlie/team,
>
> I need your expert help on one of Production issue whereas when
> pfiles <JavaappPID> command whenever runs then it cause java
> application freeze until command get completed.(it is causing long
> pauses of up-to 15 seconds)
>
> During this whole time situations appears as JVM freeze as TD’s
> for Application PID stopped generating, GC logs are getting paused
> and application stopped catering traffic.
>
> What could be the cause of it, Application Java version is
> 1.6u45.(Application PID is 18387)
>
> 3 samples from Top command when pfiles ,pldd and pstack command
> cause Application freeze(as TD’s was also not coming we treated
> this as JVM freeze).
>
> ========================================================
>
> 1^st Sample:
>
> load averages: 1.11, 1.00, 0.98; up
> 68+20:03:01 21:41:59
>
> 94 processes: 91 sleeping, 1 stopped, 2 on cpu
>
> CPU states: 85.6% idle, 0.4% user, 14.1% kernel, 0.0% iowait,
> 0.0% swap
>
> Memory: 64G phys mem, 30G free mem, 16G total swap, 16G free swap
>
> PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
>
> 5251 root 1 0 17 2872K 1396K cpu/2 0:00 6.82%pfiles
> 18387
>
> 18387rkadm 999 0 0 29G 23Gstop 878.5H 5.73%
> /usr/Java_1.6_45/bin/amd64/java
> -Djava.util.logging.manager=com.redknee.framewo
>
> ======================================================================
>
> 2^nd Sample:
>
> PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
>
> 10145 root 1 0 17 2620K 1676K cpu/2 0:00 3.90%pldd 18387
>
> 5352 root 4 6 17 31M 27M sleep 0:01 3.33%
> pkgserv -N pkgchk
>
> 18387rkadm 999 59 0 29G 23Gstop 878.5H 0.82%
> /usr/Java_1.6_45/bin/amd64/java
> -Djava.util.logging.manager=com.redknee.framewo
>
> =================================================================
>
> 3^rd Sample
>
> load averages: 1.21, 1.16, 1.15; up
> 68+18:02:37 19:41:35
>
> 88 processes: 85 sleeping, 1 stopped, 2 on cpu
>
> CPU states: 85.9% idle, 0.2% user, 13.8% kernel, 0.0% iowait,
> 0.0% swap
>
> Memory: 64G phys mem, 30G free mem, 16G total swap, 16G free swap
>
> PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
>
> 1250 rkadm 1 30 0 28M 11M cpu/1 0:09 12.49%pstack 18387
>
> 19859 rkadm 392 59 0 3322M 2342M sleep 69.6H 0.46%
> /usr/Java_1.6_45/bin/java -classpath
> /usr/Java_1.6_45/lib/tools.jar:/opt/redkne
>
> 1257 rkadm 1 59 0 3128K 1756K cpu/3 0:00 0.05% top -c
> -d 1 -s 1 100
>
> 1024 root 12 6 17 154M 109M sleep 118:09 0.01%
> /opt/IBM/SCM/client/../_jvm/bin/java -Xint -Xmx128m
> -Djlog.logCmdPort=1953 -Dsu
>
> 15036 root 32 59 0 138M 74M sleep 85:05 0.01%
> /usr/java/bin/java -Dviper.fifo.path=/var/run/smc898/boot.fifo
> -Xmx128m -Dsun.s
>
> 1075 noaccess 19 59 0 97M 91M sleep 69:05 0.01%
> /usr/java/bin/java -server -Xmx128m -XX:+UseParallelGC
> -XX:ParallelGCThreads=4
>
> 18285 rkadm 23 59 0 110M 53M sleep 42:39 0.01% orbd
> -ORBInitialPort 20000 -port 20100
>
> 18289 rkadm 21 59 0 105M 45M sleep 42:02 0.01% orbd
> -ORBInitialPort 21000 -port 21100 /opt/redknee/log/rkctl.log
>
> 185 root 1 59 0 2472K 1108K sleep 46:38 0.01%
> /usr/lib/inet/in.mpathd -a
>
> 1254 root 43 59 0 47M 22M sleep 12:32 0.00%
> /opt/IBM/ITM/sol606/ul/bin/kulagent
>
> 1467 root 44 59 0 61M 32M sleep 157:12 0.00%
> /opt/IBM/ITM/sol606/ux/bin/kuxagent
>
> 665 root 1 59 0 9084K 2596K sleep 1:41 0.00%
> /usr/lib/sendmail -bd -q15m
>
> 7 root 14 59 0 12M 10M sleep 1:58 0.00%
> /lib/svc/bin/svc.startd
>
> 502 root 1 59 0 1444K 748K sleep 0:18 0.00% /usr/lib/utmpd
>
> 676 root 1 100 -20 2652K 1424K sleep 5:32 0.00%
> /usr/lib/inet/xntpd
>
> 574 root 4 59 0 6412K 3068K sleep 4:19 0.00%
> /usr/lib/inet/inetd start
>
> 643 root 1 59 0 2468K 1304K sleep 0:00 0.00%
> /usr/lib/snmp/snmpdx -y -c /etc/snmp/conf
>
> 4933 root 2 59 0 590M 578M sleep 499:41 0.00%
> /usr/bin/dsmc schedule
>
> 5231 root 7 59 0 44M 17M sleep 2:59 0.00%
> /opt/IBM/ITM/sol606/ux/bin/kcawd
>
> 18387 rkadm 999 59 0 29G 23Gstop 877.2H 0.00%
> /usr/Java_1.6_45/bin/amd64/java
> -Djava.util.logging.manager=com.redknee.framewo
>
> Thanks,
>
> Amit Mishra
>
>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20160907/2bd6f3f4/attachment.html>
More information about the hotspot-gc-use
mailing list