100% cpu usage on CentOS/RHEL 6

[Amol Kulkarni]

Hi,

I have a java app running on CentOS 6.0. It always runs in background via
cron. Sometimes this app goes into wait state while using 100% cpu.

My java version is :

    java version "1.6.0_17"
    OpenJDK Runtime Environment (IcedTea6 1.7.4) (rhel-1.21.b17.el6-x86_64)
    OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)

Other symptoms are :

a. One thread of the process seems to be in a loop waiting for something.
When traced using strace, it shows following o/p continuously :

>     futex(0x7fb8000ac728, FUTEX_WAKE_PRIVATE, 1) = 0
>     futex(0x7fb8000ac754, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
1, {1340347489,> 822867000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

b. It seems like the process has finished working, looking at the files it
is using. Only few files are remaining. The output of 'ls /proc/pid/fd/
shows :

>     lr-x------ 1 root root 64 Jun 22 13:13 0 -> pipe:[77107601]
>     l-wx------ 1 root root 64 Jun 22 13:13 1 -> pipe:[77120162]
>     l-wx------ 1 root root 64 Jun 22 13:13 2 ->
/var/log/mithi/mcs/agent_account_mailstore_exceed_limit.sh.log
>     lr-x------ 1 root root 64 Jun 22 13:13 3 ->
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar


More specifically, are there any known problems in running openjdk based
Java processes in background on CentOS/RHEL 6?

I'm able to simulate the problem with a very simple script :

#!/bin/bash

while [ 1 ]
do
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java -version &
sleep 1s
done

When the above script is run for about 3 - 4 hours, one or two java
processes go in hang state with cpu usage touching 100%.


Has anybody faced similar situation?
Any clues or references would be very helpful.