hotspot-gc-use Digest, Vol 88, Issue 1

[Wang, Yanping]

Hi, Simone

Is the application you mentioned related to HDFS FIS and FOS?
>From the log, there are 31647 FinalReferences.
I think the first pause data after SoftReference includes overheads to Ref Proc. Maybe those FinalReferences are the problem?

One suggestion is, you can use jmap -dump:format=b,file=/home/test.hprof <PID> to collect heap profile, then use Eclipse MAT: Memory Analyzer (http://www.eclipse.org/mat/downloads.php)   to open .hprof file, Open Query Browser -> Java Basics -> References to see where those references come from. 

Thanks
-yanping

-----Original Message-----
From: hotspot-gc-use [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of hotspot-gc-use-request at openjdk.java.net
Sent: Monday, July 20, 2015 12:49 PM
To: hotspot-gc-use at openjdk.java.net
Subject: hotspot-gc-use Digest, Vol 88, Issue 1

Send hotspot-gc-use mailing list submissions to
	hotspot-gc-use at openjdk.java.net

To subscribe or unsubscribe via the World Wide Web, visit
	http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
or, via email, send a message with subject or body 'help' to
	hotspot-gc-use-request at openjdk.java.net

You can reach the person managing the list at
	hotspot-gc-use-owner at openjdk.java.net

When replying, please edit your Subject line so it is more specific
than "Re: Contents of hotspot-gc-use digest..."


Today's Topics:

   1. Re: Long Reference Processing Time (Tao Mao)
   2. G1: SoftReference, 0 refs, 31.0027203 secs (Simone Bordet)


----------------------------------------------------------------------

Message: 1
Date: Tue, 23 Jun 2015 17:42:19 -0700
From: Tao Mao <yiyeguhu at gmail.com>
To: Simone Bordet <simone.bordet at gmail.com>
Cc: "hotspot-gc-use at openjdk.java.net"
	<hotspot-gc-use at openjdk.java.net>
Subject: Re: Long Reference Processing Time
Message-ID:
	<CANrGW1zgEOX1+D=QeycMQA9mLMHbBoNCLRnwOpW3MYHvxhyvAQ at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Or, give Java Mission Control a try! -Tao

On Wed, May 20, 2015 at 3:52 PM, Simone Bordet <simone.bordet at gmail.com>
wrote:

> Hi,
>
> On Thu, May 21, 2015 at 12:41 AM, Joy Xiong <joyxiong at yahoo.com> wrote:
> > Is there other ways for this? It's a prod environment and it would be too
> > intrusive for a heap dump...
>
> We used our solution in production too, by enabling it for few minutes
> to collect data (via JMX) and then disabling it until the next restart
> (also via JMX), where it was removed.
> Required 2 restarts: one to add the instrumentation, and one to remove it.
>
> --
> Simone Bordet
> http://bordet.blogspot.com
> ---
> Finally, no matter how good the architecture and design are,
> to deliver bug-free software with optimal performance and reliability,
> the implementation technique must be flawless.   Victoria Livschitz
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20150623/83138a86/attachment-0001.html>

------------------------------

Message: 2
Date: Thu, 16 Jul 2015 10:16:34 +0200
From: Simone Bordet <simone.bordet at gmail.com>
To: "'hotspot-gc-use at openjdk.java.net'
	(hotspot-gc-use at openjdk.java.net)"	<hotspot-gc-use at openjdk.java.net>
Subject: G1: SoftReference, 0 refs, 31.0027203 secs
Message-ID:
	<CAFWmRJ2SEiFfc_tBqixN2mHGEYFNSF_ttH-XNY+o+eva7+_zQw at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

I have an application that reported very large (around 30 s) times to
process *zero* SoftReferences, for example:

6015.665: [SoftReference, 0 refs, 23.0169525 secs]6038.682:
[WeakReference, 1 refs, 0.0046033 secs]6038.687: [FinalReference,
31647 refs, 0.0090301 secs]6038.696: [PhantomReference, 241 refs,
0.0048419 secs]6038.701: [JNI Weak Reference, 0.0000463 secs],
23.2166772 secs]

We have been hit by this anomaly a few times now, and in the attached
logs (that also show the command line flags) it happened 3 times: at
uptimes 6015.512, 6074.487, 6141.161.

What happens after these long pauses is that G1 goes into "GC overhead
mode", tries to expand the heap (which fails because it's already
expanded), but keeps the Eden at a very small size, resulting in a
series of back-to-back collections that lasted almost 3 minutes where
the MMU dropped making the application almost unusable.
After that, G1 was able to recover to normal behavior.

I was wondering if anyone knows a little more about this issue (long
times to process zero soft references), or whether it has been fixed
in more recent releases.

We are not aware of any other process that could have caused this such
as busy disk I/O or swapping (the machine has plenty of memory left
and it is dedicated to the JVM), but we'll run jHiccup next time.
However, the fact that it always happened during the processing of
soft references seems suspicious.

Should I file an issue ?

Thanks !

-- 
Simone Bordet
http://bordet.blogspot.com
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz
-------------- next part --------------
A non-text attachment was scrubbed...
Name: akiba-20150713-gc.log.gz
Type: application/x-gzip
Size: 594940 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20150716/c5f58489/akiba-20150713-gc.log.gz>

------------------------------

Subject: Digest Footer

_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


------------------------------

End of hotspot-gc-use Digest, Vol 88, Issue 1
*********************************************