Extremely long parnew/cms promotion failure scenario?

[Peter B. Kessler]

IIRC, promotion failure still has to finish the evacuation attempt (and some objects may get promoted while the ones that fail get self-looped).  That part is the usual multi-threaded object graph walk, with failed PLAB allocations thrown in to slow you down.  Then you get to start the pass that deals with the self-loops, which you say is single-threaded.  Undoing the self-loops is in address order, but it walks by the object sizes, so probably it mostly misses in the cache.  40GB at the average object size (call them 40 bytes to make the math easy) is a lot of cache misses.  How fast is your memory system?  Probably faster than (10minutes / (40GB / 40bytes)) per cache miss.

Is it possible you are paging?  Maybe not when things are running smoothly, but maybe a 10 minute stall on one service causes things to back up (and grow the heap of) other services on the same machine?  I'm guessing.

			... peter

Srinivas Ramakrishna wrote:
> 
> Has anyone come across extremely long (upwards of 10 minutes) promotion 
> failure unwinding scenarios when using any of the collectors, but 
> especially with ParNew/CMS?
> I recently came across one such occurrence with ParNew/CMS that, with a 
> 40 GB young gen took upwards of 10 minutes to "unwind". I looked through 
> the code and I can see
> that the unwinding steps can be a source of slowdown as we iterate 
> single-threaded (DefNew) through the large Eden to fix up self-forwarded 
> objects, but that still wouldn't
> seem to explain such a large pause, even with a 40 GB young gen. I am 
> looking through the promotion failure paths to see what might be the 
> cause of such a large pause,
> but if anyone has experienced this kind of scenario before or has any 
> conjectures or insights, I'd appreciate it.
> 
> thanks!
> -- ramki
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use