<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Yeah, if a cache is being loaded and continuing to see heap occupancy grow … not sounding good. As you have said, other observations will not really matter if that’s the case.<div class=""><br class=""></div><div class="">thanks,</div><div class=""><br class=""></div><div class="">charlie</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On May 1, 2017, at 4:32 PM, Kirk Pepperdine <<a href="mailto:kirk@kodewerk.com" class="">kirk@kodewerk.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" class=""><div class=""><br class="Apple-interchange-newline">On May 1, 2017, at 11:19 PM, charlie hunt <<a href="mailto:charlie.hunt@oracle.com" class="">charlie.hunt@oracle.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hi Kirk,<div class=""><br class=""></div><div class="">A comment from me embedded below.</div><div class=""><br class=""></div><div class="">charlie</div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On May 1, 2017, at 3:42 PM, Kirk Pepperdine <<a href="mailto:kirk@kodewerk.com" class="">kirk@kodewerk.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">So, never mind all this.. I scrubbed all of the noise out of the log and what I see here is a Heap After GC chart that shows heap completely filled. Unless the Full GC cleaned things up, this VM isn’t going any further. Heap fills very quickly.<div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><blockquote type="cite" class=""><br class=""></blockquote></div><div class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div bgcolor="#FFFFFF" class=""><br class="">Remember set is the data structure that keep track of pointers from other regions to the region we are operating on. If there are pointers from old regions to the objects in the young regions, we can not collect that object.<br class="">After a gc pause, the live objects are evacuated to either the survivor or old gen regions. Then we need to update the Remember set. That is what I was referring to.<br class="">It is part of object copy.<br class=""><br class="">Like Kirk mentioned, the parallelism of the gc threads seems fine. Charlie.Hunt suggested maybe the object graph is very deep so that the gc threads can not steal work and end up spinning.<br class=""></div></blockquote></div></div></div></div></blockquote><div class=""><br class=""></div>I’d like to hear more from Charlie on this point because if the parallelism is ok then I would conclude that the work load is well balanced implying that there shouldn’t be any work stealing.. am prepared to be wrong on this but…. and, that said the number of attempts to shutdown suggest that there was something going on at the tail end of the collection and maybe work stealing was far to granular???? No idea….</div></div></div></div></blockquote><div class=""><br class=""></div>I haven’t had a chance to look at the logs in detail. That said, generally if we see high termination times it suggests that one or more of the GC threads are in a termination protocol where it is waiting for one or more other GC threads to finish their work. Hence my thought that there may be some part of the object graph that is really deep that is keeping one GC thread very occupied, i.e. much longer than the other GC threads.</div></div></div></div></blockquote><div class=""><br class=""></div>Ok, so we’re on the same page… If this were happening I’d expect that there would be some unbalance in the parallelization of the work load and I don’t see that in this case. However, I am using a heuristic to estimate this value and the heuristic could be a wee bit off. But then, all the GC threads seem to have trouble with termination so… don’t know what to say about this ‘cept it is a minor problem compared to object copy times but then… since heap is filling to full, I’d expect object copy times to be high and higher.<br class=""><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><div class=""><br class=""></div><div class="">Btw, thanks for jumping in and offering a detailed analysis … a full heap followed by a Full GC is not good, especially if it didn’t make much space available. :-|</div></div></div></div></blockquote><div class=""><br class=""></div>Don’t know, the record is corrupted.</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><div class=""><br class=""></div><div class="">Ooh, before I forget, did I see Nezih mention that THP was enabled? If that’s the case, I would disable it.</div></div></div></div></blockquote><div class=""><br class=""></div>Kept meaning to put that in but I don’t think it’s making a difference to the primary problem in this case.. the heap occupancy increases after each and every collection. Looks like a cache is being loaded.</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Regards,</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Kirk</div></div></blockquote></div><br class=""></div></body></html>