The GCLocker strikes again (bad remark / GCLocker interaction)

Thu Sep 4 15:20:39 UTC 2014

Hi Ramki,

Thanks for taking the time to reply! To follow-up on a couple of points 
in your e-mail:

"A ticklish dilemma here is whether the CMS thread should wait for the 
JNI CS to clear or just plough on as is the case today. The thinking 
there was that it's better to have a longish remark pause because of not 
emptying Eden than to delay the CMS collection and risk a concurrent 
mode failure which would be much more expensive."

Yeah, it's an interesting trade-off. You're right that holding up the 
remark while waiting for the GCLocker to drain could cause a concurrent 
mode failure. And, yes, the remark could be loonger too (more cards can 
be potentially dirtied while we're waiting). On the other hand, if the 
critical sections are properly written (i.e., they are bounded / they 
don't block; and in our case they definitely seem to be) the remark will 
only be delayed by a very short amount of time. Of course, as we all 
know, the above "if" is really big "IF". :-) But, we should not be 
penalizing correctly behaving code to make sure misbehaving code doesn't 
misbehave even more (read this like: we can put the new behavior on a flag).

Regarding a concurrent mode failure being more expensive than a longer 
remark: Of course! But, our remarks when a scavenge fails tend to be two 
orders of magnitude longer than the scavenge suceeds, so quite 
disruptive. From a small test run on my MacBook that's trying to cause 
this often: with a 3G young gen, normal remarks are 20ms-25ms, when the 
scavenge fails remarks last over 1 sec.

"As you alluded to in your email, the issue is a bit tricky"

Oh, yes. This is why, unlike JDK-8048556 (the other GCLocker issue), I 
didn't attempt to immediately work on a fix and opted to discuss this 
first on the list.

"I suspect the correct way to deal with this one and for all in a 
uniform manner might be to have vm ops that need a vacant gc-locker to 
be enqueued on a separate vm-ops queue whose operations are executed as 
soon as the gc-locker has been vacated"

I do like the idea of having a way for a VM op to indicate that it 
requires the GCLocker to be inactive when the threads stop. This will 
add all the logic "in one place" (OK, spread out on the GCLocker, VM 
thread, etc. but you know what I mean) and we will be able to re-use it 
wherever we need it, instead of re-inventing the wheel everywhere we 
need this.

BTW, in other but related news :-), explicit GCs also suffer from this 
problem. So, it's possible for a System.gc() to be completely ignored if 
the GCLocker is active during the safepoint (I can reproduce this with 
both CMS and ParOld). Like remark, the code that schedules the 
System.gc() VM op also doesn't synchronize properly with the GCLocker. 
And, yes, instead of a Full GC this also causes a GCLocker-induced young 
GC with a non-full eden (for the same reason it happens after the 
remarks with the failed scavenge which I described in my original 
e-mail). This is another use for the "this VM op should only be 
scheduled when the GCLocker is inactive" feature.

"All this having been said, I am slightly surprised that remark pauses 
for large Edens are so poor."

They are quite large edens. :-)

"Have you looked at the distribution of Eden scanning times during the 
remark pause? Does Eden scanning dominate the remark cost?"

I assumed it did, the long remarks only happen when the eden is not 
empty. However, I haven't looked at the scanning times. Is there 
existing instrumentation that will tell us that? We can always add some 
ourselves.

"(I was also wondering if it might be possible to avoid using whatever 
was causing such frequent gc-locker activity as a temporary workaround 
until the issue w/CMS is fixed?)"

We're looking into this.

BTW, I'll create new CRs for the remark / GCLocker interaction and the 
System.gc() / GCLocker interaction to properly track both issues.

Tony

On 9/3/14, 3:11 PM, Srinivas Ramakrishna wrote:
> Hi Tony --
>
> The scavenge-before-remark bailing if the gc-locker is active was an 
> expedient solution and one that I did not expend much thought to
> as gc-lockers were considered infrequent enough not to affect the 
> bottom-line by much. I can imagine though that with very frequent 
> gc-locker
> activity and extremely large Edens, that this can be an issue. The 
> fact that the scavenge might bail was already considered as some of the
> comments in that section of code indicate. A ticklish dilemma here is 
> whether the CMS thread should wait for the JNI CS to clear or just plough
> on as is the case today. The thinking there was that it's better to 
> have a longish remark pause because of not emptying Eden than to delay the
> CMS collection and risk a concurrent mode failure which would be much 
> more expensive.
>
> As you alluded to in your email, the issue is a bit tricky because of 
> the way scavenge before remark is currently implemented ... CMS decides to
> do a remark, stops all the mutators, then decides that it must do a 
> scavenge, which now cannot be done because the gc-locker is held, so 
> we bail from
> the scavenge and just do the remark pause (this is safe because no 
> objects are moved). The whole set-up of CMS' vm-ops was predicated on the
> assumption of non-interference with other operations because these are 
> in some sense "read-only" wrt the heap, so we can safely
> schedule the safepoint at any time without any worries about moving 
> objects.
>
> Scavenge-before-remark is the only wrinkle in this otherwise flat and 
> smooth landscape.
>
> I suspect the correct way to deal with this one and for all in a 
> uniform manner might be to have vm ops that need a vacant gc-locker to
> be enqueued on a separate vm-ops queue whose operations are executed 
> as soon as the gc-locker has been vacated (this would likely
> be all the vm-ops other than perhaps a handful of CMS vm-ops today). 
> But this would be a fairly intrusive and delicate rewrite of the
> vm-op and gc-locker subsystems.
>
> A quicker point-solution might be to split the scavenge-and-remark 
> vm-op into two separate vm ops -- one that does a (guaranteed) scavenge,
> followed by another that does a remark -- each in a separate 
> safepoint, i.e. two separate vm-ops. One way to do this might be for 
> the CMS
> thread to take the jni critical lock, set needs_gc() if the gc locker 
> is active, and then wait on the jni critical lock for it to be cleared 
> (which it
> will be by the last thread exiting a JNI CS) which would initiate the 
> scavenge. If the gc locker isn't active, the scavenge can be initiated 
> straightaway
> by the CMS thread in the same way that a JNI thread would have 
> initiated it when it was the last one exiting a JNI CS.. Once the 
> scavenge has
> happened, the CMS thread can then do the remark in the normal way. 
> Some allocation would have happened in Eden between the scavenge and
> the remark to follow, but hopefully that would be sufficiently small 
> as not to affect the performance of the remark. The delicate part here 
> is the
> synchronization between gc locker state, the cms thread initiating the 
> vm-op for scavenge/remark and the jni threads, but this protocol would
> be identical to the existing one, except that the CMS thread would now 
> be a participant in that proctocol, which it never was before (this might
> call for some scaffolding in the CMS thread so it can participate).
>
> All this having been said, I am slightly surprised that remark pauses 
> for large Edens are so poor. I would normally expect that pointers 
> from young
> to old would be quite few and with the Eden being scanned 
> multi-threaded (at sampled "top" boundaries -- perhaps this should use 
> TLAB
> boundaries instead), we would be able to scale OK to larger Edens. 
> Have you looked at the distribution of Eden scanning times during the
> remark pause? Does Eden scanning dominate the remark cost? (I was also 
> wondering if it might be possible to avoid using whatever was
> causing such frequent gc-locker activity as a temporary workaround 
> until the issue w/CMS is fixed?)
>
> -- ramki
>
>
>
> On Tue, Sep 2, 2014 at 3:21 PM, Tony Printezis <tprintezis at twitter.com 
> <mailto:tprintezis at twitter.com>> wrote:
>
>     Hi there,
>
>     In addition to the GCLocker issue I've already mentioned
>     (JDK-8048556: unnecessary young GCs due to the GCLocker) we're
>     also hitting a second one, which in some ways is more severe in
>     some cases.
>
>     We use quite large edens and we run with
>     -XX:+CMSScavengeBeforeRemark to empty the eden before each remark
>     to keep remark times reasonable. It turns out that when the remark
>     pause is scheduled it doesn't try to synchronize with the GCLocker
>     at all. The result is that, quite often, the scavenge before
>     remark aborts because the GCLocker is active. This leads to
>     substantially longer remarks.
>
>     A side-effect of this is that the remark pause with the aborted
>     scavenge is immediately followed by a GCLocker-initiated GC (with
>     the eden being half empty). The aborted scavenge checks whether
>     the GCLocker is active with check_active_before_gc() which tells
>     the GCLocker to do a young GC if it's active. And the young GC is
>     done without waiting for the eden to fill up.
>
>     The issue is very easy to reproduce with a test similar to what I
>     posted on JDK-8048556 (force concurrent cycles by adding a thread
>     that calls System.gc() every say 10 secs and set
>     -XX:+ExplicitGCInvokesConcurrent). I can reproduce this with the
>     current hotspot-gc repo.
>
>     We were wondering whether this is a known issue and whether
>     someone is working on it. FWIW, the fix could be a bit tricky.
>
>     Thanks,
>
>     Tony
>
>     -- 
>     Tony Printezis | JVM/GC Engineer / VM Team | Twitter
>
>     @TonyPrintezis
>     tprintezis at twitter.com <mailto:tprintezis at twitter.com>
>
>

-- 
Tony Printezis | JVM/GC Engineer / VM Team | Twitter

@TonyPrintezis
tprintezis at twitter.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20140904/1bd911c8/attachment.htm>