Very slow promotion failures in ParNew / ParallelGC

[Mikael Gerdin]

Hi Tony,

On 2016-01-12 19:15, Tony Printezis wrote:
> Thomas,
>
> Inline.
>
> On January 12, 2016 at 7:00:45 AM, Thomas Schatzl
> (thomas.schatzl at oracle.com <mailto:thomas.schatzl at oracle.com>) wrote:
>
>> Hi,
>>
>> On Mon, 2016-01-11 at 12:59 -0500, Tony Printezis wrote:
>> > Hi all,
>> >
>> > We have been recently investigating some very lengthy (several
>> > minutes) promotion failures in ParNew, which also appear in
>> > ParallelGC. We have identified a few issues and have some fixes to
>> > address them. Here's a quick summary:
>> >
>> > 1) There's a scalability bottleneck when adding marks to the
>> > preserved mark stack as there is only one stack, shared by all
>> > workers, and pushes to it are protected by a mutex. This essentially
>> > serializes all workers if there is a non-trivial amount of marks to
>> > be preserved. The fix is similar to what's been implemented in G1 in
>> > JDK 9, which is to introduce per-worker preserved mark stacks.
>> >
>> > 2) (More interestingly) I was perplexed by the huge number of marks
>> > that I see getting preserved during promotion failure. I did a small
>> > study with a test I can reproduce the issue with. The majority of the
>> > preserved marks were 0x5 (i.e. "anonymously biased"). According to
>> > the current logic, no mark is preserved if it's biased, presumably
>> > because it's assumed that the object is biased towards a specific
>> > thread and we want to preserve that mark as it contains the thread
>> > pointer.
>>
>> I think the reason is that nobody ever really measured the impact of
>> biased locking on promotion failures, and so never considered it.
>
>
> I bet. :-)
>
>
>>
>> > The fix is to use a different default mark value when biased locking
>> > is enabled (0x5) or disabled (0x1, as it is now). During promotion
>>
>> > failures, marks are not preserved if they are equal to the default
>> > value and the mark of forwarded objects is set to the default value
>> > post promotion failure and before the preserved marks are re
>> > -instated.
>>
>> You mean the value of the mark as it is set during promotion failure
>> for the new objects?
>
>
> Not sure what you mean by “for new objects”.
>
> Current state: When we encounter promotion failures, we check whether
> the mark is the default (0x1). If it is, we don’t preserve it. If it is
> not, we preserve it. After promotion failure, we iterate over the young
> gen and set the mark of all objects (ParNew) or all forwarded objects
> (ParallelGC) to the default (0x1), then apply all preserved marks.
>
> What I’m proposing is that in the process I just described, the default
> mark will be 0x5, if biased locking is enabled (as most objects will be
> expected to have a 0x5 mark) and 0x1, if biased locking is disabled (as
> it is the case right now).

I'm thinking that we could check if the current header matches the 
prototype header for the class then we would not need to preserve it.

This would hopefully let us avoid saving/restoring anonymously biased 
marks at least.

/Mikael

>
>
>>
>> Did some very quick measurements on the distribution of marks on a few
>> certainly also non-representative workloads and can see your point.
>
>
> I also did that for synthetic tests and I see the same. I’ll try to get
> some data from production.
>
>
>>
>> When running without biased locking, the amount of preserved marks is
>> even lower.
>
>
> Of course, because the the most populous mark will be 0x1 when biased
> locking is disabled, not 0x5. The logic of whether to preserve a mark or
> not was taken before biased locking was introduced, when most objects
> would have a 0x1 mark. Biased locking changed this behavior and most
> objects have a 0x5 mark, which invalidated the original assumptions.
>
>
>> That may be an option in some cases in addition to these suggested
>> changes.
>
>
> Not sure what you mean.
>
>
>>
>> > A few extra observations on this:
>> >
>> > - I don't know if the majority of objects we'll come across during
>> > promotion failures will be anonymously biased (it is the case for
>> > synthetic benchmarks). So, the above might pay off in certain cases
>> > but not all. But I think it's still worth doing.
>>
>> I tend to agree since after looking through the biased locking code a
>> bit, it seems that by default new objects are anonymously biased with
>> biased locking on, so this will most likely help decreasing the amount
>> of marks to preserved.
>
>
> Yes, I agree with this.
>
>
>>
>> > - Even though the per-worker preserved mark stacks eliminate the big
>> > scalability bottleneck, reducing (potentially dramatically) the
>> > number of marks that are preserved helps in a couple of ways: a)
>> > avoids allocating a lot of memory for the preserved mark stacks
>> > (which can get very, very large in some cases) and b) avoids having
>> > to scan / reclaim the preserved mark stacks post promotion failure,
>> > which reduces the overall GC time further. Even the parallel time in
>> > ParNew improves by a bit because there are a lot fewer stack pushes
>> > and malloc calls.
>>
>> ... during promotion failure.
>
>
> Yes, I’m sorry I was not clear. ParNew times improve a bit when they
> encounter promotion failures.
>
>
>>
>> > 3) In the case where lots of marks need to be preserved, we found
>> > that using 64K stack segments, instead of 4K segments, speeds up the
>> > preserved mark stack reclamation by a non-trivial amount (it's 3x/4x
>> > faster).
>>
>> In my tests some time ago, increasing stack segment size only helped a
>> little, not 3x/4x times though as reported after implementing the per
>> -thread preserved stacks.
>
>
> To be clear: it’s only the reclamation of the preserved mark stacks I’ve
> seen improve by 3x/4x. Given all the extra work we have to do (remove
> forwarding references, apply preserved marks, etc.) this is a very small
> part of the GC when a promotion failure happens. But, still...
>
>
>>
>> A larger segment size may be a better trade-off for current, larger app
>> lications though.
>
>
> Is there any way to auto-tune the segment size? So, the larger the stack
> grows, the larger the segment size?
>
>
>>
>> > We have fixes for all three issues above for ParNew. We're also going
>> > to implement them for ParallelGC. For JDK 9, 1) is already
>> > implemented, but 2) or 3) might also be worth doing.
>> >
>> > Is there interest in these changes?
>
>
> OK, as I said to Jon, I’ll have the ParNew changes ported to JDK 9 soon.
> Should I create a new CR per GC (ParNew and ParallelGC) for the
> per-worker preserved mark stacks and we’ll take it from there?
>
> Tony
>
>
>>
>> Yes.
>>
>> Thanks,
>> Thomas
>>
>
>
>
>
>
>
>
>
>
>
> -----
>
> Tony Printezis | JVM/GC Engineer / VM Team | Twitter
>
> @TonyPrintezis
> tprintezis at twitter.com <mailto:tprintezis at twitter.com>
>