High termination times pre-concurrent cycle in G1

[Thomas Schatzl]

Hi,

On Fri, 2016-09-02 at 15:08 +0200, William Good wrote:
> Thomas,
> 
> More than happy to build and test with a patched JDK. Let me know
> what I need to do, at least what sources to grab (I think I can build
> it once I've got them).
> 
> I learned this morning that my belief that significant tenuring
> wasn't taking place was possibly wrong. Our problem seems to have
> disappeared with an akka upgrade [1] and my guess for the relevant
> fix [2] indicates that unintended tenuring was occurring at some
> point (teh tenuring has never been significant enough for us to
> notice). However as long as I'm unable to reproduce with the new
> version I'm happy to continue testing using the older version I know
> to reproduce, as I don't think this G1 behavior is intended.

  this bug report makes me tend to believe that the suggested fix (for
JDK-8152438) will actually fix the issue. Let me explain:

Every G1 thread has two work queues, one fixed size public one where
others can steal from, and one resizable one that is private. Work
(references) is first put into the public one, and then if it is full,
into the private one.
Threads first process their private buffers, so that others can steal
from the public one while they are working on it.
Due to some conditions it can happen that the public queues are already
completely empty, while one thread is still busy for a long time with
its private one. I.e. the work in the private queue can be so that it
never generates more work in the public queue that others can steal and
continue work from. So they wait.

That can cause this high termination time. Of course this situation can
occur multiple times during a GC, so that every thread gets his fair
share of waiting :)

That mentioned fix has been pushed into the http://hg.openjdk.java.net/
jdk8u/jdk8u-dev/ repository. It should be a matter of pulling it.
The README-builds file in the repo has build instructions.

It would be really nice if we could track down your problem to this
issue, or at least significantly improve it. (JDK9 has more significant
patches in that area iirc, but this one is probably the most
important).

Thanks,
  Thomas