High termination times pre-concurrent cycle in G1

[yu.zhang at oracle.com]

William,

While this might not help the termination time, I hope adjusting the 
region size can get rid of some of the humongous objects.

It seems the humongous objects are filling old gen, about ~5g. The full 
gc can clean 10g. Can you try increase region size to 8m or 16m?

Thanks

Jenny


On 09/05/2016 03:02 AM, William Good wrote:
> Hi Thomas,
>
> Attached is a log file from a run with this morning's tip. I don't see
> any change unfortunately.
>
> I'll try with jdk9 tip later today since you mention there are more fixes there.
>
> configure summary looked like this, I think everything here is in order:
>
> Configuration summary:
> * Debug level:    release
> * JDK variant:    normal
> * JVM variants:   server
> * OpenJDK target: OS: linux, CPU architecture: x86, address length: 64
>
> Tools summary:
> * Boot JDK:       java version "1.8.0_101" Java(TM) SE Runtime
> Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM
> (build 25.101-b13, mixed mode)  (at /var/tmp/jdk1.8.0_101)
> * C Compiler:     gcc (GCC) 4.8.5 20150623 (Red Hat-4) version 4.8.5
> (at /usr/bin/gcc)
> * C++ Compiler:   g++ (GCC) 4.8.5 20150623 (Red Hat-4) version 4.8.5
> (at /usr/bin/g++)
>
> Best,
> William
>
>
> On Fri, Sep 2, 2016 at 4:01 PM, Thomas Schatzl
> <thomas.schatzl at oracle.com> wrote:
>> Hi,
>>
>> On Fri, 2016-09-02 at 15:08 +0200, William Good wrote:
>>> Thomas,
>>>
>>> More than happy to build and test with a patched JDK. Let me know
>>> what I need to do, at least what sources to grab (I think I can build
>>> it once I've got them).
>>>
>>> I learned this morning that my belief that significant tenuring
>>> wasn't taking place was possibly wrong. Our problem seems to have
>>> disappeared with an akka upgrade [1] and my guess for the relevant
>>> fix [2] indicates that unintended tenuring was occurring at some
>>> point (teh tenuring has never been significant enough for us to
>>> notice). However as long as I'm unable to reproduce with the new
>>> version I'm happy to continue testing using the older version I know
>>> to reproduce, as I don't think this G1 behavior is intended.
>>    this bug report makes me tend to believe that the suggested fix (for
>> JDK-8152438) will actually fix the issue. Let me explain:
>>
>> Every G1 thread has two work queues, one fixed size public one where
>> others can steal from, and one resizable one that is private. Work
>> (references) is first put into the public one, and then if it is full,
>> into the private one.
>> Threads first process their private buffers, so that others can steal
>> from the public one while they are working on it.
>> Due to some conditions it can happen that the public queues are already
>> completely empty, while one thread is still busy for a long time with
>> its private one. I.e. the work in the private queue can be so that it
>> never generates more work in the public queue that others can steal and
>> continue work from. So they wait.
>>
>> That can cause this high termination time. Of course this situation can
>> occur multiple times during a GC, so that every thread gets his fair
>> share of waiting :)
>>
>> That mentioned fix has been pushed into the http://hg.openjdk.java.net/
>> jdk8u/jdk8u-dev/ repository. It should be a matter of pulling it.
>> The README-builds file in the repo has build instructions.
>>
>> It would be really nice if we could track down your problem to this
>> issue, or at least significantly improve it. (JDK9 has more significant
>> patches in that area iirc, but this one is probably the most
>> important).
>>
>> Thanks,
>>    Thomas
>>