High termination times pre-concurrent cycle in G1

[yu.zhang at oracle.com]

William,

Thanks for the update. Gladly surprised as well.

One explanation is those humongous objects have pointers to the young 
gen. Now since we moved them to young gen by increasing region size, we 
do not need to update those pointers. In the tip.log, the update 
remember set and object copy time is not evenly distributed among the 
threads. Maybe g1 is having difficulties with those special kind of 
object arrays.

Can you get a heap dump? I am curious what kind of objects those are.

Thanks
Jenny
On 09/08/2016 08:35 AM, William Good wrote:
> Hi Jenny,
>
> Attached are logs for runs with 8m and 16m region sizes. This is with
> the old version of the application (before the akka patch). I was
> pleasantly surprised by the 16m run and tried it a few times, but it
> seems these results are reliable.
>
> Thanks,
> William
>
> On Tue, Sep 6, 2016 at 6:25 PM, yu.zhang at oracle.com <yu.zhang at oracle.com> wrote:
>> William,
>>
>> While this might not help the termination time, I hope adjusting the region
>> size can get rid of some of the humongous objects.
>>
>> It seems the humongous objects are filling old gen, about ~5g. The full gc
>> can clean 10g. Can you try increase region size to 8m or 16m?
>>
>> Thanks
>>
>> Jenny
>>
>>
>>
>> On 09/05/2016 03:02 AM, William Good wrote:
>>> Hi Thomas,
>>>
>>> Attached is a log file from a run with this morning's tip. I don't see
>>> any change unfortunately.
>>>
>>> I'll try with jdk9 tip later today since you mention there are more fixes
>>> there.
>>>
>>> configure summary looked like this, I think everything here is in order:
>>>
>>> Configuration summary:
>>> * Debug level:    release
>>> * JDK variant:    normal
>>> * JVM variants:   server
>>> * OpenJDK target: OS: linux, CPU architecture: x86, address length: 64
>>>
>>> Tools summary:
>>> * Boot JDK:       java version "1.8.0_101" Java(TM) SE Runtime
>>> Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM
>>> (build 25.101-b13, mixed mode)  (at /var/tmp/jdk1.8.0_101)
>>> * C Compiler:     gcc (GCC) 4.8.5 20150623 (Red Hat-4) version 4.8.5
>>> (at /usr/bin/gcc)
>>> * C++ Compiler:   g++ (GCC) 4.8.5 20150623 (Red Hat-4) version 4.8.5
>>> (at /usr/bin/g++)
>>>
>>> Best,
>>> William
>>>
>>>
>>> On Fri, Sep 2, 2016 at 4:01 PM, Thomas Schatzl
>>> <thomas.schatzl at oracle.com> wrote:
>>>> Hi,
>>>>
>>>> On Fri, 2016-09-02 at 15:08 +0200, William Good wrote:
>>>>> Thomas,
>>>>>
>>>>> More than happy to build and test with a patched JDK. Let me know
>>>>> what I need to do, at least what sources to grab (I think I can build
>>>>> it once I've got them).
>>>>>
>>>>> I learned this morning that my belief that significant tenuring
>>>>> wasn't taking place was possibly wrong. Our problem seems to have
>>>>> disappeared with an akka upgrade [1] and my guess for the relevant
>>>>> fix [2] indicates that unintended tenuring was occurring at some
>>>>> point (teh tenuring has never been significant enough for us to
>>>>> notice). However as long as I'm unable to reproduce with the new
>>>>> version I'm happy to continue testing using the older version I know
>>>>> to reproduce, as I don't think this G1 behavior is intended.
>>>>     this bug report makes me tend to believe that the suggested fix (for
>>>> JDK-8152438) will actually fix the issue. Let me explain:
>>>>
>>>> Every G1 thread has two work queues, one fixed size public one where
>>>> others can steal from, and one resizable one that is private. Work
>>>> (references) is first put into the public one, and then if it is full,
>>>> into the private one.
>>>> Threads first process their private buffers, so that others can steal
>>>> from the public one while they are working on it.
>>>> Due to some conditions it can happen that the public queues are already
>>>> completely empty, while one thread is still busy for a long time with
>>>> its private one. I.e. the work in the private queue can be so that it
>>>> never generates more work in the public queue that others can steal and
>>>> continue work from. So they wait.
>>>>
>>>> That can cause this high termination time. Of course this situation can
>>>> occur multiple times during a GC, so that every thread gets his fair
>>>> share of waiting :)
>>>>
>>>> That mentioned fix has been pushed into the http://hg.openjdk.java.net/
>>>> jdk8u/jdk8u-dev/ repository. It should be a matter of pulling it.
>>>> The README-builds file in the repo has build instructions.
>>>>
>>>> It would be really nice if we could track down your problem to this
>>>> issue, or at least significantly improve it. (JDK9 has more significant
>>>> patches in that area iirc, but this one is probably the most
>>>> important).
>>>>
>>>> Thanks,
>>>>     Thomas
>>>>