High termination times pre-concurrent cycle in G1

[William Good]

Hi Jenny,

Attached are logs for runs with 8m and 16m region sizes. This is with
the old version of the application (before the akka patch). I was
pleasantly surprised by the 16m run and tried it a few times, but it
seems these results are reliable.

Thanks,
William

On Tue, Sep 6, 2016 at 6:25 PM, yu.zhang at oracle.com <yu.zhang at oracle.com> wrote:
> William,
>
> While this might not help the termination time, I hope adjusting the region
> size can get rid of some of the humongous objects.
>
> It seems the humongous objects are filling old gen, about ~5g. The full gc
> can clean 10g. Can you try increase region size to 8m or 16m?
>
> Thanks
>
> Jenny
>
>
>
> On 09/05/2016 03:02 AM, William Good wrote:
>>
>> Hi Thomas,
>>
>> Attached is a log file from a run with this morning's tip. I don't see
>> any change unfortunately.
>>
>> I'll try with jdk9 tip later today since you mention there are more fixes
>> there.
>>
>> configure summary looked like this, I think everything here is in order:
>>
>> Configuration summary:
>> * Debug level:    release
>> * JDK variant:    normal
>> * JVM variants:   server
>> * OpenJDK target: OS: linux, CPU architecture: x86, address length: 64
>>
>> Tools summary:
>> * Boot JDK:       java version "1.8.0_101" Java(TM) SE Runtime
>> Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM
>> (build 25.101-b13, mixed mode)  (at /var/tmp/jdk1.8.0_101)
>> * C Compiler:     gcc (GCC) 4.8.5 20150623 (Red Hat-4) version 4.8.5
>> (at /usr/bin/gcc)
>> * C++ Compiler:   g++ (GCC) 4.8.5 20150623 (Red Hat-4) version 4.8.5
>> (at /usr/bin/g++)
>>
>> Best,
>> William
>>
>>
>> On Fri, Sep 2, 2016 at 4:01 PM, Thomas Schatzl
>> <thomas.schatzl at oracle.com> wrote:
>>>
>>> Hi,
>>>
>>> On Fri, 2016-09-02 at 15:08 +0200, William Good wrote:
>>>>
>>>> Thomas,
>>>>
>>>> More than happy to build and test with a patched JDK. Let me know
>>>> what I need to do, at least what sources to grab (I think I can build
>>>> it once I've got them).
>>>>
>>>> I learned this morning that my belief that significant tenuring
>>>> wasn't taking place was possibly wrong. Our problem seems to have
>>>> disappeared with an akka upgrade [1] and my guess for the relevant
>>>> fix [2] indicates that unintended tenuring was occurring at some
>>>> point (teh tenuring has never been significant enough for us to
>>>> notice). However as long as I'm unable to reproduce with the new
>>>> version I'm happy to continue testing using the older version I know
>>>> to reproduce, as I don't think this G1 behavior is intended.
>>>
>>>    this bug report makes me tend to believe that the suggested fix (for
>>> JDK-8152438) will actually fix the issue. Let me explain:
>>>
>>> Every G1 thread has two work queues, one fixed size public one where
>>> others can steal from, and one resizable one that is private. Work
>>> (references) is first put into the public one, and then if it is full,
>>> into the private one.
>>> Threads first process their private buffers, so that others can steal
>>> from the public one while they are working on it.
>>> Due to some conditions it can happen that the public queues are already
>>> completely empty, while one thread is still busy for a long time with
>>> its private one. I.e. the work in the private queue can be so that it
>>> never generates more work in the public queue that others can steal and
>>> continue work from. So they wait.
>>>
>>> That can cause this high termination time. Of course this situation can
>>> occur multiple times during a GC, so that every thread gets his fair
>>> share of waiting :)
>>>
>>> That mentioned fix has been pushed into the http://hg.openjdk.java.net/
>>> jdk8u/jdk8u-dev/ repository. It should be a matter of pulling it.
>>> The README-builds file in the repo has build instructions.
>>>
>>> It would be really nice if we could track down your problem to this
>>> issue, or at least significantly improve it. (JDK9 has more significant
>>> patches in that area iirc, but this one is probably the most
>>> important).
>>>
>>> Thanks,
>>>    Thomas
>>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vary_region_sizes_logs.tar.bz
Type: application/x-bzip2
Size: 75668 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20160908/f6d4990f/vary_region_sizes_logs.tar-0001.bz>