From ikelly at splunk.com  Tue Oct 27 09:23:26 2020
From: ikelly at splunk.com (Ivan Kelly)
Date: Tue, 27 Oct 2020 10:23:26 +0100
Subject: How do I find which objects are tenuring?
Message-ID: <CAKLzds+k2dW=FLqh8CCdp1PM3=89YTK6BC9A7BnoAu_-4JqR4g@mail.gmail.com>

Hi all,

I have a system that's spending about 25% of all cycles in GC, with most of
that time spent copying objects. I have a good idea of which objects these
are, but I would really like some proof so I can go to management and
justify spending a large chunk of time fixing it.

Is there any way to see what is being tenured? Any method at all would be
useful.

Cheers,
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20201027/be0e1401/attachment.htm>

From ikelly at splunk.com  Tue Oct 27 13:05:23 2020
From: ikelly at splunk.com (Ivan Kelly)
Date: Tue, 27 Oct 2020 14:05:23 +0100
Subject: How do I find which objects are tenuring? [SOLUTION]
In-Reply-To: <CAKLzds+k2dW=FLqh8CCdp1PM3=89YTK6BC9A7BnoAu_-4JqR4g@mail.gmail.com>
References: <CAKLzds+k2dW=FLqh8CCdp1PM3=89YTK6BC9A7BnoAu_-4JqR4g@mail.gmail.com>
Message-ID: <CAKLzdsL4LdBdovHSfR8+qyBXCXeiXUqyh0PopAPtJBzwaHyVVw@mail.gmail.com>

In case someone comes across this message and wonders if I ever solved it,
I managed it using bpftrace, and the following script:

<snip>
struct Symbol {
    u16 refcount;
    u16 length;
    u32 identity_hash;
    char name[10];
}

struct Klass {
    u64 vtable;
    u32 layout_helper;
    u32 super_check_offset;
    struct Symbol* symbol;
}

struct OopDesc {
    u64 markOop; // pointer really
    struct Klass* klass;
};

uretprobe:/proc/22849/root/usr/local/openjdk-8/jre/lib/amd64/server/libjvm.so:_ZN20G1ParScanThreadState22copy_to_survivor_spaceE11InCSetStateP7oopDescP11markOopDesc
{
    @copies = count();
    $symbol = ((struct OopDesc *)retval)->klass->symbol;

    $symbolnameoffset = ((uint64)$symbol) + 8;
    $name = buf($symbolnameoffset, 100);
    @klasses[$name] = count();
}

interval:s:60 {
    print(@copies); zero(@copies);
    print(@klasses, 20); clear(@klasses);
}
</snip>

You'll need to change the path to libjvm.so. In my case, I was running as
root and a machine running containers, which is why the path is so long.

It touches a lot of internals, so YMMV. I ran this against 1.8.0_265-b01.

-Ivan



On Tue, Oct 27, 2020 at 10:23 AM Ivan Kelly <ikelly at splunk.com> wrote:

> Hi all,
>
> I have a system that's spending about 25% of all cycles in GC, with most
> of that time spent copying objects. I have a good idea of which objects
> these are, but I would really like some proof so I can go to management and
> justify spending a large chunk of time fixing it.
>
> Is there any way to see what is being tenured? Any method at all would be
> useful.
>
> Cheers,
> Ivan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20201027/e151d920/attachment.htm>