Native Memory Leak

[Jacob Schlather]

Thanks Andrew & Colleen, that was really helpful. The issue ended up being
inside caffeine. In a recent upgrade they changed over the way caches are
created to use method handles and some of our tooling continually recreates
these caches. I downgraded caffeine to before the upgrade and the issue
went away. I filed an issue in caffeine
https://github.com/ben-manes/caffeine/issues/222 . Thanks for the help.

On Fri, Feb 9, 2018 at 4:46 AM, Kirk Pepperdine <kirk.pepperdine at gmail.com>
wrote:

> Hi Jacob,
>
> Out of curiosity, are you using any JDBC drivers? I believe there is a
> version of some drivers that do leak native memory buffers. Also, are you
> tracking anon block allocations with pmap?
>
> Kind regards,
> Kirk Pepperdine
>
> > On Feb 8, 2018, at 7:26 PM, Jacob Schlather <jschlather at hubspot.com>
> wrote:
> >
> > Thanks Andrew. I've looked at file handles and they don't seem to be
> > growing over time. The allocations I posted were after the JVM had been
> up
> > for one day. After another 24 hours all of the allocations have nearly
> > doubled the internal memory reservation is at 435mb, with the MemberTable
> > at 212mb and the JNI allocations around 140mb. Is this sort of growth of
> > these normal? The characteristic of this issue that I've been seeing is
> > that the memory of the JVM monotically increases over time with internal
> > memory seeming to be the culprit from native memory tracking. We're
> seeing
> > this issue across multiple services, on our higher throughput boxes
> > internal memory will grow to 3gb, while the heap is 6g.
> >
> > [0x00007efffa8d29fd] GenericGrowableArray::raw_allocate(int)+0x17d
> > [0x00007efffab86456] MemberNameTable::add_member_name(_jobject*)+0x66
> > [0x00007efffa8ffb24] InstanceKlass::add_member_name(Handle)+0x84
> > [0x00007efffab8777d] MethodHandles::init_method_MemberName(Handle,
> > CallInfo&)+0x28d
> >                             (malloc=212998KB +212577KB #19)
> >
> >
> > On Thu, Feb 8, 2018 at 12:38 PM, Andrew Haley <aph at redhat.com> wrote:
> >
> >> On 07/02/18 19:27, Jacob Schlather wrote:
> >>> Having found these I could use some guidance on what to look at next
> and
> >>> what sorts of calls could potentially be causing these issues. We don't
> >>> have any explicit uses of JNI that I can tell in our code bases. Let me
> >>> know if there's anymore information that would be helpful.
> >>
> >> These look pretty normal.  Assuming that this is Linux, look at
> >> /proc/<pid>/maps/.  You might actually be leaking memory because
> >> of some native structures like FileHandles.
> >>
> >> --
> >> Andrew Haley
> >> Java Platform Lead Engineer
> >> Red Hat UK Ltd. <https://www.redhat.com>
> >> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671
> >>
>
>