Extend Native Memory Tracking over the JDK ? (was: Proposal: track zlib native memory usage with NMT)

Wed Nov 9 07:12:59 UTC 2022

Hi Alan,

(replaced hotspot-runtime-dev with hotspot-dev, since its more of a general
topic)

thank you for your time!

I am very happy to talk this through. I think native memory observability
in the JDK (and customer code!) is sorely lacking. Witness the countless
"where did my native memory go" blog articles. At SAP we have been
struggling with this topic for a long time and have come up with a mixture
of solutions. The aforementioned tracker was one, which extended our
version of NMT across the JDK. Our SapMachine MallocTracer, which allows us
to trace uninstrumented customer code, another. We even experimented with
exchanging the allocator (using jemalloc) to gain insights. But that is a
whole different topic with deep logistical implications, I don't want to
touch it here. Exchanging the allocator does not help to observe virtual
memory or the brk segment, of course.

And to make the picture complete, another insight we currently lack is the
implicit allocator overhead, which can be very significant and is hidden by
the libc. We also have observability for that in the SapMachine, and I miss
it in OpenJDK.

As you noticed, my original intent was just to instrument Zlib and possibly
improve tracking for DBBs. Although, thinking beyond that, another
attractive instrumentation target would be mapped NIO buffers at least.

So I think native memory observability is important. Arguably we could even
extend observability to cover other OS resources, e.g. file handles. If we
shift code around, to java/Panama: data that move the java heap does not
need to be tracked, but other memory will always come from one of the basic
system APIs, regardless of who allocates it and where in the stack
allocation happens. Be it native JDK code, Panama, or even customer JNI
code.

If we agree on the importance of native memory observability, then I
believe NMT is the right tool for it. It is a good tool. The machinery is
already there. It covers both C-heap and virtual memory APIs, as well as
thread stacks, and could easily be extended to cover sbrk if needed. And I
assume that whatever shape OpenJDK takes on in the future, there always
will be a libjvm.so at its core, so we will always have it. But even if
not, NMT could be separated from libjvm.so quite easily, since it has no
deep ties with the JVM.

About coupling JVM with outside code: We don't have to directly link
against libjvm.so. We can keep things loose if the intent is to be runnable
without a JVM, or be JVM-version-agnostic. That could take the form of a
function-pointer interface like JVMTI. Or outside code could dynamically
dlsym the JVM allocation hooks. In any case gracefully falling back to
system allocation routines when necessary.

And I agree, polluting the NMT tag space with outside meaning is ugly. I
only did it because I planned to go no further than instrumenting Zlib and
possibly DBBs. But if we take this further, my preferred solution would be
a reserved tag range or -ranges for outside use, whose inner meaning would
be opaque to the JVM. Kind of like SIGRTMIN+SIGRTMAX. Then, outside code
could register tags and their meta information with the JVM, or we find a
different way to convey the tag meaning to NMT (config files, or
callbacks). That could even be opened up for customer use.

This also touches on another question, that of NMT tag space. NMT tags are
very useful since they allow cheap tracking without capturing call stacks.
However, tags are underused and show growing pains since they are too
one-dimensional and restrictive. We had competing interests in the past
about tag granularity. It is all over the place. We have coarse-grained
tags like "mtThread", and very fine-grained ones like "mtObjectMonitor".
There are several ways we could improve, e.g., by making them combinable
like UL does, or allowing for a hierarchy of them - either a hard-wired
limited one like "domain"+"tag", or an unlimited tree-like one. Technically
interesting since whatever the new encoding is, they still must fit into a
malloc header. I opened https://bugs.openjdk.org/browse/JDK-8281819 to
track ideas like these.

Instrumenting Panama allocations, including the ability to tag allocations,
would be a very good idea. For instance, if we ever remove the native Zlib
layer and convert it to java using Panama, we can do the same with Panama I
do now natively - use the Zlib zalloc interface to hook in JVM memory
allocation functions. The result could be completely identical, and the end
user looking at the NMT output need never know that anything changed.

And that goes for all instrumentation - if today we add it to JNI code, and
that code gets removed tomorrow, we can add it to Panama code too. Unless
data structures move to the heap, in which case there is no need to track
them.

You mentioned that NMT was more of an in-house support tool. Our experience
is different. Even though it was positioned as a tool for JVM developers,
and we never cared for the backward compatibility or consistency, it gets
used a *lot* by our customers. We have to explain its output frequently.
Also, many blog articles exist documenting its use. So, maybe it would be
okay to elevate it to a user-facing tool since it seems to occupy that role
anyway. We may also open up consumption of NMT results via java APIs, or
expose its results via MXBeans.

If this is to be a JEP, okay, but I'm afraid it would stall things a bit. I
am interested in getting a simpler and quicker solution for older support
releases at least, possibly based on my PR. I know that would be
unconventional though.

Thank you,

Thomas

On Sun, Nov 6, 2022 at 9:31 AM Alan Bateman <Alan.Bateman at oracle.com> wrote:

> On 04/11/2022 16:54, Thomas Stüfe wrote:
> > Hi all,
> >
> > I am currently working on https://bugs.openjdk.org/browse/JDK-8296360;
> > I was preparing the final PR [1], but then Alan did ask me to discuss
> > this on core-libs first.
> >
> > Backstory:
> >
> > NMT tracks hotspot native allocations but does not cover the JDK
> > libraries (small exception: Unsafe.AllocateMemory). However, the
> > native memory footprint of JDK libraries can be significant. We have
> > no in-VM tracker for these and need tools like valgrind or our
> > SapMachine MallocTracer [2] to observe them.
>
> Thanks for starting a discussion on this as this is a topic that
> requires agreement from several areas. If this is the start of something
> bigger, where you want to have all allocation sites in the libraries
> using NMT, then I think it needs a write-up, maybe a JEP.
>
> For starters, I think it needs some agreement on using NMT for memory
> allocated outside of libjvm. You mentioned Unsafe as an exception but
> that is implemented in the VM so you get tracking for free, albeit I
> think all allocations are in the "mtOther" category.
>
> A general concern is that it creates more coupling between the VM code
> and the libraries code. As you probably know, we've removed most of the
> dependences on JVM_* functions from non-core areas over many years. So I
> think that needs consideration as I assume we don't want
> memory/allocation.hpp declaring a dozen catagories for allocations done
> in say java.desktop module for example. Maybe your proposal will be
> strictly limited to java.base but even then, do we really want the VM
> even knowing about categories that are specific to zip compression or
> decompression?
>
> There are probably longer term trends that should be part of the
> discussion too. One general trend is that "run time" is becoming more
> and more a hybrid of code in libvm and the Java libraries. Lambdas,
> module system, virtual threads implementations are a few examples in the
> last few release. This comes with many "Java on Java" challenges,
> including serviceability where users of the platform will expect tools
> to just work and won't care where the code is. NMT is probably more for
> support teams and not something that most developers will ever use but I
> think is part of the challenge of having serviceability solutions "just
> work".
>
> In addition to having more of the Java runtime written in Java, there
> will likely be less JNI code in the future. It's very possible that the
> JNI code (including the JNI methods in libzip) will be replaced with
> code that uses Panama memory and linker APIs once they are become
> permanent. The effect of that would to have a lot of the memory
> allocations be tracked in the mtOther category again. Maybe integration
> with memory tracking should be looked at in conjunction with these APIs
> and this migration. I could imagine the proposed "Arena" API
> (MemorySession in Java 19) having some integration with NMT and it might
> be interesting to look into that.
>
> So yes, this topic does need broader discussion and it might be a bit
> premature to start with a PR for libzip without talking about the bigger
> picture first.
>
> -Alan
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-dev/attachments/20221109/4236b240/attachment-0001.htm>