<div dir="ltr"><div>Thank you for the positive encouragement, Roman :-)</div><div><br></div><div>Cheers, Thomas<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 5, 2022 at 12:03 PM Kennke, Roman <<a href="mailto:rkennke@amazon.de">rkennke@amazon.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Thomas,<br>
<br>
I very much like the idea and also your proposals how to do it. Insights <br>
in JDK's native memory usage is sorely lacking and would be very useful! <br>
I don't have all that much to add about the details beyond what you <br>
already covered, though :-)<br>
<br>
Cheers,<br>
Roman<br>
<br>
<br>
> Are there any opinions about whether or not to extend NMT across the JDK?<br>
> <br>
> This blocks <a href="https://bugs.openjdk.org/browse/JDK-8296360" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8296360</a> <br>
> <<a href="https://bugs.openjdk.org/browse/JDK-8296360" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8296360</a>>, and I had a PR prepared <br>
> as <a href="https://github.com/openjdk/jdk/pull/10988" rel="noreferrer" target="_blank">https://github.com/openjdk/jdk/pull/10988</a> <br>
> <<a href="https://github.com/openjdk/jdk/pull/10988" rel="noreferrer" target="_blank">https://github.com/openjdk/jdk/pull/10988</a>>. Originally I was hoping to <br>
> get this into JDK 20, but I don't think that is realistic anymore. I am <br>
> fine with postponing my work in favor of a baseline discussion, but so <br>
> far there is very little discussion about this topic.<br>
> <br>
> How should I proceed?<br>
> <br>
> Thanks, Thomas<br>
> <br>
> <br>
> <br>
> On Wed, Nov 9, 2022 at 8:12 AM Thomas Stüfe <<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a> <br>
> <mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a>>> wrote:<br>
> <br>
> Hi Alan,<br>
> <br>
> (replaced hotspot-runtime-dev with hotspot-dev, since its more of a<br>
> general topic)<br>
> <br>
> thank you for your time!<br>
> <br>
> I am very happy to talk this through. I think native memory<br>
> observability in the JDK (and customer code!) is sorely lacking.<br>
> Witness the countless "where did my native memory go" blog articles.<br>
> At SAP we have been struggling with this topic for a long time and<br>
> have come up with a mixture of solutions. The aforementioned tracker<br>
> was one, which extended our version of NMT across the JDK. Our<br>
> SapMachine MallocTracer, which allows us to trace uninstrumented<br>
> customer code, another. We even experimented with exchanging the<br>
> allocator (using jemalloc) to gain insights. But that is a whole<br>
> different topic with deep logistical implications, I don't want to<br>
> touch it here. Exchanging the allocator does not help to observe<br>
> virtual memory or the brk segment, of course.<br>
> <br>
> And to make the picture complete, another insight we currently lack<br>
> is the implicit allocator overhead, which can be very significant<br>
> and is hidden by the libc. We also have observability for that in<br>
> the SapMachine, and I miss it in OpenJDK.<br>
> <br>
> As you noticed, my original intent was just to instrument Zlib and<br>
> possibly improve tracking for DBBs. Although, thinking beyond that,<br>
> another attractive instrumentation target would be mapped NIO<br>
> buffers at least.<br>
> <br>
> So I think native memory observability is important. Arguably we<br>
> could even extend observability to cover other OS resources, e.g.<br>
> file handles. If we shift code around, to java/Panama: data that<br>
> move the java heap does not need to be tracked, but other memory<br>
> will always come from one of the basic system APIs, regardless of<br>
> who allocates it and where in the stack allocation happens. Be it<br>
> native JDK code, Panama, or even customer JNI code.<br>
> <br>
> If we agree on the importance of native memory observability, then I<br>
> believe NMT is the right tool for it. It is a good tool. The<br>
> machinery is already there. It covers both C-heap and virtual memory<br>
> APIs, as well as thread stacks, and could easily be extended to<br>
> cover sbrk if needed. And I assume that whatever shape OpenJDK takes<br>
> on in the future, there always will be a libjvm.so at its core, so<br>
> we will always have it. But even if not, NMT could be separated from<br>
> libjvm.so quite easily, since it has no deep ties with the JVM.<br>
> <br>
> About coupling JVM with outside code: We don't have to directly link<br>
> against libjvm.so. We can keep things loose if the intent is to be<br>
> runnable without a JVM, or be JVM-version-agnostic. That could take<br>
> the form of a function-pointer interface like JVMTI. Or outside code<br>
> could dynamically dlsym the JVM allocation hooks. In any case<br>
> gracefully falling back to system allocation routines when necessary.<br>
> <br>
> And I agree, polluting the NMT tag space with outside meaning is<br>
> ugly. I only did it because I planned to go no further than<br>
> instrumenting Zlib and possibly DBBs. But if we take this further,<br>
> my preferred solution would be a reserved tag range or -ranges for<br>
> outside use, whose inner meaning would be opaque to the JVM. Kind of<br>
> like SIGRTMIN+SIGRTMAX. Then, outside code could register tags and<br>
> their meta information with the JVM, or we find a different way to<br>
> convey the tag meaning to NMT (config files, or callbacks). That<br>
> could even be opened up for customer use.<br>
> <br>
> This also touches on another question, that of NMT tag space. NMT<br>
> tags are very useful since they allow cheap tracking without<br>
> capturing call stacks. However, tags are underused and show growing<br>
> pains since they are too one-dimensional and restrictive. We had<br>
> competing interests in the past about tag granularity. It is all<br>
> over the place. We have coarse-grained tags like "mtThread", and<br>
> very fine-grained ones like "mtObjectMonitor". There are several<br>
> ways we could improve, e.g., by making them combinable like UL does,<br>
> or allowing for a hierarchy of them - either a hard-wired limited<br>
> one like "domain"+"tag", or an unlimited tree-like one. Technically<br>
> interesting since whatever the new encoding is, they still must fit<br>
> into a malloc header. I opened<br>
> <a href="https://bugs.openjdk.org/browse/JDK-8281819" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8281819</a><br>
> <<a href="https://bugs.openjdk.org/browse/JDK-8281819" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8281819</a>> to track ideas like these.<br>
> <br>
> Instrumenting Panama allocations, including the ability to tag<br>
> allocations, would be a very good idea. For instance, if we ever<br>
> remove the native Zlib layer and convert it to java using Panama, we<br>
> can do the same with Panama I do now natively - use the Zlib zalloc<br>
> interface to hook in JVM memory allocation functions. The result<br>
> could be completely identical, and the end user looking at the NMT<br>
> output need never know that anything changed.<br>
> <br>
> And that goes for all instrumentation - if today we add it to JNI<br>
> code, and that code gets removed tomorrow, we can add it to Panama<br>
> code too. Unless data structures move to the heap, in which case<br>
> there is no need to track them.<br>
> <br>
> You mentioned that NMT was more of an in-house support tool. Our<br>
> experience is different. Even though it was positioned as a tool for<br>
> JVM developers, and we never cared for the backward compatibility or<br>
> consistency, it gets used a *lot* by our customers. We have to<br>
> explain its output frequently. Also, many blog articles exist<br>
> documenting its use. So, maybe it would be okay to elevate it to a<br>
> user-facing tool since it seems to occupy that role anyway. We may<br>
> also open up consumption of NMT results via java APIs, or expose its<br>
> results via MXBeans.<br>
> <br>
> If this is to be a JEP, okay, but I'm afraid it would stall things a<br>
> bit. I am interested in getting a simpler and quicker solution for<br>
> older support releases at least, possibly based on my PR. I know<br>
> that would be unconventional though.<br>
> <br>
> Thank you,<br>
> <br>
> Thomas<br>
> <br>
> <br>
> On Sun, Nov 6, 2022 at 9:31 AM Alan Bateman <<a href="mailto:Alan.Bateman@oracle.com" target="_blank">Alan.Bateman@oracle.com</a><br>
> <mailto:<a href="mailto:Alan.Bateman@oracle.com" target="_blank">Alan.Bateman@oracle.com</a>>> wrote:<br>
> <br>
> On 04/11/2022 16:54, Thomas Stüfe wrote:<br>
> > Hi all,<br>
> ><br>
> > I am currently working on<br>
> <a href="https://bugs.openjdk.org/browse/JDK-8296360" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8296360</a><br>
> <<a href="https://bugs.openjdk.org/browse/JDK-8296360" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8296360</a>>;<br>
> > I was preparing the final PR [1], but then Alan did ask me to<br>
> discuss<br>
> > this on core-libs first.<br>
> ><br>
> > Backstory:<br>
> ><br>
> > NMT tracks hotspot native allocations but does not cover the JDK<br>
> > libraries (small exception: Unsafe.AllocateMemory). However, the<br>
> > native memory footprint of JDK libraries can be significant.<br>
> We have<br>
> > no in-VM tracker for these and need tools like valgrind or our<br>
> > SapMachine MallocTracer [2] to observe them.<br>
> <br>
> Thanks for starting a discussion on this as this is a topic that<br>
> requires agreement from several areas. If this is the start of<br>
> something<br>
> bigger, where you want to have all allocation sites in the<br>
> libraries<br>
> using NMT, then I think it needs a write-up, maybe a JEP.<br>
> <br>
> For starters, I think it needs some agreement on using NMT for<br>
> memory<br>
> allocated outside of libjvm. You mentioned Unsafe as an<br>
> exception but<br>
> that is implemented in the VM so you get tracking for free,<br>
> albeit I<br>
> think all allocations are in the "mtOther" category.<br>
> <br>
> A general concern is that it creates more coupling between the<br>
> VM code<br>
> and the libraries code. As you probably know, we've removed most<br>
> of the<br>
> dependences on JVM_* functions from non-core areas over many<br>
> years. So I<br>
> think that needs consideration as I assume we don't want<br>
> memory/allocation.hpp declaring a dozen catagories for<br>
> allocations done<br>
> in say java.desktop module for example. Maybe your proposal will be<br>
> strictly limited to java.base but even then, do we really want<br>
> the VM<br>
> even knowing about categories that are specific to zip<br>
> compression or<br>
> decompression?<br>
> <br>
> There are probably longer term trends that should be part of the<br>
> discussion too. One general trend is that "run time" is becoming<br>
> more<br>
> and more a hybrid of code in libvm and the Java libraries. Lambdas,<br>
> module system, virtual threads implementations are a few<br>
> examples in the<br>
> last few release. This comes with many "Java on Java" challenges,<br>
> including serviceability where users of the platform will expect<br>
> tools<br>
> to just work and won't care where the code is. NMT is probably<br>
> more for<br>
> support teams and not something that most developers will ever<br>
> use but I<br>
> think is part of the challenge of having serviceability<br>
> solutions "just<br>
> work".<br>
> <br>
> In addition to having more of the Java runtime written in Java,<br>
> there<br>
> will likely be less JNI code in the future. It's very possible<br>
> that the<br>
> JNI code (including the JNI methods in libzip) will be replaced<br>
> with<br>
> code that uses Panama memory and linker APIs once they are become<br>
> permanent. The effect of that would to have a lot of the memory<br>
> allocations be tracked in the mtOther category again. Maybe<br>
> integration<br>
> with memory tracking should be looked at in conjunction with<br>
> these APIs<br>
> and this migration. I could imagine the proposed "Arena" API<br>
> (MemorySession in Java 19) having some integration with NMT and<br>
> it might<br>
> be interesting to look into that.<br>
> <br>
> So yes, this topic does need broader discussion and it might be<br>
> a bit<br>
> premature to start with a PR for libzip without talking about<br>
> the bigger<br>
> picture first.<br>
> <br>
> -Alan<br>
> <br>
> <br>
> <br>
<br>
<br>
<br>
Amazon Development Center Germany GmbH<br>
Krausenstr. 38<br>
10117 Berlin<br>
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss<br>
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B<br>
Sitz: Berlin<br>
Ust-ID: DE 289 237 879<br>
<br>
<br>
</blockquote></div>