Analyzing Efficiency Of Java Memory Usage

[Ruslan Synytsky]

Hi Volker and Thomas, thank you for the feedback.

On Fri, 17 Jul 2020 at 21:59, Volker Simonis <volker.simonis at gmail.com>
wrote:

> On Fri, Jul 17, 2020 at 5:18 PM Thomas Stüfe <thomas.stuefe at gmail.com>
> wrote:
> >
> > Did Ruslan use NMT to get those numbers?
>
> No, probably they used the JMX MemoryMXBean. But that reports both,
> "committed" and "used", and "used" should be pretty much the same like
> "touched" (or RSS). E.g.
>
> MemoryUsage G1MonitoringSupport::memory_usage() {
>   MutexLocker x(MonitoringSupport_lock, Mutex::_no_safepoint_check_flag);
>   return MemoryUsage(InitialHeapSize, _overall_used,
> _overall_committed, _g1h->max_capacity());
> }
>
Exactly, I used MemoryMXBean as I have not found another way to collect the
required metrics without interrupting the running java processes. Please
take a look at the source code for a better understanding of how the stats
were collected.

The main shell script
https://github.com/siruslan/java-memory-usage/blob/master/jmu.sh
The collector of java memory usage metrics
https://github.com/siruslan/java-memory-usage/blob/master/src/jmu/MemoryUsageCollector.java

Btw, none from the checked java processes has enabled native memory
tracking...

>>> Interesting findings that a) sometimes committed heap memory is
> >>> higher than real OS used memory

>> Yes, this is a long standing, annoying problem which is most probably
> >> caused by the fact "Committed" as reported by the VM/NMT is not really
> >> the same like RSS reported by system tools. I've opened "8249666:
> >> Improve Native Memory Tracking to report the actual RSS usage" [0] to
> >> track this. Here's an excerpt from the issue describing the problem:
> >>
> >> Currently, NMT shows allocated memory as either "Reserved" or
> >> "Committed". Reserved memory is actually just reserved, virtual
> >> address space which was mmaped with MAP_NORESERVE, while Committed
> >> memory is mapped without MAP_NORESERVE. In the output of top or pmap,
> >> both Reserved and Committed show up as "Virtual" memory until they
> >> will be used for the first time (i.e. touched). Only after a memory
> >> page (usually 4k) has been written to for the first time, it will
> >> consume physical memory and appear in the "resident set" (i.e. RSS) of
> >> top's/pmap's output.
> >>
> >> The difference between allocating memory with or without MAP_NORESERVE
> >> depends on the Linux memory overcommit configuration [1]. By default,
> >> overcommit is allowed and memory allocated with MAP_NORESERVE isn't
> >> checked against the available physical memory (see man proc(5) [2]).
> >> If the HotSpot VM tries to commit reserved memory (i.e. re-map a
> >> memory region without MAP_NORESERVE which was previously mapped with
> >> MAP_NORESERVE) and there's not enough free memory available an
> >> OutOfMemoyError will be thrown.
> >>
> >> But even committing a memory region doesn't mean that physical memory
> >> pages will be allocated for that region (and accounted in the
> >> processes RSS) until that memory will be written to for the first
> >> time. So depending on the overcommit settings, an application might
> >> still crash with a SIGBUS because it is running out of physical memory
> >> when touching memory for the first time which was committed a long
> >> time ago.
> >>
> >> The main problem with the current NMT output is that it can not
> >> distinguish between touched and untouched Committed memory. If a VM is
> >> started with -Xms1g -Xmx1g the VM will commit the whole 1g heap and
> >> NMT will report Reserved=Committed=1g. In contrast, system tools like
> >> ps/top will only show the part of the heap as RSS which has really
> >> been used (i.e. touched), usually just about 100m. This is at least
> >> confusing.
> >>
> >> But we can do better. We can use mincore() [3] to find the RSS part of
> >> the amount of memory which is accounted as Committed in NMT's output
> >> and report that instead (or in addition). Notice that this feature has
> >> already been implemented for threads stacks with "JDK-8191369: NMT:
> >> Enhance thread stack tracking" [4] and just needs to be extended to
> >> all other kinds of memory, starting with the Java heap.

>>
> >> Alternatively, instead of using mincore() we could use the information
> >> from /proc/<pid>/smaps (also accessible through the pmap [5] command
> >> line utility) directly and merge it with the NMT data to get a
> >> complete, annotated overview of the whole address space of a Java
> >> process.
>
This clarification helps. I confirm that in our case the committed memory
becomes counted as really used, and as a result billed by the cloud
platform, only after that memory is written for the first time. Then when
it's not needed anymore uncommitment should be triggered by JVM for
releasing it back to OS, so other processes or even containers can use it.


> >> > , and b) in many cases non heap committed
> >> > memory is as big as heap committed.
>

I just added to the graph a new metric NonHeapUsed+ (royal blue) which
equals the used non heap stacked on top of the used heap memory. In our
case, this metric is often relatively close to NonHeapCommitted+. It will
be very helpful to get similar stats from other companies, so we can get a
better understanding of the real world. The source code of the metrics
collector is available now for a security check. We will skip collection of
the hostname and java start options by default, so the data will be fully
anonymized. Please let me know if you can join the analysis for making the
Java world a more efficient place.

Thanks
-- 
Ruslan