NMT merges adjacent unrelated reservations

[Zhengyu Gu]

Hi Thomas,

This is Zhengyu, I am the implementer of NMT. Please see comments inline.

On 4/25/2013 10:18 AM, Thomas Schatzl wrote:
> Hi all,
>
>    while trying out NMT I found some oddity with it: the NMT summary
> seems to merge virtual memory reservations that are adjacent in memory.
NMT does not merge virtual memory reservation, but does merge committed 
regions. Hotspot sometimes reserves a large memory region, then splits 
it into pieces, in the case, you may only see one region.
>
> I.e. at the moment in G1 there are five independent virtual memory
> reservations at startup - I've seen NMT summaries with two to four
> (never five) entries in the virtual memory map. This depends on thread
> timing it seems.
NMT does lag in reporting memory usage. But if there are 5 separate 
reservations, they should eventually show up.

I am currently working on an issue that appears related to G1 - 
ArrayAllocator::allocate() does not tag virtual memory correctly.

I am not sure it is related to what you mentioned. If not, please 
provide me details, I will track it down.

> Note that four of these reservations commit their entire memory almost
> immediately.
>
> Is this intentional? If so, is there a way to disable this behavior?
> This makes determining the source of these allocations unnecessarily
> difficult.
>
> Further, is there a way to get more than one stack level in the
> allocation stack trace? Currently it always reads
> "ReservedSpace::initialize()" in all allocations. This is not really
> useful. Maybe it is possible to use the PC of the preceding call to
> MemTracker::record_virtual_memory_type() for getting the stack trace
> instead.
NMT depends on stack walking for caller, but it is not reliable, since 
different compilers have different inline strategies ... due to memory 
consumption concern, NMT only records one pc.

>
> Is there a way to add a tag to allocations for grouping purposes in the
> summary? Currently, given that there is only a single stack level in the
> source in the summary, it is very hard to assign allocations to the
> components that actually did the allocation.
Again, this is memory consumption concern. However, it is possible to 
add new tracking categories, but you have to go through source code to 
change the tags.
> For some reasons the "for
> <component>" string is also missing in the malloc allocation map.
This may be related to the issue I mentioned above. You may also see 
missing callers on Solaris, that is due to the memory allocations occur 
before stacks are walkable.

Thanks,

-Zhengyu

> Maybe you have another idea how to solve this problem?
>
> Sorry if all that has already been discussed here.
>
> Thanks,
> Thomas
>
>