NMT merges adjacent unrelated reservations

[Thomas Schatzl]

Hi,

On Thu, 2013-04-25 at 11:43 -0400, Zhengyu Gu wrote:
> Hi Thomas,
> 
> This is Zhengyu, I am the implementer of NMT. Please see comments inline.
> 
> On 4/25/2013 10:18 AM, Thomas Schatzl wrote:
> > Hi all,
> >
> >    while trying out NMT I found some oddity with it: the NMT summary
> > seems to merge virtual memory reservations that are adjacent in memory.
>
> NMT does not merge virtual memory reservation, but does merge committed 
> regions. Hotspot sometimes reserves a large memory region, then splits 
> it into pieces, in the case, you may only see one region.

This is not the case here. The G1 GC reserves this memory in completely
separate chunks as they are for different data structures.

I attached a diff which adds some messages for the actual allocations.
If you run applications with it using G1 (-XX:+UseG1GC
-XX:NativeMemoryTracking=detail -XX:+PrintNMTStatistics) you get roughly
the following output:

Alloc cardtable at 0x00007f1d6cadc000 size 1028
Alloc offsettable at 0x00007f1d6c9db000 size 1028 (reg:
0x00007f1d6c9db000/1028)
Alloc bitmap at 0x00007f1d6c1b9000 size 8196 (reg:
0x00007f1d6c1b9000/8196)
Alloc bitmap at 0x00007f1d647ff000 size 8196 (reg:
0x00007f1d647ff000/8196)
Markstack at 0x00007f1d3e000000 size 32768 (reg:
0x00007f1d3e000000/32768)

[...]

[0x00007fc17d7ff000 - 0x00007fc180000000] reserved 40964KB for GC
		from [ReservedSpace::initialize(unsigned long, unsigned long, bool,
char*, unsigned long, bool)+0x256]
	[0x00007fc17d7ff000 - 0x00007fc180000000] committed 40964KB from
[VirtualSpace::expand_by(unsigned long, bool)+0x1c9]

[...]
 
[0x00007fc1a47ff000 - 0x00007fc1a5000000] reserved 8196KB for GC
		from [ReservedSpace::initialize(unsigned long, unsigned long, bool,
char*, unsigned long, bool)+0x256]
	[0x00007fc1a47ff000 - 0x00007fc1a5270000] committed 10692KB from
[VirtualSpace::expand_by(unsigned long, bool)+0x1c9]
 
[...]
 
[0x00007fc1ac2df000 - 0x00007fc1ac4e1000] reserved 2056KB for GC
		from [ReservedSpace::initialize(unsigned long, unsigned long, bool,
char*, unsigned long, bool)+0x256]
	[0x00007fc1ac2df000 - 0x00007fc1ac3e0000] committed 1028KB from
[VirtualSpace::expand_by(unsigned long, bool)+0x1c9]
	[0x00007fc1ac3e0000 - 0x00007fc1ac4e0000] committed 1024KB from
[CardTableModRefBS::resize_covered_region(MemRegion)+0x354]
	[0x00007fc1ac4e0000 - 0x00007fc1ac4e1000] committed 4KB from
[CardTableModRefBS::CardTableModRefBS(MemRegion, int)+0x26e]

So the total sizes match (~50M), but NMT tells me the VM used three
allocations, one 40M, one 8M and one 2M one. I.e. it merged the 32M and
one of the 8M allocations. It also merged the first two allocations.

As mentioned, sometimes I get only two entries in the virtual memory
map.

Looking at the numbers again now, the reservations are not even
adjacent.

> > Further, is there a way to get more than one stack level in the
> > allocation stack trace? Currently it always reads
> > "ReservedSpace::initialize()" in all allocations. This is not really
> > useful. Maybe it is possible to use the PC of the preceding call to
> > MemTracker::record_virtual_memory_type() for getting the stack trace
> > instead.
> NMT depends on stack walking for caller, but it is not reliable, since 
> different compilers have different inline strategies ... due to memory 

That's fine, I could live with that :)

> consumption concern, NMT only records one pc.

Often, during debugging or analysis memory is not that much of a
concern. There may be other situations where it is.

> > Is there a way to add a tag to allocations for grouping purposes in the
> > summary? Currently, given that there is only a single stack level in the
> > source in the summary, it is very hard to assign allocations to the
> > components that actually did the allocation.
> Again, this is memory consumption concern. However, it is possible to 
> add new tracking categories, but you have to go through source code to 
> change the tags.

That I would not mind. Could I just add a few new categories? I saw that
the tag is just a short, where half of it is already reserved.

(Looking at the use of MEMFLAGS variables, I think due to member
alignment and padding it wouldn't matter even if you used a size_t btw).

> > For some reasons the "for
> > <component>" string is also missing in the malloc allocation map.
> This may be related to the issue I mentioned above. You may also see 
> missing callers on Solaris, that is due to the memory allocations occur 
> before stacks are walkable.

Using Linux here.

I meant e.g. the line 

[0x00007fc1ae95f8c8] G1CollectorPolicy::G1CollectorPolicy()+0x978
                            (malloc=1KB #1)

in the details. While here it is easily possible to see that this is
from the GC from the available stack contents, there are cases where
this is not as easy. I saw that all allocations are now tagged with the
originating component anyway, so an additional "for GC" like in the
virtual memory map would help a lot.

Thanks,
  Thomas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: g1allocmessages.diff
Type: text/x-patch
Size: 2483 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20130425/5f5fb459/g1allocmessages.diff