Code review request: CR 6995781 Native Memory Tracking Phase 1

[Zhengyu Gu]

Hi David,
>>> 2. In jvmg.make on all platforms we unconditionally add
>>> -D_NMT_NOINLINE_ to inform NMT that no inlining is done by the
>>> compiler. How do we know that? Is there some specific compiler flag
>>> that controls this? If so the setting of -D_NMT_NOINLINE_ should be
>>> conditional on that flag being set. As you can override flag settings
>>> at build-time it would seem possible to me set -D_NMT_NOINLINE_
>>> incorrectly.
>> It is just a hint to NMT to adjust how many frames it needs to walk to
>> get proper callsite pc. I can not tell if the flag works on all
>> platforms with all possible compiler, but seems to work for the
>> platforms I tested (windows, linux and solaris). I will file a RFE for
>> build-time override.
>
> I'm not sure I made my point clearly on this - as your suggested RFE 
> is not what I was suggesting. This would seem very compiler-settings 
> specific, yet it is set unconditionally, for jvmg builds. Are you 
> assuming that the optimization levels set for jvmg builds render this 
> assumption valid? What happens if the assumption is wrong? Would we 
> crash or just get inaccurate information?
>
So far, I can tell, it just results inaccurate information. Even for 
optimized build, there is not way to tell if a function actually will be 
inlined or not, so some functions that are assumed to be inlined by NMT, 
are not always inlined. I see the callsites not always the same on 
different platforms, but have yet see crashes due to stack walking.

I ran NMT through JPRT and other tests with tracking level = detail, 
which means to perform stack walking for callsites, I have not seen any 
crashes related to stack walking for a fair long time (3 ~ 4 months)

But again, that just small set of compilers, which we use ...

Thanks,

-Zhengyu

>>> General comment which I'm sure you've had before: it is a real shame
>>> that every NEW/FREE_C_HEAP_ARRAY etc had to be modified to add a tag -
>>> and it seems the majority of them are mtInternal. Could we not define
>>> the existing macros to add mtInternal, and then define new macros, eg
>>> NEW_C_HEAP_ARRAY_TAGGED, for explicit tagging for use when not
>>> mtInternal?
>>>
>> New macros can be a solution.
>
> Thanks for considering. I know there's a lot of schedule pressure on 
> this work.
>
> David
> -----
>
>> Thanks,
>>
>> -Zhengyu
>>
>>> Thanks,
>>> David
>>> -----
>>>
>>> On 28/06/2012 11:32 AM, Zhengyu Gu wrote:
>>>> Hi,
>>>>
>>>> Sorry for the long delay, here is the updated webrev:
>>>> http://cr.openjdk.java.net/~zgu/6995781/webrev.01
>>>>
>>>> Thanks for many people who take time to review NMT code, the new 
>>>> webrev
>>>> reflects the comments and suggestions from you, along with the 
>>>> fixes of
>>>> serviceability agent and some bugs.
>>>>
>>>> Thanks,
>>>>
>>>> -Zhengyu
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 6/12/2012 12:11 PM, Zhengyu Gu wrote:
>>>>> This is the webrev for native memory tracking phase 1, which is
>>>>> tracked by CR6995781
>>>>> (http://monaco.us.oracle.com/detail.jsf?cr=6995781) and related DCmd
>>>>> CR7151532 (http://monaco.us.oracle.com/detail.jsf?cr=7151532).
>>>>>
>>>>> Native memory tracking (NMT) phase 1, is designed to track native
>>>>> memory (malloc'd and mmap'd) usages by VM code only, it does not 
>>>>> track
>>>>> the memory usages by libraries and JNI code.
>>>>>
>>>>> On the implementation side, NMT intercepts memory related calls in os
>>>>> implementation, such as os::malloc, os::realloc, os::free,
>>>>> os::reserve_memory, os::commit_memory and etc. the caller is required
>>>>> to provide the 'memory type' information, which represents the VM
>>>>> subsystem that the memory is allocated for. The 'memory type' is
>>>>> defined in /src/share/memory/allocation.hpp. Also, a caller's pc is
>>>>> captured if NMT tracking level is set to detail.
>>>>>
>>>>> There are a few ways to tag a memory block to a memory type:
>>>>>
>>>>> 1. os::malloc takes an extra parameter for the memory type.
>>>>> 2. CHeapObj now is a template, it takes a memory type as template
>>>>> parameter to tag the object to a specified memory type.
>>>>> 3. Utility classes, such as Arena, GrowableArray and etc. the memory
>>>>> type is a parameter of 'new' operator.
>>>>> 4. Virtual memory block is tagged on its base address.
>>>>>
>>>>> When NMT intercepts the call, a memory record is created and
>>>>> serialized by a sequence number, generated by global sequence
>>>>> generator. The memory record is written to a per-thread recorder
>>>>> without a lock, when calling thread is a 'safepoint visible'
>>>>> JavaThread - a JavaThread that will block at safepoint. Otherwise,
>>>>> ThreadCritical lock is acquired to write to a global recorder. The
>>>>> recorders (or raw data) are synchronized at some safepoints, and
>>>>> global sequence generator is also reset at synchronization time, to
>>>>> avoid overflow the sequence number.
>>>>>
>>>>> A dedicated NMT worker thread is created to process the raw memory
>>>>> records. It first stages a generation of raw data (a generation is
>>>>> defined as span between resets of global sequence number), then
>>>>> promotes the staged data to a global snapshot, which maintains
>>>>> information of all 'live' memory block.
>>>>>
>>>>> NMT Tracking option has to be specified through VM command line 
>>>>> option
>>>>> -XX:NativeMemoryTracking (off by default), tracking level can not be
>>>>> altered during runtime, but it can be shutdown through jcmd tool.
>>>>>
>>>>> NMT DCmd implements a set of sub-command to control NMT runtime and
>>>>> request tracking data.
>>>>>
>>>>> Webrev: http://cr.openjdk.java.net/~zgu/6995781/webrev.00
>>>>>
>>>>> The tests and results:
>>>>>
>>>>> - Passed internal smoke tests
>>>>> - JTreg test: http://cr.openjdk.java.net/~zgu/6995781/JTreport/
>>>>>
>>>>>
>>>>> Additional notes:
>>>>> - Some classes' new operators are marked _NOINLINE_ (for the same
>>>>> purpose as above), the performance runs, I performed, indicate that
>>>>> they do not impact performance numbers.
>>>>>
>>>>> - SA agent changes are in progress
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Zhengyu
>>>>>
>>>>>
>>>>>
>>>>>