Code review request: CR 6995781 Native Memory Tracking Phase 1

[Thomas Stüfe]

Zhengyu, Vitali,

I like that __FUNCTION__ idea. Seems nice and simple compared to
walking the stack (which may not be cheap, e.g. on Win64
RtlCaptureBackTrace() needs to parse unwind information).

Do you have an idea by how much the binary would grow? I would have
thought that memory allocation is not done at *that* many different
places.

Thomas Stuefe
SAP Germany

On Thu, Jun 28, 2012 at 3:51 PM, Zhengyu Gu <zhengyu.gu at oracle.com> wrote:
> Hi Vitaly,
>
> We did, there were concerns over increasing binary sizes.
>
> Thanks,
>
> -Zhengyu
>
>
> On 6/28/2012 9:44 AM, Vitaly Davidovich wrote:
>
> Hi Zhengyu,
>
> Have you considered using compiler macros (e.g. _func_, _FUNCTION_) to
> record the name of the func that allocated the memory, rather than stack
> walking? Not sure if all compilers that you guys use support this, but I
> think the major ones do.
>
> This would sidestep the inlining issue.
>
> Thanks
>
> Sent from my phone
>
> On Jun 28, 2012 8:18 AM, "Zhengyu Gu" <zhengyu.gu at oracle.com> wrote:
>>
>> Hi David,
>>>>>
>>>>> 2. In jvmg.make on all platforms we unconditionally add
>>>>> -D_NMT_NOINLINE_ to inform NMT that no inlining is done by the
>>>>> compiler. How do we know that? Is there some specific compiler flag
>>>>> that controls this? If so the setting of -D_NMT_NOINLINE_ should be
>>>>> conditional on that flag being set. As you can override flag settings
>>>>> at build-time it would seem possible to me set -D_NMT_NOINLINE_
>>>>> incorrectly.
>>>>
>>>> It is just a hint to NMT to adjust how many frames it needs to walk to
>>>> get proper callsite pc. I can not tell if the flag works on all
>>>> platforms with all possible compiler, but seems to work for the
>>>> platforms I tested (windows, linux and solaris). I will file a RFE for
>>>> build-time override.
>>>
>>>
>>> I'm not sure I made my point clearly on this - as your suggested RFE is
>>> not what I was suggesting. This would seem very compiler-settings specific,
>>> yet it is set unconditionally, for jvmg builds. Are you assuming that the
>>> optimization levels set for jvmg builds render this assumption valid? What
>>> happens if the assumption is wrong? Would we crash or just get inaccurate
>>> information?
>>>
>> So far, I can tell, it just results inaccurate information. Even for
>> optimized build, there is not way to tell if a function actually will be
>> inlined or not, so some functions that are assumed to be inlined by NMT, are
>> not always inlined. I see the callsites not always the same on different
>> platforms, but have yet see crashes due to stack walking.
>>
>> I ran NMT through JPRT and other tests with tracking level = detail, which
>> means to perform stack walking for callsites, I have not seen any crashes
>> related to stack walking for a fair long time (3 ~ 4 months)
>>
>> But again, that just small set of compilers, which we use ...
>>
>> Thanks,
>>
>> -Zhengyu
>>
>>>>> General comment which I'm sure you've had before: it is a real shame
>>>>> that every NEW/FREE_C_HEAP_ARRAY etc had to be modified to add a tag -
>>>>> and it seems the majority of them are mtInternal. Could we not define
>>>>> the existing macros to add mtInternal, and then define new macros, eg
>>>>> NEW_C_HEAP_ARRAY_TAGGED, for explicit tagging for use when not
>>>>> mtInternal?
>>>>>
>>>> New macros can be a solution.
>>>
>>>
>>> Thanks for considering. I know there's a lot of schedule pressure on this
>>> work.
>>>
>>> David
>>> -----
>>>
>>>> Thanks,
>>>>
>>>> -Zhengyu
>>>>
>>>>> Thanks,
>>>>> David
>>>>> -----
>>>>>
>>>>> On 28/06/2012 11:32 AM, Zhengyu Gu wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Sorry for the long delay, here is the updated webrev:
>>>>>> http://cr.openjdk.java.net/~zgu/6995781/webrev.01
>>>>>>
>>>>>> Thanks for many people who take time to review NMT code, the new
>>>>>> webrev
>>>>>> reflects the comments and suggestions from you, along with the fixes
>>>>>> of
>>>>>> serviceability agent and some bugs.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -Zhengyu
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 6/12/2012 12:11 PM, Zhengyu Gu wrote:
>>>>>>>
>>>>>>> This is the webrev for native memory tracking phase 1, which is
>>>>>>> tracked by CR6995781
>>>>>>> (http://monaco.us.oracle.com/detail.jsf?cr=6995781) and related DCmd
>>>>>>> CR7151532 (http://monaco.us.oracle.com/detail.jsf?cr=7151532).
>>>>>>>
>>>>>>> Native memory tracking (NMT) phase 1, is designed to track native
>>>>>>> memory (malloc'd and mmap'd) usages by VM code only, it does not
>>>>>>> track
>>>>>>> the memory usages by libraries and JNI code.
>>>>>>>
>>>>>>> On the implementation side, NMT intercepts memory related calls in os
>>>>>>> implementation, such as os::malloc, os::realloc, os::free,
>>>>>>> os::reserve_memory, os::commit_memory and etc. the caller is required
>>>>>>> to provide the 'memory type' information, which represents the VM
>>>>>>> subsystem that the memory is allocated for. The 'memory type' is
>>>>>>> defined in /src/share/memory/allocation.hpp. Also, a caller's pc is
>>>>>>> captured if NMT tracking level is set to detail.
>>>>>>>
>>>>>>> There are a few ways to tag a memory block to a memory type:
>>>>>>>
>>>>>>> 1. os::malloc takes an extra parameter for the memory type.
>>>>>>> 2. CHeapObj now is a template, it takes a memory type as template
>>>>>>> parameter to tag the object to a specified memory type.
>>>>>>> 3. Utility classes, such as Arena, GrowableArray and etc. the memory
>>>>>>> type is a parameter of 'new' operator.
>>>>>>> 4. Virtual memory block is tagged on its base address.
>>>>>>>
>>>>>>> When NMT intercepts the call, a memory record is created and
>>>>>>> serialized by a sequence number, generated by global sequence
>>>>>>> generator. The memory record is written to a per-thread recorder
>>>>>>> without a lock, when calling thread is a 'safepoint visible'
>>>>>>> JavaThread - a JavaThread that will block at safepoint. Otherwise,
>>>>>>> ThreadCritical lock is acquired to write to a global recorder. The
>>>>>>> recorders (or raw data) are synchronized at some safepoints, and
>>>>>>> global sequence generator is also reset at synchronization time, to
>>>>>>> avoid overflow the sequence number.
>>>>>>>
>>>>>>> A dedicated NMT worker thread is created to process the raw memory
>>>>>>> records. It first stages a generation of raw data (a generation is
>>>>>>> defined as span between resets of global sequence number), then
>>>>>>> promotes the staged data to a global snapshot, which maintains
>>>>>>> information of all 'live' memory block.
>>>>>>>
>>>>>>> NMT Tracking option has to be specified through VM command line
>>>>>>> option
>>>>>>> -XX:NativeMemoryTracking (off by default), tracking level can not be
>>>>>>> altered during runtime, but it can be shutdown through jcmd tool.
>>>>>>>
>>>>>>> NMT DCmd implements a set of sub-command to control NMT runtime and
>>>>>>> request tracking data.
>>>>>>>
>>>>>>> Webrev: http://cr.openjdk.java.net/~zgu/6995781/webrev.00
>>>>>>>
>>>>>>> The tests and results:
>>>>>>>
>>>>>>> - Passed internal smoke tests
>>>>>>> - JTreg test: http://cr.openjdk.java.net/~zgu/6995781/JTreport/
>>>>>>>
>>>>>>>
>>>>>>> Additional notes:
>>>>>>> - Some classes' new operators are marked _NOINLINE_ (for the same
>>>>>>> purpose as above), the performance runs, I performed, indicate that
>>>>>>> they do not impact performance numbers.
>>>>>>>
>>>>>>> - SA agent changes are in progress
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> -Zhengyu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>