Code review request: CR 6995781 Native Memory Tracking Phase 1

Thu Jun 28 08:50:46 PDT 2012

I see, thanks.  Too bad, would make some of this easier :)

Sent from my phone
On Jun 28, 2012 9:51 AM, "Zhengyu Gu" <zhengyu.gu at oracle.com> wrote:

> **
> Hi Vitaly,
>
> We did, there were concerns over increasing binary sizes.
>
> Thanks,
>
> -Zhengyu
>
> On 6/28/2012 9:44 AM, Vitaly Davidovich wrote:
>
> Hi Zhengyu,
>
> Have you considered using compiler macros (e.g. _func_, _FUNCTION_) to
> record the name of the func that allocated the memory, rather than stack
> walking? Not sure if all compilers that you guys use support this, but I
> think the major ones do.
>
> This would sidestep the inlining issue.
>
> Thanks
>
> Sent from my phone
> On Jun 28, 2012 8:18 AM, "Zhengyu Gu" <zhengyu.gu at oracle.com> wrote:
>
>> Hi David,
>>
>>>  2. In jvmg.make on all platforms we unconditionally add
>>>>> -D_NMT_NOINLINE_ to inform NMT that no inlining is done by the
>>>>> compiler. How do we know that? Is there some specific compiler flag
>>>>> that controls this? If so the setting of -D_NMT_NOINLINE_ should be
>>>>> conditional on that flag being set. As you can override flag settings
>>>>> at build-time it would seem possible to me set -D_NMT_NOINLINE_
>>>>> incorrectly.
>>>>>
>>>> It is just a hint to NMT to adjust how many frames it needs to walk to
>>>> get proper callsite pc. I can not tell if the flag works on all
>>>> platforms with all possible compiler, but seems to work for the
>>>> platforms I tested (windows, linux and solaris). I will file a RFE for
>>>> build-time override.
>>>>
>>>
>>> I'm not sure I made my point clearly on this - as your suggested RFE is
>>> not what I was suggesting. This would seem very compiler-settings specific,
>>> yet it is set unconditionally, for jvmg builds. Are you assuming that the
>>> optimization levels set for jvmg builds render this assumption valid? What
>>> happens if the assumption is wrong? Would we crash or just get inaccurate
>>> information?
>>>
>>>  So far, I can tell, it just results inaccurate information. Even for
>> optimized build, there is not way to tell if a function actually will be
>> inlined or not, so some functions that are assumed to be inlined by NMT,
>> are not always inlined. I see the callsites not always the same on
>> different platforms, but have yet see crashes due to stack walking.
>>
>> I ran NMT through JPRT and other tests with tracking level = detail,
>> which means to perform stack walking for callsites, I have not seen any
>> crashes related to stack walking for a fair long time (3 ~ 4 months)
>>
>> But again, that just small set of compilers, which we use ...
>>
>> Thanks,
>>
>> -Zhengyu
>>
>>   General comment which I'm sure you've had before: it is a real shame
>>>>> that every NEW/FREE_C_HEAP_ARRAY etc had to be modified to add a tag -
>>>>> and it seems the majority of them are mtInternal. Could we not define
>>>>> the existing macros to add mtInternal, and then define new macros, eg
>>>>> NEW_C_HEAP_ARRAY_TAGGED, for explicit tagging for use when not
>>>>> mtInternal?
>>>>>
>>>>>  New macros can be a solution.
>>>>
>>>
>>> Thanks for considering. I know there's a lot of schedule pressure on
>>> this work.
>>>
>>> David
>>> -----
>>>
>>>  Thanks,
>>>>
>>>> -Zhengyu
>>>>
>>>>  Thanks,
>>>>> David
>>>>> -----
>>>>>
>>>>> On 28/06/2012 11:32 AM, Zhengyu Gu wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Sorry for the long delay, here is the updated webrev:
>>>>>> http://cr.openjdk.java.net/~zgu/6995781/webrev.01
>>>>>>
>>>>>> Thanks for many people who take time to review NMT code, the new
>>>>>> webrev
>>>>>> reflects the comments and suggestions from you, along with the fixes
>>>>>> of
>>>>>> serviceability agent and some bugs.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -Zhengyu
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 6/12/2012 12:11 PM, Zhengyu Gu wrote:
>>>>>>
>>>>>>> This is the webrev for native memory tracking phase 1, which is
>>>>>>> tracked by CR6995781
>>>>>>> (http://monaco.us.oracle.com/detail.jsf?cr=6995781) and related DCmd
>>>>>>> CR7151532 (http://monaco.us.oracle.com/detail.jsf?cr=7151532).
>>>>>>>
>>>>>>> Native memory tracking (NMT) phase 1, is designed to track native
>>>>>>> memory (malloc'd and mmap'd) usages by VM code only, it does not
>>>>>>> track
>>>>>>> the memory usages by libraries and JNI code.
>>>>>>>
>>>>>>> On the implementation side, NMT intercepts memory related calls in os
>>>>>>> implementation, such as os::malloc, os::realloc, os::free,
>>>>>>> os::reserve_memory, os::commit_memory and etc. the caller is required
>>>>>>> to provide the 'memory type' information, which represents the VM
>>>>>>> subsystem that the memory is allocated for. The 'memory type' is
>>>>>>> defined in /src/share/memory/allocation.hpp. Also, a caller's pc is
>>>>>>> captured if NMT tracking level is set to detail.
>>>>>>>
>>>>>>> There are a few ways to tag a memory block to a memory type:
>>>>>>>
>>>>>>> 1. os::malloc takes an extra parameter for the memory type.
>>>>>>> 2. CHeapObj now is a template, it takes a memory type as template
>>>>>>> parameter to tag the object to a specified memory type.
>>>>>>> 3. Utility classes, such as Arena, GrowableArray and etc. the memory
>>>>>>> type is a parameter of 'new' operator.
>>>>>>> 4. Virtual memory block is tagged on its base address.
>>>>>>>
>>>>>>> When NMT intercepts the call, a memory record is created and
>>>>>>> serialized by a sequence number, generated by global sequence
>>>>>>> generator. The memory record is written to a per-thread recorder
>>>>>>> without a lock, when calling thread is a 'safepoint visible'
>>>>>>> JavaThread - a JavaThread that will block at safepoint. Otherwise,
>>>>>>> ThreadCritical lock is acquired to write to a global recorder. The
>>>>>>> recorders (or raw data) are synchronized at some safepoints, and
>>>>>>> global sequence generator is also reset at synchronization time, to
>>>>>>> avoid overflow the sequence number.
>>>>>>>
>>>>>>> A dedicated NMT worker thread is created to process the raw memory
>>>>>>> records. It first stages a generation of raw data (a generation is
>>>>>>> defined as span between resets of global sequence number), then
>>>>>>> promotes the staged data to a global snapshot, which maintains
>>>>>>> information of all 'live' memory block.
>>>>>>>
>>>>>>> NMT Tracking option has to be specified through VM command line
>>>>>>> option
>>>>>>> -XX:NativeMemoryTracking (off by default), tracking level can not be
>>>>>>> altered during runtime, but it can be shutdown through jcmd tool.
>>>>>>>
>>>>>>> NMT DCmd implements a set of sub-command to control NMT runtime and
>>>>>>> request tracking data.
>>>>>>>
>>>>>>> Webrev: http://cr.openjdk.java.net/~zgu/6995781/webrev.00
>>>>>>>
>>>>>>> The tests and results:
>>>>>>>
>>>>>>> - Passed internal smoke tests
>>>>>>> - JTreg test: http://cr.openjdk.java.net/~zgu/6995781/JTreport/
>>>>>>>
>>>>>>>
>>>>>>> Additional notes:
>>>>>>> - Some classes' new operators are marked _NOINLINE_ (for the same
>>>>>>> purpose as above), the performance runs, I performed, indicate that
>>>>>>> they do not impact performance numbers.
>>>>>>>
>>>>>>> - SA agent changes are in progress
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> -Zhengyu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>