Code review request: CR 6995781 Native Memory Tracking Phase 1

[Coleen Phillimore]

Dumping the memory usage information in the hs_err file is something we 
wanted but sort of got overwhelmed by the number of other changes.  
Zhengyu I think we should file an RFE to add that as soon as the bulk of 
the changes are in.
Thanks,
Coleen

On 6/13/2012 3:16 PM, Vladimir Kozlov wrote:
> Zhengyu Gu wrote:
>> On 6/13/2012 2:52 PM, Vladimir Kozlov wrote:
>>> Zhengyu Gu wrote:
>>>> Vladimir,
>>>>
>>>> I did find that compiler sometimes uses Arenas as value or stack 
>>>> objects,  they are actually tracked through 
>>>> Arena::set_size_in_bytes(), only they can not be categorized, so 
>>>> you may see "Unknown" category.
>>>>
>>>> Thread resource areas are malloc'd arenas, they should already be 
>>>> tracked. GrowableArray is tracked through c-heap allocation (c-heap 
>>>> type), arena or resource area.
>>>
>>> Yes, they are tracked but they are not marked as mtCompiler. So 
>>> result will be is not as useful as it could be.
>>>
>> I see.  I probably can find a way to tag "embedded arena". But if 
>> compiler uses resource area, it is very hard to tell, since NMT does 
>> not track objects inside arena.
>
> Zhengyu, I think you should go with your current version and think 
> about further improvement after that. It whould be more clear how we 
> can improve it after we use it for some time.
>
> Thanks,
> Vladimir
>
>>
>> -Zhengyu
>>
>>>>
>>>> Please let me know if I missed anything, and could you point me a 
>>>> few instances?
>>>
>>> What about dumping memory usage info into hs_err file when VM crash?
>>>
>>> Vladimir
>>>
>>>>
>>>> Thanks,
>>>>
>>>> -Zhengyu
>>>>
>>>> On 6/13/2012 2:15 PM, Vladimir Kozlov wrote:
>>>>> Zhengyu,
>>>>>
>>>>> You missed a lot of places in JIT compilers, c1 and opto, where we 
>>>>> use arenas, thread resource areas, GrowableArrays. Could be 
>>>>> because some arenas are embedded like _comp_arena. Compilers can 
>>>>> eat several hundreds MBytes easily.
>>>>>
>>>>> Is it possible to implement lightweight of this to turn it ON 
>>>>> always to dump usage info into hs_err file when VM crash? You 
>>>>> don't need to track PC for that, only total usage by MemoryType. 
>>>>> It would be very useful because we see a lot of OOM crashes 
>>>>> recently (we have it in each Nightly testing).
>>>>>
>>>>> Thanks,
>>>>> Vladimir
>>>>>
>>>>> Zhengyu Gu wrote:
>>>>>> This is the webrev for native memory tracking phase 1, which is 
>>>>>> tracked by CR6995781 
>>>>>> (http://monaco.us.oracle.com/detail.jsf?cr=6995781) and related 
>>>>>> DCmd CR7151532 (http://monaco.us.oracle.com/detail.jsf?cr=7151532).
>>>>>>
>>>>>> Native memory tracking (NMT) phase 1, is designed to track native 
>>>>>> memory (malloc'd and mmap'd) usages by VM code only, it does not 
>>>>>> track the memory usages by libraries and JNI code.
>>>>>>
>>>>>> On the implementation side, NMT intercepts memory related calls 
>>>>>> in os implementation, such as os::malloc, os::realloc, os::free, 
>>>>>> os::reserve_memory, os::commit_memory and etc. the caller is 
>>>>>> required to provide the 'memory type' information, which 
>>>>>> represents the VM subsystem that the memory is allocated for. The 
>>>>>> 'memory type' is defined in /src/share/memory/allocation.hpp. 
>>>>>> Also, a caller's pc is captured if NMT tracking level is set to 
>>>>>> detail.
>>>>>>
>>>>>> There are a few ways to tag a memory block to a memory type:
>>>>>>
>>>>>> 1. os::malloc takes an extra parameter for the memory type.
>>>>>> 2. CHeapObj now is a template, it takes a memory type as template 
>>>>>> parameter to tag the object to a specified memory type.
>>>>>> 3. Utility classes, such as Arena, GrowableArray and etc. the 
>>>>>> memory type is a parameter of 'new' operator.
>>>>>> 4. Virtual memory block is tagged on its base address.
>>>>>>
>>>>>> When NMT intercepts the call, a memory record is created and 
>>>>>> serialized by a sequence number, generated by global sequence 
>>>>>> generator. The memory record is written to a per-thread recorder 
>>>>>> without a  lock, when calling thread is a 'safepoint visible' 
>>>>>> JavaThread - a JavaThread that will block at safepoint. 
>>>>>> Otherwise, ThreadCritical lock is acquired to write to a global 
>>>>>> recorder. The recorders (or raw data) are synchronized at some 
>>>>>> safepoints, and global sequence generator is also reset at 
>>>>>> synchronization time, to avoid overflow the sequence number.
>>>>>>
>>>>>> A dedicated NMT worker thread is created to process the raw 
>>>>>> memory records. It first stages a generation of raw data (a 
>>>>>> generation is defined as span between resets of global sequence 
>>>>>> number), then promotes the staged data to a global snapshot, 
>>>>>> which maintains information of all 'live' memory block.
>>>>>>
>>>>>> NMT Tracking option has to be specified through VM command line 
>>>>>> option -XX:NativeMemoryTracking (off by default), tracking level 
>>>>>> can not be altered during runtime, but it can be shutdown through 
>>>>>> jcmd tool.
>>>>>>
>>>>>> NMT DCmd implements a set of sub-command to control NMT runtime 
>>>>>> and request tracking data.
>>>>>>
>>>>>> Webrev: http://cr.openjdk.java.net/~zgu/6995781/webrev.00
>>>>>>
>>>>>> The tests and results:
>>>>>>
>>>>>>  - Passed internal smoke tests
>>>>>>  - JTreg test: http://cr.openjdk.java.net/~zgu/6995781/JTreport/
>>>>>>
>>>>>>
>>>>>> Additional notes:
>>>>>>   - Some classes' new operators are marked _NOINLINE_ (for the 
>>>>>> same purpose as above), the performance runs, I performed, 
>>>>>> indicate that they do not impact performance numbers.
>>>>>>
>>>>>>   - SA agent changes are in progress
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -Zhengyu
>>>>>>
>>>>>>
>>>>>>
>>>>>>