Updated: RFR(L) 8046598: Scalable Native memory tracking development

[Zhengyu Gu]

Coleen,

Thanks for the review. I will cleanup as you suggested.

-Zhengyu


On 8/6/2014 8:47 AM, Coleen Phillimore wrote:
>
> memReporter.hpp and cpp copyrights look messed up.  Only the latest 
> date should be changed.
>
> mallocSiteTable.hpp typo
>
>  135     // A very large negnative number. The only possibility to 
> "overflow"
>
>
> mallocTracker.cpp typo chunck
>
>   63 // from total chuncks to get total free chunck size
>
>
> virtualMemoryTracker.hpp typo
>
>
>  153   // Move virtual memory from one memorty type to another.
>
>
> virtualMemoryTracker.hpp   It looks like you couldn't remove the 
> assignment operator so you should add a copy constructor.  You could 
> add the copy constructor as private (with ShouldNotReachHere) to make 
> sure nobody calls it.
>
>  365   ReservedMemoryRegion& operator= (const ReservedMemoryRegion& 
> other) {
>
>
> Ship it!
> Coleen
>
> On 8/4/14, 2:22 PM, Zhengyu Gu wrote:
>> The update is based on code review meetings with Coleen, Christian, 
>> George and Harold.
>>
>> http://cr.openjdk.java.net/~zgu/8046598/webrev.01/ 
>> <http://cr.openjdk.java.net/%7Ezgu/8046598/webrev.01/>
>>
>>
>> Thanks,
>>
>> -Zhengyu
>>
>>
>>
>> On 7/15/2014 4:12 PM, Zhengyu Gu wrote:
>>> This is an update to previous RFR 8028541: Native Memory Tracking 
>>> enhancement, the original one is closed as duplicate of current one.
>>>
>>> The update is mainly based on feedback from Coleen and Christian:
>>>   - Refactored MemReporter to break up some large functions and 
>>> eliminate duplicated code.
>>>   - Minor change to MemBaseline for eliminating duplicated code.
>>>   - Changed MEMFLAGS type from unsigned  short => MemoryType.
>>>
>>> Also added unit tests for LinkedList.
>>>
>>> The note from  RFR 8028541:
>>> =========================
>>> This is significant rework of native memory tracking introduced in 
>>> earlier releases.
>>>
>>> The goal of this enhancement is to improve scalability, from both 
>>> tracking memory and CPU usage perspectives, so it can scale well 
>>> with increased memory allocation in large applications.
>>>
>>> The enhancement is mainly focused on malloc memory tracking, whose 
>>> activities are several magnitude higher than virtual memory, and was 
>>> the main bottleneck in early implementation.
>>>
>>> Instead of using book keeping records for tracking malloc 
>>> activities, new implementation co-locates tracking data along side 
>>> with user data by using a prefixed header. The header size is 8 
>>> bytes on 32-bit systems and 16 bytes on 64-bit systems, which ensure 
>>> that user data also align properly.
>>>
>>> Virtual memory tracking still uses book keeping records, and 
>>> ThreadCritical lock is always acquired to alter the records and 
>>> related data structures.
>>>
>>> Summary tracking data is maintained in static data structures, via 
>>> atomic operations. Malloc detail tracking call stacks are maintained 
>>> in a lock free hashtable.
>>>
>>> The key improvements:
>>>   1. Up-to-date tracking report.
>>>   2. Detail tracking now shows multiple call frames. Number of 
>>> frames is compilation time decision, currently default to 4.
>>>   3. Malloc tracking is lock free.
>>>   4. Tracking summary is reported in hs_err file when native memory 
>>> tracking is enabled.
>>>   5. Query is faster, uses little memory and need a very little 
>>> process.
>>>
>>> The drawback is that, malloc tracking header is always needed if 
>>> native memory tracking has ever been enabled, even after tracking is 
>>> shutdown.
>>>
>>> Impacts:
>>>   The most noticeable impact for JVM developers, is that Arena now 
>>> also take memory type as constructor parameter, besides the new 
>>> operators.
>>>      Arena* a = new (mtCode) Arena()  => Arena* a = new (mtCode) 
>>> Arena(mtCode)
>>>
>>> The webrev shows modification of about 60 files, but most of them 
>>> are due to tracking API changes, mainly due to tracking stack, now, 
>>> is an object, vs. a single pc.
>>>
>>> The most important files for this implementations are:
>>>
>>> memTracker.hpp/cpp
>>> mallocTracker.hpp/cpp and mallocTracker.inline.hpp
>>> virtualMemoryTracker.hpp/cpp
>>> mallocSiteTable.hpp/cpp
>>>
>>> allocationSite.hpp
>>> nativeCallStack.hpp/cpp
>>> linkedlist.hpp
>>>
>>>
>>> Tests:
>>>   - JPRT
>>>   - NMT test suite
>>>   - vm.quick.testlist
>>>   - Kitchensink stability test for 16+ days
>>>   - FMW
>>>
>>>
>>>
>>>
>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8046598
>>> Webrev: http://cr.openjdk.java.net/~zgu/8046598/webrev.00/ 
>>> <http://cr.openjdk.java.net/%7Ezgu/8046598/webrev.00/>
>>>
>>>
>>>
>>> Thanks,
>>>
>>> -Zhengyu
>>>
>>
>