Updated: RFR(L) 8046598: Scalable Native memory tracking development

[Zhengyu Gu]

Thanks, George.

I will fix them accordingly.

-Zhengyu

On 8/6/2014 12:40 PM, George Triantafillou wrote:
> Hi Zhengyu,
>
> I found some typos.  Otherwise, it looks good.
>
> -George
>
> src/share/vm/services/memTracker.hpp:
>
>     142   // Launcher only obligates to pass native memory tracking
>     142   // Launcher only obligated to pass native memory tracking
>
>     143   // option value, but not obligates to validate the value,
>     143   // option value, but not obligated to validate the value,
>
>     192   // chuncks that backing up the arena.
>     192   // chunks backing up the arena.
>
> src/share/vm/services/mallocTracker.hpp:
>
>     181   // from total chuncks to get total free chunck size
>     181   // from total chunks to get total free chunk size
>
> src/share/vm/services/virtualMemoryTracker.cpp:
>
>     75 // the two adjacent regions have to same call stack, merge them
>     75         // the two adjacent regions have the same call stack,
>     merge them
>
>     314       // Overalpped reservation.
>     314       // Overlapped reservation.
>
> src/share/vm/utilities/linkedlist.hpp:
>
>     37 // as the linked list what contains this entry.
>     37 // as the linked list that contains this entry.
>
>     57 // any storage type it uses, so all methods involves
>     57 // any storage type it uses, so all methods involving
>
> src/share/vm/services/mallocSiteTable.hpp:
>
>     73   // The operation can be failed due to contention from other
>     thread.
>     73   // The operation can fail due to contention from another thread.
>
> src/share/vm/services/mallocSiteTable.cpp:
>
>     173 // contented, other thread won
>     173       // contended, other thread won
>
>
> On 8/4/2014 2:22 PM, Zhengyu Gu wrote:
>> The update is based on code review meetings with Coleen, Christian, 
>> George and Harold.
>>
>> http://cr.openjdk.java.net/~zgu/8046598/webrev.01/ 
>> <http://cr.openjdk.java.net/%7Ezgu/8046598/webrev.01/>
>>
>>
>> Thanks,
>>
>> -Zhengyu
>>
>>
>>
>> On 7/15/2014 4:12 PM, Zhengyu Gu wrote:
>>> This is an update to previous RFR 8028541: Native Memory Tracking 
>>> enhancement, the original one is closed as duplicate of current one.
>>>
>>> The update is mainly based on feedback from Coleen and Christian:
>>>   - Refactored MemReporter to break up some large functions and 
>>> eliminate duplicated code.
>>>   - Minor change to MemBaseline for eliminating duplicated code.
>>>   - Changed MEMFLAGS type from unsigned  short => MemoryType.
>>>
>>> Also added unit tests for LinkedList.
>>>
>>> The note from  RFR 8028541:
>>> =========================
>>> This is significant rework of native memory tracking introduced in 
>>> earlier releases.
>>>
>>> The goal of this enhancement is to improve scalability, from both 
>>> tracking memory and CPU usage perspectives, so it can scale well 
>>> with increased memory allocation in large applications.
>>>
>>> The enhancement is mainly focused on malloc memory tracking, whose 
>>> activities are several magnitude higher than virtual memory, and was 
>>> the main bottleneck in early implementation.
>>>
>>> Instead of using book keeping records for tracking malloc 
>>> activities, new implementation co-locates tracking data along side 
>>> with user data by using a prefixed header. The header size is 8 
>>> bytes on 32-bit systems and 16 bytes on 64-bit systems, which ensure 
>>> that user data also align properly.
>>>
>>> Virtual memory tracking still uses book keeping records, and 
>>> ThreadCritical lock is always acquired to alter the records and 
>>> related data structures.
>>>
>>> Summary tracking data is maintained in static data structures, via 
>>> atomic operations. Malloc detail tracking call stacks are maintained 
>>> in a lock free hashtable.
>>>
>>> The key improvements:
>>>   1. Up-to-date tracking report.
>>>   2. Detail tracking now shows multiple call frames. Number of 
>>> frames is compilation time decision, currently default to 4.
>>>   3. Malloc tracking is lock free.
>>>   4. Tracking summary is reported in hs_err file when native memory 
>>> tracking is enabled.
>>>   5. Query is faster, uses little memory and need a very little 
>>> process.
>>>
>>> The drawback is that, malloc tracking header is always needed if 
>>> native memory tracking has ever been enabled, even after tracking is 
>>> shutdown.
>>>
>>> Impacts:
>>>   The most noticeable impact for JVM developers, is that Arena now 
>>> also take memory type as constructor parameter, besides the new 
>>> operators.
>>>      Arena* a = new (mtCode) Arena()  => Arena* a = new (mtCode) 
>>> Arena(mtCode)
>>>
>>> The webrev shows modification of about 60 files, but most of them 
>>> are due to tracking API changes, mainly due to tracking stack, now, 
>>> is an object, vs. a single pc.
>>>
>>> The most important files for this implementations are:
>>>
>>> memTracker.hpp/cpp
>>> mallocTracker.hpp/cpp and mallocTracker.inline.hpp
>>> virtualMemoryTracker.hpp/cpp
>>> mallocSiteTable.hpp/cpp
>>>
>>> allocationSite.hpp
>>> nativeCallStack.hpp/cpp
>>> linkedlist.hpp
>>>
>>>
>>> Tests:
>>>   - JPRT
>>>   - NMT test suite
>>>   - vm.quick.testlist
>>>   - Kitchensink stability test for 16+ days
>>>   - FMW
>>>
>>>
>>>
>>>
>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8046598
>>> Webrev: http://cr.openjdk.java.net/~zgu/8046598/webrev.00/ 
>>> <http://cr.openjdk.java.net/%7Ezgu/8046598/webrev.00/>
>>>
>>>
>>>
>>> Thanks,
>>>
>>> -Zhengyu
>>>
>>
>