Updated: RFR(L) 8046598: Scalable Native memory tracking development

[George Triantafillou]

Hi Zhengyu,

I found some typos.  Otherwise, it looks good.

-George

src/share/vm/services/memTracker.hpp:

    142   // Launcher only obligates to pass native memory tracking
    142   // Launcher only obligated to pass native memory tracking

    143   // option value, but not obligates to validate the value,
    143   // option value, but not obligated to validate the value,

    192   // chuncks that backing up the arena.
    192   // chunks backing up the arena.

src/share/vm/services/mallocTracker.hpp:

    181   // from total chuncks to get total free chunck size
    181   // from total chunks to get total free chunk size

src/share/vm/services/virtualMemoryTracker.cpp:

    75 // the two adjacent regions have to same call stack, merge them
    75         // the two adjacent regions have the same call stack,
    merge them

    314       // Overalpped reservation.
    314       // Overlapped reservation.

src/share/vm/utilities/linkedlist.hpp:

    37 // as the linked list what contains this entry.
    37 // as the linked list that contains this entry.

    57 // any storage type it uses, so all methods involves
    57 // any storage type it uses, so all methods involving

src/share/vm/services/mallocSiteTable.hpp:

    73   // The operation can be failed due to contention from other thread.
    73   // The operation can fail due to contention from another thread.

src/share/vm/services/mallocSiteTable.cpp:

    173 // contented, other thread won
    173       // contended, other thread won


On 8/4/2014 2:22 PM, Zhengyu Gu wrote:
> The update is based on code review meetings with Coleen, Christian, 
> George and Harold.
>
> http://cr.openjdk.java.net/~zgu/8046598/webrev.01/ 
> <http://cr.openjdk.java.net/%7Ezgu/8046598/webrev.01/>
>
>
> Thanks,
>
> -Zhengyu
>
>
>
> On 7/15/2014 4:12 PM, Zhengyu Gu wrote:
>> This is an update to previous RFR 8028541: Native Memory Tracking 
>> enhancement, the original one is closed as duplicate of current one.
>>
>> The update is mainly based on feedback from Coleen and Christian:
>>   - Refactored MemReporter to break up some large functions and 
>> eliminate duplicated code.
>>   - Minor change to MemBaseline for eliminating duplicated code.
>>   - Changed MEMFLAGS type from unsigned  short => MemoryType.
>>
>> Also added unit tests for LinkedList.
>>
>> The note from  RFR 8028541:
>> =========================
>> This is significant rework of native memory tracking introduced in 
>> earlier releases.
>>
>> The goal of this enhancement is to improve scalability, from both 
>> tracking memory and CPU usage perspectives, so it can scale well with 
>> increased memory allocation in large applications.
>>
>> The enhancement is mainly focused on malloc memory tracking, whose 
>> activities are several magnitude higher than virtual memory, and was 
>> the main bottleneck in early implementation.
>>
>> Instead of using book keeping records for tracking malloc activities, 
>> new implementation co-locates tracking data along side with user data 
>> by using a prefixed header. The header size is 8 bytes on 32-bit 
>> systems and 16 bytes on 64-bit systems, which ensure that user data 
>> also align properly.
>>
>> Virtual memory tracking still uses book keeping records, and 
>> ThreadCritical lock is always acquired to alter the records and 
>> related data structures.
>>
>> Summary tracking data is maintained in static data structures, via 
>> atomic operations. Malloc detail tracking call stacks are maintained 
>> in a lock free hashtable.
>>
>> The key improvements:
>>   1. Up-to-date tracking report.
>>   2. Detail tracking now shows multiple call frames. Number of frames 
>> is compilation time decision, currently default to 4.
>>   3. Malloc tracking is lock free.
>>   4. Tracking summary is reported in hs_err file when native memory 
>> tracking is enabled.
>>   5. Query is faster, uses little memory and need a very little process.
>>
>> The drawback is that, malloc tracking header is always needed if 
>> native memory tracking has ever been enabled, even after tracking is 
>> shutdown.
>>
>> Impacts:
>>   The most noticeable impact for JVM developers, is that Arena now 
>> also take memory type as constructor parameter, besides the new 
>> operators.
>>      Arena* a = new (mtCode) Arena()  => Arena* a = new (mtCode) 
>> Arena(mtCode)
>>
>> The webrev shows modification of about 60 files, but most of them are 
>> due to tracking API changes, mainly due to tracking stack, now, is an 
>> object, vs. a single pc.
>>
>> The most important files for this implementations are:
>>
>> memTracker.hpp/cpp
>> mallocTracker.hpp/cpp and mallocTracker.inline.hpp
>> virtualMemoryTracker.hpp/cpp
>> mallocSiteTable.hpp/cpp
>>
>> allocationSite.hpp
>> nativeCallStack.hpp/cpp
>> linkedlist.hpp
>>
>>
>> Tests:
>>   - JPRT
>>   - NMT test suite
>>   - vm.quick.testlist
>>   - Kitchensink stability test for 16+ days
>>   - FMW
>>
>>
>>
>>
>> Bug: https://bugs.openjdk.java.net/browse/JDK-8046598
>> Webrev: http://cr.openjdk.java.net/~zgu/8046598/webrev.00/ 
>> <http://cr.openjdk.java.net/%7Ezgu/8046598/webrev.00/>
>>
>>
>>
>> Thanks,
>>
>> -Zhengyu
>>
>