Updated: RFR(L) 8046598: Scalable Native memory tracking development

[Christian Tornqvist]

Hi Zhengyu,

Split the assert at:

src/share/vm/services/memBaseline.hpp @136

As mentioned during our code review meetings, "at" is not a good name for a
method and should be renamed to be more descriptive
(src/share/vm/services/memTracker.cpp @242)

Otherwise this looks good.

Thanks,
Christian

-----Original Message-----
From: hotspot-dev [mailto:hotspot-dev-bounces at openjdk.java.net] On Behalf Of
Zhengyu Gu
Sent: Monday, August 4, 2014 2:23 PM
To: hotspot-dev at openjdk.java.net
Subject: Updated: RFR(L) 8046598: Scalable Native memory tracking
development

The update is based on code review meetings with Coleen, Christian, George
and Harold.

http://cr.openjdk.java.net/~zgu/8046598/webrev.01/
<http://cr.openjdk.java.net/%7Ezgu/8046598/webrev.01/>


Thanks,

-Zhengyu



On 7/15/2014 4:12 PM, Zhengyu Gu wrote:
> This is an update to previous RFR 8028541: Native Memory Tracking 
> enhancement, the original one is closed as duplicate of current one.
>
> The update is mainly based on feedback from Coleen and Christian:
>   - Refactored MemReporter to break up some large functions and 
> eliminate duplicated code.
>   - Minor change to MemBaseline for eliminating duplicated code.
>   - Changed MEMFLAGS type from unsigned  short => MemoryType.
>
> Also added unit tests for LinkedList.
>
> The note from  RFR 8028541:
> =========================
> This is significant rework of native memory tracking introduced in 
> earlier releases.
>
> The goal of this enhancement is to improve scalability, from both 
> tracking memory and CPU usage perspectives, so it can scale well with 
> increased memory allocation in large applications.
>
> The enhancement is mainly focused on malloc memory tracking, whose 
> activities are several magnitude higher than virtual memory, and was 
> the main bottleneck in early implementation.
>
> Instead of using book keeping records for tracking malloc activities, 
> new implementation co-locates tracking data along side with user data 
> by using a prefixed header. The header size is 8 bytes on 32-bit 
> systems and 16 bytes on 64-bit systems, which ensure that user data 
> also align properly.
>
> Virtual memory tracking still uses book keeping records, and 
> ThreadCritical lock is always acquired to alter the records and 
> related data structures.
>
> Summary tracking data is maintained in static data structures, via 
> atomic operations. Malloc detail tracking call stacks are maintained 
> in a lock free hashtable.
>
> The key improvements:
>   1. Up-to-date tracking report.
>   2. Detail tracking now shows multiple call frames. Number of frames 
> is compilation time decision, currently default to 4.
>   3. Malloc tracking is lock free.
>   4. Tracking summary is reported in hs_err file when native memory 
> tracking is enabled.
>   5. Query is faster, uses little memory and need a very little process.
>
> The drawback is that, malloc tracking header is always needed if 
> native memory tracking has ever been enabled, even after tracking is 
> shutdown.
>
> Impacts:
>   The most noticeable impact for JVM developers, is that Arena now 
> also take memory type as constructor parameter, besides the new 
> operators.
>      Arena* a = new (mtCode) Arena()  => Arena* a = new (mtCode) 
> Arena(mtCode)
>
> The webrev shows modification of about 60 files, but most of them are 
> due to tracking API changes, mainly due to tracking stack, now, is an 
> object, vs. a single pc.
>
> The most important files for this implementations are:
>
> memTracker.hpp/cpp
> mallocTracker.hpp/cpp and mallocTracker.inline.hpp
> virtualMemoryTracker.hpp/cpp
> mallocSiteTable.hpp/cpp
>
> allocationSite.hpp
> nativeCallStack.hpp/cpp
> linkedlist.hpp
>
>
> Tests:
>   - JPRT
>   - NMT test suite
>   - vm.quick.testlist
>   - Kitchensink stability test for 16+ days
>   - FMW
>
>
>
>
> Bug: https://bugs.openjdk.java.net/browse/JDK-8046598
> Webrev: http://cr.openjdk.java.net/~zgu/8046598/webrev.00/ 
> <http://cr.openjdk.java.net/%7Ezgu/8046598/webrev.00/>
>
>
>
> Thanks,
>
> -Zhengyu
>