[PATCH] RFR Bug-pending: Enable Hotspot to Track Native Memory Usage for Direct Byte Buffers

[Zhengyu Gu]

>> Think of it as an NMT upgrade.
>>
>> Here's an example of what the output should look like:
>>
>> https://developer.ibm.com/answers/questions/288697/why-
>> does-nativememinfo-in-javacore-show-incorrect.html?sort=oldest
>>
>> - Adam
>>
>>
> I think NMT walks the stack, so we should get allocation points grouped by
> call stacks. Provided we have symbols loaded for the native library using
> Unsafe.allocateMemory(), this should give us too a fine granularity. But I
> have not yet tested this in practice. Maybe Zhengyu knows more.

Quick test shows this call site:

[0x00007f8558b26243] Unsafe_AllocateMemory0+0x93
[0x00007f8537b085cb]
                              (malloc=2KB type=Internal #1)

I will take a look why there is a frame not decoded.

Thanks,

-Zhengyu



> 
> ..Thomas
> 
> 
>>>
>>>> David
>>>>
>>>> On 14/02/2018 9:32 PM, Adam Farley8 wrote:
>>>>> Hi All,
>>>>>
>>>>> Currently, diagnostic core files generated from OpenJDK seem to lump
>> all
>>>>> of the
>>>>> native memory usages together, making it near-impossible for someone
>> to
>>>>> figure
>>>>> out *what* is using all that memory in the event of a memory leak.
>>>>>
>>>>> The OpenJ9 VM has a feature which allows it to track the allocation of
>>>>> native
>>>>> memory for Direct Byte Buffers (DBBs), and to supply that information
>>>>> into
>>>>> the
>>>>> cores when they are generated. This makes it a *lot* easier to find
>> out
>>>>> what is using
>>>>> all that native memory, making memory leak resolution less like some
>> dark
>>>>> art, and
>>>>> more like logical debugging.
>>>>>
>>>>> To use this feature, there is a native method referenced in
>> Unsafe.java.
>>>>> To open
>>>>> up this feature so that any VM can make use of it, the java code below
>>>>> sets the
>>>>> stage for it. This change starts letting people call DBB-specific
>> methods
>>>>> when
>>>>> allocating native memory, and getting into the habit of using it.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> Best Regards
>>>>>
>>>>> Adam Farley
>>>>>
>>>>> P.S. Code:
>>>>>
>>>>> diff --git
>>>>> a/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>>>> b/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>>>> ---
>> a/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>>>> +++
>> b/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>>>> @@ -85,7 +85,7 @@
>>>>>                    // Paranoia
>>>>>                    return;
>>>>>                }
>>>>> -            UNSAFE.freeMemory(address);
>>>>> +            UNSAFE.freeDBBMemory(address);
>>>>>                address = 0;
>>>>>                Bits.unreserveMemory(size, capacity);
>>>>>            }
>>>>> @@ -118,7 +118,7 @@
>>>>>            long base = 0;
>>>>>            try {
>>>>> -            base = UNSAFE.allocateMemory(size);
>>>>> +            base = UNSAFE.allocateDBBMemory(size);
>>>>>            } catch (OutOfMemoryError x) {
>>>>>                Bits.unreserveMemory(size, cap);
>>>>>                throw x;
>>>>> diff --git a/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>>>> b/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>>>> --- a/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>>>> +++ b/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>>>> @@ -632,6 +632,26 @@
>>>>>        }
>>>>>        /**
>>>>> +     * Allocates a new block of native memory for DirectByteBuffers,
>> of
>>>>> the
>>>>> +     * given size in bytes.  The contents of the memory are
>>>>> uninitialized;
>>>>> +     * they will generally be garbage.  The resulting native pointer
>>>>> will
>>>>> +     * never be zero, and will be aligned for all value types.
>> Dispose
>>>>> of
>>>>> +     * this memory by calling {@link #freeDBBMemory} or resize it
>> with
>>>>> +     * {@link #reallocateDBBMemory}.
>>>>> +     *
>>>>> +     * @throws RuntimeException if the size is negative or too large
>>>>> +     *                          for the native size_t type
>>>>> +     *
>>>>> +     * @throws OutOfMemoryError if the allocation is refused by the
>>>>> system
>>>>> +     *
>>>>> +     * @see #getByte(long)
>>>>> +     * @see #putByte(long, byte)
>>>>> +     */
>>>>> +    public long allocateDBBMemory(long bytes) {
>>>>> +        return allocateMemory(bytes);
>>>>> +    }
>>>>> +
>>>>> +    /**
>>>>>         * Resizes a new block of native memory, to the given size in
>>>>> bytes.
>>>>> The
>>>>>         * contents of the new block past the size of the old block are
>>>>>         * uninitialized; they will generally be garbage.  The resulting
>>>>> native
>>>>> @@ -687,6 +707,27 @@
>>>>>        }
>>>>>        /**
>>>>> +     * Resizes a new block of native memory for DirectByteBuffers, to
>>
>>>>> the
>>>>> +     * given size in bytes.  The contents of the new block past the
>> size
>>>>> of
>>>>> +     * the old block are uninitialized; they will generally be
>> garbage.
>>>>> The
>>>>> +     * resulting native pointer will be zero if and only if the
>>>>> requested
>>>>> size
>>>>> +     * is zero.  The resulting native pointer will be aligned for all
>>>>> value
>>>>> +     * types.  Dispose of this memory by calling {@link
>> #freeDBBMemory},
>>>>> or
>>>>> +     * resize it with {@link #reallocateDBBMemory}.  The address
>> passed
>>>>> to
>>>>> +     * this method may be null, in which case an allocation will be
>>>>> performed.
>>>>> +     *
>>>>> +     * @throws RuntimeException if the size is negative or too large
>>>>> +     *                          for the native size_t type
>>>>> +     *
>>>>> +     * @throws OutOfMemoryError if the allocation is refused by the
>>>>> system
>>>>> +     *
>>>>> +     * @see #allocateDBBMemory
>>>>> +     */
>>>>> +    public long reallocateDBBMemory(long address, long bytes) {
>>>>> +        return reallocateMemory(address, bytes);
>>>>> +    }
>>>>> +
>>>>> +    /**
>>>>>         * Sets all bytes in a given block of memory to a fixed value
>>>>>         * (usually zero).
>>>>>         *
>>>>> @@ -918,6 +959,17 @@
>>>>>            checkPointer(null, address);
>>>>>        }
>>>>> +    /**
>>>>> +     * Disposes of a block of native memory, as obtained from {@link
>>>>> +     * #allocateDBBMemory} or {@link #reallocateDBBMemory}.  The
>> address
>>>>> passed
>>>>> +     * to this method may be null, in which case no action is taken.
>>>>> +     *
>>>>> +     * @see #allocateDBBMemory
>>>>> +     */
>>>>> +    public void freeDBBMemory(long address) {
>>>>> +        freeMemory(address);
>>>>> +    }
>>>>> +
>>>>>        /// random queries
>>>>>        /**
>>>>>
>>>>> Unless stated otherwise above:
>>>>> IBM United Kingdom Limited - Registered in England and Wales with
>> number
>>>>> 741598.
>>>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>>
>>>>> 3AU
>>>>>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>