[PATCH] RFR Bug-pending: Enable Hotspot to Track Native Memory Usage for Direct Byte Buffers

[Zhengyu Gu]

On 02/14/2018 08:16 AM, Thomas Stüfe wrote:
> On Wed, Feb 14, 2018 at 1:53 PM, David Holmes <david.holmes at oracle.com>
> wrote:
> 
>> On 14/02/2018 10:43 PM, David Holmes wrote:
>>
>>> Adding in core-libs-dev as there's nothing related to hotspot directly
>>> here.
>>>
>>
>> Correction, this is of course leading to a proposed change in hotspot to
>> implement the new Unsafe methods and perform the native memory tracking. Of
>> course we already have NMT so the obvious question is how this will fit in
>> with NMT?
>>
>>
> I thought Unsafe.allocateMemory is served by hotspot os::malloc(), is it
> not? So, allocations should show up in NMT with "Unsafe_AllocateMemory0".

Could use another category? to make it earlier to identify.

Thanks,

-Zhengyu

> 
> ..Thomas
> 
> 
> 
>> David
>>
>>
>> David
>>>
>>> On 14/02/2018 9:32 PM, Adam Farley8 wrote:
>>>
>>>> Hi All,
>>>>
>>>> Currently, diagnostic core files generated from OpenJDK seem to lump all
>>>> of the
>>>> native memory usages together, making it near-impossible for someone to
>>>> figure
>>>> out *what* is using all that memory in the event of a memory leak.
>>>>
>>>> The OpenJ9 VM has a feature which allows it to track the allocation of
>>>> native
>>>> memory for Direct Byte Buffers (DBBs), and to supply that information
>>>> into
>>>> the
>>>> cores when they are generated. This makes it a *lot* easier to find out
>>>> what is using
>>>> all that native memory, making memory leak resolution less like some dark
>>>> art, and
>>>> more like logical debugging.
>>>>
>>>> To use this feature, there is a native method referenced in Unsafe.java.
>>>> To open
>>>> up this feature so that any VM can make use of it, the java code below
>>>> sets the
>>>> stage for it. This change starts letting people call DBB-specific methods
>>>> when
>>>> allocating native memory, and getting into the habit of using it.
>>>>
>>>> Thoughts?
>>>>
>>>> Best Regards
>>>>
>>>> Adam Farley
>>>>
>>>> P.S. Code:
>>>>
>>>> diff --git
>>>> a/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>>> b/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>>> --- a/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>>> +++ b/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>>> @@ -85,7 +85,7 @@
>>>>                    // Paranoia
>>>>                    return;
>>>>                }
>>>> -            UNSAFE.freeMemory(address);
>>>> +            UNSAFE.freeDBBMemory(address);
>>>>                address = 0;
>>>>                Bits.unreserveMemory(size, capacity);
>>>>            }
>>>> @@ -118,7 +118,7 @@
>>>>            long base = 0;
>>>>            try {
>>>> -            base = UNSAFE.allocateMemory(size);
>>>> +            base = UNSAFE.allocateDBBMemory(size);
>>>>            } catch (OutOfMemoryError x) {
>>>>                Bits.unreserveMemory(size, cap);
>>>>                throw x;
>>>> diff --git a/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>>> b/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>>> --- a/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>>> +++ b/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>>> @@ -632,6 +632,26 @@
>>>>        }
>>>>        /**
>>>> +     * Allocates a new block of native memory for DirectByteBuffers, of
>>>> the
>>>> +     * given size in bytes.  The contents of the memory are
>>>> uninitialized;
>>>> +     * they will generally be garbage.  The resulting native pointer
>>>> will
>>>> +     * never be zero, and will be aligned for all value types.  Dispose
>>>> of
>>>> +     * this memory by calling {@link #freeDBBMemory} or resize it with
>>>> +     * {@link #reallocateDBBMemory}.
>>>> +     *
>>>> +     * @throws RuntimeException if the size is negative or too large
>>>> +     *                          for the native size_t type
>>>> +     *
>>>> +     * @throws OutOfMemoryError if the allocation is refused by the
>>>> system
>>>> +     *
>>>> +     * @see #getByte(long)
>>>> +     * @see #putByte(long, byte)
>>>> +     */
>>>> +    public long allocateDBBMemory(long bytes) {
>>>> +        return allocateMemory(bytes);
>>>> +    }
>>>> +
>>>> +    /**
>>>>         * Resizes a new block of native memory, to the given size in
>>>> bytes.
>>>> The
>>>>         * contents of the new block past the size of the old block are
>>>>         * uninitialized; they will generally be garbage.  The resulting
>>>> native
>>>> @@ -687,6 +707,27 @@
>>>>        }
>>>>        /**
>>>> +     * Resizes a new block of native memory for DirectByteBuffers, to
>>>> the
>>>> +     * given size in bytes.  The contents of the new block past the size
>>>> of
>>>> +     * the old block are uninitialized; they will generally be garbage.
>>>> The
>>>> +     * resulting native pointer will be zero if and only if the
>>>> requested
>>>> size
>>>> +     * is zero.  The resulting native pointer will be aligned for all
>>>> value
>>>> +     * types.  Dispose of this memory by calling {@link #freeDBBMemory},
>>>> or
>>>> +     * resize it with {@link #reallocateDBBMemory}.  The address passed
>>>> to
>>>> +     * this method may be null, in which case an allocation will be
>>>> performed.
>>>> +     *
>>>> +     * @throws RuntimeException if the size is negative or too large
>>>> +     *                          for the native size_t type
>>>> +     *
>>>> +     * @throws OutOfMemoryError if the allocation is refused by the
>>>> system
>>>> +     *
>>>> +     * @see #allocateDBBMemory
>>>> +     */
>>>> +    public long reallocateDBBMemory(long address, long bytes) {
>>>> +        return reallocateMemory(address, bytes);
>>>> +    }
>>>> +
>>>> +    /**
>>>>         * Sets all bytes in a given block of memory to a fixed value
>>>>         * (usually zero).
>>>>         *
>>>> @@ -918,6 +959,17 @@
>>>>            checkPointer(null, address);
>>>>        }
>>>> +    /**
>>>> +     * Disposes of a block of native memory, as obtained from {@link
>>>> +     * #allocateDBBMemory} or {@link #reallocateDBBMemory}.  The address
>>>> passed
>>>> +     * to this method may be null, in which case no action is taken.
>>>> +     *
>>>> +     * @see #allocateDBBMemory
>>>> +     */
>>>> +    public void freeDBBMemory(long address) {
>>>> +        freeMemory(address);
>>>> +    }
>>>> +
>>>>        /// random queries
>>>>        /**
>>>>
>>>> Unless stated otherwise above:
>>>> IBM United Kingdom Limited - Registered in England and Wales with number
>>>> 741598.
>>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>>>> 3AU
>>>>
>>>>