[PATCH] RFR Bug-pending: Enable Hotspot to Track Native Memory Usage for Direct Byte Buffers

[Thomas Stüfe]

On Wed, Feb 14, 2018 at 1:53 PM, David Holmes <david.holmes at oracle.com>
wrote:

> On 14/02/2018 10:43 PM, David Holmes wrote:
>
>> Adding in core-libs-dev as there's nothing related to hotspot directly
>> here.
>>
>
> Correction, this is of course leading to a proposed change in hotspot to
> implement the new Unsafe methods and perform the native memory tracking. Of
> course we already have NMT so the obvious question is how this will fit in
> with NMT?
>
>
I thought Unsafe.allocateMemory is served by hotspot os::malloc(), is it
not? So, allocations should show up in NMT with "Unsafe_AllocateMemory0".

..Thomas



> David
>
>
> David
>>
>> On 14/02/2018 9:32 PM, Adam Farley8 wrote:
>>
>>> Hi All,
>>>
>>> Currently, diagnostic core files generated from OpenJDK seem to lump all
>>> of the
>>> native memory usages together, making it near-impossible for someone to
>>> figure
>>> out *what* is using all that memory in the event of a memory leak.
>>>
>>> The OpenJ9 VM has a feature which allows it to track the allocation of
>>> native
>>> memory for Direct Byte Buffers (DBBs), and to supply that information
>>> into
>>> the
>>> cores when they are generated. This makes it a *lot* easier to find out
>>> what is using
>>> all that native memory, making memory leak resolution less like some dark
>>> art, and
>>> more like logical debugging.
>>>
>>> To use this feature, there is a native method referenced in Unsafe.java.
>>> To open
>>> up this feature so that any VM can make use of it, the java code below
>>> sets the
>>> stage for it. This change starts letting people call DBB-specific methods
>>> when
>>> allocating native memory, and getting into the habit of using it.
>>>
>>> Thoughts?
>>>
>>> Best Regards
>>>
>>> Adam Farley
>>>
>>> P.S. Code:
>>>
>>> diff --git
>>> a/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>> b/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>> --- a/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>> +++ b/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template
>>> @@ -85,7 +85,7 @@
>>>                   // Paranoia
>>>                   return;
>>>               }
>>> -            UNSAFE.freeMemory(address);
>>> +            UNSAFE.freeDBBMemory(address);
>>>               address = 0;
>>>               Bits.unreserveMemory(size, capacity);
>>>           }
>>> @@ -118,7 +118,7 @@
>>>           long base = 0;
>>>           try {
>>> -            base = UNSAFE.allocateMemory(size);
>>> +            base = UNSAFE.allocateDBBMemory(size);
>>>           } catch (OutOfMemoryError x) {
>>>               Bits.unreserveMemory(size, cap);
>>>               throw x;
>>> diff --git a/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>> b/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>> --- a/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>> +++ b/src/java.base/share/classes/jdk/internal/misc/Unsafe.java
>>> @@ -632,6 +632,26 @@
>>>       }
>>>       /**
>>> +     * Allocates a new block of native memory for DirectByteBuffers, of
>>> the
>>> +     * given size in bytes.  The contents of the memory are
>>> uninitialized;
>>> +     * they will generally be garbage.  The resulting native pointer
>>> will
>>> +     * never be zero, and will be aligned for all value types.  Dispose
>>> of
>>> +     * this memory by calling {@link #freeDBBMemory} or resize it with
>>> +     * {@link #reallocateDBBMemory}.
>>> +     *
>>> +     * @throws RuntimeException if the size is negative or too large
>>> +     *                          for the native size_t type
>>> +     *
>>> +     * @throws OutOfMemoryError if the allocation is refused by the
>>> system
>>> +     *
>>> +     * @see #getByte(long)
>>> +     * @see #putByte(long, byte)
>>> +     */
>>> +    public long allocateDBBMemory(long bytes) {
>>> +        return allocateMemory(bytes);
>>> +    }
>>> +
>>> +    /**
>>>        * Resizes a new block of native memory, to the given size in
>>> bytes.
>>> The
>>>        * contents of the new block past the size of the old block are
>>>        * uninitialized; they will generally be garbage.  The resulting
>>> native
>>> @@ -687,6 +707,27 @@
>>>       }
>>>       /**
>>> +     * Resizes a new block of native memory for DirectByteBuffers, to
>>> the
>>> +     * given size in bytes.  The contents of the new block past the size
>>> of
>>> +     * the old block are uninitialized; they will generally be garbage.
>>> The
>>> +     * resulting native pointer will be zero if and only if the
>>> requested
>>> size
>>> +     * is zero.  The resulting native pointer will be aligned for all
>>> value
>>> +     * types.  Dispose of this memory by calling {@link #freeDBBMemory},
>>> or
>>> +     * resize it with {@link #reallocateDBBMemory}.  The address passed
>>> to
>>> +     * this method may be null, in which case an allocation will be
>>> performed.
>>> +     *
>>> +     * @throws RuntimeException if the size is negative or too large
>>> +     *                          for the native size_t type
>>> +     *
>>> +     * @throws OutOfMemoryError if the allocation is refused by the
>>> system
>>> +     *
>>> +     * @see #allocateDBBMemory
>>> +     */
>>> +    public long reallocateDBBMemory(long address, long bytes) {
>>> +        return reallocateMemory(address, bytes);
>>> +    }
>>> +
>>> +    /**
>>>        * Sets all bytes in a given block of memory to a fixed value
>>>        * (usually zero).
>>>        *
>>> @@ -918,6 +959,17 @@
>>>           checkPointer(null, address);
>>>       }
>>> +    /**
>>> +     * Disposes of a block of native memory, as obtained from {@link
>>> +     * #allocateDBBMemory} or {@link #reallocateDBBMemory}.  The address
>>> passed
>>> +     * to this method may be null, in which case no action is taken.
>>> +     *
>>> +     * @see #allocateDBBMemory
>>> +     */
>>> +    public void freeDBBMemory(long address) {
>>> +        freeMemory(address);
>>> +    }
>>> +
>>>       /// random queries
>>>       /**
>>>
>>> Unless stated otherwise above:
>>> IBM United Kingdom Limited - Registered in England and Wales with number
>>> 741598.
>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>>> 3AU
>>>
>>>