DirectByteBuffer change proposal
Biju G.S Nair
gs.biju at gmail.com
Wed Aug 12 12:53:54 UTC 2015
Thanks Peter for the detailed response. Currently we are in JDK 7 since the
underlying datastore is still being certified for moving to 8. So the
issues we are seeing is not on JDK 9 with the patch. I have requested the
jdk8u-dev and jdk9u-dev team through the dev email whether they can back
port the patch to the two versions so that we can test. Waiting to get the
patch merged. Will update you as soon as the JDK is updated.
Thanks,
Biju
On Wed, Aug 12, 2015 at 2:51 AM, Peter Levart <peter.levart at gmail.com>
wrote:
> Hi Biju,
>
> Just to clear any doubts. You have seen sleep() being called with JDK9
> which already includes the patch for 6857566 ?
>
> The cleanup of native memory for DirecByteBuffers was and after 6857566
> still is designed around sun.misc.Cleaner which is basically a
> PhantomReference pointing to the DirecByteBuffer. When DirectByteBuffer
> instance becomes phantom-reachable, VM will at some later time discover it
> as a pending reference and ReferenceHandlerThread will "clean" it (execute
> Cleaner's thunk) releasing the native memory. So ordinarily, cleanup of
> native memory is performed asynchronously short time after VM decides to
> execute code to discover pending references. This is usually performed when
> heap allocation reaches certain point (maybe also periodically?). The
> problem is that memory pressure detected by VM is based on heap memory
> allocation which does not include native memory. It can and frequently
> happens that there's no heap memory pressure, but native memory reserved
> for DirectByteBuffers is exhausted.
>
> Current strategy works by waiting until this exhaustion happens and then:
> - attempts to "help" ReferenceHandlerThread to drain the pending
> references already discovered by VM, executing any sun.misc.Cleaners on the
> way; if that does not help, it
> - triggers System.gc() which hopefully also triggers discovery of pending
> references, followed by a round of ReferenceHandlerThread helping; if that
> does not help, it
> - retries the gc/tryHandlePending cycle, introducing exponentially
> increasing delays (from 1ms up to 512ms) which also acts as some kind of
> back-pressure; if that does not help, it
> - considers native memory full and throws OutOfMemoryError
>
> You propose to add triggering of gc() when native memory occupancy reaches
> certain % of max. memory allowed for native buffers. This sounds reasonable
> as it mimics roughly what happens when heap allocation reaches certain
> point. Reference handling would still be performed asynchronously solely by
> ReferenceHandlerThread in this case (no helping with tryHandlePending())
> until native memory is exhausted. But note that System.gc() is not free. It
> usually is implemented as a blocking call which triggers safe-point VM
> processing and waits for it to complete before returning. So if you wanted
> to see low latencies for native buffer allocations, this should be
> performed by a background thread that continuously monitors current
> occupancy. You can try doing this in client code. There's a JMX bean that
> exposes native buffer allocation state (BufferPoolMXBean). Can you try
> doing this and report if it helps with allocation latencies in your
> application?
>
> Regards, Peter
>
>
> On 08/11/2015 11:43 PM, Biju G.S Nair wrote:
>
> Hi Peter,
> Thanks for the background. Yes. We have seen sleep being called since
> we are using DirectByteBuffers in GBs and the plan is to see whether we can
> go up more than 90 GB. Also we have seen in certain version of Linux kernel
> the 100 ms sleep was insufficient (looking for the real issue on the kernel
> end). This is primarily to best use all the resources on our servers which
> are mostly 32 cores and > 126 GB physical memory. By giving users the
> additional option to set the threshold that would give users like us
> additional lever to control based on the work load/pattern. That is the
> thought. Hope this helps. Please let me know if you have any further
> questions.
>
> Thanks,
> Biju
>
> On Tue, Aug 11, 2015 at 5:22 PM, Peter Levart <peter.levart at gmail.com>
> wrote:
>
>> Hi Biju,
>>
>> When I was preparing this patch for JDK9, I did the following
>> measurement: Using LongAdders (to avoid Heisenberg) I counted various exit
>> paths from Bits.reserveMemory() during a test that spawned 128 allocating
>> threads on a 4-core i7 machine, allocating direct buffers randomly sized
>> between 256KB and 1MB for 60 seconds, using -XX:MaxDirectMemorySize=512m:
>>
>> Total of 909960 allocations were performed:
>>
>> - 247993 were satisfied before attempting to help ReferenceHandler thread
>> - 660184 were satisfied while helping ReferenceHandler thread but before
>> triggering System.gc()
>> - 1783 were satisfied after triggering System.gc() but before doing any
>> sleep
>> - no sleeping has been performed
>>
>> The same test, just changing -XX:MaxDirectMemorySize=128m (that means 1MB
>> per thread each allocating direct buffers randomly sized between 256KB and
>> 1MB):
>>
>> Total of 579943 allocations were performed:
>>
>> - 131547 were satisfied before attempting to help ReferenceHandler thread
>> - 438345 were satisfied while helping ReferenceHandler thread but before
>> triggering System.gc()
>> - 10016 were satisfied after triggering System.gc() but before doing any
>> sleep
>> - 34 were satisfied after sleep(1)
>> - 1 was satisfied after sleep(1) followed by sleep(2)
>>
>>
>> Have your observations been different? Did you observe sleep() been
>> called in a real-world application?
>>
>>
>> Regards, Peter
>>
>>
>>
>> On 08/11/2015 04:11 PM, Biju G.S Nair wrote:
>>
>> Hello All,
>> While the patch https://bugs.openjdk.java.net/browse/JDK-6857566
>> currently applied to jdk 9 (which I had requested to be back ported to JDK
>> 8 & 7) fixes the OOM exception during memory allocation by exponentially
>> increasing the sleep time, this can negatively impact low latency
>> applications using DirectByteBuffers. If we are able to provide a new JVM
>> parameter which the users can set to a percentage value of DirectBuffer use
>> as threshold when the "System.gc()" call in java/nio/Bits.java to be made,
>> then the probability of sleep time being much lower is high. Also it gives
>> users some control over when the gc() need to be requested instead of
>> starting the gc() at the last moment when the direct memory is used fully.
>> Without knowing all the details, to me it looks like a straight forward
>> change. Let me know if there is any issue with the proposed change. If this
>> change is a possibility let me know how I can make a request for this
>> change.
>>
>> Thanks,
>> Biju
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20150812/f50afe3a/attachment.html>
More information about the nio-dev
mailing list