DirectByteBuffer change proposal

[Biju G.S Nair]

Peter, that sounds very good. Hope the patch will be back-ported and made
available in JDK 7 & 8 so that we can test. I will help resolve our issues
as well.

Thanks,
Biju

On Wed, Aug 12, 2015 at 10:05 AM, Peter Levart <peter.levart at gmail.com>
wrote:

> Hi Biju,
>
> Well, seeing sleep() in JDK7 is understandable, since before the patch for
> 6857566, the strategy was to simply call System.gc() followed by a constant
> sleep(100L) every time native memory exhaustion was detected. There was no
> attempt to help process references synchronously before the 1st sleep. My
> measurements after patch for 6857566 show that sleep() will only be called
> very infrequently in extreme situations provoked only with stress tests
> that do nothing but allocate direct buffers in multiple threads. I would be
> surprised if this happens in any real-world application. The amount of max.
> direct buffer memory should play a positive role here as more memory there
> is for native buffers, less probability it is that sleep() will be called.
> Sleep is called only in situations where the amount of native memory freed
> when calling gc()/tryHandlePending() - which should be all the memory
> allocated by currently unreachable buffers - is not enough to cope with
> concurrent allocation in a time slot between calling
> gc()/tryHandlePending() and attempting the allocation. At that point, sleep
> represents a kind of back-pressure to slow-down allocation and allow the
> program to continue to function over unexpected peaks (like DDOS attacks
> for example).
>
> If you size the max. memory for native buffers to be big enough compared
> to max. expected allocated and used memory at any time with some spare
> share, sleep() will not occur. If it occurs regularly, it is a sign that
> you should increase the max. native memory limit, as your program really
> needs more memory.
>
> Regards, Peter
>
>
> On 08/12/2015 02:53 PM, Biju G.S Nair wrote:
>
> Thanks Peter for the detailed response. Currently we are in JDK 7 since
> the underlying datastore is still being certified for moving to 8. So the
> issues we are seeing is not on JDK 9 with the patch. I have requested the
> jdk8u-dev and jdk9u-dev team through the dev email whether they can back
> port the patch to the two versions so that we can test. Waiting to get the
> patch merged. Will update you as soon as the JDK is updated.
>
> Thanks,
> Biju
>
> On Wed, Aug 12, 2015 at 2:51 AM, Peter Levart <peter.levart at gmail.com>
> wrote:
>
>> Hi Biju,
>>
>> Just to clear any doubts. You have seen sleep() being called with JDK9
>> which already includes the patch for 6857566 ?
>>
>> The cleanup of native memory for DirecByteBuffers was and after  6857566
>> still is designed around sun.misc.Cleaner which is basically a
>> PhantomReference pointing to the DirecByteBuffer. When DirectByteBuffer
>> instance becomes phantom-reachable, VM will at some later time discover it
>> as a pending reference and ReferenceHandlerThread will "clean" it (execute
>> Cleaner's thunk) releasing the native memory. So ordinarily, cleanup of
>> native memory is performed asynchronously short time after VM decides to
>> execute code to discover pending references. This is usually performed when
>> heap allocation reaches certain point (maybe also periodically?). The
>> problem is that memory pressure detected by VM is based on heap memory
>> allocation which does not include native memory. It can and frequently
>> happens that there's no heap memory pressure, but native memory reserved
>> for DirectByteBuffers is exhausted.
>>
>> Current strategy works by waiting until this exhaustion happens and then:
>> - attempts to "help" ReferenceHandlerThread to drain the pending
>> references already discovered by VM, executing any sun.misc.Cleaners on the
>> way; if that does not help, it
>> - triggers System.gc() which hopefully also triggers discovery of pending
>> references, followed by a round of ReferenceHandlerThread helping; if that
>> does not help, it
>> - retries the gc/tryHandlePending cycle, introducing exponentially
>> increasing delays (from 1ms up to 512ms) which also acts as some kind of
>> back-pressure; if that does not help, it
>> - considers native memory full and throws OutOfMemoryError
>>
>> You propose to add triggering of gc() when native memory occupancy
>> reaches certain % of max. memory allowed for native buffers. This sounds
>> reasonable as it mimics roughly what happens when heap allocation reaches
>> certain point. Reference handling would still be performed asynchronously
>> solely by ReferenceHandlerThread in this case (no helping with
>> tryHandlePending()) until native memory is exhausted. But note that
>> System.gc() is not free. It usually is implemented as a blocking call which
>> triggers safe-point VM processing and waits for it to complete before
>> returning. So if you wanted to see low latencies for native buffer
>> allocations, this should be performed by a background thread that
>> continuously monitors current occupancy. You can try doing this in client
>> code. There's a JMX bean that exposes native buffer allocation state
>> (BufferPoolMXBean). Can you try doing this and report if it helps with
>> allocation latencies in your application?
>>
>> Regards, Peter
>>
>>
>> On 08/11/2015 11:43 PM, Biju G.S Nair wrote:
>>
>> Hi Peter,
>>    Thanks for the background. Yes. We have seen sleep being called since
>> we are using DirectByteBuffers in GBs and the plan is to see whether we can
>> go up more than 90 GB. Also we have seen in certain version of Linux kernel
>> the 100 ms sleep was insufficient (looking for the real issue on the kernel
>> end). This is primarily to best use all the resources on our servers which
>> are mostly 32 cores and > 126 GB physical memory. By giving users the
>> additional option to set the threshold that would give users like us
>> additional lever to control based on the work load/pattern. That is the
>> thought. Hope this helps. Please let me know if you have any further
>> questions.
>>
>> Thanks,
>> Biju
>>
>> On Tue, Aug 11, 2015 at 5:22 PM, Peter Levart < <peter.levart at gmail.com>
>> peter.levart at gmail.com> wrote:
>>
>>> Hi Biju,
>>>
>>> When I was preparing this patch for JDK9, I did the following
>>> measurement: Using LongAdders (to avoid Heisenberg) I counted various exit
>>> paths from Bits.reserveMemory() during a test that spawned 128 allocating
>>> threads on a 4-core i7 machine, allocating direct buffers randomly sized
>>> between 256KB and 1MB for 60 seconds, using -XX:MaxDirectMemorySize=512m:
>>>
>>> Total of 909960  allocations were performed:
>>>
>>> - 247993 were satisfied before attempting to help ReferenceHandler thread
>>> - 660184 were satisfied while helping ReferenceHandler thread but before
>>> triggering System.gc()
>>> - 1783 were satisfied after triggering System.gc() but before doing any
>>> sleep
>>> - no sleeping has been performed
>>>
>>> The same test, just changing -XX:MaxDirectMemorySize=128m (that means
>>> 1MB per thread each allocating direct buffers randomly sized between 256KB
>>> and 1MB):
>>>
>>> Total of 579943 allocations were performed:
>>>
>>> - 131547 were satisfied before attempting to help ReferenceHandler thread
>>> - 438345 were satisfied while helping ReferenceHandler thread but before
>>> triggering System.gc()
>>> - 10016 were satisfied after triggering System.gc() but before doing any
>>> sleep
>>> - 34 were satisfied after sleep(1)
>>> - 1 was satisfied after sleep(1) followed by sleep(2)
>>>
>>>
>>> Have your observations been different? Did you observe sleep() been
>>> called in a real-world application?
>>>
>>>
>>> Regards, Peter
>>>
>>>
>>>
>>> On 08/11/2015 04:11 PM, Biju G.S Nair wrote:
>>>
>>> Hello All,
>>>    While the patch https://bugs.openjdk.java.net/browse/JDK-6857566
>>> currently applied to jdk 9 (which I had requested to be back ported to JDK
>>> 8 & 7) fixes the OOM exception during memory allocation by exponentially
>>> increasing the sleep time, this can negatively impact low latency
>>> applications using DirectByteBuffers. If we are able to provide a new JVM
>>> parameter which the users can set to a percentage value of DirectBuffer use
>>> as threshold when the "System.gc()" call in java/nio/Bits.java to be made,
>>> then the probability of sleep time being much lower is high. Also it gives
>>> users some control over when the gc() need to be requested instead of
>>> starting the gc() at the last moment when the direct memory is used fully.
>>> Without knowing all the details, to me it looks like a straight forward
>>> change. Let me know if there is any issue with the proposed change. If this
>>> change is a possibility let me know how I can make a request for this
>>> change.
>>>
>>> Thanks,
>>> Biju
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20150812/cb38bbbb/attachment.html>