Stupid question
Greg Bowyer
gbowyer at fastmail.co.uk
Tue May 22 16:19:24 PDT 2012
So the more I think about this the more it occurs to me that its obvious
that the mmap() is for the full heap size and that my mlock is in the
wrong place
I am thinking that this should be moved up to os::malloc since
hotspot/src/share/vm/memory/allocation.inline.hpp really does the
allocation and the os macros, but then the space for parallel new does
use os::commit rather than os::malloc ?
On 22/05/12 14:35, Greg Bowyer wrote:
> To clarify slightly I see this here
>
> through /proc/meminfo it does not look like the used pages are locked,
> just unevictable
>
> greg at localhost ~ $ cat /proc/meminfo
> MemTotal: 12286752 kB
> MemFree: 209512 kB
> Buffers: 21612 kB
> Cached: 487636 kB
> SwapCached: 352 kB
> Active: 1573468 kB
> Inactive: 435696 kB
> Active(anon): 1325624 kB
> Inactive(anon): 178984 kB
> Active(file): 247844 kB
> Inactive(file): 256712 kB
> Unevictable: 9602276 kB
> Mlocked: 246888 kB
>
> the process itself suggests that its fully wired
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 31019 greg 20 0 9523m 9.2g 11m S 0 78.4 0:03.96 java
>
> On 22/05/12 14:27, Greg Bowyer wrote:
>> Done, I get this
>>
>> The "Leaky program" bit is stupid simple test program that slowly
>> leaks memory, mostly to test the jvm simply
>>
>> Printf's for commint memory and anon_mmap added
>>
>> I notice that we get this one -> anon_mmap addr=0x5bae00000
>> bytes=9749659648 fixed=0
>>
>> greg at localhost
>> ~/projects/openjdk/jdk7/build/linux-amd64-fastdebug/j2sdk-image $
>> ./bin/java -Xms512m -Xmx9g -XX:+LockHeapInMemory test
>> VM option '+LockHeapInMemory'
>> commit_memory addr=0x7f2d66c49000 bytes=12288
>> anon_mmap addr=(nil) bytes=50331648 fixed=0
>> commit_memory addr=0x7f2d60bb5000 bytes=2555904
>> anon_mmap addr=(nil) bytes=786432 fixed=0
>> commit_memory addr=0x7f2d66b89000 bytes=40960
>> anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0
>> anon_mmap addr=(nil) bytes=19046400 fixed=0
>> commit_memory addr=0x7f2d60bb4000 bytes=4096
>> commit_memory addr=0x740000000 bytes=178913280
>> commit_memory addr=0x7f2d605b4000 bytes=352256
>> commit_memory addr=0x5c0000000 bytes=357957632
>> anon_mmap addr=(nil) bytes=12582912 fixed=0
>> commit_memory addr=0x7f2d5f9b4000 bytes=700416
>> commit_memory addr=0x7f2d5ed8b000 bytes=700416
>> commit_memory addr=0x5bae00000 bytes=21757952
>> anon_mmap addr=(nil) bytes=167936 fixed=0
>> commit_memory addr=0x7f2d5f98b000 bytes=45056
>> commit_memory addr=0x7f2d5ed62000 bytes=45056
>> commit_memory addr=0x7f2d5d7e5000 bytes=12288
>> commit_memory addr=0x7f2d5d6e4000 bytes=12288
>> commit_memory addr=0x7f2d5d3f5000 bytes=12288
>> commit_memory addr=0x7f2d5d2f4000 bytes=12288
>> commit_memory addr=0x7f2d5d1f3000 bytes=12288
>> commit_memory addr=0x7f2d5d0f2000 bytes=12288
>> Leaking program is slowly leaking 1
>> Leaking program is slowly leaking 2
>> Leaking program is slowly leaking 3
>> ....
>> ^Ccommit_memory addr=0x7f2d5cef0000 bytes=12288
>>
>> On 22/05/12 13:35, Vitaly Davidovich wrote:
>>>
>>> can you add a print statement to commit_memory() to see what
>>> arguments are being passed to it? I'm sure someone on this mailing
>>> list will have a better suggestion, but that's what I'd try just to
>>> establish that some code is indeed trying to expand the heap to its
>>> max allotted size.
>>>
>>> Sent from my phone
>>>
>>> On May 22, 2012 4:08 PM, "Greg Bowyer" <gbowyer at fastmail.co.uk
>>> <mailto:gbowyer at fastmail.co.uk>> wrote:
>>>
>>> Yes thats basically it
>>>
>>> java -Xms512m -Xmx9g test
>>>
>>> On 22/05/12 12:27, Vitaly Davidovich wrote:
>>>>
>>>> Hi Greg,
>>>>
>>>> As an aside, in os::commit_memory, I think the tertiary
>>>> condition that sets the flags is inverted - you want MAP_LOCKED
>>>> if LockHeapInMemory is set, I believe.
>>>>
>>>> As to your question, what args are being passed to the above
>>>> function in your run? I take it -Xms is less than Xmx in your
>>>> trial?
>>>>
>>>> Sent from my phone
>>>>
>>>> On May 22, 2012 3:07 PM, "Greg Bowyer" <gbowyer at fastmail.co.uk
>>>> <mailto:gbowyer at fastmail.co.uk>> wrote:
>>>>
>>>> Not sure if this is the right venue for this, or if I am insane
>>>>
>>>> I have been playing with the openjdk code with a view to
>>>> lock the heap in memory (or at least suggest to the OS that
>>>> it wants to be locked in memory).
>>>>
>>>> My use case is for java processes that are greedy in memory
>>>> and typically have a virtual size larger than physical ram.
>>>> This is not where an end user allocates a java heap beyond
>>>> ram but rather things like Lucene / Cassandra, where
>>>> typically the JVM heap is large but limited to say 1/4 of
>>>> the total physical ram and the rest of the process virtual
>>>> size is taken up with mmap()'d files.
>>>>
>>>> There are java projects that currently do this with a call
>>>> out via JNA / JNI to mlockall()
>>>>
>>>> Asking the OS to use the MAP_LOCKED flag in the mmap calls
>>>> in os_linux.cpp effectively does an mlock / mlockall which
>>>> I think means that when the OS chooses pages to page out;
>>>> then it should (for some measure of should) avoid paging
>>>> out the JVM heap.
>>>>
>>>> This means that horrors between CMS and paging do not cause
>>>> hateful pauses (hopefully)
>>>>
>>>> The thing I cant understand is that when I start a new VM
>>>> with this code (attached) it appears to lock the full size
>>>> of the heap (-Xmx) (even though the given memory is not
>>>> used by the VM), this seems to make the entire space resident.
>>>>
>>>> any ideas ?
>>>>
>>>> _______________________________________________
>>>> hotspot-gc-use mailing list
>>>> hotspot-gc-use at openjdk.java.net
>>>> <mailto:hotspot-gc-use at openjdk.java.net>
>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120522/d0abb5d3/attachment.html
More information about the hotspot-gc-use
mailing list