MemorySegment JVM memory leak

[Samuel Audet]

Hi, Uwe,

I have a question about what your recommendations are surrounding calls 
to madvise(). If I understand correctly, if we don't "clean up" memory 
manually that way, the "resident set size" that the process uses 
according to the kernel remains high until some undefined point in time. 
If that's the case, then which value should we check that would give us 
a better indication of how much memory the process is actually using?

In JavaCPP, what happens is that when the RSS goes beyond some 
threshold, it calls a couple of times in a row both System.gc() and 
malloc_trim(0) (which calls madvise() on all free pages), to see if it 
couldn't reduce memory consumption that way, before bailing out with an 
OutOfMemoryError. (Something smarter than that could be implemented as 
part of the JDK, but our stuff needs to be compatible with Java SE 7).

What do you think of this approach? In other words, in your opinion, 
what should the JDK do for Buffer and MemorySegment in general?

Samuel

On 4/22/20 8:15 PM, Uwe Schindler wrote:
> Hi,
> 
> Just some comments from opposite site:
> 
>>> I am doing also some testing with C using mmap and sometimes having
>>> same java issue where memory consumption very high. But still doing
>>> investigation and not yet concluded.
>>>
>> I did exactly the same to narrow down the issue, and I too was having
>> very high memory consumption with big mappings.
>>
>> This is my main loop in C:
>>
>> char * region = mmap(.......);
>>
>> for (long l = 0 ; l < SIZE ; l+= 4096) {
>>       memcpy(buf, &region[l], 4096);
>>       madvise(region, l, MADV_DONTNEED); // <--------
>>     }
> 
> This exact behavior is wanted for memory mapped files in most cases, the resident memory should be cleaned up later and the OS kernel does a good job with it. E.g., if Lucene/Solr/Elasticsearch would use MADV_DONTNEED its whole IO would go crazy. Why Lucene/Solr/Elasticsearch relies on this is the type of I/O its doing: https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html - Those servers are relying on the fact it behaves like the linux kernel handles it by default! So please don't add anything like this into MappedByteBuffer and the segment API! It's no memory leak, its just normal behaviour!
> 
> I agree, it's a problem for anonymous mapping if they can't be cleaned up, but those should be unmapped after usage. For mmapped files, the memory consumption is not higher, it's just better use of file system cache. If the kernel lacks enough space for other stuff that has no disk backend (anonymous mapping), it will free the occupied resources or add a disk backend by using swap file.
> 
> I still have some wish: Sometimes we would like to pass MADV_DONTNEED for memory mapped files, e.g. when we only read them once. With MappedByteBuffer this is not possible at the moment, there was already the proposal to allow setting those flags. Same for normal file IO, where its fadvise and should be implemented as open option in java.nio.file.Files class. So maybe add some OpenOption.WONTNEED to Files API. Thanks.
> 
> Uwe
>