MemorySegment JVM memory leak

[Uwe Schindler]

Hi, 

your screenshot is perfectly fine. What's you issue? It shows VIRT=100G (which is the amount of virtual memory assigned to the process; and more importantly this is not consuming any physical memory, unless its paged in). From the area I am coming from, this looks like a perfectly organized Elasticsearch server with 32 GiB physical RAM that has an 100 Gigabyte index open: How much memory is paged in can be seen in the columns for the file cache, but that's not process specific. A good operating system should always use as much physical memory for the file cache as its available. If you just read from those pages and don't write (Lucene is doing that), it's very easy for the OS to quickly free them if its required anywhere else. If you use the mmap for read/write things get more complicated, as uncommitted pages may need to be written to disk first (for that you can use force()).

It looks like you have to understand what TOP shows, this is all sane. VIRT is virtual memory; the amount of ADRESS SPACE (not memory) assigned to the process. Address space is there enough, no need to argue. 

Uwe

-----
Uwe Schindler
uschindler at apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
https://lucene.apache.org/

> -----Original Message-----
> From: Ahmed <ahmdprog.java at gmail.com>
> Sent: Thursday, April 23, 2020 10:01 AM
> To: 'Maurizio Cimadamore' <maurizio.cimadamore at oracle.com>; 'Uwe
> Schindler' <uschindler at apache.org>; panama-dev at openjdk.java.net
> Subject: RE: MemorySegment JVM memory leak
> 
> Hello,
> 
> Coming back to Linux mmap
> 
> The problem with madvise(&db_map[i],4096,MADV_DONTNEED); Will make
> the memory segment in OS cache/buffer. I did a lot of testing and the OS eat all
> memory and put it as cache/buffer and OS will never release it until process
> die.
> 
> The attached screenshot explains very well.
> 
> I don't think madvise(&db_map[i],4096,MADV_DONTNEED); Is the perfect
> solution for Linux. There is madvise(&db_map[i],4096,MADV_REMOVE); Which
> completely return the memory segment as a free memory which it is excellent.
> 
> But again madvise(&db_map[i],4096,MADV_REMOVE); Is slow and severely
> impact performance.
> 
> Maybe a good solution to make a queue Of segment pages and quietly free
> them by madvise(&db_map[i],4096,MADV_REMOVE);
> 
> I completely understand OS to put segment in cache/buffer if process does not
> respect the OS page size. But honestly, I can't understand why OS keep the
> memory segment in cache/buffer after written to disk in mmap. The whole idea
> of mmap is to have a direct access and avoid cache/buffering of traditional
> read/write
> 
> By the way, in MAC I did a quick test and mmap is great and politely mark
> memory segment to free pool after written to disk. Unfortunately I don't have
> access to any Solaris, but I am very curious to know how it behave there, very
> long time ago I remember some news about and perfect implementation mmap
> by Sun Microsystem that time.
> 
> How java C source code implement this in mmap. Can you please share c source
> code of mmap usage ?.
> 
> Regards, Ahmed Hamdallah.
> 
> -----Original Message-----
> From: Maurizio Cimadamore <maurizio.cimadamore at oracle.com>
> Sent: Thursday, April 23, 2020 4:48 AM
> To: Uwe Schindler <uschindler at apache.org>; 'Ahmed'
> <ahmdprog.java at gmail.com>; panama-dev at openjdk.java.net
> Subject: Re: MemorySegment JVM memory leak
> 
> 
> On 23/04/2020 01:34, Maurizio Cimadamore wrote:
> > No. But that doesn't seem to be a very important issue? If you don't
> > need the original safe segment, you can just ditch it and leave it to
> > the GC?
> 
> Ok, I see what you mean now.
> 
> You basically want that calling close() on the unsafe segment calls the cleanup
> action (unmap) of the original segment. And, it seems you don't want to write
> native code for it either, so you have no way to write a Runnable that does
> what you want.
> 
> When I issued the patch originally, I allowed the possibility of passing an
> AutoCloseable as a cleanup action (not a Runnable), which meant you could
> pass the _safe_ segment as the cleanup action, and the cleanup action would
> have been called regardless of confinement.
> 
> During the review process, it was observed that this was perhaps a bit too
> magic - e.g. calling close() would leave the original segment in an unconsistent
> state (e.g. isAlive() == true, but memory freed).
> 
> I'll think a bit more whether something like that could be achieved in a way
> that doesn't add too much complexity to the API, I think I see maybe a path to
> get there (or maybe is the late hour :-) ).
> 
> Thanks
> Maurizio