Best practices with reading/writing to a Memory Mapped File

[Johannes Lichtenberger]

Hi Maurizio,

indeed, now I just allocate basically the whole file again for reading
page-fragments. I think for 2,7 Gb of test data stored in a SirixDB
resource (I think 223_000_000 nodes roughly) it's currently about 5:50min
to traverse the whole file in preorder vs 6:30min for the RandomAccessFile
but just from running in IntelliJ. I could also see in the profiler that
the mapping and unmapping is the problem when appending (during a
transaction commit).

The page references keep track of the offsets in the file or memory mapped
region (pages of a tree of tries structure, which borrows some ideas of
ART(adaptive radix tree) and hash array mapped tries.

I think another issue when reading is, that I probably don't even need a
Cache anymore. Usually I'm setting the loaded page in
PageReference.setPage(Page) and also storing it in a cache. The cache is
for instance also used to evict entries, thus also nulling the referenced
page again (sort of pointer swizzling). I think both might not be needed
when using the mapped memory segment(s).

Regarding writing I'm not sure if I can simply delegate reads to the
MemoryMappedFileReader and doing the appending directly to the
RandomAccessFile. From what I read appending might be the worst use case
for Memory Mapped Files, as besides on Linux you probably have to
preallocate chunks, tracking the size and afterwards truncate to this size
again. On Linux there's a remap function I think.

But maybe appending can simply be done by appending to the RandomAccessFile
and reading through the mapped memory segment, what do you think? As
mentioned earlier I should probably also get rid of the cache and not
setting the on-heap Page instance, but I'm also not sure about this.

>From what I read basically every database system seems to use mmap
nowadays, some are in-memory data stores as for instance HyPer and SAP
HANA, but also MongoDB and I think Cassandra...

Kind regards

Maurizio Cimadamore <maurizio.cimadamore at oracle.com> schrieb am Mo., 29.
Juni 2020, 17:31:

> Hi Johannes,
> glad that you managed to make everything work.
>
> While I'm not an expert in mmap fine-tuning, one thing that comes to
> mind is that memory mapped files are mapped into main memory one page at
> a time, so if your pattern of access is really random/sparse, maybe
> there's not a lot to be gained by using mapped file in your use case.
>
> Also, looking at the code, it seems like you are creating a mapped
> segment for each page write, which seems odd - typically you'd want a
> mapped segment to contain all the memory you need to access, and then
> let the loading/unloading of pages to the OS, which generally knows
> better. It seems to me that your application is, instead selecting with
> PageReference to write, then creates a mapped segment for that page, and
> then persists the changes via the mapped segment; I think doing this
> probably nullifies all the advantages of keeping the contents of the
> file in memory. In fact, with your approach, since the mapped segment is
> not stashed anywhere, I don't think the file will be even kept in memory
> (you map and then discard soon after, page after page).
>
> I'd expect some state to remain cached from one write to the next (e.g.
> the mapped segment should, ideally, be stashed in some field, and only
> discarded if, for some reason, the original bounds are no longer valid -
> e.g. because the file is truncated, or expanded). But, assuming your
> file size remains stable, your code should keep accessing memory using
> _the same_ mapped segment, and the OS will load/unload pages for you as
> it sees fit (using heuristics to keep frequently used pages loaded, and
> discard the ones that have been used less frequently - all taking into
> account how much memory your system has).
>
> Maurizio
>
> On 27/06/2020 11:50, Johannes Lichtenberger wrote:
> > Hi,
> >
> > I've fixed my Memory Mapped file implementation using your Foreign Memory
> > API.
> >
> >
> https://github.com/sirixdb/sirix/tree/master/bundles/sirix-core/src/main/java/org/sirix/io/memorymapped
> >
> > Running my tests (mostly simple integration tests, which test if the
> stuff
> > I'm storing can be retrieved again or the result of queries are what I
> > expect), I can't see a clear performance difference between the
> > RandomAccessFile implementation
> >
> >
> https://github.com/sirixdb/sirix/tree/master/bundles/sirix-core/src/main/java/org/sirix/io/file
> >
> > and the new memorymapped implementation.
> >
> > So far, I have to create a new mapping everytime I'm appending to the
> > memory mapped segment of the underlying file I guess (otherwise the
> bounds
> > checks will obviously fail):
> >
> >
> https://github.com/sirixdb/sirix/blob/627fa5a57a302b04d7165aad75a780d74e14c2e9/bundles/sirix-core/src/main/java/org/sirix/io/memorymapped/MemoryMappedFileWriter.java#L141
> >
> > I'm only ever appending data when writing or reading randomly based on
> > offsets.
> >
> > I haven't done any microbenchmarks as of now and did not check bigger
> files
> > ranging from 1Gb to much more nor did I use a profiler to check what's
> > going on. However, maybe creating the mapping often times is costly and
> > maybe you can simply spot a performance issue. Or it's IntelliJ and my
> > rather small flles for testing as of now.
> >
> > Will next check if importing a 3,8 Gb JSON file is faster or iterating
> > through the whole imported file with around 400_000_000 nodes :-)
> >
> > If anyone wants to check it it's simply changing
> >
> > private static final StorageType STORAGE = StorageType.FILE;
> >
> > to
> >
> > private static final StorageType STORAGE = StorageType.MEMORY_MAPPED;
> >
> > in the class: org.sirix.access.ResourceConfiguration
> >
> > Thanks for all the suggestions and hints so far
> > Johannes
>