<div dir="ltr">Thank you very much for the advice, I will implement these suggestions =)</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 2, 2022 at 12:12 PM Radosław Smogura <<a href="mailto:mail@smogura.eu">mail@smogura.eu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg1870387414433145371">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_-8662949787305582565WordSection1">
<p class="MsoNormal">Hi Gavin,</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I see you do a good progress.</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">This is good approach. Minor improvement would be to use MemorySegment.ofBuffer(), to create memory segment from _<i>direct</i>_ byte buffer. This way you would have consistency (using only MemorySegment) and FileChannels or other methods
to manage file size.</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Most probably you would like to use MappedByteBuffer.force() to flush changes to disk (equivalent of sync in Linux) – i.e. to be sure transaction is persisted or for write ahead log.</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">In most cases if you want to work with zero-copy reads, you have to map a whole file as direct buffer / memory segment. You would need to enlarge file using (most probably file channel) or other methods, if you want to append new data (otherwise
sigbus or segfault can be generated – can result in exception or crash).</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">You can compare different approaches using JMH to measure reads and writes performance.</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Kind regards,</p>
<p class="MsoNormal">Rado Smogura</p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal" style="border:none;padding:0in"><b>From: </b><a href="mailto:ray.gavin97@gmail.com" target="_blank">Gavin Ray</a><br>
<b>Sent: </b>Friday, September 2, 2022 5:50 PM<br>
<b>To: </b><a href="mailto:lichtenberger.johannes@gmail.com" target="_blank">Johannes Lichtenberger</a><br>
<b>Cc: </b><a href="mailto:maurizio.cimadamore@oracle.com" target="_blank">Maurizio Cimadamore</a>;
<a href="mailto:panama-dev@openjdk.org" target="_blank">panama-dev@openjdk.org</a><br>
<b>Subject: </b>Re: Question: ByteBuffer vs MemorySegment for binary (de)serializiation and in-memory buffer pool</p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On a related note, is there any way to do zero-copy reads from files using MemorySegments for non-Memory-Mapped files?<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Currently I'm using "SeekableByteChannel" and wrapping the MemorySegment using ".asByteBuffer()"<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Is this the most performant way?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">========================<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">class DiskManager {<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> private final RandomAccessFile raf;<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> private final SeekableByteChannel dbFileChannel;<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> public void readPage(PageId pageId, MemorySegment pageBuffer) {<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> int pageOffset = pageId.value() * Constants.PAGE_SIZE;<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> dbFileChannel.position(pageOffset);<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> dbFileChannel.read(pageBuffer.asByteBuffer());<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> }<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> public void writePage(PageId pageId, MemorySegment pageBuffer) {<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> int pageOffset = pageId.value() * Constants.PAGE_SIZE;<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> dbFileChannel.position(pageOffset);<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> dbFileChannel.write(pageBuffer.asByteBuffer());<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> }<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">}<u></u><u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Thu, Sep 1, 2022 at 6:13 PM Johannes Lichtenberger <<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank">lichtenberger.johannes@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">I think it's a really good idea to use off-heap memory for the Buffer Manager/the pages with the stored records. In my case, I'm working on an immutable, persistent DBMS currently storing JSON and XML with only one read-write trx per resource
concurrently and if desired in parallel to N read-only trx bound to specific revisions (in the relational world the term for a resource is a relation/table). During an import of a close to 4Gb JSON file with intermediate commits, I found out that depending
on the number of records/nodes accumulated in the trx intent log (a trx private map more or less), after which a commit and thus a sync to disk with removing the pages from the log is issued, the GC runs are >= 100ms most of the times and the objects are long-lived
and are promoted to the old gen obviously, which seems to take these >= 100ms. That is I'll have to study how Shenandoah works, but in this case, it brings no advantage regarding the latency.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Maybe it would make sense to store the data in the record instances also off-head, as Gavin did with his simple Buffer Manager :-) that said lowering the max records number after which to commit and sync to disk also has a tremendous effect
and with Shenandoah, the GC times are less than a few ms at least.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm using the Foreign Memory API however already to store the data in memory-mapped files, once the pages (or page fragments) and records therein are serialized and then written to the memory segment after compression and hopefully soon
encyrption.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Kind regards<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Johannes<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Am Do., 1. Sept. 2022 um 22:52 Uhr schrieb Maurizio Cimadamore <<a href="mailto:maurizio.cimadamore@oracle.com" target="_blank">maurizio.cimadamore@oracle.com</a>>:<u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
</div>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12pt;margin-left:9.6pt">
<br>
On 01/09/2022 19:26, Gavin Ray wrote:<br>
> I think this is where my impression of verbosity is coming from, in <br>
> [1] I've linked a gist of ByteBuffer vs MemorySegment implementation <br>
> of a page header struct,<br>
> and it's the layout/varhandles that are the only difference, really.<br>
><br>
Ok, I see what you mean, of course; thanks for the Gist.<br>
<br>
In this case I think the instance accessor we added on MemorySegment <br>
will bring the code more or less to the same shape as what it used to be <br>
with the ByteBuffer API.<br>
<br>
Using var handles is very useful when you want to access elements (e.g. <br>
structs inside other structs inside arrays) as it takes all the offset <br>
computation out of the way.<br>
<br>
If you're happy enough with hardwired offsets (and I agree that in this <br>
case things might be good enough), then there's nothing wrong with using <br>
the ready-made accessor methods.<br>
<br>
Maurizio<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></blockquote></div>