Question: ByteBuffer vs MemorySegment for binary (de)serializiation and in-memory buffer pool

[Gavin Ray]

I'd never heard of this Chronicle Bytes library, I looked it up and the
whole suite of OpenHFT tools look mighty nifty for high-performance Java.
Thanks for sharing!

About the repository, I've not gotten far enough along to put something up
but I could put up a sandbox repo or something
If you want to mail me directly I'm more than happy to continue chatting.
Had a look at your DB's source and saw some interesting approaches there.


On Mon, Aug 22, 2022 at 6:21 PM Johannes Lichtenberger <
lichtenberger.johannes at gmail.com> wrote:

> That's an interesting use case of the Foreign Memory API IMHO. I haven't
> even thought about using it in this particular case. I'm also working on a
> DBMS[1] side project in my spare time. I've used the Foreign Memory API to
> memory map the storage file (for reads), but couldn't measure much impact
> over a RandomAccess file / FileChannel reader. As it's an append-only store
> writes are done via the FileChannel abstraction instead.
> The serialization/deserialization to an array of slots is done however
> using Chronicle Bytes with Bytes<ByteBuffer> mainly, as Chronicle handles
> the resizing and does some Unsafe tricks, IIRC.
>
> @Gavin, do you also have a repository? I'd love to exchange ideas as I'm
> currently the only one working on the project and sharing thoughts/ideas
> would be awesome.
>
> Kind regards
> Johannes
>
> [1] https://github.com/sirixdb/sirix and the query engine
> https://github.com/sirixdb/brackit
>
> Am Mo., 22. Aug. 2022 um 20:30 Uhr schrieb Gavin Ray <
> ray.gavin97 at gmail.com>:
>
>> Thanks Paul,
>>
>> The API is a fair bit wordier, but that's the price you pay in exchange
>> for greater expressiveness and flexibility I suppose.
>> I'll likely go forward with the MemorySegment implementation then -- I
>> appreciate the reply!
>>
>> On Mon, Aug 22, 2022 at 2:21 PM Paul Sandoz <paul.sandoz at oracle.com>
>> wrote:
>>
>>> Hi Gavin,
>>>
>>> This is the right place to reach out.
>>>
>>> Using MemorySegment will given you far more control over the description
>>> (layout) and management (freeing and pooling) than ByteBuffer. Also, if
>>> it’s an issue you will also not be constrained by ByteBuffer’s size
>>> limitation. Performance wise using MemorySegment should be as good as or
>>> better than ByteBuffer.
>>>
>>> In many respects MemorySegment is a better API to interact with native
>>> memory. ByteBuffer was introduced in Java 1.4 with NIO and had additional
>>> design constraints in mind that are less relevant today (such as an
>>> internal mutable index).
>>>
>>> Paul.
>>>
>>> > On Aug 20, 2022, at 10:35 AM, Gavin Ray <ray.gavin97 at gmail.com> wrote:
>>> >
>>> > Hiya, I hope it's okay that I ask questions here (if not, please
>>> direct me elsewhere)
>>> >
>>> > I tried to ask on StackOverflow without much luck, and there aren't
>>> many resources
>>> > available at the moment to read about the Foreign Memory API so I'm
>>> reaching out here.
>>> >
>>> > I'm implementing a basic database on the JVM and am trying to
>>> understand the
>>> > technical implications of using either ByteBuffer or MemorySegment for
>>> heap page management and in-memory buffer allocation.
>>> >
>>> > Details are in this thread:
>>> > database - Java: Performance of ByteBuffer versus
>>> jdk.incubator.foreign (Panama) Foreign Memory methods
>>> (MemoryLayout/Segment) - Stack Overflow
>>> >
>>> > Would appreciate any information folks would be willing to share =)
>>> >
>>> > Thank you,
>>> > Gavin Ray
>>> >
>>> > (By the way, I'm not subscribed to the mailing list, will I be able to
>>> reply to responses?)
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20220823/00dae293/attachment.htm>