[External] : Re: MemorySegment APIs for reading and writing strings with known lengths

[Liam Miller-Cushon]

Thanks Maurizio,

I think that looks great. It addresses the use-cases I'm aware of for
reading and writing strings of known length, without null terminators.

I have a few small questions and comments.

> The number of bytes associated with chars is platform-dependent: it's 10
bytes on Windows, but 20 bytes on Linux. By having MemorySegment::getString
work on units the only thing we need to worry about is that we specify the
correct charset when reading the string -- e.g. UTF16 (on Windows) or UTF32
(on Linux).

I want to double-check my understanding of that example. Is the idea that
you would read a code unit length from the struct Foo (5 in the example)
and then depending on whether the API expected bytes or code units, you
would have to do one of the following?

// with a code unit length
getString(0, /* code units */ 5, windows ? UTF16 : UTF32)

// vs. with a byte length
getString(0, /* bytes */ 5 * (windows ? 2 : 4), windows ? UTF16 : UTF32)

> Liam said protobuf uses byte lengths. Native strings probably use unit
lengths

One of my colleagues working on protobuf pointed out I had described that
imprecisely and offered the following clarification:

Its really that Protobuf just solely ever represents strings as UTF8 in
native heap memory (even on windows where the OS APIs will expect utf16, if
someone ever wanted to take use C++Protobuf strings to/from windows paths,
they would actually have to convert utf8<>utf16 themselves); it means that
byte and unit lengths are always the same for us when stored in native heap

Another small clarification - the doc mentions 'compressed strings', should
that be 'compact strings'? (JEP 254 mentions there was an older 'compressed
strings' feature in JDK 6?)

And one more thing - the doc discusses string lengths. In addition to the
number of UTF-16 code units in the UTF-16 encoding of the string
(String.length), and the number of unicode characters (codePointCount), it
is sometimes desirable to know the number of code units in the encoded
string. It's possible to do this with getBytes(charset).length, but similar
to the other performance discussions here, it's expensive to create the
array and throw it away, and an implementation outside the JDK can't
benefit from the JDK's fast paths in StringSupport. In theory there could
be a method like String#getEncodedLength(Charset) that returned the same
value as getBytes(charset).length with better performance. Is that
something you think could be interesting to consider, either as part of
this proposal, or separately?

On Fri, Nov 7, 2025 at 5:40 PM Maurizio Cimadamore <
maurizio.cimadamore at oracle.com> wrote:

> Hi,
> here's a document that outlines a more general strategy to read and write
> strings from/to memory segments:
>
> https://cr.openjdk.org/~mcimadamore/panama/strings_ffm.html
>
> As outlined in this thread, the enabling primitive for the writing side is
> a factory that allows to view a Java string as a read-only heap memory
> segment.
>
> We believe the proposed strategy should provide better usability, while at
> the same time leaving room for the implementation to "cut corners", to make
> performance as good as possible.
>
> Let us know what you think.
>
> Cheers
> Maurizio
> On 05/11/2025 12:45, Liam Miller-Cushon wrote:
>
> Thanks Maurizio, considering all of this holistically makes sense. I look
> forward to hearing your thoughts on it once you've had time to consider the
> options :)
>
> On Tue, Nov 4, 2025 at 11:21 PM Maurizio Cimadamore <
> maurizio.cimadamore at oracle.com> wrote:
>
>>
>> On 04/11/2025 15:23, Jorn Vernee wrote:
>>
>> Do you have thoughts on the best way to proceed here? Do you think it
>> makes sense to do incrementally, or would you prefer to see all of these
>> related changes happen together under a single issue?
>>
>> I don't have a preference. Since you've already started a PR for
>> enhancing getString, maybe you can focus on that for now, and we'll file
>> followup issues for the others. Splitting things up might be nice since
>> there's probably some benchmarking work involved for each. I think the copy
>> and allocateFrom overload can be done in one patch though.
>>
>> I think I sort of do have a preference :-)
>>
>> My feeling is that we're dealing with 2-3 different APIs that are all
>> tightly interconnected. We have some ideas on how to solve some (I think
>> getString with explicit length seems the most settled), but we're still
>> playing with other ideas for the others (like copy, or memory segment
>> views). And we also have to think about relationship with SegmentAllocator.
>>
>> For this reason, I'd prefer to think about it more holistically and think
>> about all the related APIs at once.
>>
>> Once we get a consensus on how to proceed we can decide whether to pursue
>> them all in a single PR, or split them into separate PRs.
>>
>> But it would be unfortunate, I think, if the first PR would later reveal
>> to be a dead end for the other use cases.
>>
>> I plan to think about this some more (but I need some more time).
>>
>> Cheers
>> Maurizio
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20251110/d7fa4680/attachment-0001.htm>