[External] : Re: MemorySegment APIs for reading and writing strings with known lengths

Mon Nov 10 14:12:41 UTC 2025

On 10/11/2025 13:05, Liam Miller-Cushon wrote:
> Thanks Maurizio,
>
> I think that looks great. It addresses the use-cases I'm aware of for 
> reading and writing strings of known length, without null terminators.
>
> I have a few small questions and comments.
>
> > The number of bytes associated with chars is platform-dependent: 
> it's 10 bytes on Windows, but 20 bytes on Linux. By having 
> MemorySegment::getString work on units the only thing we need to worry 
> about is that we specify the correct charset when reading the string 
> -- e.g. UTF16 (on Windows) or UTF32 (on Linux).
>
> I want to double-check my understanding of that example. Is the idea 
> that you would read a code unit length from the struct Foo (5 in the 
> example) and then depending on whether the API expected bytes or code 
> units, you would have to do one of the following?
>
> // with a code unit length
> getString(0, /* code units */ 5, windows ? UTF16 : UTF32)
>
> // vs. with a byte length
> getString(0, /* bytes */ 5 * (windows ? 2 : 4), windows ? UTF16 : UTF32)
Yes, note that you can do `Linker.canonicalLayouts().get("wchar_t")` to 
discover the platform dependent size of `wchar_t` and select the 
appropriate charset.
>
> > Liam said protobuf uses byte lengths. Native strings probably use 
> unit lengths
I hope this comment was not in my doc?
>
> One of my colleagues working on protobuf pointed out I had described 
> that imprecisely and offered the following clarification:
>
> Its really that Protobuf just solely ever represents strings as UTF8 
> in native heap memory (even on windows where the OS APIs will expect 
> utf16, if someone ever wanted to take use C++Protobuf strings to/from 
> windows paths, they would actually have to convert utf8<>utf16 
> themselves); it means that byte and unit lengths are always the same 
> for us when stored in native heap
Makes sense -- so units seems a good choice there (and always use UTF8 
on protobuf side).
>
> Another small clarification - the doc mentions 'compressed strings', 
> should that be 'compact strings'? (JEP 254 mentions there was an older 
> 'compressed strings' feature in JDK 6?)
Whoops - yes, confused with compressed oops :-)
>
> And one more thing - the doc discusses string lengths. In addition to 
> the number of UTF-16 code units in the UTF-16 encoding of the string 
> (String.length), and the number of unicode characters 
> (codePointCount), it is sometimes desirable to know the number of code 
> units in the encoded string. It's possible to do this with 
> getBytes(charset).length, but similar to the other performance 
> discussions here, it's expensive to create the array and throw it 
> away, and an implementation outside the JDK can't benefit from the 
> JDK's fast paths in StringSupport. In theory there could be a method 
> like String#getEncodedLength(Charset) that returned the same value as 
> getBytes(charset).length with better performance. Is that something 
> you think could be interesting to consider, either as part of this 
> proposal, or separately?

You mean the _byte size_ of the encoded string (rather than number of 
code units?)

Something like this might be interesting. That said... if the charset 
matches, then creating the segment view, then obtaining its byte size is 
O(1) (e.g. no decoding). And if the charset doesn't match, you'll need 
to decode anyway -- at which point I'm not sure the array creation is 
really the bottleneck?

Maurizio

>
> On Fri, Nov 7, 2025 at 5:40 PM Maurizio Cimadamore 
> <maurizio.cimadamore at oracle.com> wrote:
>
>     Hi,
>     here's a document that outlines a more general strategy to read
>     and write strings from/to memory segments:
>
>     https://cr.openjdk.org/~mcimadamore/panama/strings_ffm.html
>
>     As outlined in this thread, the enabling primitive for the writing
>     side is a factory that allows to view a Java string as a read-only
>     heap memory segment.
>
>     We believe the proposed strategy should provide better usability,
>     while at the same time leaving room for the implementation to "cut
>     corners", to make performance as good as possible.
>
>     Let us know what you think.
>
>     Cheers
>     Maurizio
>
>     On 05/11/2025 12:45, Liam Miller-Cushon wrote:
>>     Thanks Maurizio, considering all of this holistically makes
>>     sense. I look forward to hearing your thoughts on it once you've
>>     had time to consider the options :)
>>
>>     On Tue, Nov 4, 2025 at 11:21 PM Maurizio Cimadamore
>>     <maurizio.cimadamore at oracle.com> wrote:
>>
>>
>>         On 04/11/2025 15:23, Jorn Vernee wrote:
>>>>         Do you have thoughts on the best way to proceed here? Do
>>>>         you think it makes sense to do incrementally, or would you
>>>>         prefer to see all of these related changes happen together
>>>>         under a single issue?
>>>>
>>>         I don't have a preference. Since you've already started a PR
>>>         for enhancing getString, maybe you can focus on that for
>>>         now, and we'll file followup issues for the others.
>>>         Splitting things up might be nice since there's probably
>>>         some benchmarking work involved for each. I think the copy
>>>         and allocateFrom overload can be done in one patch though.
>>>
>>         I think I sort of do have a preference :-)
>>
>>         My feeling is that we're dealing with 2-3 different APIs that
>>         are all tightly interconnected. We have some ideas on how to
>>         solve some (I think getString with explicit length seems the
>>         most settled), but we're still playing with other ideas for
>>         the others (like copy, or memory segment views). And we also
>>         have to think about relationship with SegmentAllocator.
>>
>>         For this reason, I'd prefer to think about it more
>>         holistically and think about all the related APIs at once.
>>
>>         Once we get a consensus on how to proceed we can decide
>>         whether to pursue them all in a single PR, or split them into
>>         separate PRs.
>>
>>         But it would be unfortunate, I think, if the first PR would
>>         later reveal to be a dead end for the other use cases.
>>
>>         I plan to think about this some more (but I need some more time).
>>
>>         Cheers
>>         Maurizio
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20251110/ec4e5d12/attachment.htm>