Ability to extend a MemorySegment

[Maurizio Cimadamore]

On 21/01/2020 11:22, Michael Zucchi wrote:
> I think MemorySegment is way over-engineered already, adding more 
> complexity doesn't sound great.  I wish it was just a 1:1 mapping to a 
> malloc block (or slice thereof), and had no support for arrays or 
> array bytebuffers. 

Well, you come from the perspective of someone who needs ABI support - 
fine. But are you saying that, because MemorySegment have to play nice 
with ABI, it is not important to support other related use cases which 
have _nothing to do with ABI_ ?

One cool thing you can do with segments is:

* create a memory segment
* map it into a ByteBuffer
* use the ByteBuffer as usual
* close the segment

This gives you same old good ByteBuffer API (no var handles!) but with 
deterministic deallocation on top (which is something that Java users 
have been asking for ages, but that, given restrictions of the BB API we 
couldn't fully deliver).

Another thing segments allow you to do is to take a big file (over 2G) 
and map slices of it to different BB, effectively allowing you to use 
the BB API on stuff that is bigger than 2GB.

What you call over-engineering to me sounds more like you are probably 
just not interested in the other half of use cases that this API is 
about. With this I'm not saying the API as is is perfect, but in some of 
the feedback we have received so far the recurrent theme seems to be "I 
don't need XYZ, so it is just silly to have it in the API", or "I really 
badly need XYZ so why didn't you just add it to the API" - and all I'm 
saying here is that there are principles upon which some of these 
decisions have been made.

> And also that MemoryAddress was literally just an address inside of 
> it's segment - and not an 'offset' which may or may not actually be an 
> address (with no direct way to find out - from java). 
If you really want to get a low-level address from a MemoryAddress you 
can call ForeignUnsafe::getUnsafeOffset(MemoryAddress) and get the 
'long' address you need.
> The need to support this other stuff just makes the api weird and 
> confusing since the naming conventions don't make sense if it was just 
> backed by malloc and for all that you can never actually use these 
> array-based segments with "foreign" functions anyway.
See above. There's more to memory segments than just ABI.
> It this stuff there just as a (clumsy) mechanism to allow for the 
> general copy() method?
Please expand a bit more - are you referring to MemoryAddress::copy? Why 
is it clumsy (same interface as System.arrayCopy which has served us 
well for quite a bit).
> I'm sure there are reasons it's the way it is but the whole thing is 
> just so, well, 'foreign', to anything from either C or Java.

> And an even more restricted allocator?  Ouch. 

Again, you are jumping ahead and misreading what I said. What I was 
trying to explain is that there are a number of considerations which 
might affect the decision of whether something like 'realloc' really 
belongs to the MemorySegment API such as:

* which allocator segments ended up being based upon
* the fact that, like it or not, not all segments will be able to 
support this operation anyway
* whether it's important enough to deserve a place in the API anyway 
(seems like, e.g. you and Ty disagree on how much important realloc 
actually is)

Sure, we can pretend all these choices do not exist - but in reality 
they do. And again, are you saying "ouch" to having an allocator which 
will give you faster performances and better scalability than malloc, 
and better integration with the VM ecosystem as a whole? On what basis? 
Do you know that real-world projects like e.g. Netty [1] are building 
their own allocators anyway as they find malloc/free (and, by 
consequence, ByteBufer) to be hugely unfeasible for them?

Maurizio

[1] - 
https://blog.twitter.com/engineering/en_us/a/2013/netty-4-at-twitter-reduced-gc-overhead.html