Creating a high level Pointer API

[Maurizio Cimadamore]

Can we please agree to disagree? I've shown in the document how the 
fragmentation you are afraid of is already the status quo, by linking at 
least 3 different pointer implementation, with radically different 
properties, both at in terms of the type system choices and in terms of 
how they are implemented. Some of these libraries even have different 
pointer classes to denote different state - e.g. pointer with size vs. 
unchecked pointer. I'm under no illusion that "one pointer will rule 
them all".

My definition of higher level API is, to put it more clearly, an API 
which captures some of the properties of a pointer in the type system, 
perhaps through generics (e.g. Pointer<Integer>). And, some sprinkle of 
accessor methods, so that users of the higher-level API need not to 
worry about constructing VarHandle accessors. So, to me something like 
roughly like this:

class Pointer<X> {
    MemoryAddress _addr;
    VarHandle handle;

    Pointer(MemoryAddress addr, MemoryLayout layout) { ... init var 
handle ... }

    X get() { return handle.get() }

    void set(X x) { return handle.set(x); }

    ...

}

Would constitute a possible sketch for an higher-level API.

But we're getting ahead of ourselves - the way we will proceed will be 
to (1) roll out the minimal extract (2) use it to rewrite the existing 
foreign examples (3) assess its usability, and add some usability boosts 
where needed. Here, it seems to me we're jumping to discuss (3), without 
having even tried the thing in the first place. So, I'd prefer to hold 
off this discussion, at least until we can make it more concrete as I 
feel it's going in circles.

As for memory segment not exposing layouts, that's by design - memory 
layouts are optional - you can use them, or not, depending on your need. 
A segment doesn't carry around the layout it was used for its creation, 
also because there might not be one in the first place (e.g. 
MemorySegment.allocateNative(100)). Also, when slicing and dicing a 
segment, it would be hard, if not impossible, to keep the layout 
information up to date (what happens if you slice in the middle of, say 
a 4 byte ValueLayout?).

Maurizio


On 11/01/2020 00:17, Ty Young wrote:
>
> On 1/10/20 4:48 AM, Maurizio Cimadamore wrote:
>> Hi think some of the points in this emails have been discussed to 
>> death (such as the enum translation). 
>
>
> Which you largely ignore entirely or by saying:
>
>
>> The point of having an API is precisely that you can decide whatever 
>> translation you see fit.
>
>
> Which I've explained multiple times now why that's a bad idea, and you 
> neither are willing to recognize it as a possible problem or refute it.
>
>
>> As for your claims about higher-level APIs, again, I think an higher 
>> level API would need to _hide_ the memory access API, so I see very 
>> little value in having an higher-level API implements memory access 
>> interfaces directly. You seem caught in the middle - you want 
>> something slightly higher level than memory address, but you want to 
>> use raw bindings which use MemoryAddress and friends.
>
>
> There is no concrete definition on what constitutes a "higher-level" 
> API. It's a purposely vague, open ended terminology that at best means 
> "something more than the original and simplified in some way". What is 
> "more" or "original"? In what way is it "simplified"? That's up to the 
> person doing the "higher-level" API to decide and *that's the problem* 
> and why exposing lower level Memory Access APIs is a must, *so you 
> aren't bound to one person's API*.
>
>
>>
>> If you introduce an higher level abstraction - call it HighPointer, 
>> then you can, through the jextract API, wrap all MemoryAddress coming 
>> from the library into HighPointer, and back. The user will never have 
>> even to think about MemoryAddress. That's how I'd go about doing an 
>> higher level pointer API.
>
>
> Like I've said in the past(and you seem to just ignore), this is going 
> to cause fragmentation, compatibility headaches, and more memory 
> usage(more objects). You're going to end up using library A that has 
> its own form of Pointer API with library B that has its own Pointer 
> API and no way to cross-communicate besides *maybe* taking the data 
> stored in API A and storing into API B instead of just plugging a 
> single Pointer interface implementation from the JDK itself into both. 
> No one has refuted this possibility and you seem to be putting your 
> head in the sand whenever it's brought up and shouting "just use the 
> jextract API!".
>
>
> Better yet, higher-level abstraction APIs can't integrate with Memory 
> Access because the interfaces are going to be sealed. Not only does 
> this create more boilerplate(everyone loves pointless boilerplate, 
> right?) but creates a disconnect between native API documentation and 
> what actually goes on in Java land with no way to fix that at a higher 
> API level. Not wanting to provide the API in the first place is bad 
> as-is but then to not provide the means to allow integrated(with 
> Memory Access, that is) APIs by sealing the interfaces is a giant 
> middle finger.
>
>
> Is the jextract API only going to apply a higher level abstraction to 
> a specific library or can you define broad API that can be applied to 
> any native library? Either could be bad for different reasons.
>
>
>>
>> As for the rest, distinguishing between long/double, and selecting 
>> SystemABI, these are rough edges of the API we are aware of. As you 
>> point out, this is early days and we're still actively hacking on 
>> that API (while the memory access API is more stable now) - we know 
>> we have to do better in order to make some of the choices the ABI 
>> does more transparent. Opening up the ABI classes, and also opening 
>> up the layout metadata infrastructure is the likely way for us to get 
>> there.
>
>
> Good to hear but (IMO) Memory Access could use work too in terms of 
> information it exposes... like the layout used to create a segment for 
> example. It would be nice if I could reconstruct a pointer based 
> solely on its segment.
>