[foreign-memaccess+abi] RFR: 8291639: Improve the ability to visualize a MemorySegment in human readable forms [v2]

[Maurizio Cimadamore]

On Fri, 26 May 2023 15:24:56 GMT, Maurizio Cimadamore <mcimadamore at openjdk.org> wrote:

>> Is it possible that we will get something like this in the future:
>> 
>> C Header:
>> 
>> struct Point {
>>     int x, y;
>> };
>> 
>> extern void f(struct Point p);
>> 
>> 
>> Java Code:
>> 
>> record Point(int x, int y) {}
>> 
>> ValueLayout pointLayout = MemoryLayout.structLayout(
>>                 JAVA_INT.withName("x"),
>>                 JAVA_INT.withName("y"))
>>         .withCarrier(Point.class);
>> 
>> MethodHandle f = nativeLinker.downcallHandle(lookup.find("f").get(), FunctionDescriptor.ofVoid(pointLayout));
>> f.invoke(new Point(10, 20)); // ok
>
>> Is it possible that we will get something like this in the future:
> 
> I believe record mapper is a "high-level" kind of mapping, so supporting this mapping from the linker out of the box would be too magic. Note also that, if you pull on that string, you might also want to express memory dereference directly using records. We thought about this kind of stuff but found it problematic because, while it's easy to read a segment "as a record" the other way around is not easy. As you point out there are lifetime issues, but also issues when it comes to writing unions (which fields of the record/union do we write?).
> 
> One of the strength of the Linker API (and FFM API) is that it is built out of very simple primitives, but it also allows composition. So, if you have a downcall which returns a segment, it is quite easy to adapt it to one that returns a record (same for var handles).
> 
> But records are only _one_ of the possible ways to map foreign types into Java-land (and one that has few limitations, as for unions). Other clients might prefer mapping structs/unions with simple classes which wrap a memory segment (which might not be too bad if these classes are Valhalla values). From a design perspective, the linker should be relatively agnostic about these kinds of high-level mappings (but should allow for tools that desire to do so, to provide said mappings).

> @mcimadamore I realized that establishing a mapping between record and MemorySegment was an overly high-level operation, so I later modified the above example to focus on mapping between record and struct.
> 
> > ```java
> > record Point(int x, int y) {}
> > 
> > ValueLayout pointLayout = MemoryLayout.structLayout(
> >                 JAVA_INT.withName("x"),
> >                 JAVA_INT.withName("y"))
> >         .withCarrier(Point.class);
> > 
> > MethodHandle f = nativeLinker.downcallHandle(lookup.find("f").get(), FunctionDescriptor.ofVoid(pointLayout));
> > f.invoke(new Point(10, 20)); // ok
> > ```
> 
> Now we do not have any type that directly corresponds to the struct type of C, so we have to convert the value of the structure to MemorySegment before passing it.
> 
> I think this status quo is unsatisfactory. Because of this, I think providing the ability to establish a mapping between record and struct is meaningful.

Having bi-directional conversions from memory segments to records is attractive, and it is something that we have explored (as mentioned above), but abandoned because, as stated above, it cannot be generalized to all by-value arguments. While structs are the most common case, it is smelly that we can only support "half" of the composite types with the system you envision.

The other issue is that you say that it is suboptimal that we need to go from the record to the segment when calling a native function. Well, yes, but it is similarly suboptimal to just use records directly - as you can see from the RecordMapper implementation, there's quite a lot of "plumbing" that needs to occur so that fields can be extracted. And, in many cases, the Linker will have to bulk-copy the struct or union somewhere else (e.g. in the stack). So, a record class with indirections is not in a form that is "easy" to use for the linker (unlike a flat memory segment). This is the main reason as to why the Linker API likes its by-value composite arguments to be passed as memory segments.

If a tool or a client would like to provide different mappings for structs, they are certainly free to do so (and the Linker API should not block them). But if clients are after performance, a record is a a handy, but lousy carrier for passing arguments to the linker (or to dereference APIs). A much better carrier is (as stated before) a class that wraps a memory segment, so that the segment can be extracted immediately, and then used by the linker to do what is required.

Ultmately, I think you are reading too much into what this API is trying to provide (but I understand the angle you came from, as we were tempted in similar directions). This is a way for clients to see what's inside a memory segment. It replaces other experimental APIs that have been provided in the past to - e.g. generate an hex dump from a segment, or generate a string representation of the contents of a segment. We realized that, by mapping a segment into a record, we could provide a much better description of "what's in the segment", in a way that is transparent to Java code, can be pattern-matched, etc.

-------------

PR Comment: https://git.openjdk.org/panama-foreign/pull/833#issuecomment-1564616859