Foreign memory access with classes

Red IO redio.development at gmail.com
Wed Nov 9 18:59:44 UTC 2022 

I must admit there are some very valid points against it. Some I though
about and might be solvable. But others that I simply overlooked like the
fact that it locks the syntax "native class" forever to that feature and
potentially hardwireing it to a failed feature. I think I will go back to
the drawing board and focus on simple mostly class based solutions. But I
am convinced that the great flexible api the foreign function interface is
needs a good "frontend" like the proposed native classes would have
provided. A simple template that allows simple creation of high level class
bindings for low level structs (unions, arrays). The most technical part of
that equation is done (the connection from struct to
GroupLayout/MemorySegment) all that's left is a public api that is very
flexible enough to easily create the bindings like getX, setX, getY, setY.

Great regards
RedIODev

On Wed, Nov 9, 2022, 18:32 Radosław Smogura <mail at rsmogura.net> wrote:

> Hi all,
>
> I would add one more reason why this can be hard to implement as language
> feature.
>
> Java is multi platform, but native code is not. This means that particular
> structure can have different layout depending on cpu, operating system or
> compiler switches (ie.: packed structures).
>
> I guess it would be very obscuring for Java to add all possible options
> from C related to structure shape.
>
> Kind regards,
> Radoslaw Smogura
> ------------------------------
> *From:* panama-dev <panama-dev-retn at openjdk.org> on behalf of Maurizio
> Cimadamore <maurizio.cimadamore at oracle.com>
> *Sent:* Wednesday, November 9, 2022 5:09:51 PM
> *To:* redio.development at gmail.com <redio.development at gmail.com>
> *Cc:* 'panama-dev' <panama-dev at openjdk.org>
> *Subject:* Re: AW: Foreign memory access with classes
>
>
> Hi,
> I still think that leaning on the language to capture this morally wrong.
> For better or worse, the audience that requires native interop is a (very)
> small fraction of the entire Java ecosystem.
>
> Having a top-level "native" declaration would be very confusing. Try to
> put yourself in the shoes of a poor CS teacher that will have to explain
> what a native class is. Simply saying that you can ignore it if you don't
> need it is not good enough for a language/platform as popular as this.
>
> There are also many issues with the proposed approach. Some are:
>
> * the layout of the native class seems to be determined by the record
> components. This means no way to add padding (or slippery slope to add
> padding via annotations).
> * more on layouts, mapping int to JAVA_INT is ok, but only gets you so far
> - at some point you will need a layout that is 1-1 with the C layout, and,
> to do that, you need a tool which can understand the C code (unless you
> suggest to dump all this on javac)
> * structs are great - what about unions? And arrays?
> * the naming convention for getter and setter is hardwired in the
> language. Records specifically settled on using an accessor name that has
> the same name as the component, to avoid ambiguities as to what the "right"
> name should be.
>
> And, all the code you write can _still_ be generated by a tool.
>
> So, rather than brainstorming on potential Java features which might never
> see the light of the day, why don't we instead try to discuss what is the
> problem we're trying to solve?
>
> You mention, for instance, that:
>
> *The bindings created by jextract can look confusing and scary to many
> less experienced java developers.*
>
> There are two issues in this sentence: first, shape of jextract bindings
> is not generated in stone. If we'd prefer an approach where structs are
> modelled with record classes, we could go there (but of course we'd need to
> provide some escape hatch for folks that want to operate with segments
> directly).
>
> Secondly, it assumes that jextract will be a tool ran by every Java
> developer out there, or that said developers will need to be able to
> "understand" what comes out of the tool. Which seems like a dubious claim.
> E.g. if the API generated by jextract is super friendly (static functions
> and records), does the user really care how jextract gets there?
>
> *I think the ability to describe a native struct in 1 line instead of
> using an extra tool would simplify the process of interfacing with native
> structs a lot. *
>
>
> But, where do these structs come from? If they come from a native
> function, well, you still need a method handle to get there. Surely method
> handles are "harder" to understand that a plain wrapper around a memory
> segment (again, assuming that developers care about what the generated
> bindings look like) ?
>
> Concluding, besides one-liner declaration convenience (which, again, is a
> questionable use case), I'm failing to see the smoking gun as to why this
> has to be a _language_ feature understood by the javac compiler and
> specified in the JLS (!!). This feature will have to be supported forever
> and will have to interoperate nicely with all the other features in the
> language. For instance, what does it mean to have a _generic_ native class?
>
> I believe you are grossly understimating the requirement bar for adding
> new features to the Java language.
>
> Maurizio
>
>
>
> On 09/11/2022 14:17, redio.development at gmail.com wrote:
>
> Prototype for native classes
>
>
>
> *//example C-Struct Point:*
>
> struct Point {
>
>     int x,y;
>
> };
>
>
>
> *//the base class Native:*
>
> import java.lang.foreign.MemorySegment;
>
> import java.util.Objects;
>
>
>
> public abstract class Native {
>
>
>
>     protected final MemorySegment segment;
>
>
>
>     protected Native(MemorySegment segment) {
>
>         this.segment = Objects.requireNonNull(segment);
>
>         if (segment.byteSize() != layout().byteSize())
>
>             throw new IllegalArgumentException("Segment does not match
> native class.");
>
>     }
>
>
>
>     public MemorySegment segment() {
>
>         return segment;
>
>     }
>
>
>
>     public abstract GroupLayout layout();
>
> }
>
>
>
>
>
> *//the declaration of the native class Point:*
>
> public native class Point(public int x, public int y) {}
>
>
>
> *//the expansion of the syntax above:*
>
> public final class Point extends Native {
>
>
>
>     private static final GroupLayout $LAYOUT = MemoryLayout.structLayout(
>
>         ValueLayout.JAVA_INT.withBitAlignment(32).withName("x"),
>
>         ValueLayout.JAVA_INT.withBitAlignment(32).withName("y"));
>
>
>
>     private static final VarHandle X$HANDLE = $LAYOUT.varHandle(
> MemoryLayout.PathElement.groupElement("x"));
>
>     private static final VarHandle Y$HANDLE = $LAYOUT.varHandle(
> MemoryLayout.PathElement.groupElement("y"));
>
>
>
>     public Point(MemorySegment segment) {
>
>         super(segment);
>
>     }
>
>
>
>     public int getX() {
>
>         return (int)X$HANDLE.get(segment);
>
>     }
>
>
>
>     public void setX(int x) {
>
>         X$HANDLE.set(segment, x);
>
>     }
>
>
>
>     public int getY() {
>
>         return (int)Y$HANDLE.get(segment);
>
>     }
>
>
>
>     public void setY(int y) {
>
>         Y$HANDLE.set(segment, y);
>
>     }
>
>
>
>     @Override
>
>     public GroupLayout layout() {
>
>         return $LAYOUT;
>
>     }
>
> }
>
>
>
> *The expansion would be done by the compiler at compile time just like
> records.*
>
> *As said this is just a prototype and different or additional ideas are
> very welcome.*
>
> *The visibility of the accessor methods can be set in the field
> declaration brackets.*
>
> *To change visibility of only getter or setter you would simply choose the
> lower visibility and define a custom one instead of the default.*
>
> *To access the default in the custom method super could be used.*
>
> *As you might have notices this looks similar to the class jextract
> creates and that’s intentional.*
>
> *If this would be implemented jextract could just create native classes as
> bindings to native structs.*
>
> *This would practically shift the modeling of the concrete c struct from
> jextract to the javac compiler.*
>
> *Since the native struct declaration is very simplistic the need for
> jextract could be reduced.*
>
> *I think the ability to describe a native struct in 1 line instead of
> using an extra tool would simplify the process of interfacing with native
> structs a lot. *
>
> *The bindings created by jextract can look confusing and scary to many
> less experienced java developers.*
>
> *Working with a native class is the exact same as working with a mutable
> record like class with the benefits that it can be used to interact with
> native memory.*
>
> *Thinking further jextract could actually insert the generated native
> classes in signatures of the java representation of native functions.*
>
> *This idea would promote native structs to first class citizens in java.*
>
>
>
> *In great regards*
>
> *RedIODev*
>
>
>
>
>
> *Von:* Red IO <redio.development at gmail.com> <redio.development at gmail.com>
> *Gesendet:* Dienstag, 8. November 2022 14:01
> *An:* Maurizio Cimadamore <maurizio.cimadamore at oracle.com>
> <maurizio.cimadamore at oracle.com>
> *Cc:* panama-dev <panama-dev at openjdk.org> <panama-dev at openjdk.org>
> *Betreff:* Re: Foreign memory access with classes
>
>
>
> JPassport looks like a great take based on the raw Panama way of doing
> this. But it's still a pain to do and requires a precompiler. But both feel
> uncomfortable using strings to point to mark things.
>
> Also my idea would still bahave like a normal class being on the heap
> (could be declared as primitive or value class but let's not focus on
> Valhalla). The only difference would be their fields, as they would
> technically not be part of the class. The only field part of the object
> would be the hidden MemorySegment managing the access of the fields.
>
> My biggest problem with the current solution is that they try to simulate
> a simple struct with a complex factory. I understand your concerns but
> there is no split in the type system like you feared. Native objects are
> simply different, that their fields are stored somewhere else. Other than
> that they are plain objects. If projecting method calls from fields is too
> complicated to implement we could go for a record like syntax and only
> exposing the fake fields through methods. Most java developers would
> probably never create their own native classes but use those declared in
> libraries and returned by library methods. In this scenario they would be
> indistinguishable from normal objects.
>
>
>
> Regards
>
> RedIODev
>
>
>
> On Tue, Nov 8, 2022, 11:44 Maurizio Cimadamore <
> maurizio.cimadamore at oracle.com> wrote:
>
> I believe adding Java syntax to model objects backed by memory segments
> might be pushing things a little too far. From a pedagogical perspective
> you would now have to explain to _every_ Java developers that there are
> two different kinds of objects, plain and native, and the rules which
> govern their access would be different (plain objects are
> garbage-collected, native objects are not).
>
> Seems like a nightmare, for a relatively little pay off. Note that, if
> you declare a record class with components X and Y, it is relatively
> easy to construct a function that can read a segment, with given struct
> layout into a record with matching components. I believe JPassport [1]
> does something in this direction. With something like this you get
> basically 90% of what you are aiming for, without the need to change the
> language by adding a new keyword, and making the programming model more
> complex for all the developers out there, including those who do not
> care about off-heap access.
>
> Cheers
> Maurizio
>
> [1] - https://github.com/boulder-on/JPassport/
> <https://urldefense.com/v3/__https://github.com/boulder-on/JPassport/__;!!ACWV5N9M2RV99hQ!PFL9I9iPIa7J3be47BPIHVN388hZLLr4dNMQ7LC0EPYnjKXvT5DXKRBtBkJ32-ZJ3AmFMJW5ARAraEGTMTov-QkyEgMr40HT5Q$>
>
>
> On 08/11/2022 06:52, Red IO wrote:
> > I just had a crazy idea on how to make foreign memory access work and
> > feel like java native class member access.
> > The idea is that you define special classes maybe with an interface or
> > a special class keyword. This class can then be "constructed" using a
> > MemorySegment being bound by its special and temporal bounds. The
> > object would be either null or every memory access throws an exception
> > when the bounds are breached. The class would be implicitly final and
> > would extend a class Native.
> > This could look something like this :
> >
> > //c struct
> > typedef struct {
> > int x;
> > int y;
> > } Point;
> >
> > //java native class
> > public native class Point {
> >
> > int x;
> > int y;
> >
> > //implicit and forced private constructor
> > //normal creation would make things difficult
> > }
> >
> > MemorySegment data =... //native memory
> > Point p = data.object(Point.class);
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20221109/d05a3b17/attachment-0001.htm>

More information about the panama-dev mailing list