AW: Foreign memory access with classes

[Maurizio Cimadamore]

Hi,
I still think that leaning on the language to capture this morally 
wrong. For better or worse, the audience that requires native interop is 
a (very) small fraction of the entire Java ecosystem.

Having a top-level "native" declaration would be very confusing. Try to 
put yourself in the shoes of a poor CS teacher that will have to explain 
what a native class is. Simply saying that you can ignore it if you 
don't need it is not good enough for a language/platform as popular as this.

There are also many issues with the proposed approach. Some are:

* the layout of the native class seems to be determined by the record 
components. This means no way to add padding (or slippery slope to add 
padding via annotations).
* more on layouts, mapping int to JAVA_INT is ok, but only gets you so 
far - at some point you will need a layout that is 1-1 with the C 
layout, and, to do that, you need a tool which can understand the C code 
(unless you suggest to dump all this on javac)
* structs are great - what about unions? And arrays?
* the naming convention for getter and setter is hardwired in the 
language. Records specifically settled on using an accessor name that 
has the same name as the component, to avoid ambiguities as to what the 
"right" name should be.

And, all the code you write can _still_ be generated by a tool.

So, rather than brainstorming on potential Java features which might 
never see the light of the day, why don't we instead try to discuss what 
is the problem we're trying to solve?

You mention, for instance, that:

> *The bindings created by jextract can look confusing and scary to many 
> less experienced java developers.*
>
There are two issues in this sentence: first, shape of jextract bindings 
is not generated in stone. If we'd prefer an approach where structs are 
modelled with record classes, we could go there (but of course we'd need 
to provide some escape hatch for folks that want to operate with 
segments directly).

Secondly, it assumes that jextract will be a tool ran by every Java 
developer out there, or that said developers will need to be able to 
"understand" what comes out of the tool. Which seems like a dubious 
claim. E.g. if the API generated by jextract is super friendly (static 
functions and records), does the user really care how jextract gets there?

> *I think the ability to describe a native struct in 1 line instead of 
> using an extra tool would simplify the process of interfacing with 
> native structs a lot. *
>

But, where do these structs come from? If they come from a native 
function, well, you still need a method handle to get there. Surely 
method handles are "harder" to understand that a plain wrapper around a 
memory segment (again, assuming that developers care about what the 
generated bindings look like) ?

Concluding, besides one-liner declaration convenience (which, again, is 
a questionable use case), I'm failing to see the smoking gun as to why 
this has to be a _language_ feature understood by the javac compiler and 
specified in the JLS (!!). This feature will have to be supported 
forever and will have to interoperate nicely with all the other features 
in the language. For instance, what does it mean to have a _generic_ 
native class?

I believe you are grossly understimating the requirement bar for adding 
new features to the Java language.

Maurizio



On 09/11/2022 14:17, redio.development at gmail.com wrote:
>
> Prototype for native classes
>
> *//example C-Struct Point:*
>
> structPoint{
>
> intx,y;
>
> };
>
> **
>
> *//the base class Native:*
>
> importjava.lang.foreign.MemorySegment;
>
> importjava.util.Objects;
>
> publicabstractclassNative{
>
> protectedfinalMemorySegmentsegment;
>
> protectedNative(MemorySegmentsegment) {
>
> this.segment= Objects.requireNonNull(segment);
>
> if(segment.byteSize() != layout().byteSize())
>
> thrownewIllegalArgumentException("Segment does not match native class.");
>
>   }
>
> publicMemorySegmentsegment() {
>
> returnsegment;
>
>   }
>
> publicabstractGroupLayoutlayout();
>
> }
>
> **
>
> *//the declaration of the native class Point:*
>
> publicnativeclassPoint(publicint x, publicint y) {}
>
> **
>
> *//the expansion of the syntax above:*
>
> publicfinalclassPointextendsNative{
>
> privatestaticfinalGroupLayout$LAYOUT= MemoryLayout.structLayout(
>
> ValueLayout.JAVA_INT.withBitAlignment(32).withName("x"),
>
> ValueLayout.JAVA_INT.withBitAlignment(32).withName("y"));
>
> privatestaticfinalVarHandleX$HANDLE= 
> $LAYOUT.varHandle(MemoryLayout.PathElement.groupElement("x"));
>
> privatestaticfinalVarHandleY$HANDLE= 
> $LAYOUT.varHandle(MemoryLayout.PathElement.groupElement("y"));
>
> publicPoint(MemorySegmentsegment) {
>
> super(segment);
>
>   }
>
> publicintgetX() {
>
> return(int)X$HANDLE.get(segment);
>
>   }
>
> publicvoidsetX(intx) {
>
> X$HANDLE.set(segment, x);
>
>   }
>
> publicintgetY() {
>
> return(int)Y$HANDLE.get(segment);
>
>   }
>
> publicvoidsetY(inty) {
>
> Y$HANDLE.set(segment, y);
>
>   }
>
>   @Override
>
> publicGroupLayoutlayout() {
>
> return$LAYOUT;
>
>   }
>
> }
>
> **
>
> *The expansion would be done by the compiler at compile time just like 
> records.*
>
> *As said this is just a prototype and different or additional ideas 
> are very welcome.*
>
> *The visibility of the accessor methods can be set in the field 
> declaration brackets.*
>
> *To change visibility of only getter or setter you would simply choose 
> the lower visibility and define a custom one instead of the default.*
>
> *To access the default in the custom method super could be used.*
>
> *As you might have notices this looks similar to the class jextract 
> creates and that’s intentional.*
>
> *If this would be implemented jextract could just create native 
> classes as bindings to native structs.*
>
> *This would practically shift the modeling of the concrete c struct 
> from jextract to the javac compiler.*
>
> *Since the native struct declaration is very simplistic the need for 
> jextract could be reduced.*
>
> *I think the ability to describe a native struct in 1 line instead of 
> using an extra tool would simplify the process of interfacing with 
> native structs a lot. *
>
> *The bindings created by jextract can look confusing and scary to many 
> less experienced java developers.*
>
> *Working with a native class is the exact same as working with a 
> mutable record like class with the benefits that it can be used to 
> interact with native memory.*
>
> *Thinking further jextract could actually insert the generated native 
> classes in signatures of the java representation of native functions.*
>
> *This idea would promote native structs to first class citizens in java.*
>
> **
>
> *In great regards*
>
> *RedIODev*
>
> **
>
> **
>
> *Von:*Red IO <redio.development at gmail.com>
> *Gesendet:* Dienstag, 8. November 2022 14:01
> *An:* Maurizio Cimadamore <maurizio.cimadamore at oracle.com>
> *Cc:* panama-dev <panama-dev at openjdk.org>
> *Betreff:* Re: Foreign memory access with classes
>
> JPassport looks like a great take based on the raw Panama way of doing 
> this. But it's still a pain to do and requires a precompiler. But both 
> feel uncomfortable using strings to point to mark things.
>
> Also my idea would still bahave like a normal class being on the heap 
> (could be declared as primitive or value class but let's not focus on 
> Valhalla). The only difference would be their fields, as they would 
> technically not be part of the class. The only field part of the 
> object would be the hidden MemorySegment managing the access of the 
> fields.
>
> My biggest problem with the current solution is that they try to 
> simulate a simple struct with a complex factory. I understand your 
> concerns but there is no split in the type system like you feared. 
> Native objects are simply different, that their fields are stored 
> somewhere else. Other than that they are plain objects. If projecting 
> method calls from fields is too complicated to implement we could go 
> for a record like syntax and only exposing the fake fields through 
> methods. Most java developers would probably never create their own 
> native classes but use those declared in libraries and returned by 
> library methods. In this scenario they would be indistinguishable from 
> normal objects.
>
> Regards
>
> RedIODev
>
> On Tue, Nov 8, 2022, 11:44 Maurizio Cimadamore 
> <maurizio.cimadamore at oracle.com> wrote:
>
>     I believe adding Java syntax to model objects backed by memory
>     segments
>     might be pushing things a little too far. From a pedagogical
>     perspective
>     you would now have to explain to _every_ Java developers that
>     there are
>     two different kinds of objects, plain and native, and the rules which
>     govern their access would be different (plain objects are
>     garbage-collected, native objects are not).
>
>     Seems like a nightmare, for a relatively little pay off. Note
>     that, if
>     you declare a record class with components X and Y, it is relatively
>     easy to construct a function that can read a segment, with given
>     struct
>     layout into a record with matching components. I believe JPassport
>     [1]
>     does something in this direction. With something like this you get
>     basically 90% of what you are aiming for, without the need to
>     change the
>     language by adding a new keyword, and making the programming model
>     more
>     complex for all the developers out there, including those who do not
>     care about off-heap access.
>
>     Cheers
>     Maurizio
>
>     [1] - https://github.com/boulder-on/JPassport/
>     <https://urldefense.com/v3/__https://github.com/boulder-on/JPassport/__;!!ACWV5N9M2RV99hQ!PFL9I9iPIa7J3be47BPIHVN388hZLLr4dNMQ7LC0EPYnjKXvT5DXKRBtBkJ32-ZJ3AmFMJW5ARAraEGTMTov-QkyEgMr40HT5Q$>
>
>
>     On 08/11/2022 06:52, Red IO wrote:
>     > I just had a crazy idea on how to make foreign memory access
>     work and
>     > feel like java native class member access.
>     > The idea is that you define special classes maybe with an
>     interface or
>     > a special class keyword. This class can then be "constructed"
>     using a
>     > MemorySegment being bound by its special and temporal bounds. The
>     > object would be either null or every memory access throws an
>     exception
>     > when the bounds are breached. The class would be implicitly
>     final and
>     > would extend a class Native.
>     > This could look something like this :
>     >
>     > //c struct
>     > typedef struct {
>     > int x;
>     > int y;
>     > } Point;
>     >
>     > //java native class
>     > public native class Point {
>     >
>     > int x;
>     > int y;
>     >
>     > //implicit and forced private constructor
>     > //normal creation would make things difficult
>     > }
>     >
>     > MemorySegment data =... //native memory
>     > Point p = data.object(Point.class);
>     >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20221109/52933e17/attachment-0001.htm>