[foreign] jextract, distilled

[Ty Young]

On 11/22/19 3:49 PM, Maurizio Cimadamore wrote:
>
> On 22/11/2019 20:05, Ty Young wrote:
>>
>> I understand that this API is more for lower level memaccess usage, 
>> but personally I feel like a jextract API could be useful in general 
>> for just generating and validating bindings from within Java.
> On the contrary - the API is to generate bindings, in whatever shape 
> and form you like! Our own jextract tool might use this API to 
> generate low level bindings, based on method handles and var handles, 
> but the story doesn't have to stop there!
>>
>>
>> Such issues IMO with jextract are:
>>
>>
>> A. Command line can be very long and error-prone as each argument can 
>> just blend in with each other in more complicated bindings commands.
> Couldn't agree more
>>
>>
>> B. There is no way to generate bindings that are guaranteed to work 
>> for all platforms(and sub-platforms, like Linux distros). Generating 
>> cross-platform bindings really sucks and sometimes requires a 
>> different OS than what the bindings are intended for.
> This is more of a distribution problem - e.g. how do we reason about a 
> platform-dependent Java artifact; but I agree that jextract doesn't do 
> much in that respect. The fact that bindings are platform-dependent is 
> unavoidable; in some case the same set of bindings could work across 
> multiple platforms/OS, in other case you might have very wild 
> differences between the various platforms/OS which can be hard, if not 
> impossible to reconcile. This looks like one of those areas where the 
> library packager might be in a better position to know what to do in 
> terms of providing an uniform API regardless of platforms/OS.
>>
>>
>> C. There is no way(AFAIK) to reverse validate(in other words, after 
>> the bindings are generated) the validness of bindings produced by 
>> jextract. If the bindings are broken for some reason you can't, as a 
>> higher level API dev, provide any information as to why it's broken 
>> or even know that the bindings are broken until it's too late.
> Writing a tool which compares the native library entry points with the 
> set of bindings mechanically generated by jextract, and spot issues 
> seems like a very good idea, and pretty much in the real of what would 
> be possible to do with this API (combined with some classfile parsing 
> handling, e.g. via ASM).
>>
>>
>> Both A and B can be fixed with an interface API for defining the 
>> correct way to generate the required bindings and a way to execute 
>> that interface implementation from within Java. C could be fixed by 
>> providing a way to get the way the bindings where originally 
>> generated via meta data.
>
> Not sure I get what you are proposing in terms of A/B (they seem very 
> different problems to me).


They are, however I think both A and B can be fixed(or smoothed over) by 
this new API if it provided a standardized(in the literal sense) Java 
interface(s) for defining a jextract binding generation.


For example:


// returns a list of libraries required by this binding extraction

public List<File> getLibraries();


and


// returns a list of header files required by this binding extraction

public List<File> getHeaders();


etc.


The idea here is to define a standardized, repeatable way of generating 
bindings for higher level devs's targeted platform and add a way to 
submit them to jextract via its API. Currently it seems like everything 
is done via String instead of something more robust.


It might not be possible to generate "universal" bindings that Just 
Work(TM) on every platform, but defining a standardized interface 
implementations to specify various jextract commands *for* particular 
platforms is. Once these are defined they aren't likely to change often 
if ever.


Of course, it's possible for this to be done yourself. I just think it'd 
be nice if this was included by default as per the API.


> As for C, we have discussed internally to maybe sprinkle some metadata 
> on the bindings as a way to keep a 'link' to the place where they came 
> from (which is also why the API has this Position abstraction - 
> reifying that into, say, a java annotation is doable).


The type of validation I was thinking of(relates to the above 
standardized interface API somewhat) would required everything used to 
generate the bindings be part of the meta data. Basically I'm  thinking 
of comparing the provided binding generation vs. a list of valid, good 
generations and if it fails then an exception is thrown.


Comparing the library vs. the bindings kinda slipped my mind since the 
native API I'm working with is both backwards compatible and 
cross-platform compatible(except the library name itself, go figure), 
but yeah that's be really nice to have too.


Would a properties file inside the bindings jar not work better and 
easier than annotations?


>
> Maurizio
>
>>
>>
>> Could this maybe be added?
>>
>>
>>
>> On 11/22/19 10:14 AM, Maurizio Cimadamore wrote:
>>> Hi,
>>> as we got foreign memory access and foreign function access on a 
>>> much more solid footing --- using VarHandle and MethodHandle, 
>>> respectively --- I thought it was time to look again at jextract and 
>>> see how we could make these new primitives play a more central part 
>>> in the extraction process. The writeup below captures my thinking on 
>>> this topic:
>>>
>>> http://cr.openjdk.java.net/~mcimadamore/panama/jextract_distilled.html
>>>
>>> Cheers
>>> Maurizio
>>>
>>>