jextract woes

[Maurizio Cimadamore]

On 27/01/2020 01:52, Michael Zucchi wrote:
>
> Ok, using the platform clang headers - by default - I get 8000 lines 
> of this:
>
> WARNING: Using incubator modules: jdk.incubator.foreign, 
> jdk.incubator.jextract
> java.lang.UnsupportedOperationException
>         at 
> jdk.incubator.jextract/jdk.incubator.jextract.tool.TypeTranslator.visitArray(TypeTranslator.java:98)
>         at 
> jdk.incubator.jextract/jdk.incubator.jextract.tool.TypeTranslator.visitArray(TypeTranslator.java:35)
>         at 
> jdk.incubator.jextract/jdk.internal.jextract.impl.TypeImpl$ArrayImpl.accept(TypeImpl.java:231)
Noticed those too
>
>
> But using --filter=cl.h I get something similar to my current 
> generator, only it's much much slower to create.
Yeah - after some experimenting I found this filter to be optimal in the 
set of things that are generated. If you pull in platform.h, you get a 
lot of data type definitions which I'm not sure if they are useful - at 
least for the common examples.
>
> But it looks interesting and will try using it - although i'm confused 
> now about which panama branch should I be working against?  Or should 
> i just run the tool from one and use it with the foreign-abi?
foreign-jextract is, essentially, foreign-abi + the tool. So you can 
just stay on foreign-jextract. Every change pushed to foreign-abi is 
automatically propagated to foreign-jextract.
>
> Some quick observations before I try to go outside:
>
>   * One would usually want to export defines and enums to the library
>     user (CL_* ints for opencl) but almost definitely not the
>     low-level calls and definitely not the variable accessors.  Having
>     them in separate files would make this much easier.
>
Ty made a similar comment - this is all very raw and experimental at the 
moment. I suspect in few weeks it will look a bit better
>
>   * +1 for outputting source rather than a jar.
>
This isn't set in stone, at least the source vs. classfile generation. 
Most notably, the fact that we're forced to generate sources, right know 
put too many constraints which affect performances. Basically, the 
generated class has a huge (and I mean huge) static initializer which is 
suboptimal for two reasons:

* even if you are interested in calling 4-5 functions, you have to 
initialize _everything_

* it is easy to go beyond the code size limit of the static initializer

Both issues can be addressed by having direct classfile generation; if, 
in the future, Java will support something like 'lazy statics' we could 
consider emitting soucefiles again - but at this point of time, 
sourcefile generation is much less optimal.

>   * I know it's probably necessary for namespace reasons but those
>     struct accessor names are nasty.  Some libraries are going to want
>     to auto-generate concrete classes some other way  so it wont
>     matter (vulkan definitely), others perhaps not (ffmpeg).
>   * specifically with cl.h, _cl_image_desc has generated no way to
>     access the unnamed union fields "buffer" and "mem_object".  In C
>     these would just appear as "cl_image_desc.buffer" and
>     "cl_image_desc.mem_object" (i.e. unqualified) with the same struct
>     offset.
>
Odd, probably a bug.
>
>  *
>
>
>   * Should all the calls catch Throwable and then assert? e.g.
>     something that calls c that calls java, can the java throw an
>     exception to be passed back up as one could with jni?  This isn't
>     really something that can be automated though.
>
Perhaps we could catch checked exceptions, and leave runtime exceptions be?
>
> Some simple ideas which I don't think are out of scope (when the time 
> comes to consider them):
>
>   * option to disable get/setter generation and if so, optionally
>     varhandles (for libraries where many fields exist that aren't public).
>   * option to put enum/#defines in another class
>   * option to provide files with specific list of functions / structs
>     / enum / defines to generate and not just a regex.
>
I think we need to do a better job with grouping things. And, the old 
jextract had more filtering options, more similar to what you describe, 
which could be resurrected - although it was also very complex.
>
> I guess It has the same problem I have with my generator - which is 
> both a benefit and a curse - there's just no type checking beyond 
> primitive types.  When you write jni you have all the facilities of 
> the C language and the Java language at your disposal.  Here you have 
> ... "every pointer is a 'void *'".  It works in practice but it is 
> definitely error-prone. I don't have anything to suggest though 
> because either you have to put a big layer of common abstraction over 
> everything which is both a cost and a constraint (and gets cumbersome 
> real fast), or you make something tighter but application-specific 
> outside of jextract.
>
Big layer of abstraction was the route the old jextract went. While in 
some cases it was helpful, in other cases it was more an hindrance than 
anything else. In my (brief) experience converting OpenCL examples to 
Panama, I've seen exactly the same issues I had with the old jextract - 
that is, the added abstraction in the older jextract didn't really help 
solving some of the issues when using this API - actually it made it 
worse (IMHO). As you pointed out in another email, OpenCL is quite void* 
happy. And, in the old world, turning a void* into e.g. a 
Pointer<CL_Int> was not as easy as in C - and added overhead. Here, if 
you have memory address, you can dereference it any way you like, which 
is unsafe (e.g. you could attempt to dereference a string as a cl_int) 
but it's handy as well.
>
> Finally, this fails to compile presumably due to the bitfields (just 
> something i used to test a generator):
Bitfields are not supported at the moment.
>
> struct data {
>         struct data *next;
>
>         int a;
>         int b;
>         int c:3;
>         unsigned d:5;
>         int (*test_a)(int a, int b);
>
>         struct data **list;
> };
>
> void print_data(struct data *data);
>
> void *api_func(const char *name);
>
> I hate bitfields, but they get used even in public apis (or at least, 
> their header files).
Maurizio
>
>
> On 27/1/20 10:43 am, Maurizio Cimadamore wrote:
>> I noticed that problem too - I was trying to use jextract on opencl 
>> headers too to take a look at what came out and found that issue.
>>
>> This is something we fixed in the old jextract by having it include 
>> some clang headers by default - Sundar, I think this change is 
>> missing from current foreign-jextract?
>>
>> For now you can workaround like this (at least, this is what I used):
>>
>> jextract --filter cl.h -I 
>> /opt/clang/clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-16.04/lib/clang/9.0.0/include/ 
>> /usr/include/CL/cl.h
>>
>> This gave me something that was enough to Panama-ize some basic 
>> OpenCL examples I found on the web.
>>
>> (The clang path you see is the one you use to build the Panama JDK)
>>
>> Maurizio
>