[External] : Re: Feedback / query on jextract for Windows 10

[Duncan Gittins]

Understood.  The behaviour of "--source" is different to the class 
generation in this respect too, as the "--source" compile version 
creates all method handles at once. [I confirmed this by adding 
System.out.println("downcallHandle "+name); into the 
RuntimeHelper.downcallHandle method and then merged the 
modified+compiled --source class with the generated classes to verify 
on-demand creates there]

It would be nice to view the generated source to study the issue - will 
the --source option be maintained in future releases? Perhaps it's time 
that I downloaded the JDK source repository and find out myself.

Duncan

On 27/01/2021 22:13, Maurizio Cimadamore wrote:
> On Wed, 2021-01-27 at 17:56 +0000, Maurizio Cimadamore wrote:
>> I'll keep investigating to see if this is expected behavior, or if
>>
>> there's something that can be improved, either in how we generated
>>
>> code, or in the constant dynamic support.
> As it turns out, there's not much we can do in this kind of situation.
> Basically, we're in a case where we have a deep hierarchy like this:
>
> A1 extends A2 extends A3 ... extends An
>
> (we need to do this so that all symbols will be seen as "members" of
> A1, while at the same time not blowing up the classfile limits).
>
> Now, if you call a static method, like A1::m, the VM has still to load
> the entire hierarchy A1 ... An. This seems counter-intuitive at first,
> after all, we want to call a static method, not an instance one, so
> there's no vtable needed. But - VM method resolution is designed to
> look into superclasses not just for instance methods, but also for
> static ones! There's also the detail that the VM must know the
> superclasses in order to correctly layout _static_ fields in the
> hierarchy (again, there are no fields in jextract generated hierarchy -
> but to discover that you have to walk the hierarchy anyway, so that
> info cannot be exploited at load time).
>
> So, when dealing with enormous headers (like Windows.h), filtering is
> the way to go. Our approach is already trying its best at being as lazy
> as possible (and I've verified that it only creates the layouts and
> method handles that are used by the client - so that aspect works) -
> but we cannot take away the cost of class loading.
>
> Of course, if jextract bucketed the various symbols in separate files
> which you had to manually include, we could reduce the cost associated
> to loading - but at a big cost in terms of usability: as a user of a
> big library L, how are you expected to know in which header is a given
> symbol defined? We have tried breaking up by headers in the past, and
> it just doesn't work.
>
> The scheme we have now works very well for all "normal" libraries - it
> adds zero-to-very-minimal overhead, and it makes things easy for
> clients which do not have to guess where symbols are (just import the
> main `foo_h` header file, and everything works). Unfortunately this
> means that clients that want to interact with huge, monolithic
> libraries, will have to resort to filtering, or maybe define some
> headers manually (as you have done) to keep footprint down. I don't
> think there's a silver bullet here (*): we can only hope to come up
> with something that works well in the 90% of cases, and then offer
> workarounds for the remaining 10%.
>
> Cheers
> Maurizio
>
> (*) It occurs to me that most of the symbols required by a client
> interacting with a jextracted API are scrutable and foldable symbols:
> method handles, var handles, layouts, ...
> This could mean that, at least in principle, in the future it would be
> possible for javac to remove calls and dependencies on the jextract
> generated binding, and "inline" the var/method handle creation (as it
> does for other kind of "constants"). This generalization of constant
> folding has been discussed in this JEP:
>
> https://openjdk.java.net/jeps/303
>
> So, there might be, one day, a solution which will allow javac to
> "embed" only the contents from a jextract bindings that are required in
> order to execute your application. But it might take a while to get
> there.
>
>