[External] : Re: Feedback / query on jextract for Windows 10

[Maurizio Cimadamore]

On Thu, 2021-01-28 at 12:21 +0000, Duncan Gittins wrote:
> Understood.  The behaviour of "--source" is different to the class 
> generation in this respect too, as the "--source" compile version 
> creates all method handles at once. [I confirmed this by adding 
> System.out.println("downcallHandle "+name); into the 
> RuntimeHelper.downcallHandle method and then merged the 
> modified+compiled --source class with the generated classes to
> verify 
> on-demand creates there]

Yes, the source generation is eager, while the class generation is as
lazy as possible, using every inch of VM features we have available
there. We could do more on the source side, to make things lazier
there, but we're at the fence - kind of hoping for this:

https://openjdk.java.net/jeps/8209964

If we had this, then we could just use same strategy for both class AND
source and they would be both "naturally" lazy.

We do not have plans to remove the --source flag.

Cheers
Maurizio
> 
> It would be nice to view the generated source to study the issue -
> will 
> the --source option be maintained in future releases? Perhaps it's
> time 
> that I downloaded the JDK source repository and find out myself.
> 
> Duncan
> 
> On 27/01/2021 22:13, Maurizio Cimadamore wrote:
> > On Wed, 2021-01-27 at 17:56 +0000, Maurizio Cimadamore wrote:
> > > I'll keep investigating to see if this is expected behavior, or
> > > if
> > > 
> > > there's something that can be improved, either in how we
> > > generated
> > > 
> > > code, or in the constant dynamic support.
> > As it turns out, there's not much we can do in this kind of
> > situation.
> > Basically, we're in a case where we have a deep hierarchy like
> > this:
> > 
> > A1 extends A2 extends A3 ... extends An
> > 
> > (we need to do this so that all symbols will be seen as "members"
> > of
> > A1, while at the same time not blowing up the classfile limits).
> > 
> > Now, if you call a static method, like A1::m, the VM has still to
> > load
> > the entire hierarchy A1 ... An. This seems counter-intuitive at
> > first,
> > after all, we want to call a static method, not an instance one, so
> > there's no vtable needed. But - VM method resolution is designed to
> > look into superclasses not just for instance methods, but also for
> > static ones! There's also the detail that the VM must know the
> > superclasses in order to correctly layout _static_ fields in the
> > hierarchy (again, there are no fields in jextract generated
> > hierarchy -
> > but to discover that you have to walk the hierarchy anyway, so that
> > info cannot be exploited at load time).
> > 
> > So, when dealing with enormous headers (like Windows.h), filtering
> > is
> > the way to go. Our approach is already trying its best at being as
> > lazy
> > as possible (and I've verified that it only creates the layouts and
> > method handles that are used by the client - so that aspect works)
> > -
> > but we cannot take away the cost of class loading.
> > 
> > Of course, if jextract bucketed the various symbols in separate
> > files
> > which you had to manually include, we could reduce the cost
> > associated
> > to loading - but at a big cost in terms of usability: as a user of
> > a
> > big library L, how are you expected to know in which header is a
> > given
> > symbol defined? We have tried breaking up by headers in the past,
> > and
> > it just doesn't work.
> > 
> > The scheme we have now works very well for all "normal" libraries -
> > it
> > adds zero-to-very-minimal overhead, and it makes things easy for
> > clients which do not have to guess where symbols are (just import
> > the
> > main `foo_h` header file, and everything works). Unfortunately this
> > means that clients that want to interact with huge, monolithic
> > libraries, will have to resort to filtering, or maybe define some
> > headers manually (as you have done) to keep footprint down. I don't
> > think there's a silver bullet here (*): we can only hope to come up
> > with something that works well in the 90% of cases, and then offer
> > workarounds for the remaining 10%.
> > 
> > Cheers
> > Maurizio
> > 
> > (*) It occurs to me that most of the symbols required by a client
> > interacting with a jextracted API are scrutable and foldable
> > symbols:
> > method handles, var handles, layouts, ...
> > This could mean that, at least in principle, in the future it would
> > be
> > possible for javac to remove calls and dependencies on the jextract
> > generated binding, and "inline" the var/method handle creation (as
> > it
> > does for other kind of "constants"). This generalization of
> > constant
> > folding has been discussed in this JEP:
> > 
> > https://openjdk.java.net/jeps/303
> > 
> > So, there might be, one day, a solution which will allow javac to
> > "embed" only the contents from a jextract bindings that are
> > required in
> > order to execute your application. But it might take a while to get
> > there.
> > 
> >