[External] : Re: Feedback / query on jextract for Windows 10

Maurizio Cimadamore maurizio.cimadamore at oracle.com
Wed Jan 27 14:45:35 UTC 2021 

Hi Duncan,

On Wed, 2021-01-27 at 14:36 +0000, Duncan Gittins wrote:
> Thanks, that's very helpful. Use of repeated --filter option solves
> my 
> problem with huge jars - now down to 800k and cuts the first-time 
> startup delay vs handcoded to around 50-150ms depending on the calls
> made.

I'm happy that the filter option worked for you! I still have to
investigate on the startup issue.
> 
> It might be helpful if there was a command line option to set the
> output 
> class name so the top level classname isn't derived from the header
> file 
> name. I avoid static imports, and would prefer to call say 
> "Ole32.CoInitializeEx()" than "shlobj_core.CoInitializeEx()" for
> code 
> derived from shlobj_core.h.

I think this is a good suggestion. While inferring from header file is
a good "default" I agree that it should be possible to override this,
especially now that a single header file is generated (the reason we
don't have this is that it was harder to to in the previous design).

> 
> This would also help for projects that had one master header file
> which 
> set up #define for other includes. eg. Consider winuser.h:
> 
>     set "WINKIT=c:\Program Files (x86)\Windows
> Kits\10\Include\10.0.18362.0"
>     jextract -luser32 -t duncan.win.user32 -d winuser.classes --
> filter 
> winuser.h "%WINKIT%/um/winuser.h"
>     => Fails error: "No Target Architecture"
> 
>     jextract -luser32 -t duncan.win.user32 -d winuser.classes --
> filter 
> winuser.h "%WINKIT%/um/shlobj_core.h"
>     => Works fine, but the top level class name is shlobj_core_h
> which 
> would same class name as for this jextract:
> 
>     jextract -lshell32 -t duncan.win.shell32 -d shlobj_core.classes 
> --filter shlobj_core.h "%WINKIT%/um/shlobj_core.h"
> 
> However there is a very simple workaround: I can create a User32.h
> file 
> which contains "#include <shlobj_core.h>"

Yes! I expect this kind of workaround to be unavoidable in the general
cases - especially when dealing with very large monolithic headers such
as Windows.h.

Ideally, one day, IDEs might provide some integration for this, so that
maybe when you extract you get a nice dialog with all the "discovered
symbols" and you can manually click on the ones you want to import.
This is why our aim is to provide not just a tool, but also some kind
of API, so that this kind of IDE plugin will, one day, be possible.

CheersMaurizio


> 
>      jextract -luser32 -t duncan.win.user32 -d winuser.classes --
> filter 
> winuser.h  -I "%WINKIT%/um" User32.h
> 
> Duncan
> 
> On 26/01/2021 17:25, Maurizio Cimadamore wrote:
> > Another fly by comment - have you tried using the --filter option
> > so 
> > that jextract will just extract the header you want?
> > 
> > e.g. you might try adding `--filter shlobj_core` and see what
> > happens 
> > (but it might be too narrow of a filter).
> > 
> > Maurizio
> > 
> > 
> > On 26/01/2021 15:43, Duncan Gittins wrote:
> > > I don't know what causes the speed difference, which may be down
> > > to 
> > > the jar size as you suggest. I'll re-test after the time that 
> > > "--source" generates same code as seen in the .class files when
> > > it 
> > > may be easier to understand why there is a difference. I am
> > > using 
> > > 16-panama+3-385.
> > > 
> > > This isn't a very robust test, but I also tried deleted all the 
> > > un-necessary .class files from the 11MB jextract generated
> > > classes 
> > > (not produced using --source) down to 2MB jar, then I moved my 
> > > handcoded method into a separate class and added a dependency on
> > > a 
> > > different 2MB jar. Neither of those changes affected the typical 
> > > speeds of first call times - I still consistently see that the
> > > hand 
> > > coded method handle version of my app is ~300-500ms faster on
> > > first 
> > > call, and no difference for 2nd+ calls.
> > > 
> > > Duncan
> > > 
> > > On 25/01/2021 16:18, Maurizio Cimadamore wrote:
> > > > On 25/01/2021 16:13, Duncan Gittins wrote:
> > > > > Thanks for the information.  As you suspected that bug must
> > > > > be 
> > > > > related and there is a minor difference in the --source
> > > > > output 
> > > > > because the problem does go away when removing the jextract 
> > > > > "--source" flag and making jar directly from generated
> > > > > classes 
> > > > > rather than just compiling from the generated source code.
> > > > > 
> > > > > My code samples using jextract bindings are ~300ms slower at 
> > > > > startup time (but same speed runtime). I'm not certain if
> > > > > this 
> > > > > overhead is down to initialisation of static constants or
> > > > > simply to 
> > > > > be expected when adding 11MB jar footprint (compared to 56k
> > > > > jar for 
> > > > > my handwritten MH bindings). I will follow-up if I find out
> > > > > more.
> > > > If you see the startup slowdown even when omitting --source,
> > > > then 
> > > > the issue is with loading the jar, rather than initializing -
> > > > as 
> > > > explaining no initializing should happen outside for stuff that
> > > > is 
> > > > actually used by your code - so I'd expect that initialization 
> > > > overhead for jextract code and your manually written versions
> > > > should 
> > > > be roughly the same.
> > > > > Looking at the generated code for "xxx$constants$nn.java"
> > > > > there 
> > > > > seems to be a degree of unnecessary initialisation per 
> > > > > xxx$constants$nn class as it contains static variables for 
> > > > > FunctionDescriptor / MemoryLayout which would be initialised
> > > > > every 
> > > > > time that particular class was loaded and not used in my
> > > > > code. I 
> > > > > felt that my own jar was slow starting so switched foreign
> > > > > linker 
> > > > > references to a lazy load Supplier for every library / symbol
> > > > > / mh 
> > > > > / vh so that only the referenced types get created.
> > > > 
> > > > Sure - but that's only when --source is used - when you
> > > > generated 
> > > > classes directly, everything is lazy (and in a more general way
> > > > than 
> > > > using Suppliers).
> > > > 
> > > > So, can you please confirm, to be clear, that you still see
> > > > startup 
> > > > regression even when NOT using "--source" ?
> > > > 
> > > > Thanks!
> > > > Maurizio
> > > > 
> > > > > I will have a look at future builds - this project is
> > > > > extremely 
> > > > > useful so I hope it gets into a full JDK release soon. 
> > > > > Unfortunately my issue with a huge jar in Eclipse means I
> > > > > can't 
> > > > > easily develop using the generated code variant right now,
> > > > > but in 
> > > > > the meantime the jextract code is still very helpful to work
> > > > > out 
> > > > > how to call Windows APIs correctly.
> > > > > 
> > > > > Duncan
> > > > > 
> > > > > On 25/01/2021 10:20, Maurizio Cimadamore wrote:
> > > > > > On 23/01/2021 18:34, Duncan Gittins wrote:
> > > > > > > I've been using Panama on Windows 10 to eliminate my C#
> > > > > > > apps 
> > > > > > > which I would normally call via ProcessBuilder. I'm very 
> > > > > > > impressed so far and looking forward to newer versions: 
> > > > > > > everything I have written in Windows C# is now replaced
> > > > > > > with 
> > > > > > > foreign memory / linker calls to native code using the
> > > > > > > latest 
> > > > > > > Java 16 EA.  I have examples for various Win32 API and
> > > > > > > COM 
> > > > > > > objects, HWND objects and screensaver. However I'm
> > > > > > > struggling to 
> > > > > > > get the code running with output from jextract build.
> > > > > > > Perhaps I 
> > > > > > > overlooked something very obvious in how to setup?
> > > > > > > 
> > > > > > > One example app is attached - this calls
> > > > > > > SystemParametersInfoA to 
> > > > > > > set the Windows desktop background. There are two
> > > > > > > implementations 
> > > > > > > of setImage: one with hardwired MethodHandle works fine,
> > > > > > > but the 
> > > > > > > jextract version gives UnsupportedOperationException for
> > > > > > > layout 
> > > > > > > unrelated to SystemParametersInfoA. If I edit the
> > > > > > > generated code 
> > > > > > > to comment out all lines unrelated to
> > > > > > > SystemParametersInfoA then 
> > > > > > > both versions of setImage work.
> > > > > > 
> > > > > > I think the culprit is this:
> > > > > > 
> > > > > > ```
> > > > > > Caused by: java.lang.UnsupportedOperationException:
> > > > > > Invalid 
> > > > > > alignment requirements for layout 
> > > > > > b64(ArbitraryUserPointer)[abi/kind=POINTER,layout/name=Arbi
> > > > > > traryUserPointer]
> > > > > >         at 
> > > > > > jdk.incubator.foreign/jdk.internal.foreign.LayoutPath.check
> > > > > > Alignment(LayoutPath.java:273)
> > > > > >         at 
> > > > > > jdk.incubator.foreign/jdk.internal.foreign.LayoutPath.deref
> > > > > > erenceHandle(LayoutPath.java:159)
> > > > > >         at 
> > > > > > jdk.incubator.foreign/jdk.incubator.foreign.MemoryLayout.la
> > > > > > mbda$varHandle$2(MemoryLayout.java:488)
> > > > > >         at 
> > > > > > jdk.incubator.foreign/jdk.incubator.foreign.MemoryLayout.co
> > > > > > mputePathOp(MemoryLayout.java:534)
> > > > > >         at 
> > > > > > jdk.incubator.foreign/jdk.incubator.foreign.MemoryLayout.va
> > > > > > rHandle(MemoryLayout.java:488)
> > > > > >         at 
> > > > > > duncan.win32.shobjidl_core.shlobj_core_h$constants$16.<clin
> > > > > > it>(shlobj_core_h$constants$16.java:1844)
> > > > > > ```
> > > > > > 
> > > > > > This seems a duplicate of another issue that has been
> > > > > > reported 
> > > > > > last week:
> > > > > > 
> > > > > > https://bugs.openjdk.java.net/browse/JDK-8259832
> > > > > > 
> > > > > > E.g. jextract ignoring alignment requirements. Note that
> > > > > > jextract 
> > > > > > generates lots of constants, and it takes only one bad one
> > > > > > to make 
> > > > > > initialization fail.
> > > > > > 
> > > > > > > The jextract command on Windows 10 (with fresh Vis
> > > > > > > Studio 
> > > > > > > installation) expands shlobj_core.h because I my app also
> > > > > > > uses 
> > > > > > > IShellLink.
> > > > > > > 
> > > > > > >      set "WINKIT=c:\Program Files (x86)\Windows 
> > > > > > > Kits\10\Include\10.0.18362.0"
> > > > > > >      jextract --source -t duncan.win32.shobjidl_core -d 
> > > > > > > shlobj_core.h/java     "%WINKIT%/um/shlobj_core.h"
> > > > > > > 
> > > > > > > Another issue is that this single windows header file
> > > > > > > gives rise 
> > > > > > > to 11MB jar and is unusable inside Eclipse (on my slow
> > > > > > > PC), and 
> > > > > > > is slow startup at runtime. Have you considered changing
> > > > > > > the 
> > > > > > > generated code to use lazy Supplier<MethodHandle> instead
> > > > > > > of 
> > > > > > > MethodHandle to ensure only required handles are created
> > > > > > > on-demand?
> > > > > > 
> > > > > > Uhm... we are already using dynamic constants, so you
> > > > > > should only 
> > > > > > pay for the MHs and VHs you use - e.g. in fact I'm even
> > > > > > surprised 
> > > > > > you get the failure inside the <clinit> above - since that 
> > > > > > constant should not be initialized unless it's used. I
> > > > > > wonder if 
> > > > > > there's a regression here... are you using the `--source`
> > > > > > flag? 
> > > > > > With that flag we revert back to a monolithic header, so
> > > > > > that will 
> > > > > > explain both failures. If so, try removing that flag - and
> > > > > > see 
> > > > > > what happens (if everything is working as it should I don't
> > > > > > think 
> > > > > > you should even see the error above, assuming you never use
> > > > > > that 
> > > > > > layout).
> > > > > > 
> > > > > > Maurizio
> > > > > > 
> > > > > > > Kind regards
> > > > > > > 
> > > > > > > Duncan
> > > > > > > 
> > > > > > >

More information about the panama-dev mailing list