[External] : Re: Feedback / query on jextract for Windows 10

Wed Jan 27 14:36:09 UTC 2021

Thanks, that's very helpful. Use of repeated --filter option solves my 
problem with huge jars - now down to 800k and cuts the first-time 
startup delay vs handcoded to around 50-150ms depending on the calls made.

It might be helpful if there was a command line option to set the output 
class name so the top level classname isn't derived from the header file 
name. I avoid static imports, and would prefer to call say 
"Ole32.CoInitializeEx()" than "shlobj_core.CoInitializeEx()" for code 
derived from shlobj_core.h.

This would also help for projects that had one master header file which 
set up #define for other includes. eg. Consider winuser.h:

    set "WINKIT=c:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0"
    jextract -luser32 -t duncan.win.user32 -d winuser.classes --filter 
winuser.h "%WINKIT%/um/winuser.h"
    => Fails error: "No Target Architecture"

    jextract -luser32 -t duncan.win.user32 -d winuser.classes --filter 
winuser.h "%WINKIT%/um/shlobj_core.h"
    => Works fine, but the top level class name is shlobj_core_h which 
would same class name as for this jextract:

    jextract -lshell32 -t duncan.win.shell32 -d shlobj_core.classes 
--filter shlobj_core.h "%WINKIT%/um/shlobj_core.h"

However there is a very simple workaround: I can create a User32.h file 
which contains "#include <shlobj_core.h>"

     jextract -luser32 -t duncan.win.user32 -d winuser.classes --filter 
winuser.h  -I "%WINKIT%/um" User32.h

Duncan

On 26/01/2021 17:25, Maurizio Cimadamore wrote:
> Another fly by comment - have you tried using the --filter option so 
> that jextract will just extract the header you want?
>
> e.g. you might try adding `--filter shlobj_core` and see what happens 
> (but it might be too narrow of a filter).
>
> Maurizio
>
>
> On 26/01/2021 15:43, Duncan Gittins wrote:
>> I don't know what causes the speed difference, which may be down to 
>> the jar size as you suggest. I'll re-test after the time that 
>> "--source" generates same code as seen in the .class files when it 
>> may be easier to understand why there is a difference. I am using 
>> 16-panama+3-385.
>>
>> This isn't a very robust test, but I also tried deleted all the 
>> un-necessary .class files from the 11MB jextract generated classes 
>> (not produced using --source) down to 2MB jar, then I moved my 
>> handcoded method into a separate class and added a dependency on a 
>> different 2MB jar. Neither of those changes affected the typical 
>> speeds of first call times - I still consistently see that the hand 
>> coded method handle version of my app is ~300-500ms faster on first 
>> call, and no difference for 2nd+ calls.
>>
>> Duncan
>>
>> On 25/01/2021 16:18, Maurizio Cimadamore wrote:
>>>
>>> On 25/01/2021 16:13, Duncan Gittins wrote:
>>>> Thanks for the information.  As you suspected that bug must be 
>>>> related and there is a minor difference in the --source output 
>>>> because the problem does go away when removing the jextract 
>>>> "--source" flag and making jar directly from generated classes 
>>>> rather than just compiling from the generated source code.
>>>>
>>>> My code samples using jextract bindings are ~300ms slower at 
>>>> startup time (but same speed runtime). I'm not certain if this 
>>>> overhead is down to initialisation of static constants or simply to 
>>>> be expected when adding 11MB jar footprint (compared to 56k jar for 
>>>> my handwritten MH bindings). I will follow-up if I find out more.
>>> If you see the startup slowdown even when omitting --source, then 
>>> the issue is with loading the jar, rather than initializing - as 
>>> explaining no initializing should happen outside for stuff that is 
>>> actually used by your code - so I'd expect that initialization 
>>> overhead for jextract code and your manually written versions should 
>>> be roughly the same.
>>>>
>>>> Looking at the generated code for "xxx$constants$nn.java" there 
>>>> seems to be a degree of unnecessary initialisation per 
>>>> xxx$constants$nn class as it contains static variables for 
>>>> FunctionDescriptor / MemoryLayout which would be initialised every 
>>>> time that particular class was loaded and not used in my code. I 
>>>> felt that my own jar was slow starting so switched foreign linker 
>>>> references to a lazy load Supplier for every library / symbol / mh 
>>>> / vh so that only the referenced types get created.
>>>
>>> Sure - but that's only when --source is used - when you generated 
>>> classes directly, everything is lazy (and in a more general way than 
>>> using Suppliers).
>>>
>>> So, can you please confirm, to be clear, that you still see startup 
>>> regression even when NOT using "--source" ?
>>>
>>> Thanks!
>>> Maurizio
>>>
>>>>
>>>> I will have a look at future builds - this project is extremely 
>>>> useful so I hope it gets into a full JDK release soon. 
>>>> Unfortunately my issue with a huge jar in Eclipse means I can't 
>>>> easily develop using the generated code variant right now, but in 
>>>> the meantime the jextract code is still very helpful to work out 
>>>> how to call Windows APIs correctly.
>>>>
>>>> Duncan
>>>>
>>>> On 25/01/2021 10:20, Maurizio Cimadamore wrote:
>>>>>
>>>>> On 23/01/2021 18:34, Duncan Gittins wrote:
>>>>>> I've been using Panama on Windows 10 to eliminate my C# apps 
>>>>>> which I would normally call via ProcessBuilder. I'm very 
>>>>>> impressed so far and looking forward to newer versions: 
>>>>>> everything I have written in Windows C# is now replaced with 
>>>>>> foreign memory / linker calls to native code using the latest 
>>>>>> Java 16 EA.  I have examples for various Win32 API and COM 
>>>>>> objects, HWND objects and screensaver. However I'm struggling to 
>>>>>> get the code running with output from jextract build. Perhaps I 
>>>>>> overlooked something very obvious in how to setup?
>>>>>>
>>>>>> One example app is attached - this calls SystemParametersInfoA to 
>>>>>> set the Windows desktop background. There are two implementations 
>>>>>> of setImage: one with hardwired MethodHandle works fine, but the 
>>>>>> jextract version gives UnsupportedOperationException for layout 
>>>>>> unrelated to SystemParametersInfoA. If I edit the generated code 
>>>>>> to comment out all lines unrelated to SystemParametersInfoA then 
>>>>>> both versions of setImage work.
>>>>>
>>>>> I think the culprit is this:
>>>>>
>>>>> ```
>>>>> Caused by: java.lang.UnsupportedOperationException: Invalid 
>>>>> alignment requirements for layout 
>>>>> b64(ArbitraryUserPointer)[abi/kind=POINTER,layout/name=ArbitraryUserPointer]
>>>>>         at 
>>>>> jdk.incubator.foreign/jdk.internal.foreign.LayoutPath.checkAlignment(LayoutPath.java:273)
>>>>>         at 
>>>>> jdk.incubator.foreign/jdk.internal.foreign.LayoutPath.dereferenceHandle(LayoutPath.java:159)
>>>>>         at 
>>>>> jdk.incubator.foreign/jdk.incubator.foreign.MemoryLayout.lambda$varHandle$2(MemoryLayout.java:488)
>>>>>         at 
>>>>> jdk.incubator.foreign/jdk.incubator.foreign.MemoryLayout.computePathOp(MemoryLayout.java:534)
>>>>>         at 
>>>>> jdk.incubator.foreign/jdk.incubator.foreign.MemoryLayout.varHandle(MemoryLayout.java:488)
>>>>>         at 
>>>>> duncan.win32.shobjidl_core.shlobj_core_h$constants$16.<clinit>(shlobj_core_h$constants$16.java:1844)
>>>>> ```
>>>>>
>>>>> This seems a duplicate of another issue that has been reported 
>>>>> last week:
>>>>>
>>>>> https://bugs.openjdk.java.net/browse/JDK-8259832
>>>>>
>>>>> E.g. jextract ignoring alignment requirements. Note that jextract 
>>>>> generates lots of constants, and it takes only one bad one to make 
>>>>> initialization fail.
>>>>>
>>>>>>
>>>>>> The jextract command on Windows 10 (with fresh Vis Studio 
>>>>>> installation) expands shlobj_core.h because I my app also uses 
>>>>>> IShellLink.
>>>>>>
>>>>>>      set "WINKIT=c:\Program Files (x86)\Windows 
>>>>>> Kits\10\Include\10.0.18362.0"
>>>>>>      jextract --source -t duncan.win32.shobjidl_core -d 
>>>>>> shlobj_core.h/java     "%WINKIT%/um/shlobj_core.h"
>>>>>>
>>>>>> Another issue is that this single windows header file gives rise 
>>>>>> to 11MB jar and is unusable inside Eclipse (on my slow PC), and 
>>>>>> is slow startup at runtime. Have you considered changing the 
>>>>>> generated code to use lazy Supplier<MethodHandle> instead of 
>>>>>> MethodHandle to ensure only required handles are created on-demand?
>>>>>
>>>>> Uhm... we are already using dynamic constants, so you should only 
>>>>> pay for the MHs and VHs you use - e.g. in fact I'm even surprised 
>>>>> you get the failure inside the <clinit> above - since that 
>>>>> constant should not be initialized unless it's used. I wonder if 
>>>>> there's a regression here... are you using the `--source` flag? 
>>>>> With that flag we revert back to a monolithic header, so that will 
>>>>> explain both failures. If so, try removing that flag - and see 
>>>>> what happens (if everything is working as it should I don't think 
>>>>> you should even see the error above, assuming you never use that 
>>>>> layout).
>>>>>
>>>>> Maurizio
>>>>>
>>>>>>
>>>>>> Kind regards
>>>>>>
>>>>>> Duncan
>>>>>>
>>>>>>
>>>>
>>