Provide API points for implementing linkers with non-standard calling conventions

[Maurizio Cimadamore]

Hi,
as you noticed, while the Linker javadoc alludes at the fact that there 
might be other calling conventions supported in the future, at the 
moment there's no API to expose this. What we had in mind the last time 
we discussed this was not too dissimilar to what you propose here - 
basically just keep calling convention open, by using strings, and then 
allow the "nativeLinker" factory to accept a calling convention string.

Another possibility would be to use linker options - e.g. have a 
CallingConvention linker option that can be passed to 
downcallHandle/upcallStub. This would allow to keep a single linker, but 
to support downcalls with different calling conventions. Both approaches 
are equally expressive, at least in terms of allowing to call functions 
using different argument shuffling. That said, on some platforms, like 
PowerPC support for instance different kind of endianness. So perhaps it 
would be good to have a way to ask for the "big endian" Linker, whose 
canonical layouts will be... big endian. That is, a Linker is about 
functions as much as it is about the definition of fundamental data 
types. So, perhaps when adding support for different Linker "flavors" it 
would be good to keep this in mind.

The reason we left this out in 22 was that we wanted to learn more use 
cases where this was useful. For instance, while it's true that x86 
supported several calling conventions, modern systems seems to have 
evolved a bit, so that each major platform tend to gravitate towards one 
main set of calling convention, typically specified in that platform's 
ABI (e.g. SysV for Linux). It seems to me that even in your case, the 
main driver for selecting an alternate calling convention is x86 really. 
So I'm still not 100% sure that this is something worth pursuing. I 
would feel more at ease if we had more cases where this was useful.

Cheers
Maurizio


On 10/10/2024 20:14, Владимир Козелков wrote:
> Greetings,
>
> The documentation for the Linker.nativeLinker() method says: "It is 
> not currently possible to obtain a linker for a different combination 
> of OS and processor."
>
> This is indeed true for hotspot, but what if another implementation 
> could provide the ability to create a linker for a different calling 
> convention? Even if the implementation wanted to do this, it would 
> fail because the API does not provide any points through which this 
> could be done.
>
> As an example - android allows us to use binaries for arm in aarch64 
> and for x86 in x86_64 with JNI. In the current implementation, I have 
> to filter the output of SymbolLookup.loaderLookup() so that the user 
> does not get symbols with a different calling convention, although the 
> platform really allows to use them.
>
> Additionally, I would like to note that the x86 and x86_64 platforms 
> have several "native" calling conventions, such as cdecl (which is 
> actually used now), fastcall, vectorcall, etc. Even if a hotspot does 
> not allow these calling conventions, it would be useful to have at 
> least the potential to implement them.
>
> I can suggest a not very good and naive method for solving the problem 
> - it is inspired by target-triple from LLVM:
>
> interface Linker ... {
>     static List<String> supportedConventions() {return ... ;}
>     static String defaultConvention() {return ... ;}
>     static boolean isSupportedConvention(String convention) {return ... ;}
>     static Linker linkerForConvention(String  convention) {return ... ;}
>     static Linker nativeLinker() {
>         return linkerForConvention(defaultConvention());
>     }
> }
>
> For android aarch64 defaultConvention() will return something like 
> "aarch64-android-cdecl"
>
> Thanks for reading