RFR: 8303524: Check FunctionDescriptor byte order when linking

[Jorn Vernee]

On Thu, 23 Mar 2023 16:47:08 GMT, Jorn Vernee <jvernee at openjdk.org> wrote:

> Linkers are strongly tied to a particular byte order, because they are tied to a particular platform. So, the linker should reject layouts that have another byte order. This patch implements that check.

> > > The code changes look good, but the javadoc should be updated to reflect the new constraints.
> > 
> > 
> > We currently have this:
> > ```
> > @throws IllegalArgumentException if the provided function descriptor is not supported by this linker.
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Which technically seems enough to cover this as well, as byte order is part of the layouts that are part of the function descriptor. I'm not sure if we want to get very detailed here. Different linker implementations might make different decisions, at least theoretically, but this is also the case for the fallback linker which e.g. rejects unions and accepts layouts with any byte order.
> 
> I'm noting an asymmetry between this and #13164. I think the same arguments apply for/against both.

Ok. I think there is room for some text in the class javadoc, since that is talking specifically about the native linker. But I don't think we should add more text on the `downcallHandle`/`upcallStub` methods.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13161#issuecomment-1485405979