A pure Java example with reflective access behaving differently with invokeWithArguments() compared to core reflection's invoke() (Re: Strange observation: MethodHandle.invokeWithArguments() would not work, whereas Method.invoke() would with the very same arguments
Rony G. Flatscher
Rony.Flatscher at wu.ac.at
Sun Jun 10 12:16:54 UTC 2018
Dear Peter:
thank you very much for your thorough and extensive analysis taking the evolvement of Java into account!
You use the Java compiler and its behaviour (which adds syntax sugar to the Java language and is
known to stick strictly to the Java language specification) for explanations and as reference, which
is also what I have been doing in the past to counter-check the implemented behaviour of the bridge.
The Java compiler has a very different outset: *at compilation time* it knows everything it needs to
know for compiling. If something is missing or wrong at compilation time it is able to stop
compilation raising appropriate errors, thereby inhibiting the successful creation of Java classes
as long as the Java programmer does not correct erroneous code or supplies missing vital
information. Java being a strictly typed language allows one to cast arguments to force the compiler
to pick the method the Java programmer had in mind in the case that the compiler could choose one
from a set of a signatures.
By contrast a bridge for an interpreted, dynamically typed language (like ooRexx) needs to reflect
and infer at *runtime* in order to be able to choose the Java methods with the appropriate
signature. In this particular case all primitive datatypes are (ooRexx/C) strings in the dynamically
typed language such that at runtime each argument needs to be checked whether they contain valid
values for conversion to primitive datatypes in the signature of a Java method candidate.
In the dynamically typed language there is no means available for casting that would be needed to
help solve the presented problem (there is optionally a box()-routine in the bridge that allows the
ooRexx programmer to explicitly determine which primitive type is to be used for picking an
appropriate Java method).
The new code in the bridge (had to rewrite that part from scratch because of Java 9 and later) uses
java.lang.reflect to pick an appropriate Constructor, Method or Field object. For Java 1.6 and 1.7
(although java.lang.invoke/MethodHandle got introduced with 1.7 I found cases where inaccessible
types did not have accessible ancestors in the JRE) java.lang.reflect invocation will be carried
out, for Java 1.8 and later java.lang.invoke/MethodHandle invocations gets employed.
Comparing the speed between invocations using java.lang.reflect and java.lang.invoke,
java.lang.invoke wins, but by a quite small margin.
Upon further testunit tests the rewritten bridge using java.lang.invoke would execute all ooRexx
programs like java.lang.reflect since Java 1.1 with one single exception so far which I reported
here, where invocation via java.lang.reflect behaves differently to java.lang.invoke.
Because of this discrepancy the question would be: is this considered a bug that will be corrected?
Would there be otherwise a solution possible in the bridge that could take care of situations like
this, where the bridge has no context-information other than referring to Constructor, Method or
Field objects with the supplied arguments?
A few coarse ideas to address this:
* if the unreflect() method gets used then MH.invoke() should behave like in the java.lang.reflect
case; this means that the same assumptions should govern MH.invoke(): reason being that if a
java.lang.reflect object gets unreflected, also the established java.lang.reflect rules should
keep working in this case for the MH.invoke* in order to remain fully compatible with each other,
o if a MH gets created without unreflect() then this would not be necessary
* the bridge could restrain itself to only use java.lang.reflect for invocations on Java 8 and
higher instead of java.lang.invoke
o clearly, the current Java development efforts in this corner are concentrated on
java.lang.invoke, so it would be desirable to switch from java.lang.reflect to
java.lang.invoke which is not possible as long as this problem (different behaviour in j.l.r
vs. j.l.i) persists
* do not fix this, but document this as an incompatibility with the ramification that existing
programs in dynamically typed languages need to be rewritten (e.g. in this case creating an
Array object from the List and supply that instead)
* ... ?
---rony
P.S.: It would be great if Java implemented reflective invocations for dynamically typed languages
as this would solve the problem for all such languages once and forever in a standard manner, rather
than have every bridge implementor create his own implementation, which is challenging, effortful
and possibly error prone. In my case the assumptions that governed the original implementation for
reflective Constructor, Method and Field access for over fifteen years got broken by Java 9, because
j.l.r (notable 'setAccessible') changed in a fundamental aspect. If Java could be emplyoed instead,
the Java supplied implementation would be able to adopt changes in the inner workings accordingly
thereby insulating bridges from such details and problems.
On 08.06.2018 18:23, Peter Levart wrote:
> I think what you found is a discrepancy between handling of varargs methods using reflection vs.
> method handle's .invokeWithArguments().
>
> Reflection basically ignores the fact that some methods are varargs methods. It treats them
> exactly the same as if they had an array parameter (not a varargs parameter).
>
> For example, Arrays.asList() is a varargs method:
>
> public static <T> List<T> asList(T ... elements)
>
> But reflection treats is exactly the same as the following method:
>
> public static <T> List<T> asList(T[] elements)
>
> That's because varargs were introduced to Java in Java 5 as a kind of compilation sugar,
> implemented as an array parameter, while reflection is an older beast and wasn't updated to behave
> differently with varargs methods.
>
> MethodHandle(s) came later, in Java 7, and took into consideration varargs methods.
>
> So where's the problem? When you say in Java:
>
> Method asListMethod = Arrays.class.getMethod("asList", Object[].class);
> String[] elements = {"a", "b", "c"};
>
> System.out.println(
> asListMethod.invoke(null, elements)
> );
>
> You get: IllegalArgumentException: wrong number of arguments
>
> Because Method.invoke() is also a varargs method:
>
> public Object invoke(Object obj, Object... args)
>
> ... so java compiler (in order to be backwards source compatible with pre-varargs methods) treats
> this invocation in a way that just passes the String[] elements to the invoke method via the args
> parameter without any conversion (because String[] is a subtype of Object[]). Mathod.invoke then
> treats elements of the args[] array as individual verbatim parameters to be passed to the method
> when invoking it. As reflection treats Arrays.asList() as a normal no-varargs method which accepts
> a single Objet[] parameter and args[] contains 3 elements to be passed to a method with 3
> parameters, exception occurs.
>
>
> But when you use method handles and .invokeWithArguments():
>
> Method asListMethod = Arrays.class.getMethod("asList", Object[].class);
> MethodHandle asListMH = MethodHandles.lookup().unreflect(asListMethod);
>
> String[] elements = {"a", "b", "c"};
>
> System.out.println(
> asListMH.invokeWithArguments(elements)
> );
>
> ... the following happens: MethodHandle.invokeWithArguments() is also a varargs method:
>
> public Object invokeWithArguments(Object... arguments)
>
> ... so java compiler (in order to be backwards source compatible with pre-varargs methods) treats
> this invocation in a way that just passes the String[] elements to the invokeWithArguments method
> via the single 'arguments' parameter without any conversion (because String[] is a subtype of
> Object[]). MethodHandle.invokeWithArguments therefore takes 3 array elements and tries to invoke
> the underlying asList method with them. It observes that Arrays.asList is a varargs method, so it
> wraps the 3 "wannabe parameters" with the Object[] and passes the array to the asList method as
> single parameter. The result of above code is therefore:
>
> [a, b, c]
>
> If you want MethodHandle.invokeWithArguments() to treat "wannabe parameters" passed to it as
> verbatim parameters regardless of the variable/fixed arity of the method to be invoked, then you
> can transform the variable arity MethodHandle to a fixed arity MH:
>
> Method asListMethod = Arrays.class.getMethod("asList", Object[].class);
>
> String[] elements = {"a", "b", "c"};
>
> System.out.println(
> asListMethod.invoke(null, (Object) elements)
> );
>
> MethodHandle asListMH = MethodHandles.lookup().unreflect(asListMethod);
> asListMH = asListMH.asFixedArity();
>
> System.out.println(
> asListMH.invokeWithArguments((Object) elements)
> );
>
> Here you have it. Both of the above invocations produce equal output:
>
> [a, b, c]
> [a, b, c]
>
>
> What happens here is the following (will only describe the MH case - Method case is similar):
>
> MethodHandle.invokeWithArguments() is a varargs method:
>
> public Object invokeWithArguments(Object... arguments)
>
> ... so java compiler this time wraps the '(Object) elements' value with an Object[] because the
> value is of Object type (which is not a subtype of Object[]) - no backwards compatibility needed
> here, varargs conversion kicks in in the javac. MethodHandle.invokeWithArguments therefore takes
> an array with 1 element (the element being 'elements' array) and tries to invoke the underlying
> asList method with it. This time it observes that the method is not a varargs method, (because
> MethodHandle was transformed to fixed arity method handle) so it passes the single 'elements'
> wannabee parameter to the Arrays.asList single Object[] parameter - this invocation works because
> fixed arity asList() takes Object[] and 'elements' is a String[]...
>
> Hope this helps you understand what's going on.
>
> Regards, Peter
>
> On 06/08/18 12:07, Rony G. Flatscher wrote:
>> On 11.03.2018 20:22, Rony G. Flatscher wrote:
>>> Well, still trying to find out what the reason is, that core reflection's invoke behaves
>>> differently to MethodHandle's invokeWithArguments in one single case so far (using the method
>>> java.utli.Arrays.asList(...)).
>>>
>>> Here is a little Java program that excercises reflective access to
>>> "java.util.Arrays.asList(T... a)" using core reflection, i.e. "Method.invoke(Object obj,
>>> Object... args)" and "MethodHandle.invokeWithArguments(Object... arguments)":
>>>
>>> import java.util.*;
>>>
>>> import java.lang.reflect.*;
>>> import java.lang.invoke.*;
>>>
>>> class DemoAsListProblem
>>> {
>>> public static void main (String args[])
>>> {
>>> String arrNames[]=new String[] { "anne", "bert", "celine"};
>>> System.out.println("[1] (main) arrNames=\""+arrNames+"\", .toString()=\""+arrNames.toString()+"\"");
>>> List listNames=Arrays.asList(arrNames);
>>> System.out.println("[2] (main) after invoking Arrays.asList(arrNames), listNames=\""+listNames+"\"");
>>> System.out.println("\ninvoking testReflective() ...\n");
>>>
>>> testReflective();
>>> }
>>>
>>> public static void testReflective()
>>> {
>>> String arrNames[]=new String[] { "anne", "bert", "celine"};
>>> System.out.println("[3] (testReflective) arrNames=\""+arrNames+"\", .toString()=\""+arrNames.toString()+"\"");
>>>
>>> Method methAsList=null;
>>> List listNames=null;
>>> try {
>>> Class paramTypes[]=new Class[] { (new Object[0]).getClass() };
>>> methAsList=Arrays.class.getDeclaredMethod("asList", paramTypes);
>>> System.out.println("--- (core reflection) Method object asList: "+methAsList);
>>> System.out.println("\n--- (core reflection) now invoking Arrays.asList() via Method.invoke(...):");
>>>
>>> listNames=(List) methAsList.invoke( null, (Object[]) new Object[]{arrNames} ); // static method
>>> System.out.println("[4a] --- (CR) methAsList.invoke( null, (Object[]) new Object[]{arrNames} ) -> listNames=\""+listNames+"\"");
>>>
>>> listNames=(List) methAsList.invoke( null, (Object) new Object[]{arrNames} ); // static method
>>> System.out.println("[4b] --- (CR) methAsList.invoke( null, (Object) new Object[]{arrNames} ) -> listNames=\""+listNames+"\"");
>>>
>>> // "java.lang.IllegalArgumentException: wrong number of arguments":
>>> // listNames=(List) methAsList.invoke( null, (Object[]) arrNames ); // static method
>>> // System.out.println("[5a] --- (CR) methAsList.invoke( null, (Object[]) arrNames ) -> listNames=\""+listNames+"\"");
>>>
>>> listNames=(List) methAsList.invoke( null, (Object) arrNames ); // static method
>>> System.out.println("[5b] --- (CR) methAsList.invoke( null, (Object) arrNames ) -> listNames=\""+listNames+"\"");
>>> }
>>> catch (Throwable t)
>>> {
>>> System.err.println("oops #1: "+t);
>>> t.printStackTrace();
>>> System.exit(-1);
>>> }
>>>
>>> System.out.println("\n--- (MH) now invoking Arrays.asList() via MethodHandle.invokeWithArguments(...):");
>>> MethodHandles.Lookup lookup=MethodHandles.lookup();
>>> MethodHandle mh=null;
>>> try {
>>> mh=lookup.unreflect(methAsList);
>>> System.out.println("--- (MH) unreflected MethodHandle mh: "+mh);
>>>
>>> listNames=(List) mh.invokeWithArguments( (Object[]) new Object[]{arrNames} );
>>> System.out.println("[6a] --- (MH) mh.invokeWithArguments( (Object[]) new Object[]{arrNames} ) -> listNames=\""+listNames+"\"");
>>>
>>> listNames=(List) mh.invokeWithArguments( (Object) new Object[]{arrNames} );
>>> System.out.println("[6b] --- (MH) mh.invokeWithArguments( (Object) new Object[]{arrNames} ) -> listNames=\""+listNames+"\"");
>>>
>>> listNames=(List) mh.invokeWithArguments( (Object[]) arrNames );
>>> System.out.println("[7a] --- (MH) mh.invokeWithArguments( (Object[]) arrNames ) -> listNames=\""+listNames+"\"");
>>>
>>> listNames=(List) mh.invokeWithArguments( (Object) arrNames );
>>> System.out.println("[7b] --- (MH) mh.invokeWithArguments( (Object) arrNames ) -> listNames=\""+listNames+"\"");
>>>
>>> }
>>> catch (Throwable t)
>>> {
>>> System.err.println("oops #2: "+t);
>>> t.printStackTrace();
>>> System.exit(-2);
>>> }
>>> }
>>> }
>>>
>>> Compiling and running it under 9.0.4 yields the following output:
>>>
>>> [1] (main) arrNames="[Ljava.lang.String;@27ddd392", .toString()="[Ljava.lang.String;@27ddd392"
>>> [2] (main) after invoking Arrays.asList(arrNames), listNames="[anne, bert, celine]"
>>>
>>> invoking testReflective() ...
>>>
>>> [3] (testReflective) arrNames="[Ljava.lang.String;@2a18f23c", .toString()="[Ljava.lang.String;@2a18f23c"
>>> --- (core reflection) Method object asList: public static java.util.List java.util.Arrays.asList(java.lang.Object[])
>>>
>>> --- (core reflection) now invoking Arrays.asList() via Method.invoke(...):
>>> [4a] --- (CR) methAsList.invoke( null, (Object[]) new Object[]{arrNames} ) -> listNames="[anne, bert, celine]"
>>> [4b] --- (CR) methAsList.invoke( null, (Object) new Object[]{arrNames} ) -> listNames="[[Ljava.lang.String;@2a18f23c]"
>>> [5b] --- (CR) methAsList.invoke( null, (Object) arrNames ) -> listNames="[anne, bert, celine]"
>>>
>>> --- (MH) now invoking Arrays.asList() via MethodHandle.invokeWithArguments(...):
>>> --- (MH) unreflected MethodHandle mh: MethodHandle(Object[])List
>>> [6a] --- (MH) mh.invokeWithArguments( (Object[]) new Object[]{arrNames} ) -> listNames="[[Ljava.lang.String;@2a18f23c]"
>>> [6b] --- (MH) mh.invokeWithArguments( (Object) new Object[]{arrNames} ) -> listNames="[[Ljava.lang.Object;@13a57a3b]"
>>> [7a] --- (MH) mh.invokeWithArguments( (Object[]) arrNames ) -> listNames="[anne, bert, celine]"
>>> [7b] --- (MH) mh.invokeWithArguments( (Object) arrNames ) -> listNames="[[Ljava.lang.String;@2a18f23c]"
>>>
>>> So a String array is turned into a List using Arrays.asList(strArray). Doing it with core
>>> reflection yields different results to doing it with invokeWithArguments().
>>>
>>> I would have expected that [4a] and [6a] would behave the same.
>>>
>>> Using reflective invoke() and invokeWithArguments() has been working for my bridge for all the
>>> test units (using literally the same arguments in both cases) interchangeably, the one exception
>>> is Arrays.asList().
>>>
>>> Maybe I am not seeing the obvious (having done too much "close-up tests" for quite some time
>>> now). So any hint, insight, help would be really appreciated!
>>>
>>> ---rony
>> As a few months have gone by without any follow-ups, I have been wondering whether there is a
>> solution at all, if this is a problem rooted in the current implementation of MethodHandle
>> methods not being 100% compatible with reflective invoke (which may imply that one needs to stick
>> to use reflective invoke and forgo MethodHandle invokes for good).
>>
>> ---rony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20180610/41d176ff/attachment-0001.html>
More information about the mlvm-dev
mailing list