New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages

[Andrew Dinn]

Hi John,

I'm reviving an old (by 2-3 weeks) conversation here because it seems
the right place to ask about how to deal with a disparity I found
between the use of reflection vs method handles. I'm have retained the
cc to jigsaw-dev because i) the conversation started there and ii) it
relates to the question of how well Lookups and MethdoHandles can stand
in for reflection. Perhaps that's too tenuous to merit a CC but I'm
adopting the excuse that those who are only interested in more direct
Jigsaw content can simply ignore the rest of this thread.

The disparity I am seeing looks like it might just be an argument for
Maurizio's new reflection API more than anything else but it may just be
that I am doing something wrong. I'll present the problem in as reduced
form as I can manage to extract from the Byteman implementation.

Byteman includes a test to ensure that the type checker allows duck
typing when a compatible array type is passed as input to a method. So,
we have a test method

  public class TestArrayArgTypeCheck extends Test {
    . . .
    public void testArrayCall(String[] args)
      {
          log("inside testArrayCall");
      }
  }

where log is defined by the parent class Test.

I use a Byteman rule as follows

  RULE test array arg type check
  CLASS TestArrayArgTypeCheck
  METHOD testArrayCall
  AT ENTRY
  BIND test : Test = $this;
  IF TRUE
  DO test.log("args : " + java.util.Arrays.asList($args) );
  ENDRULE

This rule is 'injected' at the start of the code for method
testArrayCall -- it's not actually executed as inline bytecode, rather a
callout to the rule interpreter executes the rule (out of line bytecode
execution is also an option but I don't [yet?] need to add that
complexity to the mix). In case you are unfamiliar with how Byteman
operates I'll summarise its operation.

  $this and $args name bindings for values passed into the interpreter,
respectively:
    the instance of TestArrayArgTypeCheck fielding the call to the
method (which Byteman knows is of type TestArrayArgTypeCheck)
    the argument to testArrayCall (which Byteman knows is of type String[])

  the BIND clause binds a /rule-local/ variable test (of type Test) to
the instance of TestArrayArgTypeCheck fielding the call to the method

 the DO clause
   passes $this to Arrays.asList()
   pastes the result into a String
   passes the String to Test.log

So, the test ensures that Byteman's type checker accepts that the
String[] argument passed to Arrays.asList(Object[]) is a legitimate
argument without throwing a type exception. It doesn't really matter
what the log output is but the test does check for an expected output
and that is where the disparity arises.

The call to the target method is made by the test code as follows

  . . .
  String[] ordinals = { "first", "second", "third" };
  log("calling testArrayCall");
  testArrayCall(ordinals);
  log("called testArrayCall");
  . . .

When this is run on JDK8[-] I see this output which matches expectation:

<log>
calling testArrayCall
args : [first, second, third]
inside testArrayCall
called testArrayCall
</log>

When I run this on JDK9 using a modified version of Byteman that relies
on MethodHandles I see this output:

<log>
calling testArrayCall
args : [[Ljava.lang.String;@36bc55de]
inside testArrayCall
called testArrayCall
</log>

So, the String[] array appears to have been wrapped in an Object[]
before being passed on to Arrays.asList().


On JDK8[-] I use reflection to execute the method call. So, essentially
the code looks like this

class MethodExpression {
  Method method;
  Expression recipient;
  List<Expression> arguments;
  . . .
  Object interpret(...)
    . . .
    Object recipientValue =
      (recipient != null
       ? recipient.interpret(...)
       : null);
    int argCount = arguments.size();
    Object[] argValues = new Object[argCount];
    for (int i = 0; i < argCount; i++) {
      argValues[i] = arguments.get(i).interpret(...);
    }
    . . .
    return method.invoke(recipientValue, argValues);

The essential differences in the code that gets executed on JDK9
(ignoring that I am inlining code here from different branches) are as
follows:

class MethodExpression {
  Method method;
  MethodHandle handle = getMethodHandle(method);
  List<Expression> arguments;
  . . .
  Object interpret(...)
    . . .
    Object recipientValue =
      (recipient != null
       ? recipient.interpret(...)
       : null);
    int argCount = arguments.size();
    Object[] argValues = new Object[argCount];
    for (int i = 0; i < argCount; i++) {
      argValues[i] = arguments.get(i).interpret(helper);
    }
    . . .
    if (recipient == null) {
      handle.invoke(argValues);
    } else {
      handle.invokeWithArguments(recipientValue, argValues);
    }

The key point is how I construct that method handle. In essence it is as
follows

  Lookup theLookup = ...;

  MethodHandle getMethodHandle(Method method) {
    try {
      MethodType methodType =
        MethodType.methodType(method.getReturnType(),
                              method.getParameterTypes());
      isStatic = Modifier.isStatic(method.getModifiers());
      if (isStatic) {
        handle = theLookup.findStatic(method.getDeclaringClass(),
                                           method.getName(),
                                           methodType);
      } else {
        MethodHandle h =
          theLookup.findVirtual(method.getDeclaringClass(),
                                method.getName(),
                                methodType);
        handle = h.asSpreader(1,
                              Object[].class,
                              method.getParameterCount());
      }
      return handle;
    } catch (...) {
      throw new RuntimeException(...)
    }
  }

[n.b. theLookup is a suitable MethodHandles.Lookup that I finagle out of
class Lookup -- it's provenance doesn't really matter just now]

The problem appears to relate to the argument coercing that is done
under the handle.invoke call. Clearly a String[] argument does not match
the Object[].class parameter type occurring in the parameters returned
by method.getParameterTypes() and passed to findStatic.

I have tried varying this by applying an explicit spreader

      if (isStatic) {
        MethodHandle h =
          theLookup.findStaticmethod.getDeclaringClass(),
                                method.getName(),
                                methodType);
        handle = h.asSpreader(0,
                              Object[].class,
                              method.getParameterCount());
      } else {
        . . .
      }

and using handle.invokeWithArguments in place of handle.invoke but that
produces the same result (indeed looking at the code I think the
original invoke call is already treated as a spread call.

I'm not really sure why this is operating the way it does by wrapping
the String[] input in an Object[]. It seems that it may perhaps be an
artefact of trying to combine inexact invocation with duck typing for
generics. Am I doing something wrong in the method handle lookup or in
the invocation? Or is this a corner case where invoke is performing the
wrong type of coercion?

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander