RFR: 8013527: calling MethodHandles.lookup on itself leads to errors

[Mandy Chung]

JDK-8013527: calling MethodHandles.lookup on itself leads to errors
JDK-8257874: MethodHandle injected invoker doesn't have necessary private access

Johannes Kuhn is also a contributor to this patch.

A caller-sensitive method can behave differently depending on the identity
of its immediate caller.  If a method handle for a caller-sensitive method is
requested, this resulting method handle behaves as if it were called from an
instruction contained in the lookup class.  The current implementation injects
a trampoline class (aka the invoker class) which is the caller class invoking
such caller-sensitive method handle.  It works in all CSMs except `MethodHandles::lookup`
because the caller-sensitive behavior depends on the module of the caller class,
the class loader of the caller class, the accessibility of the caller class, or
the protection domain of the caller class.  The invoker class is a hidden class
defined in the same runtime package with the same protection domain as the
lookup class, which is why the current implementation works for all CSMs except
`MethodHandles::lookup` which uses the caller class as the lookup class.

Two issues with current implementation:
1.  The invoker class only has the package access as the lookup class.  It cannot
    access private members of the lookup class and its nest members.

    The fix is to make the invoker class as a nestmate of the lookup class.

2.  `MethodHandles::lookup` if invoked via a method handle produces a `Lookup`
    object of an injected invoker class which is a bug.

There are two alternatives:
- define the invoker class with the lookup class as the class data such that
  `MethodHandles::lookup` will get the caller class from the class data if
  it's the injected invoker
- Johannes' proposal [1]: detect if an alternate implementation with an additional
  trailing caller class parameter is present, use the alternate implementation
  and bind the method handle with the lookup class as the caller class argument.

There has been several discussions on the improvement to support caller sensitive
methods for example the calling sequences and security implication.   I have
looked at how each CSM uses the caller class.  The second approach (i.e. 
defining an alternate implementation for a caller-sensitive method taking
an additional caller class parameter) does not work for non-static non-final
caller-sensitive method.  In addition, it is not ideal to pollute the source
code to provide an alternatve implementation for all 120+ caller-sensitive methods
whereas the injected invoker works for all except `MethodHandles::lookup`.

I propose to use both approaches.   We can add an alternative implementation for
a caller-sensitive method when desirable such as `MethodHandles::lookup` in
this PR.  For the injected invoker case, one could extract the original lookup
class from class data if needed.

test/jdk/jdk/internal/reflect/CallerSensitive/CheckCSM.java ensures that
no new non-static non-final caller-sensitive method will be added to the JDK.
I extend this test to catch that non-static non-final caller-sensitive method
cannot have the alternate implementation taking the additional caller class
parameter.

This fix for JDK-8013527 is needed by the prototype for JDK-6824466 I'm working on.

[1] https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-January/073184.html

-------------

Commit messages:
 - clean up whitespace
 - revise the helper method
 - clean up
 - JDK-8013527: calling MethodHandles.lookup on itself leads to errors

Changes: https://git.openjdk.java.net/jdk/pull/2367/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=2367&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8013527
  Stats: 662 lines in 10 files changed: 620 ins; 9 del; 33 mod
  Patch: https://git.openjdk.java.net/jdk/pull/2367.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/2367/head:pull/2367

PR: https://git.openjdk.java.net/jdk/pull/2367