Proposal for JPMS issue #ReflectiveAccessByInstrumentationAgents

[Andrew Dinn]

Summary

Regarding the proposed solution of adding method redefineModule to class
Instrumentation:

  I have a Proof Of Concept implementation which shows that that this
solution works for my agent and does not impose unworkable restrictions
on its operation or use.

  The implementation can be generalised so that it will very likely work
just as well for other agents -- indeed, the current implementation has
already been implemented for the most part to address this general use case.

  The current implementation involves some clumsiness interfacing the
agent to the additional code needed for the solution when (as will
likely be the case for most existing agents) the agent code needs to be
compiled to target pre-Jigsaw JDK runtimes.

  I am still working on the current implementation to make it easier to
use, to fully document how it can be used and how it works and to
package it so it can employed by other agents.

More details of of what has been achieved (and how) plus the location of
the implementation and some of the changes I need to make are included
below in case they are of wider interest.

regards,


Andrew Dinn
-----------

What Does This POC Prove?

I have successfully modified my agent to achieve reflective access to
unexported packages using the redefineModule method of class
Instrumentation. This modification satisfies all the desired design goals:

1) Decoupling from non-Jigsaw operation:

  the agent jar retains all the functionality required to operate on
non-Jigsaw JDKs (it is compiled target 6)

  all functionality required to operate on a Jigsaw-enabled JDK is
located in a single supplementary jar (it is compiled target 9 and is
/not/ a module jar)

2) Simple deployment:

  when deploying the agent on the command line full reflective access
under Jigsaw can be achieved either by adding the supplementary jar to
the system classpath or by requesting the agent to add it to the
classpath at install time (using method appendToSystemClassLoaderSearch
of class Instrumentation)

  when deploying the agent dynamically the supplementary jar can be
installed as needed by requesting the agent to add it to the classpath
at install time (using method appendToSystemClassLoaderSearch of class
Instrumentation)

  the convenience scripts used to manage both JVM-startup deployment
(bmjava) and dynamic deployment (bminstall) of the agent have been
supplemented with a simple command option to control installation of the
supplementary jar

  the annotations used to configure auto-loading of the agent as part of
JUnit/TestNG testing can be (but have not yet been) extended with a
simple configuration field to control installation of the supplementary jar

3) Fallback operation:

  if the supplementary jar is not deployed the agent defaults to being
able to access exported non-public classes + members -- it will throw an
exception when an attempt is made to access unexported non-public
classes + members

4) No prejudice to existing module visibility

  the supplementary jar uses the Layer API to create a custom module
used as target for all exports installed by calls to redefineModules,
ensuring that module accessibility for all other (JDK or application)
modules is unaffected by the presence of the agent

  n.b. the custom module is /not/ populated from a disk-based module jar
-- instead the agent must provide the module's classloader with byte[]
class definitions on demand (the expectation is that the agent will
synthesise the required byte[] definitions using ASM but it can arrange
to load definitions from disk if it wishes), the class load process
being driven by the agent by calling the classloader's loadClass method.

5) Controlled access to enabled Members

  the sole class installed into the custom module by /my/ agent provides
the agent (and only the agent) with a method which will call setEnabled
for a specific Member on demand, if necessary first establishing
whatever export relation is required from the module of the Member's
owner class


Where Is The Implementation?

The current implementation is a full implementation of the functionality
required by my agent. It is almost in a state where other agents could
pick up and use the supplementary jar to provide themselves with a
similar capability. The code still needs a small amount of cleaning up
and quite a bit of javadoc needs to be added. There are also a few
details that I am unsure about the correctness of (although they don't
seem to inhibit it from working), most specifically the use of URLs in
the ModuleFinder and ModuleReader.

The code is located in my personal Byteman github repo in its own branch

  https://github.com/adinn/byteman/tree/jigsaw-enablement

This branch builds the normal agent if you built with JDK{6-8}. It
builds the Jigsaw capable agent if you build with Jigsaw release
9-ea+126. I have no idea what happens if you use the current trunk jdk9
to build it (it may be missing critical Jigsaw features so, basically,
don't :-).

The branch contains two extra maven modules not present in the master
branch:

  jigsawlayer provides a means to hoist a custom module into the runtime
without needing to add a module jar to the module path (in fact without
needing a jar at all)

  jigsaw contains source for a class injected into the module by the
agent -- the source is just expository since the agent actually
synthesises an equivalent class, avoiding various problems which were
encountered in trying to make the module classes available from a module
jar on disk

  maven module agent includes some changes to the agent's Transformer
class to configure Jigsaw operation plus a few minor changes to the
agent code to ensure that Member accesses obtains a correctly enabled
handle.


What Code Was Added Or Changed?

module jigsawlayer:

Maven submodule jigsawlayer contains all the code which goes into the
supplementary jar byteman-jigsawlayer.jar. Essentially, it provides the
code used to create the custom module and leaves it up to the agent to
synthesise classes which go into the module.

The API is a single method

  JigsawLayer.installModule(String, String, Map<String, byte[]>)

The String arguments are the name of the custom module and the name of a
single package for the module to export. Ideally, the API should accept
a list of exported package names and a list of modules to import (it
currently hard wires import of java.instrument which my setEnabled
caller needs)

The Map provides a way for the agent to configure the module classloader
so it can retrieve a byte[] definition for any given (String) classname.
This avoids the need for the igsawlayer code to be involved in loading
or synthesising bytecode. The ModuleReader used by the classloader calls
the Map's get method when asked to load a class. In my case I provide a
pre-populated HashMap but you can provide a Map implementation whose get
method generates the required class definitons on demand.

Ideally, the Map argument would have type Function<String, byte[]> but
that doesn't work when the agent is compiled target JDK6. So, I settled
for Map. Note also that it is not possible for the agent to define an
equivalent to class Function for use in the module API to use because
the custom module cannot see classes defined by the agent.

[I am considering making the argument an Object and having the module
API check for a Function -- in which case it casts it and uses it -- or
a class which implements apply -- in which case it wraps it with a
standard wrapper which implements Function and calls the apply method
reflectively.]

module agent:

The agent checks during agent startup whether the runtime is
Jigsaw-enabled (it uses a call to ClassLoader.loadClass() to detect
whether platform class Module exists). If so it calls method
Transformer.initForJigsaw(). This is where the jigsawlayer installModule
API method gets called.

Note that the installModule call is done reflectively because you cannot
build the code as separate maven modules if you use direct linkage --
the agent code has to be compiled target JDK6 and the jigsawlayer API
method has to be compiled target JDK9.

The agent only needs to define one class, JigsawAccessEnabler, in the
custom module so it passes a HashMap into the installModule call
populated with a byte[] corresponding to the class definition. The code
synthesised for this class (by method getJigsawAccessEnablerBytes() of
agent class Transformer) is based on the source provided in (dummy)
module jigsaw. The generating method (getJigsawAccessEnablerBytes) is
annotated with the originating source lines.

Having created the Map and installed the module the agent calls
classloader.loadClass() to drive installation of class
JigsawAccessEnabler into the custom module. It uses reflection to create
an instance of this class. This instance is passed to each Byteman Rule
where it can be used to enable access to Members. The
JigsawAccessEnabler instance is given access to the agent's
Instrumentation instance at create time, allowing it to export packages
to it's own module as needed (n.b JigsawAccessEnabler throws an
exception if it finds it is not located in a named module).

jigsaw module:

If Jigsaw is not available or the supplementary jar has not been
configured the agent uses an instance of the default class
DefaultAccessEnabler to enable access. Ideally DefaultAccessEnabler and
JigsawAccessEnabler should both implement the common interface
AccessEnabler used by class Rule. However, this is not possible because
of the need for target JDK6/JDK9 deployment.

An interface deployed in the agent jar cannot be seen (and hence used)
from a class defined in the custom module. So, that explains why
JigsawAccessEnabler does not implement AccessEnabler. The alternative of
placing the interface in the supplementary jar or in the custom module
is not an option either. They are both compiled target JDK9 so are of no
use when the agent is deployed on JDK{6-8}.

So, the agent has to wrap the JigsawAccessEnabler instance with an
instance of a proxy class (JigsawAccessEnablerWrapper) which does
implement AccessEnabler. The proxy operates by forwarding method
invocations to the wrapped JigsawAccessEnabler using reflection.