Review Request JDK-8200559: Java agents doing instrumentation need a means to define auxiliary classes

[mandy chung]

Background:

Java agents support both load time and dynamic instrumentation. At load 
time,
the agent's ClassFileTransformer is invoked to transform class bytes.  
There is
no Class objects at this time.  Dynamic instrumentation is when 
redefineClasses
or retransformClasses is used to redefine an existing loaded class.  The
ClassFileTransformer is invoked with class bytes where the Class object 
is present.

Java agent doing instrumentation needs a means to define auxiliary classes
that are visible and accessible to the instrumented class. Existing agents
have been using sun.misc.Unsafe::defineClass to define aux classes directly
or accessing protected ClassLoader::defineClass method with setAccessible to
suppress the language access check (see [1] where this issue was brought 
up).

Instrumentation::appendToBootstrapClassLoaderSearch and 
appendToSystemClassLoaderSearch
APIs are existing means to supply additional classes.  It's too limited
for example it can't inject a class in the same runtime package as the class
being transformed.

Proposal:

This proposes to add a new ClassFileTransformer.transform method taking 
additional ClassDefiner parameter.  A transformer can define additional
classes during the transformation process, i.e.
when ClassFileTransformer::transform is invoked. Some details:

1. ClassDefiner::defineClass defines a class in the same runtime package
    as the class being transformed.
2. The class is defined in the same thread as the transformers are being
    invoked.   ClassDefiner::defineClass returns Class object directly
    before the transformed class is defined.
3. No transformation is applied to classes defined by 
ClassDefiner::defineClass.

The first prototype we did is to collect the auxiliary classes and define
them  until all transformers are invoked and have these aux classes to go
through the transformation pipeline.  Several complicated issues would
need to be resolved for example timing whether the auxiliary classes should
be defined before the transformed class (otherwise a potential race where
some other thread references the transformed class and cause the code to
execute that in turn reference the auxiliary classes.  The current
implementation has a native reentrancy check that ensure one class is being
transformed to avoid potential circularity issues.  This may need JVM TI
support to be reliable.

This proposal would allow java agents to migrate from internal API and 
ClassDefiner to be enhanced in the future.

Webrev:
http://cr.openjdk.java.net/~mchung/jdk11/webrevs/8200559/webrev.00/

Mandy
[1] http://mail.openjdk.java.net/pipermail/jdk-dev/2018-January/000405.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20180415/d6de4d7e/attachment.html>