RFR(S) 8230956: Should disable Escape Analysis when JVMTI capability can_tag_objects is taken

[Reingruber, Richard]

Hi,

could I please get reviews for

Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230956/webrev.0/
Bug:    https://bugs.openjdk.java.net/browse/JDK-8230956

JVMTI provides functions to follow references beginning at the roots of the object graph and it
provides functions to iterate all objects on the heap[1][2]. These functions are means to access
objects which are otherwise local to a Java thread. In terms of escape analysis these local objects
escape through these JVMTI functions invalidating optimizations based on escape analysis.

Example:

- Let J be a JavaThread that calls a compiled method M with a NoEscape instance I of class C that is
  scalar replaced.

- JVMTI agent A uses JVMTI FollowReferences() to iterate the objects in the object graph tagging all
  instances of C.

- A uses GetObjectsWithTags() to retrieve the tagged instances of C.

- Error: I is missing because its allocation was eliminated / scalar replaced.

Agents are required to possess the capability can_tag_objects in order to call the JVMTI heap
functions that let objects escape.  Currently it is not possible to revert EA based optimizations
just before objects escape through JVMTI therefore escape analysis should be disabled as soon as the
JVMTI capability can_tag_objects is taken.

But this is not sufficient, because there may be compiled frames on stack with EA based
optimizations when a JVMTI agent takes can_tag_objects (see included exclusive test cases), and then
it does not help to disable escape analysis or invalidate compiled methods with ea based
optimizations. In general it is still an improvement to do so. JDK-8227745 would be a complete
solution to the issue.

An further improvement could be to invalidate methods compiled by c2 when can_tag_objects gets
added, but I'd rather suggest to integrated the implementation for JDK-8227745. Note also that after
calling JVMTI AddCapabilities(), even with an empty set of capabilities,
JvmtiExport::can_walk_any_space() will return true.

I've run tier1 tests.

Thanks, Richard.

[1] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap
[2] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap_1_0