RFR(S) 8230956: Should disable Escape Analysis when JVMTI capability can_tag_objects is taken

[Reingruber, Richard]

Hello Vladimir,

thanks for looking at this.

  > can_tag_objects is "always" capability.

That's correct.

  > If it is true then EA will be disabled in all cases when JVMTI agent is used. It is too broad.
  >
  > Am I missing something?

No that's correct too. If you include jvmti as hotspot feature in your build then you should disable
escape analysis by default.

That's really too hard. My proposed improvement has the effect that escape analysis is disabled if
any agent calls AddCapabilities() during the OnLoad phase. Because calling AddCapabilities(), even
with an empty set of capabilities, has the effect that JvmtiExport::can_walk_any_space() will return
always true after that. Not sure if this is a bug or feature though. But only disabling EA by
default or JDK-8227745 would be a real fix.

  > It is also not clear to me that it is bug. Based on all description this functionality is used to 
  > catch leaks in Java heap. But scalar replaced objects do not exists. JVMTI should not even see them.

I do think, JVMTI heap functions must report scalar replaced objects for formal and for practical
reasons.

JVM spec defines for the new bytecode [1]

  Operation

    Create new object

  Description

    [...] Memory for a new instance of that class is allocated from the garbage-collected heap, [...]

So I think JVMTI heap functions have to report that new instance on the virtual heap.

If they don't do it you can get contradicting data, too, from agents that employ bytecode
instrumentation to count instances.

Also there can be a leak rooted at a chain of scalar replaced objects. This is currently not
reported either. If an agent traverses references it won't arrive at the leaked objects. If it
itereates objects on the heap then the leaking objects are found, but it will remain unknown what's
keeping them alive.

Thanks, Richard.

[1] https://docs.oracle.com/javase/specs/jvms/se13/html/jvms-6.html#jvms-6.5.new

-----Original Message-----
From: Vladimir Kozlov <vladimir.kozlov at oracle.com> 
Sent: Mittwoch, 25. September 2019 03:04
To: Reingruber, Richard <richard.reingruber at sap.com>; hotspot-compiler-dev at openjdk.java.net; OpenJDK Serviceability <serviceability-dev at openjdk.java.net>
Subject: Re: RFR(S) 8230956: Should disable Escape Analysis when JVMTI capability can_tag_objects is taken

It is also not clear to me that it is bug. Based on all description this functionality is used to 
catch leaks in Java heap. But scalar replaced objects do not exists. JVMTI should not even see them.

Thanks,
Vladimir

On 9/24/19 3:37 PM, Vladimir Kozlov wrote:
> can_tag_objects is "always" capability.
> 
> If it is true then EA will be disabled in all cases when JVMTI agent is used. It is too broad.
> 
> Am I missing something?
> 
> Thanks,
> Vladimir
> 
> On 9/13/19 7:12 AM, Reingruber, Richard wrote:
>> Hi,
>>
>> could I please get reviews for
>>
>> Webrev: http://cr.openjdk.java.net/~rrich/webrevs/2019/8230956/webrev.0/
>> Bug:    https://bugs.openjdk.java.net/browse/JDK-8230956
>>
>> JVMTI provides functions to follow references beginning at the roots of the object graph and it
>> provides functions to iterate all objects on the heap[1][2]. These functions are means to access
>> objects which are otherwise local to a Java thread. In terms of escape analysis these local objects
>> escape through these JVMTI functions invalidating optimizations based on escape analysis.
>>
>> Example:
>>
>> - Let J be a JavaThread that calls a compiled method M with a NoEscape instance I of class C that is
>>    scalar replaced.
>>
>> - JVMTI agent A uses JVMTI FollowReferences() to iterate the objects in the object graph tagging all
>>    instances of C.
>>
>> - A uses GetObjectsWithTags() to retrieve the tagged instances of C.
>>
>> - Error: I is missing because its allocation was eliminated / scalar replaced.
>>
>> Agents are required to possess the capability can_tag_objects in order to call the JVMTI heap
>> functions that let objects escape.  Currently it is not possible to revert EA based optimizations
>> just before objects escape through JVMTI therefore escape analysis should be disabled as soon as the
>> JVMTI capability can_tag_objects is taken.
>>
>> But this is not sufficient, because there may be compiled frames on stack with EA based
>> optimizations when a JVMTI agent takes can_tag_objects (see included exclusive test cases), and then
>> it does not help to disable escape analysis or invalidate compiled methods with ea based
>> optimizations. In general it is still an improvement to do so. JDK-8227745 would be a complete
>> solution to the issue.
>>
>> An further improvement could be to invalidate methods compiled by c2 when can_tag_objects gets
>> added, but I'd rather suggest to integrated the implementation for JDK-8227745. Note also that after
>> calling JVMTI AddCapabilities(), even with an empty set of capabilities,
>> JvmtiExport::can_walk_any_space() will return true.
>>
>> I've run tier1 tests.
>>
>> Thanks, Richard.
>>
>> [1] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap
>> [2] https://docs.oracle.com/en/java/javase/11/docs/specs/jvmti.html#Heap_1_0
>>