Contribution: Lock Contention Profiler for HotSpot

[Peter Hofer]

Hi Jeremy,

On 2016-11-04 19:24, Jeremy Manson wrote:
> Why aren't these extensions to JVMTI, which already has
> MonitorContendedEnter and MonitorContendedEntered events? You could
> just add a MonitorContendedRelease event to cover what you want.
> Then the bulk of the tracking work can be done in JVMTI.

  One of our main goals was to make profiling very lightweight so that 
there is a chance that the profiler can be used on production systems. 
In the HotSpot code, we can record events and maintain state very 
efficiently.

I agree that a profiler that uses only JVMTI and extension methods would 
be more modular. We actually tried to implement a comparable profiler 
using JVMTI. It performs very frequent state transitions to the agent 
and back, requires wrapping all references and data structures, and 
needs tagging to associate state with objects. Moreover, it cannot 
efficiently cache stack traces without always resolving inlined methods 
from the compiler's debug information (which makes a lot of difference 
in our HotSpot-internal profiler). The JVMTI-based profiler turned out 
to be rather inefficient, which is why we didn't pursue this approach 
further.

As Alex pointed out, there used to be a MonitorContendedExit event in 
early versions of JVMTI. It was eliminated because the context of a 
monitor exit is not really safe for invoking a JVMTI callback, which is 
another issue that would need to be addressed first.

Cheers,
  Peter

> At Google, we've built on these JVMTI primitives quite successfully.
> The only internal enhancements we've had to make is to make them support
> j.u.c locks.
>
> (We've also done the hotspot lock contention work, but it has been less
> directly useful.)
>
> Jeremy
>
> On Fri, Nov 4, 2016 at 4:26 AM, David Gnedt <David.Gnedt at jku.at
> <mailto:David.Gnedt at jku.at>> wrote:
>
>     Hello,
>
>     I am one of the authors of this work and I gladly support this
>     contribution.
>
>     Best regards,
>     David Gnedt
>
>     >>> Peter Hofer <peter.hofer at jku.at <mailto:peter.hofer at jku.at>>
>     04.11.16 11.01 Uhr >>>
>     Hello everyone,
>
>     we are researchers at the University of Linz and have worked on a lock
>     contention profiler that is built into HotSpot. We would like to
>     contribute this work to the OpenJDK community.
>
>     Our profiler records an event when a thread fails to acquire a contended
>
>     lock and also when a thread releases a contended lock. It further
>     efficiently records the stack traces where these events occur. We
>     devised a versatile visualization tool that analyzes the recorded events
>
>     and determines when and where threads _cause_ contention by holding a
>     contended lock. The visualization tool can show the contention by stack
>     trace, by lock, by lock class, by thread, and by any combination of
>     those aspects.
>
>     We described our profiler in more detail in a research paper at ICPE
>     2016. [1] In our evaluation, we found that the overhead is typically
>     below 10% for common multi-threaded Java benchmarks. Please find a free
>     download of the paper on our website:
>     > http://mevss.jku.at/lct/
>
>     I contribute this work on behalf of Dynatrace Austria (the sponsor of
>     this research), my colleagues David Gnedt and Andreas Schoergenhumer,
>     and myself. The necessary OCAs have already been submitted.
>
>     We provide two patches:
>
>     Patch 1. A patch for OpenJDK 8u102-b14 with the profiler that we
>     described and evaluated in our paper, plus minor improvements. It
>     records events for Java intrinsic locks (monitors) and for
>     java.util.concurrent locks (ReentrantLock and ReentrantReadWriteLock).
>     We support only Linux on 64-bit x86 hardware.
>
>     >
>     http://cr.openjdk.java.net/~tschatzl/phofer/webrev_hotspot_jdk8u102b14/
>     <http://cr.openjdk.java.net/~tschatzl/phofer/webrev_hotspot_jdk8u102b14/>
>     >
>     http://cr.openjdk.java.net/~tschatzl/phofer/webrev_jdk_jdk8u102b14/
>     <http://cr.openjdk.java.net/~tschatzl/phofer/webrev_jdk_jdk8u102b14/>
>
>     Patch 2. A patch for OpenJDK 9+140 with a profiler for VM-internal
>     native locks only. We consider this to be useful for HotSpot developers
>     to find locking bottlenecks in HotSpot itself. We tested this patch only
>
>     on Linux on 64-bit x86 hardware, but it should require few changes for
>     other platforms.
>
>     >
>     http://cr.openjdk.java.net/~tschatzl/phofer/webrev_nativelocksonly_hotspot_jdk9%2b140/
>     <http://cr.openjdk.java.net/~tschatzl/phofer/webrev_nativelocksonly_hotspot_jdk9%2b140/>
>     >
>     http://cr.openjdk.java.net/~tschatzl/phofer/webrev_nativelocksonly_jdk_jdk-9%2b140/
>     <http://cr.openjdk.java.net/~tschatzl/phofer/webrev_nativelocksonly_jdk_jdk-9%2b140/>
>
>     With both patches, the profiler is enabled with -XX:+EnableEventTracing.
>
>     By default, an uncompressed event trace is written to file "output.trc".
>
>     More detailed usage information and a download of the corresponding
>     visualization tool is available on our website,
>     http://mevss.jku.at/lct/.
>
>     Kind regards,
>       Peter Hofer
>
>
>     --
>     Peter Hofer
>     Christian Doppler Laboratory on Monitoring and Evolution of
>     Very-Large-Scale Software Systems / Institute for System Software
>     University of Linz
>
>
>     [1] Peter Hofer, David Gnedt, Andreas Schoergenhumer, Hanspeter
>     Moessenboeck. Efficient Tracing and Versatile Analysis of Lock
>     Contention in Java Applications on the Virtual Machine Level.
>     Proceedings of the 7th ACM/SPEC International Conference on Performance
>     Engineering (ICPE’16), Delft, Netherlands, 2016.
>
>
>