RFC: Java/Systemtap examples

[Mark Wielaard]

On Fri, 2009-12-18 at 10:20 +0000, Andrew Haley wrote:
> jon.vanalten at redhat.com wrote:
> 
> > I've added a small tapset which builds on the hotspot.stp and
> > hotspot_jni.stp to help keep track of java/native threads, and a few
> > example systemtap scripts that demo these tapsets.  I'd like them to
> > go out in the next release, if people think they are potentially
> > useful (even if only as demonstration for how to make use of the
> > other tapsets).  Diff attached.  Comments?
> 
> What does it do?  How do we use it?  I'd love to use SystemTap but I
> fear I'd have to spend a week reading docs before I could do anything.

Inside the big patch there was a README. I have added it below. Could
you go over that and tell us if that is enough of an (practical)
introduction to using Systemtap with java? We might think it is all
obvious, but I am sure that for someone that has never used it there are
some things we forget to mention.

It might be good to have a minimal concrete example in the README, so
someone can just step in directly. The current README only gives
abstract examples. Maybe something like:

  [Make sure systemtap is installed and setup for your user account.]

  To show when threads are started and stopped go into the
  j2sdk-image directory and do:
  $ stap -I tapset -e \
    'probe hotspot.thread_* { printf("%s: %s %s\n",
                              ctime(gettimeofday_s()),
                              name, thread_name) }' \
     -c 'bin/java -classpath ~/src/tests Hello'

  This uses the tapsets in the directory given by -I tapset,
  it will execute (-e) the given stap script, where the probes
  will trigger for the given program (-c). The script sets up
  a handler on all hotspot.thread_* probes (start and stop). The
  handler (between { and }) prints the time (gettimeofday_s) in
  human readable form (ctime), the name of the probe and the name
  of the thread:

  Fri Dec 18 13:03:39 2009: thread_start Reference Handler
  Fri Dec 18 13:03:39 2009: thread_start Finalizer
  Fri Dec 18 13:03:39 2009: thread_start Signal Dispatcher
  Fri Dec 18 13:03:39 2009: thread_start CompilerThread0
  Fri Dec 18 13:03:39 2009: thread_start CompilerThread1
  Fri Dec 18 13:03:39 2009: thread_start Low Memory Detector
  Fri Dec 18 13:03:40 2009: thread_start Thread-0
  Fri Dec 18 13:03:40 2009: thread_stop Thread-0
  [...]

  If you already have icedtea installed systemwide you don't need to
  add the -I and can just invoke java (the classpath and the Hello
  class are just examples that you can replace with any java program
  you want to inspect).

Would that be a simple/easy introduction to the rest of the scripts and
using systemtap/java? Should we instead of a handwritten example use one
of the actual example scripts Jon has added?

I'll review Jon's actual tapset examples in a bit.

> Maybe mjw should present at FOSDEM...  :-)

I definitely will.

Cheers,

Mark

+diff -ruN openjdkold/jdk/src/share/sample/systemtap/README openjdk/jdk/src/share/sample/systemtap/README
+--- openjdkold/jdk/src/share/sample/systemtap/README   1969-12-31 19:00:00.000000000 -0500
++++ openjdk/jdk/src/share/sample/systemtap/README      2009-12-14 17:35:35.000000000 -0500
+@@ -0,0 +1,85 @@
++*******************************************************************************
++Copyright (C) 2009, Red Hat Inc.
++
++This file is part of IcedTea.
++
++IcedTea is free software; you can redistribute it and/or modify
++it under the terms of the GNU General Public License as published by
++the Free Software Foundation; either version 2, or (at your option)
++any later version
++
++IcedTea is distributed in the hope that it will be useful, but
++WITHOUT ANY WARRANTY; without even the implied warranty of
++MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
++General Public License for more details.
++
++You should have received a copy of the GNU General Public License
++along with IcedTea; see the file COPYING. If not, write to the
++Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
++02110-1301 USA.
++*******************************************************************************
++
++These are some examples of systemtap scripts that demonstrate the use of some
++of the probes available in the Hotspot JVM and JNI code.  For more information
++about systemtap, see:
++
++   http://sourceware.org/systemtap/index.html
++
++This README assumes that the reader is familiar with, or will use the above
++URL to become familiar with, how to run systemtap scripts in the general case.
++The probes available are the same as those provided for dtrace.  These are
++documented at:
++
++   http://java.sun.com/javase/6/docs/technotes/guides/vm/dtrace.html
++
++In order to use these scripts, systemtap must be able to see the tapsets
++hotspot.stp, hotspot_jni.stp, and jthread.stp.  Your installation of java may
++have installed these files in the standard systemtap tapsets directory.  If
++not, you can specify the argument: 
++
++   -I <PATH_TO_JDK_INSTALLATION>/tapset
++
++to the stap executable.  The tapsets noted should be found in this directory.
++
++Each of these scripts begins with some documentation.  You should take a look
++at this before running.  In the general case, you will probably be doing this:
++
++   $ stap [-I path] <script.stp> -c 'java <-XX:JVMARGUMENT> <Classname>'
++
++Each script requires different -XX arguments; these details are noted in the
++comments at the top of each script file.  You can also run these scripts
++attached to java programs that are launched without calling the "java"
++executable explicitly, provided that the program provides a way to pass the
++necessary argument(s) to the JVM.  For example, attaching to eclipse:
++
++   $ stap <script.stp> -c 'eclipse -vmargs <-XX:JVMARGUMENT>'
++
++Systemtap also supports attaching to an existing process.  These examples are
++intended to provide contrast between java threads (which in most cases are
++implemented as OS threads), so this might not be as interesting as attaching
++to an entire java program right as it starts.  The jmethod.stp script in
++particular simply will not work if it is not started along with the program.
++Your mileage may vary with the other examples, but since one might conceivably
++write a script that is more useful for such a case, you would attach by:
++
++   $ stap <script.stp> -x <PID>
++
++If things are not working properly, you may wish to add verbosity with the 
++-v[vv...] stap argument and examine the output as systemtap makes its passes
++over the script, or use the -t argument to see some statistics on the firing
++(and skipping) of probes once the stap run is complete.  These scripts have
++been tuned to (mostly) stay within the default safety and security limits set
++by systemtap, but have not been used on a wide variety of java applications
++so your mileage may vary.  Script values that can be adjusted to consume less
++resources are noted in comments.
++
++While hotspot.stp and hotspot_jni.stp expose probes in the JVM and JNI code,
++jthread.stp actually provides wrapper aliases for a selection of probes within
++the other tapsets.  These, along with a handful of global variables and a
++couple of helper functions, provide some logic for keeping information
++straight between java threads and native threads.  This may be useful for
++scripts that use probes from both the hotspot and the jni tapsets, because the
++hotspot probes all provide the java thread ID while the jni probes are only
++able to provide the native thread ID.  The reader is encouraged to read the
++comments in jthread.stp for more details, and examine the source of these
++examples to understand the use of this tapset.