Breakpoint Performance (JVMTI)

[Nicolas Michael]

Hi all --

we are using JVMTI for tracing/debugging purposes, especially to get notified
about method entries/exits. Since the MethodEntry/MethodExit probes are
unconditional and fire for *any* method entries/exits (which disqualifies this
technology for performance critical workloads), we're using breakpoints instead,
which we just set for the methods we're interested in. We've noticed that this
is becoming really expensive when we have lots of breakpoints in one class. 

For example, setting breakpoints on method entries and exits for all methods in
one class gives us about 1,000 breakpoints for that particular class. For our
benchmark workload, 400 of them fire per request. Considering what our
application is doing per request, this should be more or less nothing. But our
process is becoming a lot more expensive. Sun Studio profiling shows that we
spend almost 30% of the CPU time in handling breakpoints and related effort
(e.g. running interpreted instead of JIT-compiled code for this class's
methods). About 1/4th of this overhead is spent in our own agent library (which
we are currently optimizing), but *half* of it is spent in orig_bytecode_at(int)
getting the original bytecode where the breakpoint was:

Excl. User CPU    Incl. User CPU     Name  
                                           
     sec.      %       sec.      %    
31744.035 100.00  31744.035 100.00   <Total>
 4969.196  15.65   4969.196  15.65   methodOopDesc::orig_bytecode_at(int)
  890.083   2.80  16522.197  52.05   Interpreter
  366.857   1.16   2743.579   8.64  
JvmtiExport::post_raw_breakpoint(JavaThread*,methodOopDesc*,unsigned char*)

It seems, orig_bytecode_at(int) is iterating linearly over a list of all
breakpoints of this class (in our case: about 1,000) to find the matching
breakpoint:

Bytecodes::Code methodOopDesc::orig_bytecode_at(int bci) {
  BreakpointInfo* bp = instanceKlass::cast(method_holder())->breakpoints();
  for (; bp != NULL; bp = bp->next()) {
    if (bp->match(this, bci)) {
      return bp->orig_bytecode();
    }
  }
  ShouldNotReachHere();
  return Bytecodes::_shouldnotreachhere;
}

I guess the breakpoint mechanism wasn't really intended for what we are doing?
;-) But do you think this might be optimized?

Thanks,
Nick.