Can a long period of prior load prevent JIT compilation of a method which would otherwise have been compiled?

[Benedict Russell]

This mailing list looks like the best bet to get an answer to my
question. However, please let me know if there's somewhere more
appropriate for me to post this question.

I'm investigating a performance problem in my application (a
tomcat-based webserver) which has the following characteristics:

- On a server which has just been restarted, uploading a given data
file takes around 5 seconds. I can upload several hundred such files
and the performance remains consistently good.
- On a server which has been heavily loaded for several days but with
a different load profile (i.e. has handled a very large number of
other requests but NOT any data uploads of this kind), uploading the
same data file takes around 40 seconds.  I've now uploaded nearly 300
such files and the performance remains consistently bad.
- On the slow servers, the CPU is maxed out while handling the uploads
and thread dumps consistently show threads spending lots of time in
runnable state in one particular method -
com.sun.mail.util.BASE64EncoderStream.write.
- jstat output confirms that very little time is being spent in GC in
either case.

One possible explanation would be that the
com.sun.mail.util.BASE64EncoderStream.write method is not being
compiled on the servers which have been running for several days,
which might happen if the JIT compiler's view of which methods should
be compiled has become skewed after a few days of different load. To
test whether this could explain the difference, I ran two tests:
- Restarted a server with "-XX:+PrintCompilation" and uploaded some
data files. This showed the server compiling the
com.sun.mail.util.BASE64EncoderStream.write method very early on (as
expected), and showing good upload performance (~5s per file).
- Restarted the same server with
"-XX:CompileCommand=exclude,com/sun/mail/util/BASE64EncoderStream,write"
to exclude this method from compilation. In this case, the performance
was very similar to that on the servers which have handled several
days of other load (~40s per file).

So the circumstantial evidence is consistent with the theory, but that
obviously doesn't prove anything.

A few questions then:

1) Is it possible that the JIT compiler is selecting methods for
compilation very differently on the two servers (one freshly restarted
and one after several days of load), or is this fundamentally not how
it works?
2) Is there any way of checking which methods have been compiled on a
running server? Could I easily prove or disprove the theory without
having to restart all the servers and reproduce this scenario with
extra debugging options?
3) If the theory I've suggested isn't plausible, does anyone have any
other ideas what might cause this behaviour?

If it makes a difference, the JVM I'm using is

java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
Java HotSpot(TM) Server VM (build 10.0-b23, mixed mode)

Thanks in advance!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20120312/6ef865a5/attachment.html