Number of Parallel GC Threads

[Nicolas Michael]

Hi,

I didn't expect to kick off that many emails! :-) So first of all: 
Thanks a lot! I'll try to reply to some of them in this "compound" mail.

I was referring to Java 1.6.0_07 (and some Java 5 builds): With these 
releases, we really *do* have 256 gc threads on our T5440! I didn't know 
that you had changed that in 1.6.0_10. Having "just" 80 gc threads with 
the 5/8 rule sounds already much better, but would still be overkill for 
an Xmx128m process (see below).

Tony wrote:
 > If someone is running several JVMs per box, then I assume that they
 > know what they are doing, so playing around with the # of parallel GC
 > threads should be straightforward to them!

Hmmm... I'll try to explain in a little more detail: You know our 
configuration, and that's exactly what we do: Without going into 
details, we configure the number of parallel gc threads over *all* our 
JVM's that are part of our workload such that they don't exceed the 
number of cpus. Works very well!

But that's just part of the story. Nowadays, you have lots of programs 
that are developed in Java. And you run such programs in the background 
on the same servers you run your workload on. Those programs are not 
necessarily part of your workload, but perform monitoring/administrative 
tasks in the background. They are often not developed by your own 
department, but either in other parts of your company or are coming from 
OEM partners. So you often don't have the possibility to change JVM 
settings for them. (I'm writing this in a very general fashion since 
this is a public list...)

As an example, take the management and supervision tools of Sun's 
StorageTek arrays (CAM software). There's this "Sun StorageTek Fault 
Management Services" process, running Java 1.5.0_11-b03 (of course, it's 
coming with its own JVM...).

$ jinfo 763
java.vm.version = 1.5.0_11-b03
java.vm.name = Java HotSpot(TM) Server VM
VM Flags:
-Xms8m -Xmx128m ...
...

Currently, this FM service agent has been doing 792 Young GC's within 3 
days (that's about once every 6 minutes) -- with 256 parallel gc threads:

$ pstack 763 | grep __1cMGCTaskThreadDrun6M_v_  | wc -l
      256

Clemens wrote:
 > It should be a rather rare case that two JVMs running run a gc-cycle
 > at the same time for which performance will be a bit worse

If a server runs during peak load at high cpu utilizations (there are 
applications that can do this even with 256 cpus in the server...), a 
256-threaded gc cycle of a monitoring agent could be quite disruptive 
for such a workload (especially when it is sensitive to response times).

I'm not saying that we *do* have a critical problem here (in my 
particular case). I was writing my first mail just to point out that in 
my opinion there are situations where JVM's are maybe using too many gc 
threads on large systems. Of course, I totally agree that it's difficult 
(if not even impossible) for the JVM to come up with best default 
settings for *any* imaginable situation. Usually the user will need to 
adjust some of the settings. In some case as I'm describing above, this 
is unfortunately not always possible: Afaik, there is no external 
interface to change the JVM settings for this CAM software agent. (Ok, 
we could consider this the fault of the agent, which could in a startup 
script detect how many cpus there are in the system, and limit its 
number of gc threads to let's say 4 (which should be sufficient for such 
a process with 128m heap). Or use the Client VM instead...) 
Unfortunately, I believe there are many programs around which fail to do 
such things...

Therefore I thought it might help if the JVM would limit the number of 
gc threads or large systems if it is likely that the process would not 
benefit from more gc threads. It's sure difficult to tell, but I've seen 
lots of ideas in these mails. 128m heap certainly don't need 256 gc 
threads (or 80 with the 5/8 rule). Btw, this CAM agent has 274 threads 
in total. Subtracting 256 gc threads leaves 18 threads. I don't know how 
many of them really do anything, but an application with < 18 mutator 
threads, a heap of 128m, minor gc intervals of 6 minutes certainly 
doesn't need many gc threads -- not even on a 256-way system... ;-) Of 
course, number of mutator threads and gc intervals are dynamic 
parameters and can't be determined by the JVM at startup.

Nick.