Number of Parallel GC Threads

Tony Printezis Antonios.Printezis at sun.com
Sat Jan 24 22:13:09 UTC 2009


Nick,

Hi. You do bring up, as always!!!, a good point. Maybe we can come up 
with some better defaults, but my guess would be that it might deal a 
bit better with a small percentage of cases, but it will not be the 
magic wand that will automagically solve all related issues. We have 
considered doing something like trying to monitor how many JVMs are 
running on a particular machine and increase / decrease the resources 
dedicated to each as needed. But that's not trivial and I don't think 
it's high on our list.

Regarding the CAM agent, you said correctly that they should somehow 
tune its parameters a bit better. And you are right; I would have 
reported it as a bug.

Tony

Nicolas Michael wrote:
> Hi,
>
> I didn't expect to kick off that many emails! :-) So first of all: 
> Thanks a lot! I'll try to reply to some of them in this "compound" mail.
>
> I was referring to Java 1.6.0_07 (and some Java 5 builds): With these 
> releases, we really *do* have 256 gc threads on our T5440! I didn't 
> know that you had changed that in 1.6.0_10. Having "just" 80 gc 
> threads with the 5/8 rule sounds already much better, but would still 
> be overkill for an Xmx128m process (see below).
>
> Tony wrote:
> > If someone is running several JVMs per box, then I assume that they
> > know what they are doing, so playing around with the # of parallel GC
> > threads should be straightforward to them!
>
> Hmmm... I'll try to explain in a little more detail: You know our 
> configuration, and that's exactly what we do: Without going into 
> details, we configure the number of parallel gc threads over *all* our 
> JVM's that are part of our workload such that they don't exceed the 
> number of cpus. Works very well!
>
> But that's just part of the story. Nowadays, you have lots of programs 
> that are developed in Java. And you run such programs in the 
> background on the same servers you run your workload on. Those 
> programs are not necessarily part of your workload, but perform 
> monitoring/administrative tasks in the background. They are often not 
> developed by your own department, but either in other parts of your 
> company or are coming from OEM partners. So you often don't have the 
> possibility to change JVM settings for them. (I'm writing this in a 
> very general fashion since this is a public list...)
>
> As an example, take the management and supervision tools of Sun's 
> StorageTek arrays (CAM software). There's this "Sun StorageTek Fault 
> Management Services" process, running Java 1.5.0_11-b03 (of course, 
> it's coming with its own JVM...).
>
> $ jinfo 763
> java.vm.version = 1.5.0_11-b03
> java.vm.name = Java HotSpot(TM) Server VM
> VM Flags:
> -Xms8m -Xmx128m ...
> ...
>
> Currently, this FM service agent has been doing 792 Young GC's within 
> 3 days (that's about once every 6 minutes) -- with 256 parallel gc 
> threads:
>
> $ pstack 763 | grep __1cMGCTaskThreadDrun6M_v_  | wc -l
>      256
>
> Clemens wrote:
> > It should be a rather rare case that two JVMs running run a gc-cycle
> > at the same time for which performance will be a bit worse
>
> If a server runs during peak load at high cpu utilizations (there are 
> applications that can do this even with 256 cpus in the server...), a 
> 256-threaded gc cycle of a monitoring agent could be quite disruptive 
> for such a workload (especially when it is sensitive to response times).
>
> I'm not saying that we *do* have a critical problem here (in my 
> particular case). I was writing my first mail just to point out that 
> in my opinion there are situations where JVM's are maybe using too 
> many gc threads on large systems. Of course, I totally agree that it's 
> difficult (if not even impossible) for the JVM to come up with best 
> default settings for *any* imaginable situation. Usually the user will 
> need to adjust some of the settings. In some case as I'm describing 
> above, this is unfortunately not always possible: Afaik, there is no 
> external interface to change the JVM settings for this CAM software 
> agent. (Ok, we could consider this the fault of the agent, which could 
> in a startup script detect how many cpus there are in the system, and 
> limit its number of gc threads to let's say 4 (which should be 
> sufficient for such a process with 128m heap). Or use the Client VM 
> instead...) Unfortunately, I believe there are many programs around 
> which fail to do such things...
>
> Therefore I thought it might help if the JVM would limit the number of 
> gc threads or large systems if it is likely that the process would not 
> benefit from more gc threads. It's sure difficult to tell, but I've 
> seen lots of ideas in these mails. 128m heap certainly don't need 256 
> gc threads (or 80 with the 5/8 rule). Btw, this CAM agent has 274 
> threads in total. Subtracting 256 gc threads leaves 18 threads. I 
> don't know how many of them really do anything, but an application 
> with < 18 mutator threads, a heap of 128m, minor gc intervals of 6 
> minutes certainly doesn't need many gc threads -- not even on a 
> 256-way system... ;-) Of course, number of mutator threads and gc 
> intervals are dynamic parameters and can't be determined by the JVM at 
> startup.
>
> Nick.

-- 
----------------------------------------------------------------------
| Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          |
|                                   | MS BUR02-311                   |
| e-mail: tony.printezis at sun.com    | 35 Network Drive               |
| office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  |
----------------------------------------------------------------------
e-mail client: Thunderbird (Solaris)





More information about the hotspot-gc-dev mailing list