monitors explosion

[Peter Levart]

On 05/25/11, Y. Srinivas Ramakrishna wrote:
> Hi Peter --
> 
> Which version of the JDK elicits this behaviour? The monitor leakage
> sounds vaguely like a problem that i encountered before and which
> was fixed a while back, but this may well be a new problem,
> and i can't dig up the CR for the problem that I recall in
> this code, or where the fix went to. (Karen might recall
> the details because she pushed the fix/workaround for that one.)
> 

The production environment that we first experienced this on is:

SunOS nobiex1 5.10 Generic_141444-09 sun4v sparc SUNW,Sun-Blade-T6320 with 32GB RAM

and JVM was 1.6.0_23 with the following arguments:

-Xmx28672m
-XX:MaxPermSize=640m
-XX:+DisableExplicitGC
-XX:+UseParallelGC
-XX:ParallelGCThreads=32
-XX:+UseParallelOldGC
-XX:LargePageSizeInBytes=256m
-XX:+UseLargePages
-XX:+UseCompressedOops
-XX:+MaxFDLimit
-XX:ReservedCodeCacheSize=256m

Recently we upgraded to:

java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)

with no observed difference in behavior.

We also tried with the latest:

java version "1.7.0-ea"
Java(TM) SE Runtime Environment (build 1.7.0-ea-b142)
Java HotSpot(TM) Server VM (build 21.0-b12, mixed mode)

with no observed difference.

But when I tried with  1.6.0_25-b06 on amd64/Linux and 1.6.0_25-b06 on amd64/Solaris 11, I did not experience the JVM slow-down. But this amd64 HW platform was actually a virtual machine on a VMware host and I think I did not have more than 2 virtual CPUs assigned so this might be the reason (when I did those tests I hadn't yet established the reason for JVM slow-down).

The work-arround we were are using in production (until I learned about the -XX:MonitorBound option) and was found by chance is awkward: loading the Yourkit Java profiler agent into the JVM. This somehow inhibits the monitors explosion (we boot-up with approx 16000 exant semaphores with profiler agent loaded).

> This is the CR that i was thinking of:-
> 
> 6852873 Increase in delta between application stopped time and ParNew GC time over application lifetime
> 
> I can't tell much on how or if the fix/workaround got
> into specific releases, but i am guessing it must be in
> the latest 6uXX public release out there (6u25?).
> 
> Also relevant might be this CR:
> 
> 6988353 refactor sync subsystem
> 
> By the way, that was such a nice bug report you filed
> (we just need the JVM version info from you that's all);
> i wish all of our customers were like you. Thanks!

It was a learning excercise where I got intimate with DTrace and some JVM internals...

> 
> -- ramki


On 05/26/11, Karen Kinnear wrote:
> Peter,includes
> 
> Yes, I've been running your test case with some existing flags we have
> to help with this kind of problem. Thank you for the test case - that  
> really
> helps.
> 
> There is a flag -XX:MonitorBound=<int> which can limit the number of
> monitors that a thread can hang on to. For the test case I randomly
> tried -XX:MonitorBound=80 and it reduced the final deflating idel  
> monitors
> reported time from
> 0.624620 secs to
> 0.0007840 secs

It helps for our application too. I don't have to resort to loading the Yourkit agent into the JVM any more. I set the value to 16000 since this is the amount that gets allocated when there is no bound specified and yourkit agent is loaded.

> 
> The number can be tuned for your specific application. If you set it
> lower, it reduces the number of monitors tied-up, and therefore
> reduces the amount of time spent deflating idle monitors. The trade-off
> is that it increases the frequency of GC in order to free the monitors.

Does it? It can increase frequency of safepointing, but do induced safepoints trigger GC?

> So you have to tune the number to your specific application needs,
> so as to minimize overall pause time. A recommended place to start would
> be with # threads * # locks per thread at a steady state, where # locks
> is actually number of allocated monitors, which reflects contended locks
> and locks for which you use wait/notify.

There are not many in our code, but I have a feeling that Oracle Coherence might be using much more in some situations. For example, it has a feature where you can "atomically" modify multiple entries in a Map and we use that to isolate "transactions". This feature works by placing "locks" on each modifying entry in some defined order. All in all it's hard to determine the maximum amount of monitors that could at some time be needed at once. I'll start with a big limit that still doesn't make deflation pauses noticable...

> 
> The flag is available in JDK7. I believe it was also backported to  
> JDK6u19.
> 
> hope this helps,

It does. Thank you.

> Karen

On 05/26/11, David Holmes wrote:
> Is there some specific reason that taking the hashCode of a locked 
> object forces inflation here? We may need to look at why inflation is 
> happening and whether there is something the application and/or 
> libraries can do to mitigate it. (As well as considering what happens in 
> the VM of course).

I used hashCode() just to prevent compiler to optimize the code away. I didn't realize that invoking hashCode() could interfere with allocation of object monitors though. Could it?

What I tried to achieve with the code was to establish a situation where one thread (Allocator) inflated/allocated the monitor and another thread (Collector) freed it so that it was returned to it's private list. But I don't know if that is what is going on. 1st I don't know exactly when the monitor has to be inflated/allocated: at the first uncontended entry into the synchronized block or at the first contended synchronyzation. 2nd I don't know exactly when the monitor is freed (returned to the private free list) and to which thread's free list (the one that allocated it or the one that is freeing it). Is there any documentation about what code is generated by JIT? There are helpfull comments in synchronize.cpp but I haven't yet been able to grasp the whole picture about monitors.

> 
> David


Regards, Peter