monitors explosion

[Karen Kinnear]

Peter,

Thanks for the correction - yes the flag increase the number of  
safepoints, not
the number of GCs.

In terms of when you do inflation - the basic reasons are:
1) If you do a wait or notify
2) If a lock is contended
3) JNI monitors
4) If one thread is setting the hash code while another thread is  
updating a lock

In terms of deflation:
1) when a thread dies, the thread's local free list goes to the global  
free list
2) At a safepoint, we walk the lists of monitors, deflating those that  
are not in use,
and putting them back on the global free list
3) The MonitorBound flag causes a more frequent safepoint
- and yes, it is hard to know how best to tune it

thanks,
Karen



On May 26, 2011, at 7:05 AM, Peter Levart wrote:

> On 05/25/11, Y. Srinivas Ramakrishna wrote:
>> Hi Peter --
>>
>> Which version of the JDK elicits this behaviour? The monitor leakage
>> sounds vaguely like a problem that i encountered before and which
>> was fixed a while back, but this may well be a new problem,
>> and i can't dig up the CR for the problem that I recall in
>> this code, or where the fix went to. (Karen might recall
>> the details because she pushed the fix/workaround for that one.)
>>
>
> The production environment that we first experienced this on is:
>
> SunOS nobiex1 5.10 Generic_141444-09 sun4v sparc SUNW,Sun-Blade- 
> T6320 with 32GB RAM
>
> and JVM was 1.6.0_23 with the following arguments:
>
> -Xmx28672m
> -XX:MaxPermSize=640m
> -XX:+DisableExplicitGC
> -XX:+UseParallelGC
> -XX:ParallelGCThreads=32
> -XX:+UseParallelOldGC
> -XX:LargePageSizeInBytes=256m
> -XX:+UseLargePages
> -XX:+UseCompressedOops
> -XX:+MaxFDLimit
> -XX:ReservedCodeCacheSize=256m
>
> Recently we upgraded to:
>
> java version "1.6.0_25"
> Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)
>
> with no observed difference in behavior.
>
> We also tried with the latest:
>
> java version "1.7.0-ea"
> Java(TM) SE Runtime Environment (build 1.7.0-ea-b142)
> Java HotSpot(TM) Server VM (build 21.0-b12, mixed mode)
>
> with no observed difference.
>
> But when I tried with  1.6.0_25-b06 on amd64/Linux and 1.6.0_25-b06  
> on amd64/Solaris 11, I did not experience the JVM slow-down. But  
> this amd64 HW platform was actually a virtual machine on a VMware  
> host and I think I did not have more than 2 virtual CPUs assigned so  
> this might be the reason (when I did those tests I hadn't yet  
> established the reason for JVM slow-down).
>
> The work-arround we were are using in production (until I learned  
> about the -XX:MonitorBound option) and was found by chance is  
> awkward: loading the Yourkit Java profiler agent into the JVM. This  
> somehow inhibits the monitors explosion (we boot-up with approx  
> 16000 exant semaphores with profiler agent loaded).
>
>> This is the CR that i was thinking of:-
>>
>> 6852873 Increase in delta between application stopped time and  
>> ParNew GC time over application lifetime
>>
>> I can't tell much on how or if the fix/workaround got
>> into specific releases, but i am guessing it must be in
>> the latest 6uXX public release out there (6u25?).
>>
>> Also relevant might be this CR:
>>
>> 6988353 refactor sync subsystem
>>
>> By the way, that was such a nice bug report you filed
>> (we just need the JVM version info from you that's all);
>> i wish all of our customers were like you. Thanks!
>
> It was a learning excercise where I got intimate with DTrace and  
> some JVM internals...
>
>>
>> -- ramki
>
>
> On 05/26/11, Karen Kinnear wrote:
>> Peter,includes
>>
>> Yes, I've been running your test case with some existing flags we  
>> have
>> to help with this kind of problem. Thank you for the test case - that
>> really
>> helps.
>>
>> There is a flag -XX:MonitorBound=<int> which can limit the number of
>> monitors that a thread can hang on to. For the test case I randomly
>> tried -XX:MonitorBound=80 and it reduced the final deflating idel
>> monitors
>> reported time from
>> 0.624620 secs to
>> 0.0007840 secs
>
> It helps for our application too. I don't have to resort to loading  
> the Yourkit agent into the JVM any more. I set the value to 16000  
> since this is the amount that gets allocated when there is no bound  
> specified and yourkit agent is loaded.
>
>>
>> The number can be tuned for your specific application. If you set it
>> lower, it reduces the number of monitors tied-up, and therefore
>> reduces the amount of time spent deflating idle monitors. The trade- 
>> off
>> is that it increases the frequency of GC in order to free the  
>> monitors.
>
> Does it? It can increase frequency of safepointing, but do induced  
> safepoints trigger GC?
>
>> So you have to tune the number to your specific application needs,
>> so as to minimize overall pause time. A recommended place to start  
>> would
>> be with # threads * # locks per thread at a steady state, where #  
>> locks
>> is actually number of allocated monitors, which reflects contended  
>> locks
>> and locks for which you use wait/notify.
>
> There are not many in our code, but I have a feeling that Oracle  
> Coherence might be using much more in some situations. For example,  
> it has a feature where you can "atomically" modify multiple entries  
> in a Map and we use that to isolate "transactions". This feature  
> works by placing "locks" on each modifying entry in some defined  
> order. All in all it's hard to determine the maximum amount of  
> monitors that could at some time be needed at once. I'll start with  
> a big limit that still doesn't make deflation pauses noticable...
>
>>
>> The flag is available in JDK7. I believe it was also backported to
>> JDK6u19.
>>
>> hope this helps,
>
> It does. Thank you.
>
>> Karen
>
> On 05/26/11, David Holmes wrote:
>> Is there some specific reason that taking the hashCode of a locked
>> object forces inflation here? We may need to look at why inflation is
>> happening and whether there is something the application and/or
>> libraries can do to mitigate it. (As well as considering what  
>> happens in
>> the VM of course).
>
> I used hashCode() just to prevent compiler to optimize the code  
> away. I didn't realize that invoking hashCode() could interfere with  
> allocation of object monitors though. Could it?
>
> What I tried to achieve with the code was to establish a situation  
> where one thread (Allocator) inflated/allocated the monitor and  
> another thread (Collector) freed it so that it was returned to it's  
> private list. But I don't know if that is what is going on. 1st I  
> don't know exactly when the monitor has to be inflated/allocated: at  
> the first uncontended entry into the synchronized block or at the  
> first contended synchronyzation. 2nd I don't know exactly when the  
> monitor is freed (returned to the private free list) and to which  
> thread's free list (the one that allocated it or the one that is  
> freeing it). Is there any documentation about what code is generated  
> by JIT? There are helpfull comments in synchronize.cpp but I haven't  
> yet been able to grasp the whole picture about monitors.
>
>>
>> David
>
>
> Regards, Peter