Increased safepoint cleanup tasks in Java 11 (vs 8)

[Daniel D. Daugherty]

On 6/5/19 7:03 PM, Vitaly Davidovich wrote:
>
>
> On Wed, Jun 5, 2019 at 6:55 PM David Holmes <david.holmes at oracle.com 
> <mailto:david.holmes at oracle.com>> wrote:
>
>     On 6/06/2019 2:31 am, Vitaly Davidovich wrote:
>     > On Wed, Jun 5, 2019 at 12:15 PM Daniel D. Daugherty <
>     > daniel.daugherty at oracle.com
>     <mailto:daniel.daugherty at oracle.com>> wrote:
>     >
>     >> On 6/5/19 12:04 PM, Vitaly Davidovich wrote:
>     >>
>     >>
>     >>
>     >> On Wed, Jun 5, 2019 at 11:44 AM Daniel D. Daugherty <
>     >> daniel.daugherty at oracle.com
>     <mailto:daniel.daugherty at oracle.com>> wrote:
>     >>
>     >>> On 6/5/19 11:37 AM, Vitaly Davidovich wrote:
>     >>>> On Wed, Jun 5, 2019 at 8:58 AM Vitaly Davidovich
>     <vitalyd at gmail.com <mailto:vitalyd at gmail.com>>
>     >>> wrote:
>     >>>>
>     >>>>> Hey Aleksey,
>     >>>>>
>     >>>>> On Wed, Jun 5, 2019 at 8:52 AM Aleksey Shipilev
>     <shade at redhat.com <mailto:shade at redhat.com>>
>     >>> wrote:
>     >>>>>
>     >>>>>> On 6/5/19 2:33 PM, Vitaly Davidovich wrote:
>     >>>>>>> Does anyone have ideas on pinpointing this further?
>     >>>>>> I have the educated guess:
>     >>>>>> https://bugs.openjdk.java.net/browse/JDK-8181859
>     >>>>>>
>     >>>>>> Which means you can try if
>     -XX:MonitorUsedDeflationThreshold=0 reduces
>     >>>>>> the number of "guaranteed"
>     >>>>>> safepoints.
>     >>>>>>
>     >>>>> Hmm, the application doesn't use JVM monitors (in any meaningful
>     >>>>> capacity).  But interesting JBS entry - I will try it out,
>     thanks! The
>     >>>>> description of safepoint clean up in there is/was consistent
>     with my
>     >>>>> understanding as well.  On Java 8, we'd see the "no vm
>     operation"
>     >>> safepoint
>     >>>>> operations, and I always assumed it was IC buffer cleaning
>     (even had a
>     >>> mail
>     >>>>> thread on that topic a few years ago, either on this list or
>     >>>>> hotspot-compiler).  That tended to stabilize (i.e. they went
>     away)
>     >>> after
>     >>>>> the application reached steady state, whereas I'm seeing
>     essentially
>     >>>>> non-stop safepoint cleanups on 11.
>     >>>>>
>     >>>> Ok, adding -XX:MonitorUsedDeflationThreshold=0 seems to
>     alleviate the
>     >>> issue
>     >>>> - # of entered safepoint cleanups looks on par with Java 8,
>     although I'm
>     >>>> still assessing a bit more.
>     >>>
>     >>> This observation doesn't really match with your earlier comment:
>     >>>
>     >>>> Hmm, the application doesn't use JVM monitors (in any meaningful
>     >>>> capacity).
>     >>>
>     >>> The default value for MonitorUsedDeflationThreshold is 90. So
>     a cleanup
>     >>> safepoint should only be triggered if monitor use exceeds 90%
>     of the
>     >>> monitor pool.
>     >>>
>     >> Hmm, ok - I see what you mean:
>     >>
>     https://github.com/openjdk/jdk11u/blob/d46b56eb81b07b452141c7dcd078d33fc8e49d63/src/hotspot/share/runtime/safepoint.cpp#L604
>     >>
>     >> So there's either monitor usage that I'm not aware of (or
>     forgot about) or
>     >> the ratio is > 90 but the absolute # of them is small (which
>     I'd consider
>     >> "don't use in any meaningful capacity").  Is there a way to
>     print out
>     >> monitor population so I can see?
>     >>
>     >>
>     >> I've been adding a lot of logging support to the Java Monitor
>     subsystem,
>     >> but those changes are in JDK13 and not in JDK11. I'm trying to
>     remember
>     >> what there is in JDK11, but ...
>     >>
>     > Yeah, I'm having a hard time keeping track of what
>     features/changes are in
>     > what Java release or update version or upstream/downstream
>     backport or ...
>     >
>     > By the way, do JNI (specifically, Java calling into native)
>     calls cause
>     > monitor inflation? The app in question does have substantial JNI
>     usage.  I
>     > guess by not using java monitors extensively, I meant the
>     vanilla flavor of
>     > Java code with synchronized blocks/methods.
>
>     JNI calls do not in themselves impact monitors. However if native
>     code
>     uses the JNI MonitorEnter then that does trigger inflation.
>
>     David
>     -----
>
> Thanks for clarifying, David.
>
> Pretty certain that JNI code I have is not calling MonitorEnter, but 
> will double check.
>
> So beyond monitor contention via vanilla Java code, are there other 
> scenarios resulting in inflation? Is there perhaps a list of these 
> that someone could point to? I can spelunk through hotspot code, but 
> maybe someone can spare me that exercise :).

Java Monitor inflation is caused by:

- contention on the same monitor between 2 or more threads
- monitor.wait() call
- synchronized (obj) { obj.hashCode(); }

The last one is specific to the HotSpot VM.

Dan


>
> Thanks!
>
>
>
>     >
>     >>
>     >> Dan
>     >>
>     >>
>     >>
>     >> Thanks Dan
>     >>
>     >>>
>     >>> Dan
>     >>>
>     >>>
>     >>>>
>     >>>> Of course now the trouble is this is an experimental option, and
>     >>> enabling
>     >>>> things like that in production raises eyebrows.  Is it safe
>     to assume,
>     >>>> however, that setting this to 0 essentially behaves like Java
>     8? It
>     >>> appears
>     >>>> so, based on that JBS Aleksey linked above.  But just double
>     checking.
>     >>>>
>     >>>> If that's the case, then perhaps we can overlook the
>     experimental bit
>     >>> and
>     >>>> wait until this is done outside safepoints, per Robbin's email.
>     >>>>
>     >>>> Thanks to everyone that responded on this thread! Very helpful.
>     >>>>
>     >>>>>> --
>     >>>>>> Thanks,
>     >>>>>> -Aleksey
>     >>>>>>
>     >>>>>>
>     >>>
>     >>>
>     >>
>
> -- 
> Sent from my phone