Increased safepoint cleanup tasks in Java 11 (vs 8)

[Vitaly Davidovich]

On Wed, Jun 5, 2019 at 7:10 PM Daniel D. Daugherty <
daniel.daugherty at oracle.com> wrote:

> On 6/5/19 7:03 PM, Vitaly Davidovich wrote:
>
>
>
> On Wed, Jun 5, 2019 at 6:55 PM David Holmes <david.holmes at oracle.com>
> wrote:
>
>> On 6/06/2019 2:31 am, Vitaly Davidovich wrote:
>> > On Wed, Jun 5, 2019 at 12:15 PM Daniel D. Daugherty <
>> > daniel.daugherty at oracle.com> wrote:
>> >
>> >> On 6/5/19 12:04 PM, Vitaly Davidovich wrote:
>> >>
>> >>
>> >>
>> >> On Wed, Jun 5, 2019 at 11:44 AM Daniel D. Daugherty <
>> >> daniel.daugherty at oracle.com> wrote:
>> >>
>> >>> On 6/5/19 11:37 AM, Vitaly Davidovich wrote:
>> >>>> On Wed, Jun 5, 2019 at 8:58 AM Vitaly Davidovich <vitalyd at gmail.com>
>> >>> wrote:
>> >>>>
>> >>>>> Hey Aleksey,
>> >>>>>
>> >>>>> On Wed, Jun 5, 2019 at 8:52 AM Aleksey Shipilev <shade at redhat.com>
>> >>> wrote:
>> >>>>>
>> >>>>>> On 6/5/19 2:33 PM, Vitaly Davidovich wrote:
>> >>>>>>> Does anyone have ideas on pinpointing this further?
>> >>>>>> I have the educated guess:
>> >>>>>>     https://bugs.openjdk.java.net/browse/JDK-8181859
>> >>>>>>
>> >>>>>> Which means you can try if -XX:MonitorUsedDeflationThreshold=0
>> reduces
>> >>>>>> the number of "guaranteed"
>> >>>>>> safepoints.
>> >>>>>>
>> >>>>> Hmm, the application doesn't use JVM monitors (in any meaningful
>> >>>>> capacity).  But interesting JBS entry - I will try it out, thanks!
>> The
>> >>>>> description of safepoint clean up in there is/was consistent with my
>> >>>>> understanding as well.  On Java 8, we'd see the "no vm operation"
>> >>> safepoint
>> >>>>> operations, and I always assumed it was IC buffer cleaning (even
>> had a
>> >>> mail
>> >>>>> thread on that topic a few years ago, either on this list or
>> >>>>> hotspot-compiler).  That tended to stabilize (i.e. they went away)
>> >>> after
>> >>>>> the application reached steady state, whereas I'm seeing essentially
>> >>>>> non-stop safepoint cleanups on 11.
>> >>>>>
>> >>>> Ok, adding -XX:MonitorUsedDeflationThreshold=0 seems to alleviate the
>> >>> issue
>> >>>> - # of entered safepoint cleanups looks on par with Java 8, although
>> I'm
>> >>>> still assessing a bit more.
>> >>>
>> >>> This observation doesn't really match with your earlier comment:
>> >>>
>> >>>> Hmm, the application doesn't use JVM monitors (in any meaningful
>> >>>> capacity).
>> >>>
>> >>> The default value for MonitorUsedDeflationThreshold is 90. So a
>> cleanup
>> >>> safepoint should only be triggered if monitor use exceeds 90% of the
>> >>> monitor pool.
>> >>>
>> >> Hmm, ok - I see what you mean:
>> >>
>> https://github.com/openjdk/jdk11u/blob/d46b56eb81b07b452141c7dcd078d33fc8e49d63/src/hotspot/share/runtime/safepoint.cpp#L604
>> >>
>> >> So there's either monitor usage that I'm not aware of (or forgot
>> about) or
>> >> the ratio is > 90 but the absolute # of them is small (which I'd
>> consider
>> >> "don't use in any meaningful capacity").  Is there a way to print out
>> >> monitor population so I can see?
>> >>
>> >>
>> >> I've been adding a lot of logging support to the Java Monitor
>> subsystem,
>> >> but those changes are in JDK13 and not in JDK11. I'm trying to remember
>> >> what there is in JDK11, but ...
>> >>
>> > Yeah, I'm having a hard time keeping track of what features/changes are
>> in
>> > what Java release or update version or upstream/downstream backport or
>> ...
>> >
>> > By the way, do JNI (specifically, Java calling into native) calls cause
>> > monitor inflation? The app in question does have substantial JNI
>> usage.  I
>> > guess by not using java monitors extensively, I meant the vanilla
>> flavor of
>> > Java code with synchronized blocks/methods.
>>
>> JNI calls do not in themselves impact monitors. However if native code
>> uses the JNI MonitorEnter then that does trigger inflation.
>>
>> David
>> -----
>
> Thanks for clarifying, David.
>
> Pretty certain that JNI code I have is not calling MonitorEnter, but will
> double check.
>
> So beyond monitor contention via vanilla Java code, are there other
> scenarios resulting in inflation? Is there perhaps a list of these that
> someone could point to? I can spelunk through hotspot code, but maybe
> someone can spare me that exercise :).
>
>
> Java Monitor inflation is caused by:
>
> - contention on the same monitor between 2 or more threads
> - monitor.wait() call
> - synchronized (obj) { obj.hashCode(); }
>
> The last one is specific to the HotSpot VM.
>
> Dan
>
Awesome, thanks Dan! In lieu of more precise monitor usage logging in 11,
I’ll look at our code for these.

Only remaining input I’d appreciate from the list is whether setting
MonitorUsedDeflationThreshold=0 “restores” 8 behavior, or whether I’d run
the risk of something unfortunate as a result (again, with respect to 8).

Thanks

>
>
>
>
> Thanks!
>
>>
>>
>> >
>> >>
>> >> Dan
>> >>
>> >>
>> >>
>> >> Thanks Dan
>> >>
>> >>>
>> >>> Dan
>> >>>
>> >>>
>> >>>>
>> >>>> Of course now the trouble is this is an experimental option, and
>> >>> enabling
>> >>>> things like that in production raises eyebrows.  Is it safe to
>> assume,
>> >>>> however, that setting this to 0 essentially behaves like Java 8? It
>> >>> appears
>> >>>> so, based on that JBS Aleksey linked above.  But just double
>> checking.
>> >>>>
>> >>>> If that's the case, then perhaps we can overlook the experimental bit
>> >>> and
>> >>>> wait until this is done outside safepoints, per Robbin's email.
>> >>>>
>> >>>> Thanks to everyone that responded on this thread! Very helpful.
>> >>>>
>> >>>>>> --
>> >>>>>> Thanks,
>> >>>>>> -Aleksey
>> >>>>>>
>> >>>>>>
>> >>>
>> >>>
>> >>
>>
> --
> Sent from my phone
>
>
> --
Sent from my phone