RFR(L) 8153224 Monitor deflation prolong safepoints (CR14/v2.14/17-for-jdk15)

Daniel D. Daugherty daniel.daugherty at oracle.com
Tue Jun 2 17:54:20 UTC 2020


Hi Carsten,

See replies below...

David, Erik and Robbin, if you folks could also check out the revised
comment below that would be appreciated.


On 6/2/20 9:39 AM, Carsten Varming wrote:
> Hi Dan,
>
> See inline.
>
> On Mon, Jun 1, 2020 at 11:32 PM Daniel D. Daugherty 
> <daniel.daugherty at oracle.com <mailto:daniel.daugherty at oracle.com>> wrote:
>
>     Hi Carsten,
>
>     Thanks for chiming in on this review thread!!
>
>
> It is my pleasure. You know the code is solid when the discussion is 
> focused on the comments.

So true, so very true!


>     On 6/1/20 10:41 PM, Carsten Varming wrote:
>>     Hi Dan,
>>
>>     I like the new protocol, but I had to think about how the extra
>>     increment to _contentions replaced the check on _owner that I
>>     originally added.
>
>     Right. The check on _owner was described in detail in the OpenJDK wiki
>     subsection that was called "T-enter Wins By A-B-A". It can still be
>     found by going thru the wiki's history links.
>
>     That subsection was renamed and rewritten and can be found here:
>
>     https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation#AsyncMonitorDeflation-T-enterWinsByCancellationViaDEFLATER_MARKERSwap
>
>
>>     I am thinking that the increased _contention value is a little
>>     mark left on the ObjectMonitor to signal to the deflater thread
>>     (which must be in the middle of trying to acquire the object
>>     monitor as _owner was set to DEFLATER_MARKER) that the deflater
>>     thread lost the race.
>
>     That is exactly what the extra increment is being used for.
>
>     In my reply to David H. that you quoted below, I describe the
>     progression
>     of contention values thru the two possible race scenarios. The
>     progression
>     shows the T-enter thread winning the race and marking the
>     contention field
>     with the extra increment while the T-deflater thread recognizes
>     that it has
>     lost the race and unmarks the contention field with an extra
>     decrement.
>
>
> I noticed that. Looks like David and I were racing and David won. :)
>
>>     That little mark stays with the object monitor long after the
>>     thread is done with the monitor.
>
>     The "little mark" stays with the ObjectMonitor after T-enter is done
>     entering until the T-deflater thread recognizes that the async
>     deflation
>     was canceled and does an extra decrement. I don't think I would
>     describe
>     it as "long after".
>
>
> Sorry about the use of "long after". When I think about the 
> correctness of protocols, like the deflation protocol, I end up 
> thinking about sequences of instructions and the relevant 
> interleavings. In that context I often end up using phrases like "long 
> after" and "after" to mean anything after a particular instruction. I 
> did not mean to imply anything about the relative speed of the 
> execution of the code.

It's okay. I do something similar in the transaction diagrams that
I use to work out timing issues: <thread stalls> ... <thread resumes>

The only point that I was trying to make is that the T-deflate thread
is responsible for cleaning up the extra mark and it's committed to
the code path that will result in the cleanup. Yes, there may be a
<thread stalls> between the time that T-deflate recognizes that async
deflation was canceled and when T-deflate does the extra decrement,
but I don't see any harm in it.


>>     It might be worth adding a comment to the code explaining that
>>     after the increment, the _contention field can only be set to 0
>>     by a corresponding decrement in the async deflater thread,
>>     ensuring that the Atomic::cmpxchg(&mid->_contentions, (jint)0,
>>     -max_jint) on line 2166 fails. In particular, the comment:
>>     +. // .... We bump contentions an
>>     + // extra time to prevent the async deflater thread from temporarily
>>     + // changing it to -max_jint and back to zero (no flicker to confuse
>>     + // is_being_async_deflated()
>>     confused me as after the deflater thread sets _contentions to
>>     -max_jint, the deflater thread has won the race and the object
>>     monitor is about to be deflated.
>
>     For context, here's the code and comment being discussed:
>
>>       527   if (AsyncDeflateIdleMonitors &&
>>       528       try_set_owner_from(DEFLATER_MARKER, Self) == DEFLATER_MARKER) {
>>     529 // Cancelled the in-progress async deflation. We bump
>>     contentions an
>>     530 // extra time to prevent the async deflater thread from
>>     temporarily
>>     531 // changing it to -max_jint and back to zero (no flicker to
>>     confuse
>>     532 // is_being_async_deflated()). The async deflater thread will
>>     533 // decrement contentions after it recognizes that the async
>>     534 // deflation was cancelled.
>>     535 add_to_contentions(1);
>
>     This part of the new comment:
>
>      532     // ...  The async deflater thread will
>      533     // decrement contentions after it recognizes that the async
>      534     // deflation was cancelled.
>
>     makes it clear that the async deflater thread does the
>     corresponding decrement
>     to the increment done by the T-enter thread so that covers this
>     part of your
>     comment above:
>
>         the _contention field can only be set to 0 by a corresponding
>     decrement
>         in the async deflater thread
>
>     This part of the new comment:
>
>      529     // ...  We bump contentions an
>      530     // extra time to prevent the async deflater thread from
>     temporarily
>      531     // changing it to -max_jint and back to zero (no flicker
>     to confuse
>      532     // is_being_async_deflated()).
>
>     makes it clear that we're keeping make-contentions-negative part
>     of the
>     async deflation protocol from happening so that covers this part
>     of your
>     comment above:
>
>         ensuring that the Atomic::cmpxchg(&mid->_contentions, (jint)0,
>     -max_jint)
>         on line 2166 fails.
>
>     This part of your comment above makes it clear where the confusion
>     arises:
>
>         confused me as after the deflater thread sets _contentions to
>     -max_jint,
>         the deflater thread has won the race and the object monitor is
>     about to
>         be deflated.
>
>     Your original algorithm is a three-part async deflation protocol:
>
>     Part 1 - set owner field to DEFLATER marker
>     Part 2 - make a zero contentions field -max_jint
>     Part 3 - check to see if the owner field is still DEFLATER_MARKER
>
>     If part 3 fails, then the contentions field that is currently negative
>     has max_jint added to it to complete the bail out process. It's that
>     third part that makes the contentions field flicker from:
>
>         0 -> -max_jint -> 0
>
>     And the extra contentions increment in the new two part protocol
>     solves
>     that flicker and allows us to treat (contentions < 0) as a
>     linearization
>     point.
>
>     Please let me know if this clarifies your concern.
>
>
> I am no longer confused, but the cause of my confusion is still 
> present in the comment.
>
> This group knows about the three part algorithm,  but when the code is 
> pushed there is no representation of the three part algorithm in the 
> code or repository.

That's a really good point and a side effect of my living with this
code for a very long time...


> I forgot the details of the algorithm and read the latest version of 
> the code to figure out what the flickering was about. As you would 
> expect, I found that there is no way the code can cause the flicker 
> mentioned. That made me worried. I started to question myself: What 
> can cause the behavior that is described in the comments? What am I 
> missing? As a result, I think it is best if we keep the flickering to 
> ourselves and update the comment to describe that because _owner was 
> DEFLATER_MARKER the deflation thread must be in the middle of the 
> protocol for deflating the object monitor, and in particular, 
> incrementing _contentions ensures the failure of the final CAS in the 
> deflation protocol (final in the protocol implemented in the code).

The above is a more clear expression of your concerns and I agree.


> To be clear:
>
> > 529 // Cancelled the in-progress async deflation.
>
> I would expend this comment by mentioning that the deflator thread 
> cannot win the last part of the 2-part deflation protocol as 0 < 
> _contentions (pre-condition to this method).
>
> > We bump contentions an
> > 530 // extra time to prevent the async deflater thread from temporarily
> > 531 // changing it to -max_jint and back to zero (no flicker to confuse
> > 532 // is_being_async_deflated()).
>
> I would replace this part with something along the lines of: We bump 
> contentions an extra time to prevent the deflator thread from winning 
> the last part of the (2-part) deflation protocol after this thread 
> decrements _contentions as part of the release of the object monitor.
>
> > The async deflater thread will
> > 533 // decrement contentions after it recognizes that the async
> > 534 // deflation was cancelled.
>
> I would keep this part.

So here's my rewrite of the code and comment block:

   if (AsyncDeflateIdleMonitors &&
       try_set_owner_from(DEFLATER_MARKER, Self) == DEFLATER_MARKER) {
     // Cancelled the in-progress async deflation by changing owner from
     // DEFLATER_MARKER to Self. As part of the contended enter protocol,
     // contentions was incremented to a positive value before EnterI()
     // was called and that prevents the deflater thread from winning the
     // last part of the 2-part async deflation protocol. After EnterI()
     // returns to enter(), contentions is decremented because the caller
     // now owns the monitor. We bump contentions an extra time here to
     // prevent the deflater thread from winning the last part of the
     // 2-part async deflation protocol after the regular decrement
     // occurs in enter(). The deflater thread will decrement contentions
     // after it recognizes that the async deflation was cancelled.
     add_to_contentions(1);

I've made this change to both places in EnterI() that had the original
confusing comment.

Please let me know if this rewrite works for everyone.

Since I've already pushed 8153224, I'll file a new bug to push this
clarification once we're all in agreement here.

Dan


>
> I hope this helps,
> Carsten
>
>>     Otherwise, the code looks great. I am looking forward to seeing
>>     in the repo.
>
>     Thanks! The code should be there soon.
>
>     Dan
>
>
>>
>>     Carsten
>>
>>     On Mon, Jun 1, 2020 at 8:32 PM Daniel D. Daugherty
>>     <daniel.daugherty at oracle.com
>>     <mailto:daniel.daugherty at oracle.com>> wrote:
>>
>>         Hi David,
>>
>>         On 6/1/20 7:58 PM, David Holmes wrote:
>>         > Hi Dan,
>>         >
>>         > Sorry for the delay.
>>
>>         No worries. It's always worth waiting for your code review in
>>         general
>>         and, with the complexity of this project, it's on my must-do
>>         list!
>>
>>
>>         >
>>         > On 28/05/2020 3:20 am, Daniel D. Daugherty wrote:
>>         >> Greetings,
>>         >>
>>         >> Erik O. had an idea for changing the three part async
>>         deflation protocol
>>         >> into a two part async deflation protocol where the second
>>         part (setting
>>         >> the contentions field to -max_jint) is a linearization
>>         point. I've taken
>>         >> Erik's proposal (which was relative to
>>         CR12/v2.12/15-for-jdk15), merged
>>         >> it with CR13/v2.13/16-for-jdk15, and made a few minor tweaks.
>>         >>
>>         >> I have attached the change list from CR13 to CR14 and I've
>>         also added a
>>         >> link to the CR13-to-CR14-changes file to the webrevs so it
>>         should be
>>         >> easy
>>         >> to find.
>>         >>
>>         >> Main bug URL:
>>         >>
>>         >>      JDK-8153224 Monitor deflation prolong safepoints
>>         >> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>
>>         >> The project is currently baselined on jdk-15+24.
>>         >>
>>         >> Here's the full webrev URL for those folks that want to
>>         see all of the
>>         >> current Async Monitor Deflation code in one go (v2.14 full):
>>         >>
>>         >>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/17-for-jdk15+24.v2.14.full/
>>
>>         >>
>>         >>
>>         >> Some folks might want to see just what has changed since
>>         the last review
>>         >> cycle so here's a webrev for that (v2.14 inc):
>>         >>
>>         >>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/17-for-jdk15+24.v2.14.inc/
>>
>>         >
>>         >
>>         > src/hotspot/share/runtime/synchronizer.cpp
>>         >
>>         > I'm having a little trouble keeping the _contentions
>>         relationships in
>>         > my head. In particular with this change I can't quite grok the:
>>         >
>>         > // Deferred decrement for the JT EnterI() that cancelled
>>         the async
>>         > deflation.
>>         > mid->add_to_contentions(-1);
>>         >
>>         > change. I kind of get EnterI() does an extra increment and the
>>         > deflator thread does the above matching decrement. But
>>         given the two
>>         > changes can happen in any order I'm not sure what the
>>         possible visible
>>         > values for _contentions will be and how that might affect
>>         other code
>>         > inspecting it?
>>
>>         I have a sub-section in the OpenJDK wiki dedicated to this
>>         particular race:
>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation#AsyncMonitorDeflation-T-enterWinsByCancellationViaDEFLATER_MARKERSwap
>>
>>         In order for this race condition to manifest, the T-enter
>>         thread has to
>>         successfully swap the owner field's DEFLATER_MARKER value for
>>         Self. That
>>         swap will eventually cause the T-deflate thread to realize
>>         that the async
>>         deflation that it started has been canceled.
>>
>>         The diagram shows the progression of contentions values:
>>
>>         - ObjectMonitor box 1 shows contentions == 1 because T-enter
>>         incremented
>>            the contentions field
>>
>>         - ObjectMonitor box 2 shows contentions == 2 because EnterI()
>>         did the
>>            extra increment.
>>
>>         - ObjectMonitor box 3 shows contentions == 1 because T-enter
>>         did the
>>            regular contentions decrement.
>>
>>         - ObjectMonitor box 4 shows contentions == 0 because
>>         T-deflate did the
>>            extra contentions decrement.
>>
>>         Now it is possible for T-deflate to do the extra decrement
>>         before T-enter
>>         does the extra increment. If I were to add another diagram to
>>         show that
>>         variant of the race, that progression of contentions values
>>         would be:
>>
>>         - ObjectMonitor box 1 shows contentions == 1 because T-enter
>>         incremented
>>            the contentions field
>>
>>         - ObjectMonitor box 2 shows contentions == 0 because
>>         T-deflate did the
>>            extra contentions decrement.
>>
>>         - ObjectMonitor box 3 shows contentions == 1 because EnterI()
>>         did the
>>            extra increment.
>>
>>         - ObjectMonitor box 4 shows contentions == 0 because T-enter
>>         did the
>>            regular contentions decrement.
>>
>>         Notice that in this second scenario the contentions field
>>         never goes
>>         negative so there's nothing to confuse a potential caller of
>>         is_being_async_deflated():
>>
>>         inline bool ObjectMonitor::is_being_async_deflated() {
>>            return AsyncDeflateIdleMonitors && contentions() < 0;
>>         }
>>
>>         It is not possible for T-deflate's extra decrement of the
>>         contentions
>>         field to make the contentions field negative. That decrement
>>         only happens
>>         when T-deflate detects that the async deflation has been
>>         canceled and
>>         async deflation can only be canceled after T-enter has
>>         already made the
>>         contentions field > 0.
>>
>>         Please let me know if this resolves your concern about:
>>
>>         > // Deferred decrement for the JT EnterI() that cancelled
>>         the async
>>         > deflation.
>>         > mid->add_to_contentions(-1);
>>
>>         I'm not planning to update the OpenJDK wiki to add a second
>>         variant of
>>         the cancellation race. Please let me know if that is okay.
>>
>>         >
>>         > But otherwise the changes in this version seem good and
>>         overall the
>>         > protocol seems simpler.
>>
>>         This sounds like a thumbs up, but I'm looking for something
>>         more definitive.
>>
>>
>>         > I'm still going to spend some more time going over the
>>         complete webrev
>>         > to get a fuller sense of things.
>>
>>         As always, if you find something after I've pushed, we'll
>>         deal with it.
>>
>>         Thanks for your many re-reviews for this project!!
>>
>>         Dan
>>
>>
>>         >
>>         > Thanks,
>>         > David
>>         >
>>         >>
>>         >>
>>         >> The OpenJDK wiki has been updated for v2.14.
>>         >>
>>         >>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>         >>
>>         >> The jdk-15+24 based v2.14 version of the patch has gone
>>         thru Mach5
>>         >> Tier[1-5]
>>         >> testing with no related failures; Mach5 Tier[67] are
>>         running now and
>>         >> so far
>>         >> have no related failures. I'll kick off Mach5 Tier8 after
>>         the other
>>         >> tiers
>>         >> have finished since Mach5 is a bit busy right now.
>>         >>
>>         >> I'm also running my usual inflation stress testing on
>>         Linux-X64 and
>>         >> macOSX
>>         >> and so far there are no issues.
>>         >>
>>         >> Thanks, in advance, for any questions, comments or
>>         suggestions.
>>         >>
>>         >> Dan
>>         >>
>>         >>
>>         >> On 5/21/20 2:53 PM, Daniel D. Daugherty wrote:
>>         >>> Greetings,
>>         >>>
>>         >>> I have made changes to the Async Monitor Deflation code
>>         in response to
>>         >>> the CR12/v2.12/15-for-jdk15 code review cycle. Thanks to
>>         David H. and
>>         >>> Erik O. for their OpenJDK reviews in the v2.12 round!
>>         >>>
>>         >>> I have attached the change list from CR12 to CR13 and
>>         I've also added a
>>         >>> link to the CR12-to-CR13-changes file to the webrevs so
>>         it should be
>>         >>> easy
>>         >>> to find.
>>         >>>
>>         >>> Main bug URL:
>>         >>>
>>         >>>     JDK-8153224 Monitor deflation prolong safepoints
>>         >>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>
>>         >>> The project is currently baselined on jdk-15+24.
>>         >>>
>>         >>> Here's the full webrev URL for those folks that want to
>>         see all of the
>>         >>> current Async Monitor Deflation code in one go (v2.13 full):
>>         >>>
>>         >>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/16-for-jdk15%2b24.v2.13.full/
>>
>>         >>>
>>         >>>
>>         >>> Some folks might want to see just what has changed since
>>         the last
>>         >>> review
>>         >>> cycle so here's a webrev for that (v2.13 inc):
>>         >>>
>>         >>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/16-for-jdk15%2b24.v2.13.inc/
>>
>>         >>>
>>         >>>
>>         >>>
>>         >>> The OpenJDK wiki is currently at v2.13 and might require
>>         minor
>>         >>> tweaks for v2.12
>>         >>> and v2.13. Yes, I need to make yet another crawl thru
>>         review of it...
>>         >>>
>>         >>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>         >>>
>>         >>> The jdk-15+24 based v2.13 version of the patch is going
>>         thru the usual
>>         >>> Mach5 testing right now. It is also going thru my usual
>>         inflation
>>         >>> stress
>>         >>> testing on Linux-X64 and macOSX.
>>         >>>
>>         >>> Thanks, in advance, for any questions, comments or
>>         suggestions.
>>         >>>
>>         >>> Dan
>>         >>>
>>         >>> On 5/14/20 5:40 PM, Daniel D. Daugherty wrote:
>>         >>>> Greetings,
>>         >>>>
>>         >>>> I have made changes to the Async Monitor Deflation code
>>         in response to
>>         >>>> the CR11/v2.11/14-for-jdk15 code review cycle. Thanks to
>>         David H.,
>>         >>>> Erik O.,
>>         >>>> and Robbin for their OpenJDK reviews in the v2.11 round!
>>         >>>>
>>         >>>> I have attached the change list from CR11 to CR12 and
>>         I've also
>>         >>>> added a
>>         >>>> link to the CR11-to-CR12-changes file to the webrevs so
>>         it should
>>         >>>> be easy
>>         >>>> to find.
>>         >>>>
>>         >>>> Main bug URL:
>>         >>>>
>>         >>>>     JDK-8153224 Monitor deflation prolong safepoints
>>         >>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>
>>         >>>> The project is currently baselined on jdk-15+23.
>>         >>>>
>>         >>>> Here's the full webrev URL for those folks that want to
>>         see all of the
>>         >>>> current Async Monitor Deflation code in one go (v2.12 full):
>>         >>>>
>>         >>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/15-for-jdk15%2b23.v2.12.full/
>>
>>         >>>>
>>         >>>>
>>         >>>> Some folks might want to see just what has changed since
>>         the last
>>         >>>> review
>>         >>>> cycle so here's a webrev for that (v2.12 inc):
>>         >>>>
>>         >>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/15-for-jdk15%2b23.v2.12.inc/
>>
>>         >>>>
>>         >>>>
>>         >>>>
>>         >>>> The OpenJDK wiki is currently at v2.11 and might require
>>         minor
>>         >>>> tweaks for v2.12:
>>         >>>>
>>         >>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>         >>>>
>>         >>>> The jdk-15+23 based v2.12 version of the patch is going
>>         thru the usual
>>         >>>> Mach5 testing right now.
>>         >>>>
>>         >>>> Thanks, in advance, for any questions, comments or
>>         suggestions.
>>         >>>>
>>         >>>> Dan
>>         >>>>
>>         >>>>
>>         >>>> On 5/7/20 1:08 PM, Daniel D. Daugherty wrote:
>>         >>>>> Greetings,
>>         >>>>>
>>         >>>>> I have made changes to the Async Monitor Deflation code in
>>         >>>>> response to
>>         >>>>> the CR10/v2.10/13-for-jdk15 code review cycle and
>>         DaCapo-h2 perf
>>         >>>>> testing.
>>         >>>>> Thanks to Erik O., Robbin and David H. for their
>>         OpenJDK reviews
>>         >>>>> in the
>>         >>>>> v2.10 round! Thanks to Eric C. for his help in
>>         isolating the
>>         >>>>> DaCapo-h2
>>         >>>>> performance regression.
>>         >>>>>
>>         >>>>> With the removal of ref_counting and the
>>         ObjectMonitorHandle
>>         >>>>> class, the
>>         >>>>> Async Monitor Deflation project is now closer to
>>         Carsten's original
>>         >>>>> prototype. While ref_counting gave us ObjectMonitor*
>>         safety
>>         >>>>> enforced by
>>         >>>>> code, I saw a ~22.8% slow down with
>>         -XX:-AsyncDeflateIdleMonitors
>>         >>>>> ("off"
>>         >>>>> mode). The slow down with "on" mode
>>         -XX:+AsyncDeflateIdleMonitors
>>         >>>>> is ~17%.
>>         >>>>>
>>         >>>>> I have attached the change list from CR10 to CR11
>>         instead of
>>         >>>>> putting it in
>>         >>>>> the body of this email. I've also added a link to the
>>         >>>>> CR10-to-CR11-changes
>>         >>>>> file to the webrevs so it should be easy to find.
>>         >>>>>
>>         >>>>> Main bug URL:
>>         >>>>>
>>         >>>>>     JDK-8153224 Monitor deflation prolong safepoints
>>         >>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>
>>         >>>>> The project is currently baselined on jdk-15+21.
>>         >>>>>
>>         >>>>> Here's the full webrev URL for those folks that want to
>>         see all of
>>         >>>>> the
>>         >>>>> current Async Monitor Deflation code in one go (v2.11
>>         full):
>>         >>>>>
>>         >>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/14-for-jdk15%2b21.v2.11.full/
>>
>>         >>>>>
>>         >>>>>
>>         >>>>> Some folks might want to see just what has changed
>>         since the last
>>         >>>>> review
>>         >>>>> cycle so here's a webrev for that (v2.11 inc):
>>         >>>>>
>>         >>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/14-for-jdk15%2b21.v2.11.inc/
>>
>>         >>>>>
>>         >>>>>
>>         >>>>> Because of the removal of ref_counting and the
>>         ObjectMonitorHandle
>>         >>>>> class, the
>>         >>>>> incremental webrev is a bit noisier than I would have
>>         preferred.
>>         >>>>>
>>         >>>>>
>>         >>>>> The OpenJDK wiki has NOT YET been updated for this
>>         round of changes:
>>         >>>>>
>>         >>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>         >>>>>
>>         >>>>> The jdk-15+21 based v2.11 version of the patch has been
>>         thru Mach5
>>         >>>>> tier[1-6]
>>         >>>>> testing on Oracle's usual set of platforms. Mach5
>>         tier[78] are
>>         >>>>> still running.
>>         >>>>> I'm running the v2.11 patch through my usual set of
>>         stress testing on
>>         >>>>> Linux-X64 and macOSX.
>>         >>>>>
>>         >>>>> I'm planning to do a SPECjbb2015, DaCapo-h2 and volano
>>         round on the
>>         >>>>> CR11/v2.11/14-for-jdk15 bits.
>>         >>>>>
>>         >>>>> Thanks, in advance, for any questions, comments or
>>         suggestions.
>>         >>>>>
>>         >>>>> Dan
>>         >>>>>
>>         >>>>>
>>         >>>>> On 2/26/20 5:22 PM, Daniel D. Daugherty wrote:
>>         >>>>>> Greetings,
>>         >>>>>>
>>         >>>>>> I have made changes to the Async Monitor Deflation
>>         code in
>>         >>>>>> response to
>>         >>>>>> the CR9/v2.09/12-for-jdk14 code review cycle. Thanks
>>         to Robbin
>>         >>>>>> and Erik O.
>>         >>>>>> for their comments in this round!
>>         >>>>>>
>>         >>>>>> With the extraction and push of
>>         {8235931,8236035,8235795} to
>>         >>>>>> JDK15, the
>>         >>>>>> Async Monitor Deflation code is back to "just" async
>>         deflation
>>         >>>>>> changes!
>>         >>>>>>
>>         >>>>>> I have attached the change list from CR9 to CR10
>>         instead of
>>         >>>>>> putting it in
>>         >>>>>> the body of this email. I've also added a link to the
>>         >>>>>> CR9-to-CR10-changes
>>         >>>>>> file to the webrevs so it should be easy to find.
>>         >>>>>>
>>         >>>>>> Main bug URL:
>>         >>>>>>
>>         >>>>>>     JDK-8153224 Monitor deflation prolong safepoints
>>         >>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>
>>         >>>>>> The project is currently baselined on jdk-15+11.
>>         >>>>>>
>>         >>>>>> Here's the full webrev URL for those folks that want
>>         to see all
>>         >>>>>> of the
>>         >>>>>> current Async Monitor Deflation code in one go (v2.10
>>         full):
>>         >>>>>>
>>         >>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/13-for-jdk15+11.v2.10.full/
>>
>>         >>>>>>
>>         >>>>>>
>>         >>>>>> Some folks might want to see just what has changed
>>         since the last
>>         >>>>>> review
>>         >>>>>> cycle so here's a webrev for that (v2.10 inc):
>>         >>>>>>
>>         >>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/13-for-jdk15+11.v2.10.inc/
>>
>>         >>>>>>
>>         >>>>>>
>>         >>>>>> Since we backed out the
>>         HandshakeAfterDeflateIdleMonitors option
>>         >>>>>> and the
>>         >>>>>> C2 ref_count changes and updated the copyright years,
>>         the "inc"
>>         >>>>>> webrev has
>>         >>>>>> a bit more noise in it than usual. Sorry about that!
>>         >>>>>>
>>         >>>>>> The OpenJDK wiki has been updated for this round of
>>         changes:
>>         >>>>>>
>>         >>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>
>>         >>>>>>
>>         >>>>>> The jdk-15+11 based v2.10 version of the patch has
>>         been thru
>>         >>>>>> Mach5 tier[1-7]
>>         >>>>>> testing on Oracle's usual set of platforms. Mach5
>>         tier8 is still
>>         >>>>>> running.
>>         >>>>>> I'm running the v2.10 patch through my usual set of
>>         stress
>>         >>>>>> testing on
>>         >>>>>> Linux-X64 and macOSX.
>>         >>>>>>
>>         >>>>>> I'm planning to do a SPECjbb2015 round on the
>>         >>>>>> CR10/v2.20/13-for-jdk15 bits.
>>         >>>>>>
>>         >>>>>> Thanks, in advance, for any questions, comments or
>>         suggestions.
>>         >>>>>>
>>         >>>>>> Dan
>>         >>>>>>
>>         >>>>>>
>>         >>>>>> On 2/4/20 9:41 AM, Daniel D. Daugherty wrote:
>>         >>>>>>> Greetings,
>>         >>>>>>>
>>         >>>>>>> This project is no longer targeted to JDK14 so this
>>         is NOT an
>>         >>>>>>> urgent code
>>         >>>>>>> review request.
>>         >>>>>>>
>>         >>>>>>> I've extracted the following three fixes from the
>>         Async Monitor
>>         >>>>>>> Deflation
>>         >>>>>>> project code:
>>         >>>>>>>
>>         >>>>>>>     JDK-8235931 add OM_CACHE_LINE_SIZE and use
>>         smaller size on
>>         >>>>>>> SPARCv9 and X64
>>         >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8235931
>>         >>>>>>>
>>         >>>>>>>     JDK-8236035 refactor ObjectMonitor::set_owner()
>>         and _owner
>>         >>>>>>> field setting
>>         >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8236035
>>         >>>>>>>
>>         >>>>>>>     JDK-8235795 replace monitor list
>>         >>>>>>> mux{Acquire,Release}(&gListLock) with spin locks
>>         >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8235795
>>         >>>>>>>
>>         >>>>>>> Each of these has been reviewed separately and will
>>         be pushed to
>>         >>>>>>> JDK15
>>         >>>>>>> in the near future (possibly by the end of this
>>         week). Of
>>         >>>>>>> course, there
>>         >>>>>>> were improvements during these review cycles and the
>>         purpose of
>>         >>>>>>> this
>>         >>>>>>> e-mail is to provided updated webrevs for this fix
>>         >>>>>>> (CR9/v2.09/12-for-jdk14)
>>         >>>>>>> within the revised context provided by {8235931,
>>         8236035, 8235795}.
>>         >>>>>>>
>>         >>>>>>> Main bug URL:
>>         >>>>>>>
>>         >>>>>>>     JDK-8153224 Monitor deflation prolong safepoints
>>         >>>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>
>>         >>>>>>> The project is currently baselined on jdk-14+34.
>>         >>>>>>>
>>         >>>>>>> Here's the full webrev URL for those folks that want
>>         to see all
>>         >>>>>>> of the
>>         >>>>>>> current Async Monitor Deflation code along with
>>         {8235931,
>>         >>>>>>> 8236035, 8235795}
>>         >>>>>>> in one go (v2.09b full):
>>         >>>>>>>
>>         >>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/12-for-jdk14.v2.09b.full/
>>
>>         >>>>>>>
>>         >>>>>>>
>>         >>>>>>> Compare the open.patch file in
>>         12-for-jdk14.v2.09.full and
>>         >>>>>>> 12-for-jdk14.v2.09b.full
>>         >>>>>>> using your favorite file comparison/merge tool to see
>>         how Async
>>         >>>>>>> Monitor Deflation
>>         >>>>>>> evolved due to {8235931, 8236035, 8235795}.
>>         >>>>>>>
>>         >>>>>>> Some folks might want to see just the Async Monitor
>>         Deflation
>>         >>>>>>> code on top of
>>         >>>>>>> {8235931, 8236035, 8235795} so here's a webrev for
>>         that (v2.09b
>>         >>>>>>> inc):
>>         >>>>>>>
>>         >>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/12-for-jdk14.v2.09b.inc/
>>
>>         >>>>>>>
>>         >>>>>>>
>>         >>>>>>> These webrevs have gone thru several Mach5 Tier[1-8]
>>         runs along
>>         >>>>>>> with
>>         >>>>>>> my usual stress testing and SPECjbb2015 testing and
>>         there aren't
>>         >>>>>>> any
>>         >>>>>>> surprises relative to CR9/v2.09/12-for-jdk14.
>>         >>>>>>>
>>         >>>>>>> Thanks, in advance, for any questions, comments or
>>         suggestions.
>>         >>>>>>>
>>         >>>>>>> Dan
>>         >>>>>>>
>>         >>>>>>>
>>         >>>>>>> On 12/11/19 3:41 PM, Daniel D. Daugherty wrote:
>>         >>>>>>>> Greetings,
>>         >>>>>>>>
>>         >>>>>>>> I have made changes to the Async Monitor Deflation
>>         code in
>>         >>>>>>>> response to
>>         >>>>>>>> the CR8/v2.08/11-for-jdk14 code review cycle. Thanks
>>         to David
>>         >>>>>>>> H., Robbin
>>         >>>>>>>> and Erik O. for their comments!
>>         >>>>>>>>
>>         >>>>>>>> This project is no longer targeted to JDK14 so this
>>         is NOT an
>>         >>>>>>>> urgent code
>>         >>>>>>>> review request. The primary purpose of this webrev
>>         is simply to
>>         >>>>>>>> close the
>>         >>>>>>>> CR8/v2.08/11-for-jdk14 code review loop and to let
>>         folks see
>>         >>>>>>>> how I resolved
>>         >>>>>>>> the code review comments from that round.
>>         >>>>>>>>
>>         >>>>>>>> Most of the comments in the CR8/v2.08/11-for-jdk14
>>         code review
>>         >>>>>>>> cycle were
>>         >>>>>>>> on the monitor list changes so I'm going to take a
>>         look at
>>         >>>>>>>> extracting those
>>         >>>>>>>> changes into a standalone patch. Switching from
>>         >>>>>>>> Thread::muxAcquire(&gListLock)
>>         >>>>>>>> and Thread::muxRelease(&gListLock) to finer grained
>>         internal
>>         >>>>>>>> spin locks needs
>>         >>>>>>>> to be thoroughly reviewed and the best way to do
>>         that is
>>         >>>>>>>> separately from the
>>         >>>>>>>> Async Monitor Deflation changes. Thanks to Coleen for
>>         >>>>>>>> suggesting doing this
>>         >>>>>>>> extraction earlier.
>>         >>>>>>>>
>>         >>>>>>>> I have attached the change list from CR8 to CR9
>>         instead of
>>         >>>>>>>> putting it in
>>         >>>>>>>> the body of this email. I've also added a link to the
>>         >>>>>>>> CR8-to-CR9-changes
>>         >>>>>>>> file to the webrevs so it should be easy to find.
>>         >>>>>>>>
>>         >>>>>>>> Main bug URL:
>>         >>>>>>>>
>>         >>>>>>>>     JDK-8153224 Monitor deflation prolong safepoints
>>         >>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>>
>>         >>>>>>>> The project is currently baselined on jdk-14+26.
>>         >>>>>>>>
>>         >>>>>>>> Here's the full webrev URL for those folks that want
>>         to see all
>>         >>>>>>>> of the
>>         >>>>>>>> current Async Monitor Deflation code in one go
>>         (v2.09 full):
>>         >>>>>>>>
>>         >>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/12-for-jdk14.v2.09.full/
>>
>>         >>>>>>>>
>>         >>>>>>>>
>>         >>>>>>>> Some folks might want to see just what has changed
>>         since the
>>         >>>>>>>> last review
>>         >>>>>>>> cycle so here's a webrev for that (v2.09 inc):
>>         >>>>>>>>
>>         >>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/12-for-jdk14.v2.09.inc/
>>
>>         >>>>>>>>
>>         >>>>>>>>
>>         >>>>>>>> The OpenJDK wiki has NOT yet been updated for this
>>         round of
>>         >>>>>>>> changes:
>>         >>>>>>>>
>>         >>>>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>>>
>>         >>>>>>>>
>>         >>>>>>>> The jdk-14+26 based v2.09 version of the patch has
>>         been thru
>>         >>>>>>>> Mach5 tier[1-7]
>>         >>>>>>>> testing on Oracle's usual set of platforms. Mach5
>>         tier8 is
>>         >>>>>>>> still running.
>>         >>>>>>>> A slightly older version of the v2.09 patch has also
>>         been
>>         >>>>>>>> through my usual
>>         >>>>>>>> set of stress testing on Linux-X64 and macOSX with
>>         the addition
>>         >>>>>>>> of Robbin's
>>         >>>>>>>> "MoCrazy 1024" test running in parallel on Linux-X64
>>         with the
>>         >>>>>>>> other tests in
>>         >>>>>>>> my lab. The "MoCrazy 1024" has been going for > 5
>>         days and
>>         >>>>>>>> 6700+ iterations
>>         >>>>>>>> without any failures.
>>         >>>>>>>>
>>         >>>>>>>> I'm planning to do a SPECjbb2015 round on the
>>         >>>>>>>> CR9/v2.09/12-for-jdk14 bits.
>>         >>>>>>>>
>>         >>>>>>>> Thanks, in advance, for any questions, comments or
>>         suggestions.
>>         >>>>>>>>
>>         >>>>>>>> Dan
>>         >>>>>>>>
>>         >>>>>>>>
>>         >>>>>>>> On 11/4/19 4:03 PM, Daniel D. Daugherty wrote:
>>         >>>>>>>>> Greetings,
>>         >>>>>>>>>
>>         >>>>>>>>> I have made changes to the Async Monitor Deflation
>>         code in
>>         >>>>>>>>> response to
>>         >>>>>>>>> the CR7/v2.07/10-for-jdk14 code review cycle.
>>         Thanks to David
>>         >>>>>>>>> H., Robbin
>>         >>>>>>>>> and Erik O. for their comments!
>>         >>>>>>>>>
>>         >>>>>>>>> JDK14 Rampdown phase one is coming on Dec. 12, 2019
>>         and the
>>         >>>>>>>>> Async Monitor
>>         >>>>>>>>> Deflation project needs to push before Nov. 12,
>>         2019 in order
>>         >>>>>>>>> to allow
>>         >>>>>>>>> for sufficient bake time for such a big change.
>>         Nov. 12 is
>>         >>>>>>>>> _next_ Tuesday
>>         >>>>>>>>> so we have 8 days from today to finish this code
>>         review cycle
>>         >>>>>>>>> and push
>>         >>>>>>>>> this code for JDK14.
>>         >>>>>>>>>
>>         >>>>>>>>> Carsten and Roman! Time for you guys to chime in
>>         again on the
>>         >>>>>>>>> code reviews.
>>         >>>>>>>>>
>>         >>>>>>>>> I have attached the change list from CR7 to CR8
>>         instead of
>>         >>>>>>>>> putting it in
>>         >>>>>>>>> the body of this email. I've also added a link to the
>>         >>>>>>>>> CR7-to-CR8-changes
>>         >>>>>>>>> file to the webrevs so it should be easy to find.
>>         >>>>>>>>>
>>         >>>>>>>>> Main bug URL:
>>         >>>>>>>>>
>>         >>>>>>>>>     JDK-8153224 Monitor deflation prolong safepoints
>>         >>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>>>
>>         >>>>>>>>> The project is currently baselined on jdk-14+21.
>>         >>>>>>>>>
>>         >>>>>>>>> Here's the full webrev URL for those folks that
>>         want to see
>>         >>>>>>>>> all of the
>>         >>>>>>>>> current Async Monitor Deflation code in one go
>>         (v2.08 full):
>>         >>>>>>>>>
>>         >>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/11-for-jdk14.v2.08.full
>>
>>         >>>>>>>>>
>>         >>>>>>>>>
>>         >>>>>>>>> Some folks might want to see just what has changed
>>         since the
>>         >>>>>>>>> last review
>>         >>>>>>>>> cycle so here's a webrev for that (v2.08 inc):
>>         >>>>>>>>>
>>         >>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/11-for-jdk14.v2.08.inc/
>>
>>         >>>>>>>>>
>>         >>>>>>>>>
>>         >>>>>>>>> The OpenJDK wiki did not need any changes for this
>>         round:
>>         >>>>>>>>>
>>         >>>>>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>>>>
>>         >>>>>>>>>
>>         >>>>>>>>> The jdk-14+21 based v2.08 version of the patch has
>>         been thru
>>         >>>>>>>>> Mach5 tier[1-8]
>>         >>>>>>>>> testing on Oracle's usual set of platforms. It has
>>         also been
>>         >>>>>>>>> through my usual
>>         >>>>>>>>> set of stress testing on Linux-X64, macOSX and
>>         Solaris-X64
>>         >>>>>>>>> with the addition
>>         >>>>>>>>> of Robbin's "MoCrazy 1024" test running in parallel
>>         with the
>>         >>>>>>>>> other tests in
>>         >>>>>>>>> my lab. Some testing is still running, but so far
>>         there are no
>>         >>>>>>>>> new regressions.
>>         >>>>>>>>>
>>         >>>>>>>>> I have not yet done a SPECjbb2015 round on the
>>         >>>>>>>>> CR8/v2.08/11-for-jdk14 bits.
>>         >>>>>>>>>
>>         >>>>>>>>> Thanks, in advance, for any questions, comments or
>>         suggestions.
>>         >>>>>>>>>
>>         >>>>>>>>> Dan
>>         >>>>>>>>>
>>         >>>>>>>>>
>>         >>>>>>>>> On 10/17/19 5:50 PM, Daniel D. Daugherty wrote:
>>         >>>>>>>>>> Greetings,
>>         >>>>>>>>>>
>>         >>>>>>>>>> The Async Monitor Deflation project is reaching
>>         the end game.
>>         >>>>>>>>>> I have no
>>         >>>>>>>>>> changes planned for the project at this time so
>>         all that is
>>         >>>>>>>>>> left is code
>>         >>>>>>>>>> review and any changes that results from those
>>         reviews.
>>         >>>>>>>>>>
>>         >>>>>>>>>> Carsten and Roman! Time for you guys to chime in
>>         again on the
>>         >>>>>>>>>> code reviews.
>>         >>>>>>>>>>
>>         >>>>>>>>>> I have attached the list of fixes from CR6 to CR7
>>         instead of
>>         >>>>>>>>>> putting it
>>         >>>>>>>>>> in the main body of this email.
>>         >>>>>>>>>>
>>         >>>>>>>>>> Main bug URL:
>>         >>>>>>>>>>
>>         >>>>>>>>>> JDK-8153224 Monitor deflation prolong safepoints
>>         >>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>>>>
>>         >>>>>>>>>> The project is currently baselined on jdk-14+19.
>>         >>>>>>>>>>
>>         >>>>>>>>>> Here's the full webrev URL for those folks that
>>         want to see
>>         >>>>>>>>>> all of the
>>         >>>>>>>>>> current Async Monitor Deflation code in one go
>>         (v2.07 full):
>>         >>>>>>>>>>
>>         >>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/10-for-jdk14.v2.07.full
>>
>>         >>>>>>>>>>
>>         >>>>>>>>>>
>>         >>>>>>>>>> Some folks might want to see just what has changed
>>         since the
>>         >>>>>>>>>> last review
>>         >>>>>>>>>> cycle so here's a webrev for that (v2.07 inc):
>>         >>>>>>>>>>
>>         >>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/10-for-jdk14.v2.07.inc/
>>
>>         >>>>>>>>>>
>>         >>>>>>>>>>
>>         >>>>>>>>>> The OpenJDK wiki has been updated to match the
>>         >>>>>>>>>> CR7/v2.07/10-for-jdk14 changes:
>>         >>>>>>>>>>
>>         >>>>>>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>>>>>
>>         >>>>>>>>>>
>>         >>>>>>>>>> The jdk-14+18 based v2.07 version of the patch has
>>         been thru
>>         >>>>>>>>>> Mach5 tier[1-8]
>>         >>>>>>>>>> testing on Oracle's usual set of platforms. It has
>>         also been
>>         >>>>>>>>>> through my usual
>>         >>>>>>>>>> set of stress testing on Linux-X64, macOSX and
>>         Solaris-X64
>>         >>>>>>>>>> with the addition
>>         >>>>>>>>>> of Robbin's "MoCrazy 1024" test running in
>>         parallel with the
>>         >>>>>>>>>> other tests in
>>         >>>>>>>>>> my lab.
>>         >>>>>>>>>>
>>         >>>>>>>>>> The jdk-14+19 based v2.07 version of the patch has
>>         been thru
>>         >>>>>>>>>> Mach5 tier[1-3]
>>         >>>>>>>>>> test on Oracle's usual set of platforms. Mach5
>>         tier[4-8] are
>>         >>>>>>>>>> in process.
>>         >>>>>>>>>>
>>         >>>>>>>>>> I did another round of SPECjbb2015 testing in
>>         Oracle's Aurora
>>         >>>>>>>>>> Performance lab
>>         >>>>>>>>>> using using their tuned SPECjbb2015 Linux-X64 G1
>>         configs:
>>         >>>>>>>>>>
>>         >>>>>>>>>>     - "base" is jdk-14+18
>>         >>>>>>>>>>     - "v2.07" is the latest version and includes C2
>>         >>>>>>>>>> inc_om_ref_count() support
>>         >>>>>>>>>>       on LP64 X64 and the new
>>         >>>>>>>>>> HandshakeAfterDeflateIdleMonitors option
>>         >>>>>>>>>>     - "off" is with -XX:-AsyncDeflateIdleMonitors
>>         specified
>>         >>>>>>>>>>     - "handshake" is with
>>         >>>>>>>>>> -XX:+HandshakeAfterDeflateIdleMonitors specified
>>         >>>>>>>>>>
>>         >>>>>>>>>> hbIR           hbIR
>>         >>>>>>>>>>     (max attempted)  (settled)  max-jOPS
>>         critical-jOPS runtime
>>         >>>>>>>>>> ---------------  ---------  -------- -------------
>>         -------
>>         >>>>>>>>>> 34282.00   30635.90  28831.30 20969.20 3841.30 base
>>         >>>>>>>>>> 34282.00   30973.00  29345.80 21025.20 3964.10 v2.07
>>         >>>>>>>>>> 34282.00   31105.60  29174.30 21074.00 3931.30
>>         >>>>>>>>>> v2.07_handshake
>>         >>>>>>>>>> 34282.00   30789.70  27151.60 19839.10 3850.20
>>         >>>>>>>>>> v2.07_off
>>         >>>>>>>>>>
>>         >>>>>>>>>>     - The Aurora Perf comparison tool reports:
>>         >>>>>>>>>>
>>         >>>>>>>>>> Comparison              max-jOPS critical-jOPS
>>         >>>>>>>>>> ---------------------- --------------------
>>         >>>>>>>>>> --------------------
>>         >>>>>>>>>> base vs 2.07            +1.78% (s, p=0.000) +0.27%
>>         >>>>>>>>>> (ns, p=0.790)
>>         >>>>>>>>>> base vs 2.07_handshake  +1.19% (s, p=0.007) +0.58%
>>         >>>>>>>>>> (ns, p=0.536)
>>         >>>>>>>>>> base vs 2.07_off        -5.83% (ns, p=0.394) -5.39%
>>         >>>>>>>>>> (ns, p=0.347)
>>         >>>>>>>>>>
>>         >>>>>>>>>>         (s) - significant  (ns) - not-significant
>>         >>>>>>>>>>
>>         >>>>>>>>>>     - For historical comparison, the Aurora Perf
>>         comparision
>>         >>>>>>>>>> tool
>>         >>>>>>>>>> reported for v2.06 with a baseline of jdk-13+31:
>>         >>>>>>>>>>
>>         >>>>>>>>>> Comparison              max-jOPS critical-jOPS
>>         >>>>>>>>>> ---------------------- --------------------
>>         >>>>>>>>>> --------------------
>>         >>>>>>>>>> base vs 2.06            -0.32% (ns, p=0.345) +0.71%
>>         >>>>>>>>>> (ns, p=0.646)
>>         >>>>>>>>>> base vs 2.06_off        +0.49% (ns, p=0.292) -1.21%
>>         >>>>>>>>>> (ns, p=0.481)
>>         >>>>>>>>>>
>>         >>>>>>>>>>         (s) - significant  (ns) - not-significant
>>         >>>>>>>>>>
>>         >>>>>>>>>> Thanks, in advance, for any questions, comments or
>>         suggestions.
>>         >>>>>>>>>>
>>         >>>>>>>>>> Dan
>>         >>>>>>>>>>
>>         >>>>>>>>>>
>>         >>>>>>>>>> On 8/28/19 5:02 PM, Daniel D. Daugherty wrote:
>>         >>>>>>>>>>> Greetings,
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> The Async Monitor Deflation project has rebased
>>         to JDK14 so
>>         >>>>>>>>>>> it's time
>>         >>>>>>>>>>> for our first code review in that new context!!
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> I've been focused on changing the monitor list
>>         management
>>         >>>>>>>>>>> code to be
>>         >>>>>>>>>>> lock-free in order to make SPECjbb2015 happier.
>>         Of course
>>         >>>>>>>>>>> with a change
>>         >>>>>>>>>>> like that, it takes a while to chase down all the
>>         new and
>>         >>>>>>>>>>> wonderful
>>         >>>>>>>>>>> races. At this point, I have the code back to the
>>         same
>>         >>>>>>>>>>> stability that
>>         >>>>>>>>>>> I had with CR5/v2.05/8-for-jdk13.
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> To lay the ground work for this round of review,
>>         I pushed
>>         >>>>>>>>>>> the following
>>         >>>>>>>>>>> two fixes to jdk/jdk earlier today:
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> JDK-8230184 rename, whitespace, indent and comments
>>         >>>>>>>>>>> changes in preparation
>>         >>>>>>>>>>>             for lock free Monitor lists
>>         >>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8230184
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> JDK-8230317
>>         serviceability/sa/ClhsdbPrintStatics.java
>>         >>>>>>>>>>> fails after 8230184
>>         >>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8230317
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> I have attached the list of fixes from CR5 to CR6
>>         instead of
>>         >>>>>>>>>>> putting
>>         >>>>>>>>>>> in the main body of this email.
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> Main bug URL:
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> JDK-8153224 Monitor deflation prolong safepoints
>>         >>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> The project is currently baselined on jdk-14+11
>>         plus the
>>         >>>>>>>>>>> fixes for
>>         >>>>>>>>>>> JDK-8230184 and JDK-8230317.
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> Here's the full webrev URL for those folks that
>>         want to see
>>         >>>>>>>>>>> all of the
>>         >>>>>>>>>>> current Async Monitor Deflation code in one go
>>         (v2.06 full):
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/9-for-jdk14.v2.06.full/
>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> The primary focus of this review cycle is on the
>>         lock-free
>>         >>>>>>>>>>> Monitor List
>>         >>>>>>>>>>> management changes so here's a webrev for just
>>         that patch
>>         >>>>>>>>>>> (v2.06c):
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/9-for-jdk14.v2.06c.inc/
>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> The secondary focus of this review cycle is on
>>         the bug fixes
>>         >>>>>>>>>>> that have
>>         >>>>>>>>>>> been made since CR5/v2.05/8-for-jdk13 so here's a
>>         webrev for
>>         >>>>>>>>>>> just that
>>         >>>>>>>>>>> patch (v2.06b):
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/9-for-jdk14.v2.06b.inc/
>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> The third and final bucket for this review cycle
>>         is the
>>         >>>>>>>>>>> rename, whitespace,
>>         >>>>>>>>>>> indent and comments changes made in preparation
>>         for lock
>>         >>>>>>>>>>> free Monitor list
>>         >>>>>>>>>>> management. Almost all of that was extracted into
>>         >>>>>>>>>>> JDK-8230184 for the
>>         >>>>>>>>>>> baseline so this bucket now has just a few
>>         comment changes
>>         >>>>>>>>>>> relative to
>>         >>>>>>>>>>> CR5/v2.05/8-for-jdk13. Here's a webrev for the
>>         remainder
>>         >>>>>>>>>>> (v2.06a):
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/9-for-jdk14.v2.06a.inc/
>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> Some folks might want to see just what has
>>         changed since the
>>         >>>>>>>>>>> last review
>>         >>>>>>>>>>> cycle so here's a webrev for that (v2.06 inc):
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/9-for-jdk14.v2.06.inc/
>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> Last, but not least, some folks might want to see
>>         the code
>>         >>>>>>>>>>> before the
>>         >>>>>>>>>>> addition of lock-free Monitor List management so
>>         here's a
>>         >>>>>>>>>>> webrev for
>>         >>>>>>>>>>> that (v2.00 -> v2.05):
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/9-for-jdk14.v2.05.inc/
>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> The OpenJDK wiki will need minor updates to match
>>         the CR6
>>         >>>>>>>>>>> changes:
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> but that should only be changes to describe
>>         per-thread list
>>         >>>>>>>>>>> async monitor
>>         >>>>>>>>>>> deflation being done by the ServiceThread.
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> (I did update the OpenJDK wiki for the CR5
>>         changes back on
>>         >>>>>>>>>>> 2019.08.14)
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> This version of the patch has been thru Mach5
>>         tier[1-8]
>>         >>>>>>>>>>> testing on
>>         >>>>>>>>>>> Oracle's usual set of platforms. It has also been
>>         through my
>>         >>>>>>>>>>> usual set
>>         >>>>>>>>>>> of stress testing on Linux-X64, macOSX and
>>         Solaris-X64.
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> I did a bunch of SPECjbb2015 testing in Oracle's
>>         Aurora
>>         >>>>>>>>>>> Performance lab
>>         >>>>>>>>>>> using using their tuned SPECjbb2015 Linux-X64 G1
>>         configs.
>>         >>>>>>>>>>> This was using
>>         >>>>>>>>>>> this patch baselined on jdk-13+31 (for stability):
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>           hbIR           hbIR
>>         >>>>>>>>>>> (max attempted)  (settled)  max-jOPS
>>         critical-jOPS runtime
>>         >>>>>>>>>>> ---------------  ---------  --------
>>         ------------- -------
>>         >>>>>>>>>>>             34282.00   28837.20  27905.20
>>         19817.40 3658.10 base
>>         >>>>>>>>>>>             34965.70   29798.80  27814.90
>>         19959.00 3514.60
>>         >>>>>>>>>>> v2.06d
>>         >>>>>>>>>>>             34282.00   29100.70  28042.50
>>         19577.00 3701.90
>>         >>>>>>>>>>> v2.06d_off
>>         >>>>>>>>>>>             34282.00   29218.50  27562.80
>>         19397.30 3657.60
>>         >>>>>>>>>>> v2.06d_ocache
>>         >>>>>>>>>>>             34965.70   29838.30  26512.40
>>         19170.60 3569.90
>>         >>>>>>>>>>> v2.05
>>         >>>>>>>>>>>             34282.00   28926.10  27734.00
>>         19835.10 3588.40
>>         >>>>>>>>>>> v2.05_off
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> The "off" configs are with
>>         -XX:-AsyncDeflateIdleMonitors
>>         >>>>>>>>>>> specified and
>>         >>>>>>>>>>> the "ocache" config is with 128 byte cache line
>>         sizes
>>         >>>>>>>>>>> instead of 64 byte
>>         >>>>>>>>>>> cache lines sizes. "v2.06d" is the last set of
>>         changes that
>>         >>>>>>>>>>> I made before
>>         >>>>>>>>>>> those changes were distributed into the "v2.06a",
>>         "v2.06b"
>>         >>>>>>>>>>> and "v2.06c"
>>         >>>>>>>>>>> buckets for this review recycle.
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> Thanks, in advance, for any questions, comments
>>         or suggestions.
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> Dan
>>         >>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>> On 7/11/19 3:49 PM, Daniel D. Daugherty wrote:
>>         >>>>>>>>>>>> Greetings,
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> I've been focused on chasing down and fixing the
>>         rare test
>>         >>>>>>>>>>>> failures
>>         >>>>>>>>>>>> that only pop up rarely. So this round is
>>         primarily fixes
>>         >>>>>>>>>>>> for races
>>         >>>>>>>>>>>> with a few additional fixes that came from
>>         Karen's review
>>         >>>>>>>>>>>> of CR4.
>>         >>>>>>>>>>>> Thanks Karen!
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> I have attached the list of fixes from CR4 to
>>         CR5 instead
>>         >>>>>>>>>>>> of putting
>>         >>>>>>>>>>>> in the main body of this email.
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> Main bug URL:
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> JDK-8153224 Monitor deflation prolong safepoints
>>         >>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> The project is currently baselined on jdk-13+29.
>>         This will
>>         >>>>>>>>>>>> likely be
>>         >>>>>>>>>>>> the last JDK13 baseline for this project and
>>         I'll roll to
>>         >>>>>>>>>>>> the JDK14
>>         >>>>>>>>>>>> (jdk/jdk) repo soon...
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> Here's the full webrev URL:
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/8-for-jdk13.full/
>>
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> Here's the incremental webrev URL:
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/8-for-jdk13.inc/
>>
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> I have not yet checked the OpenJDK wiki to see
>>         if it needs
>>         >>>>>>>>>>>> any updates
>>         >>>>>>>>>>>> to match the CR5 changes:
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> (I did update the OpenJDK wiki for the CR4
>>         changes back on
>>         >>>>>>>>>>>> 2019.06.26)
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> This version of the patch has been thru Mach5
>>         tier[1-3]
>>         >>>>>>>>>>>> testing on
>>         >>>>>>>>>>>> Oracle's usual set of platforms. Mach5 tier[4-6]
>>         is running
>>         >>>>>>>>>>>> now and
>>         >>>>>>>>>>>> Mach5 tier[78] will follow. I'll kick off the
>>         usual stress
>>         >>>>>>>>>>>> testing
>>         >>>>>>>>>>>> on Linux-X64, macOSX and Solaris-X64 as those
>>         machines
>>         >>>>>>>>>>>> become available.
>>         >>>>>>>>>>>> Since I haven't made any performance changes in
>>         this round,
>>         >>>>>>>>>>>> I'll only
>>         >>>>>>>>>>>> be running SPECjbb2015 to gather the latest
>>         >>>>>>>>>>>> monitorinflation logs.
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> Next up:
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> - We're still seeing 4-5% lower performance with
>>         >>>>>>>>>>>> SPECjbb2015 on
>>         >>>>>>>>>>>> Linux-X64 and we've determined that some of that
>>         comes from
>>         >>>>>>>>>>>> contention on the gListLock. So I'm going to
>>         investigate
>>         >>>>>>>>>>>> removing
>>         >>>>>>>>>>>> the gListLock. Yes, another lock free set of
>>         changes is
>>         >>>>>>>>>>>> coming!
>>         >>>>>>>>>>>> - Of course, going lock free often causes new
>>         races and new
>>         >>>>>>>>>>>> failures
>>         >>>>>>>>>>>> so that's a good reason for make those changes
>>         isolated
>>         >>>>>>>>>>>> in their
>>         >>>>>>>>>>>> own round (and not holding up CR5/v2.05/8-for-jdk13
>>         >>>>>>>>>>>> anymore).
>>         >>>>>>>>>>>> - I finally have a potential fix for the Win*
>>         failure with
>>         >>>>>>>>>>>> gc/g1/humongousObjects/TestHumongousClassLoader.java
>>         >>>>>>>>>>>> but I haven't run it through Mach5 yet so it'll
>>         be in the
>>         >>>>>>>>>>>> next round.
>>         >>>>>>>>>>>> - Some RTM tests were recently re-enabled in
>>         Mach5 and I'm
>>         >>>>>>>>>>>> seeing some
>>         >>>>>>>>>>>> monitor related failures there. I suspect that I
>>         need to
>>         >>>>>>>>>>>> go take a
>>         >>>>>>>>>>>> look at the C2 RTM macro assembler code and look
>>         for
>>         >>>>>>>>>>>> things that might
>>         >>>>>>>>>>>> conflict if Async Monitor Deflation. If you're
>>         interested
>>         >>>>>>>>>>>> in that kind
>>         >>>>>>>>>>>> of issue, then see the macroAssembler_x86.cpp
>>         sanity
>>         >>>>>>>>>>>> check that I
>>         >>>>>>>>>>>> added in this round!
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> Thanks, in advance, for any questions, comments or
>>         >>>>>>>>>>>> suggestions.
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> Dan
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>> On 5/26/19 8:30 PM, Daniel D. Daugherty wrote:
>>         >>>>>>>>>>>>> Greetings,
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> I have a fix for an issue that came up during
>>         performance
>>         >>>>>>>>>>>>> testing.
>>         >>>>>>>>>>>>> Many thanks to Robbin for diagnosing the issue
>>         in his
>>         >>>>>>>>>>>>> SPECjbb2015
>>         >>>>>>>>>>>>> experiments.
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> Here's the list of changes from CR3 to CR4. The
>>         list is a bit
>>         >>>>>>>>>>>>> verbose due to the complexity of the issue, but
>>         the changes
>>         >>>>>>>>>>>>> themselves are not that big.
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> Functional:
>>         >>>>>>>>>>>>>   - Change
>>         SafepointSynchronize::is_cleanup_needed() from
>>         >>>>>>>>>>>>> calling
>>         >>>>>>>>>>>>> ObjectSynchronizer::is_cleanup_needed() to calling
>>         >>>>>>>>>>>>>
>>         ObjectSynchronizer::is_safepoint_deflation_needed():
>>         >>>>>>>>>>>>>     - is_safepoint_deflation_needed() returns
>>         the result of
>>         >>>>>>>>>>>>>       monitors_used_above_threshold() for
>>         safepoint based
>>         >>>>>>>>>>>>>       monitor deflation
>>         (!AsyncDeflateIdleMonitors).
>>         >>>>>>>>>>>>>     - For AsyncDeflateIdleMonitors, it only
>>         returns true if
>>         >>>>>>>>>>>>>       there is a special deflation request,
>>         e.g., System.gc()
>>         >>>>>>>>>>>>>       - This solves a bug where there are a
>>         bunch of Cleanup
>>         >>>>>>>>>>>>>         safepoints that simply request async
>>         deflation which
>>         >>>>>>>>>>>>>         keeps the async JavaThreads from making
>>         progress on
>>         >>>>>>>>>>>>>         their async deflation work.
>>         >>>>>>>>>>>>>   - Add AsyncDeflationInterval diagnostic option.
>>         >>>>>>>>>>>>> Description:
>>         >>>>>>>>>>>>>       Async deflate idle monitors every so many
>>         >>>>>>>>>>>>> milliseconds when
>>         >>>>>>>>>>>>>       MonitorUsedDeflationThreshold is exceeded
>>         (0 is off).
>>         >>>>>>>>>>>>>   - Replace
>>         >>>>>>>>>>>>>
>>         ObjectSynchronizer::gOmShouldDeflateIdleMonitors() with
>>         >>>>>>>>>>>>> ObjectSynchronizer::is_async_deflation_needed():
>>         >>>>>>>>>>>>>     - is_async_deflation_needed() returns true when
>>         >>>>>>>>>>>>>       is_async_cleanup_requested() is true or when
>>         >>>>>>>>>>>>>       monitors_used_above_threshold() is true
>>         (but no more
>>         >>>>>>>>>>>>> often than
>>         >>>>>>>>>>>>>       AsyncDeflationInterval).
>>         >>>>>>>>>>>>>     - if AsyncDeflateIdleMonitors
>>         Service_lock->wait() now
>>         >>>>>>>>>>>>> waits for
>>         >>>>>>>>>>>>>       at most GuaranteedSafepointInterval millis:
>>         >>>>>>>>>>>>>       - This allows is_async_deflation_needed()
>>         to be
>>         >>>>>>>>>>>>> checked at
>>         >>>>>>>>>>>>>         the same interval as
>>         GuaranteedSafepointInterval.
>>         >>>>>>>>>>>>>         (default is 1000 millis/1 second)
>>         >>>>>>>>>>>>>       - Once is_async_deflation_needed() has
>>         returned
>>         >>>>>>>>>>>>> true, it
>>         >>>>>>>>>>>>>         generally cannot return true for
>>         >>>>>>>>>>>>> AsyncDeflationInterval.
>>         >>>>>>>>>>>>>         This is to prevent async deflation from
>>         swamping the
>>         >>>>>>>>>>>>>         ServiceThread.
>>         >>>>>>>>>>>>>   - The ServiceThread still handles async
>>         deflation of the
>>         >>>>>>>>>>>>> global
>>         >>>>>>>>>>>>>     in-use list and now it also marks
>>         JavaThreads for
>>         >>>>>>>>>>>>> async deflation
>>         >>>>>>>>>>>>>     of their in-use lists.
>>         >>>>>>>>>>>>>     - The ServiceThread will check for async
>>         deflation
>>         >>>>>>>>>>>>> work every
>>         >>>>>>>>>>>>>       GuaranteedSafepointInterval.
>>         >>>>>>>>>>>>>     - A safepoint can still cause the
>>         ServiceThread to
>>         >>>>>>>>>>>>> check for
>>         >>>>>>>>>>>>>       async deflation work via
>>         is_async_deflation_requested.
>>         >>>>>>>>>>>>>   - Refactor code from
>>         >>>>>>>>>>>>> ObjectSynchronizer::is_cleanup_needed() into
>>         >>>>>>>>>>>>>     monitors_used_above_threshold() and remove
>>         >>>>>>>>>>>>> is_cleanup_needed().
>>         >>>>>>>>>>>>>   - In addition to System.gc(), the VM_Exit VM
>>         op and the
>>         >>>>>>>>>>>>> final
>>         >>>>>>>>>>>>>     VMThread safepoint now set the
>>         >>>>>>>>>>>>> is_special_deflation_requested
>>         >>>>>>>>>>>>>     flag to reduce the in-use monitor
>>         population that is
>>         >>>>>>>>>>>>> reported by
>>         >>>>>>>>>>>>>
>>         ObjectSynchronizer::log_in_use_monitor_details() at VM exit.
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> Test update:
>>         >>>>>>>>>>>>>   - test/hotspot/gtest/oops/test_markOop.cpp is
>>         updated to
>>         >>>>>>>>>>>>> work with
>>         >>>>>>>>>>>>>     AsyncDeflateIdleMonitors.
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> Collateral:
>>         >>>>>>>>>>>>>   - Add/clarify/update some logging messages.
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> Cleanup:
>>         >>>>>>>>>>>>>   - Updated comments based on Karen's code review.
>>         >>>>>>>>>>>>>   - Change 'special cleanup' -> 'special
>>         deflation' and
>>         >>>>>>>>>>>>>     'async cleanup' -> 'async deflation'.
>>         >>>>>>>>>>>>>     - comment and function name changes
>>         >>>>>>>>>>>>>   - Clarify MonitorUsedDeflationThreshold
>>         description;
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> Main bug URL:
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>>     JDK-8153224 Monitor deflation prolong
>>         safepoints
>>         >>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> The project is currently baselined on jdk-13+22.
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> Here's the full webrev URL:
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/7-for-jdk13.full/
>>
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> Here's the incremental webrev URL:
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/7-for-jdk13.inc/
>>
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> I have not updated the OpenJDK wiki to reflect
>>         the CR4
>>         >>>>>>>>>>>>> changes:
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> The wiki doesn't say a whole lot about the
>>         async deflation
>>         >>>>>>>>>>>>> invocation
>>         >>>>>>>>>>>>> mechanism so I have to figure out how to add
>>         that content.
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> This version of the patch has been thru Mach5
>>         tier[1-8]
>>         >>>>>>>>>>>>> testing on
>>         >>>>>>>>>>>>> Oracle's usual set of platforms. My Solaris-X64
>>         stress kit
>>         >>>>>>>>>>>>> run is
>>         >>>>>>>>>>>>> running now. Kitchensink8H on product,
>>         fastdebug, and
>>         >>>>>>>>>>>>> slowdebug bits
>>         >>>>>>>>>>>>> are running on Linux-X64, MacOSX and
>>         Solaris-X64. I still
>>         >>>>>>>>>>>>> have to run
>>         >>>>>>>>>>>>> my stress kit on Linux-X64. I still have to run
>>         the
>>         >>>>>>>>>>>>> SPECjbb2015
>>         >>>>>>>>>>>>> baseline and CR4 runs on Linux-X64, MacOSX and
>>         Solaris-X64.
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> Thanks, in advance, for any questions, comments or
>>         >>>>>>>>>>>>> suggestions.
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> Dan
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>> On 5/6/19 11:52 AM, Daniel D. Daugherty wrote:
>>         >>>>>>>>>>>>>> Greetings,
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> I had some discussions with Karen about a race
>>         that was
>>         >>>>>>>>>>>>>> in the
>>         >>>>>>>>>>>>>> ObjectMonitor::enter() code in
>>         CR2/v2.02/5-for-jdk13.
>>         >>>>>>>>>>>>>> This race was
>>         >>>>>>>>>>>>>> theoretical and I had no test failures due to
>>         it. The fix
>>         >>>>>>>>>>>>>> is pretty
>>         >>>>>>>>>>>>>> simple: remove the special case code for async
>>         deflation
>>         >>>>>>>>>>>>>> in the
>>         >>>>>>>>>>>>>> ObjectMonitor::enter() function and rely
>>         solely on the
>>         >>>>>>>>>>>>>> ref_count
>>         >>>>>>>>>>>>>> for ObjectMonitor::enter() protection.
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> During those discussions Karen also floated
>>         the idea of
>>         >>>>>>>>>>>>>> using the
>>         >>>>>>>>>>>>>> ref_count field instead of the contentions
>>         field for the
>>         >>>>>>>>>>>>>> Async
>>         >>>>>>>>>>>>>> Monitor Deflation protocol. I decided to go
>>         ahead and
>>         >>>>>>>>>>>>>> code up that
>>         >>>>>>>>>>>>>> change and I have run it through the usual
>>         stress and
>>         >>>>>>>>>>>>>> Mach5 testing
>>         >>>>>>>>>>>>>> with no issues. It's also known as v2.03 (for
>>         those for
>>         >>>>>>>>>>>>>> with the
>>         >>>>>>>>>>>>>> patches) and as webrev/6-for-jdk13 (for those
>>         with webrev
>>         >>>>>>>>>>>>>> URLs).
>>         >>>>>>>>>>>>>> Sorry for all the names...
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> Main bug URL:
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>     JDK-8153224 Monitor deflation prolong
>>         safepoints
>>         >>>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> The project is currently baselined on jdk-13+18.
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> Here's the full webrev URL:
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/6-for-jdk13.full/
>>
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> Here's the incremental webrev URL:
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/6-for-jdk13.inc/
>>
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> I have also updated the OpenJDK wiki to
>>         reflect the CR3
>>         >>>>>>>>>>>>>> changes:
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> This version of the patch has been thru Mach5
>>         tier[1-8]
>>         >>>>>>>>>>>>>> testing on
>>         >>>>>>>>>>>>>> Oracle's usual set of platforms. My
>>         Solaris-X64 stress
>>         >>>>>>>>>>>>>> kit run had
>>         >>>>>>>>>>>>>> no issues. Kitchensink8H on product,
>>         fastdebug, and
>>         >>>>>>>>>>>>>> slowdebug bits
>>         >>>>>>>>>>>>>> had no failures on Linux-X64; MacOSX fastdebug
>>         and
>>         >>>>>>>>>>>>>> slowdebug and
>>         >>>>>>>>>>>>>> Solaris-X64 release had the usual "Too large
>>         time diff"
>>         >>>>>>>>>>>>>> complaints.
>>         >>>>>>>>>>>>>> 12 hour Inflate2 runs on product, fastdebug
>>         and slowdebug
>>         >>>>>>>>>>>>>> bits on
>>         >>>>>>>>>>>>>> Linux-X64, MacOSX and Solaris-X64 had no
>>         failures. My
>>         >>>>>>>>>>>>>> Linux-X64
>>         >>>>>>>>>>>>>> stress kit is running right now.
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> I've done the SPECjbb2015 baseline and CR3
>>         runs. I need
>>         >>>>>>>>>>>>>> to gather
>>         >>>>>>>>>>>>>> the results and analyze them.
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> Thanks, in advance, for any questions,
>>         comments or
>>         >>>>>>>>>>>>>> suggestions.
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> Dan
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>> On 4/25/19 12:38 PM, Daniel D. Daugherty wrote:
>>         >>>>>>>>>>>>>>> Greetings,
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> I have a small but important bug fix for the
>>         Async
>>         >>>>>>>>>>>>>>> Monitor Deflation
>>         >>>>>>>>>>>>>>> project ready to go. It's also known as v2.02
>>         (for those
>>         >>>>>>>>>>>>>>> for with the
>>         >>>>>>>>>>>>>>> patches) and as webrev/5-for-jdk13 (for those
>>         with
>>         >>>>>>>>>>>>>>> webrev URLs). Sorry
>>         >>>>>>>>>>>>>>> for all the names...
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> JDK-8222295 was pushed to jdk/jdk two days
>>         ago so that
>>         >>>>>>>>>>>>>>> baseline patch
>>         >>>>>>>>>>>>>>> is out of our hair.
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> Main bug URL:
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> JDK-8153224 Monitor deflation prolong safepoints
>>         >>>>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> The project is currently baselined on jdk-13+17.
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> Here's the full webrev URL:
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/5-for-jdk13.full/
>>
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> Here's the incremental webrev URL (JDK-8153224):
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/5-for-jdk13.inc/
>>
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> I still have to update the OpenJDK wiki to
>>         reflect the
>>         >>>>>>>>>>>>>>> CR2 changes:
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> This version of the patch has been thru Mach5
>>         tier[1-6]
>>         >>>>>>>>>>>>>>> testing on
>>         >>>>>>>>>>>>>>> Oracle's usual set of platforms. Mach5
>>         tier[7-8] is
>>         >>>>>>>>>>>>>>> running now.
>>         >>>>>>>>>>>>>>> My stress kit is running on Solaris-X64 now.
>>         >>>>>>>>>>>>>>> Kitchensink8H is running
>>         >>>>>>>>>>>>>>> now on product, fastdebug, and slowdebug bits on
>>         >>>>>>>>>>>>>>> Linux-X64, MacOSX
>>         >>>>>>>>>>>>>>> and Solaris-X64. 12 hour Inflate2 runs are
>>         running now
>>         >>>>>>>>>>>>>>> on product,
>>         >>>>>>>>>>>>>>> fastdebug and slowdebug bits on Linux-X64,
>>         MacOSX and
>>         >>>>>>>>>>>>>>> Solaris-X64.
>>         >>>>>>>>>>>>>>> I'll start my my stress kit on Linux-X64
>>         sometime on
>>         >>>>>>>>>>>>>>> Sunday (after
>>         >>>>>>>>>>>>>>> my jdk-13+18 stress run is done).
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> I'll do SPECjbb2015 baseline and CR2 runs
>>         after all the
>>         >>>>>>>>>>>>>>> stress
>>         >>>>>>>>>>>>>>> testing is done.
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> Thanks, in advance, for any questions,
>>         comments or
>>         >>>>>>>>>>>>>>> suggestions.
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> Dan
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>> On 4/19/19 11:58 AM, Daniel D. Daugherty wrote:
>>         >>>>>>>>>>>>>>>> Greetings,
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> I finally have CR1 for the Async Monitor
>>         Deflation
>>         >>>>>>>>>>>>>>>> project ready to
>>         >>>>>>>>>>>>>>>> go. It's also known as v2.01 (for those for
>>         with the
>>         >>>>>>>>>>>>>>>> patches) and as
>>         >>>>>>>>>>>>>>>> webrev/4-for-jdk13 (for those with webrev
>>         URLs). Sorry
>>         >>>>>>>>>>>>>>>> for all the
>>         >>>>>>>>>>>>>>>> names...
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> Main bug URL:
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> JDK-8153224 Monitor deflation prolong safepoints
>>         >>>>>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> Baseline bug fixes URL:
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> JDK-8222295 more baseline cleanups from Async
>>         >>>>>>>>>>>>>>>> Monitor Deflation project
>>         >>>>>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8222295
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> The project is currently baselined on jdk-13+15.
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> Here's the webrev for the latest baseline
>>         changes
>>         >>>>>>>>>>>>>>>> (JDK-8222295):
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/4-for-jdk13.8222295
>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> Here's the full webrev URL (JDK-8153224 only):
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/4-for-jdk13.full/
>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> Here's the incremental webrev URL (JDK-8153224):
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/4-for-jdk13.inc/
>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> So I'm looking for reviews for both
>>         JDK-8222295 and the
>>         >>>>>>>>>>>>>>>> latest version
>>         >>>>>>>>>>>>>>>> of JDK-8153224...
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> I still have to update the OpenJDK wiki to
>>         reflect the
>>         >>>>>>>>>>>>>>>> CR changes:
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> This version of the patch has been thru
>>         Mach5 tier[1-3]
>>         >>>>>>>>>>>>>>>> testing on
>>         >>>>>>>>>>>>>>>> Oracle's usual set of platforms. Mach5
>>         tier[4-6] is
>>         >>>>>>>>>>>>>>>> running now and
>>         >>>>>>>>>>>>>>>> Mach5 tier[78] will be run later today. My
>>         stress kit
>>         >>>>>>>>>>>>>>>> on Solaris-X64
>>         >>>>>>>>>>>>>>>> is running now. Linux-X64 stress testing
>>         will start on
>>         >>>>>>>>>>>>>>>> Sunday. I'm
>>         >>>>>>>>>>>>>>>> planning to do Kitchensink runs, SPECjbb2015
>>         runs and
>>         >>>>>>>>>>>>>>>> my monitor
>>         >>>>>>>>>>>>>>>> inflation stress tests on Linux-X64, MacOSX and
>>         >>>>>>>>>>>>>>>> Solaris-X64.
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> Thanks, in advance, for any questions,
>>         comments or
>>         >>>>>>>>>>>>>>>> suggestions.
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> Dan
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>> On 3/24/19 9:57 AM, Daniel D. Daugherty wrote:
>>         >>>>>>>>>>>>>>>>> Greetings,
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> Welcome to the OpenJDK review thread for my
>>         port of
>>         >>>>>>>>>>>>>>>>> Carsten's work on:
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> JDK-8153224 Monitor deflation prolong
>>         safepoints
>>         >>>>>>>>>>>>>>>>>
>>         https://bugs.openjdk.java.net/browse/JDK-8153224
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> Here's a link to the OpenJDK wiki that
>>         describes my port:
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>>
>>         https://wiki.openjdk.java.net/display/HotSpot/Async+Monitor+Deflation
>>
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> Here's the webrev URL:
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~dcubed/8153224-webrev/3-for-jdk13/
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> Here's a link to Carsten's original webrev:
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>>
>>         http://cr.openjdk.java.net/~cvarming/monitor_deflate_conc/0/
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> Earlier versions of this patch have been
>>         through
>>         >>>>>>>>>>>>>>>>> several rounds of
>>         >>>>>>>>>>>>>>>>> preliminary review. Many thanks to Carsten,
>>         Coleen,
>>         >>>>>>>>>>>>>>>>> Robbin, and
>>         >>>>>>>>>>>>>>>>> Roman for their preliminary code review
>>         comments. A
>>         >>>>>>>>>>>>>>>>> very special
>>         >>>>>>>>>>>>>>>>> thanks to Robbin and Roman for building and
>>         testing
>>         >>>>>>>>>>>>>>>>> the patch in
>>         >>>>>>>>>>>>>>>>> their own environments (including specJBB2015).
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> This version of the patch has been thru Mach5
>>         >>>>>>>>>>>>>>>>> tier[1-8] testing on
>>         >>>>>>>>>>>>>>>>> Oracle's usual set of platforms. Earlier
>>         versions have
>>         >>>>>>>>>>>>>>>>> been run
>>         >>>>>>>>>>>>>>>>> through my stress kit on my Linux-X64 and
>>         Solaris-X64
>>         >>>>>>>>>>>>>>>>> servers
>>         >>>>>>>>>>>>>>>>> (product, fastdebug, slowdebug).Earlier
>>         versions have
>>         >>>>>>>>>>>>>>>>> run Kitchensink
>>         >>>>>>>>>>>>>>>>> for 12 hours on MacOSX, Linux-X64 and
>>         Solaris-X64
>>         >>>>>>>>>>>>>>>>> (product, fastdebug
>>         >>>>>>>>>>>>>>>>> and slowdebug). Earlier versions have run
>>         my monitor
>>         >>>>>>>>>>>>>>>>> inflation stress
>>         >>>>>>>>>>>>>>>>> tests for 12 hours on MacOSX, Linux-X64 and
>>         >>>>>>>>>>>>>>>>> Solaris-X64 (product,
>>         >>>>>>>>>>>>>>>>> fastdebug and slowdebug).
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> All of the testing done on earlier versions
>>         will be
>>         >>>>>>>>>>>>>>>>> redone on the
>>         >>>>>>>>>>>>>>>>> latest version of the patch.
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> Thanks, in advance, for any questions,
>>         comments or
>>         >>>>>>>>>>>>>>>>> suggestions.
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> Dan
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>> P.S.
>>         >>>>>>>>>>>>>>>>> One subtest in
>>         >>>>>>>>>>>>>>>>>
>>         gc/g1/humongousObjects/TestHumongousClassLoader.java
>>         >>>>>>>>>>>>>>>>> is currently failing in -Xcomp mode on Win*
>>         only. I've
>>         >>>>>>>>>>>>>>>>> been trying
>>         >>>>>>>>>>>>>>>>> to characterize/analyze this failure for
>>         more than a
>>         >>>>>>>>>>>>>>>>> week now. At
>>         >>>>>>>>>>>>>>>>> this point I'm convinced that Async Monitor
>>         Deflation
>>         >>>>>>>>>>>>>>>>> is aggravating
>>         >>>>>>>>>>>>>>>>> an existing bug. However, I plan to have a
>>         better
>>         >>>>>>>>>>>>>>>>> handle on that
>>         >>>>>>>>>>>>>>>>> failure before these bits are pushed to the
>>         jdk/jdk repo.
>>         >>>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>>
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>>
>>         >>>>>>>>>>>>
>>         >>>>>>>>>>>
>>         >>>>>>>>>>
>>         >>>>>>>>>
>>         >>>>>>>>
>>         >>>>>>>
>>         >>>>>>
>>         >>>>>
>>         >>>>
>>         >>>
>>         >>
>>
>



More information about the hotspot-runtime-dev mailing list