RFR(S): 8027593: performance drop with constrained codecache starting with hs25 b111
Vladimir Kozlov
vladimir.kozlov at oracle.com
Fri Nov 8 10:08:57 PST 2013
Hi, Ramki
I doubt we will backport to 7uXX CodeCache work we did in jdk8. We start working on jdk9 soon and jdk7u is moved to
sustaining state. Unless our main 'customer' ask us to do that but I did not hear complains from them.
For 7u the only solution is increase reserved CodeCache and use UseCodeCacheFlushing (which is ON by default since
7u40). You may also have to increase CodeCacheFlushingMinimumFreeSpace (1500K default). We saw cases when there was no
space left for compiler's temp buffers so we can't compile but left space was a little > 1500K so flushing was not
triggered. There are other product flushing flags which you can play with.
Note, flushing is triggered when no space left in whole reserved space. It is not triggered when you used up initial space.
We will work on logging in jdk9 and 8u.
Regards,
Vladimir
On 11/8/13 9:12 AM, Srinivas Ramakrishna wrote:
> Thank you Albert, I appreciate it. I have one follow-up question below:-
> (Note that my comments are modulo the fact that I have not played with
> jdk 8 at all (althouygh i do have access to the source code for hotspot trun, so i can just look at the code --
> still this area is relatively unfamiliar to me in the low-level details and now, following recent rewrites and fixes,
> also in the high level structure)
>
>
> On Fri, Nov 8, 2013 at 3:06 AM, Albert Noll <albert.noll at oracle.com <mailto:albert.noll at oracle.com>> wrote:
>
> Hi Ramki,
>
> I have done the most recent changes to the code cache, so I might help in answering your questions.
> Please see comments inline.
>
>
>
>
> On 11/08/2013 09:59 AM, Srinivas Ramakrishna wrote:
>> Some of this is slightly off-topic, but here goes ...
>>
>> I haven't looked at the code or the patch/webrev, but I would definitely vote for a "Print" flag for CodeCache
>> state, analogous to PrintGCDetails for
>> Java heap space state, which will periodically (say at each GC for lack of a better metronomic signal) print the
>> size and occupancy of the code cache.
>>
> The flag -XX:+PrintCodeCacheOnCompilation prints the status of the code cache every time memory is allocated from
> the code cache.
>
>
> This is fine, but my concern is that this is at too fine a level of granularity, at least in the initial phase or in a
> phase change when
> lots of methods are typically compiled. Why not also have a somewhat asynchronous style of logging at some long period
> of granularity
> the size of the code cache. My guess (without looking at the code) is that the occupancy and size can be determined at
> very low
> cost. (Hence the suggestion of a gc as a possible place to log such information.)
>
>
>> I have separately made that request to this list in an earlier email this week, and would like to reiterate that
>> request.
>>
>> In general, what is the advice for pre-jdk8 (i.e. 7uXX vintage) of JVM's wrt the setting of UseCodeCacheFlushing?
>> Does having a suitably large code cache ensure that any of the myriad issues that seem to have been logged in
>> JDK jira/bugzilla will not affect us, or is it generally recommended that we both increase the size of the code
>> cache to a suitably safe value as well as switch off UseCodeCacheFlushing?
>>
> In general, we recommend to use code cache flushing. The reason is that if you do not use it and
> you run out of code cache, the VM is not able to compile hot methods bytecode to machine code
> anymore. I.e., all methods that have not been compiled will run in interpreted mode. If your application
> runs out of code cache and you incur a performance regression due to that, we recommend to
> increase the code cache size (-XX:ReservedCodeCacheSize=).
>
>
> Great; thanks for that advice. We have increased the reserved as well as initial code cache size. Not having looked at the
> code, one concern was that having a low initial size might (again without looking at the code) cause a flush cycle which
> might
> extract a performance penalty, before the cache is expanded (as it works its way to the max).
>
> One other concern was whether a sufficiently high occupancy of the code cache will cause the flush code to kick in
> and extract a performance drop. The strategy while we are unsure about this state of affairs has been to try and
> altogether avoid code cache flushing from kicking in by using a sufficiently large code cache size.
>
> If the code cache occupancy stabilizes over a period of time, this might seem to be a reasonable strategy until
> we are able to move to jdk 8 where these issues are hopefully all fixed.
>
>
>> Finally, does anyone on this list know where one might find the hotspot sources for various jdk7uXX releases? I am
>> interested
>> in looking at many of them. Given the rapid changes in code cache flushing and related code in the last few
>> months, I'd like to
>> understand, from looking at the code and exercising it, as to what kinds of performance bugs we might be open to
>> in this area
>> in a specific 7uXX release. (Unless someone on this list already has a crisp description of that.) I'll verify my
>> findings
>> with folks on this list and share my findings once I have looked through specific 7uXX releases.
>>
> You should be able to get the sources from http://openjdk.java.net/
>
>
>
> I was looking for pointers to the exact URL for the hsx 7uxx release repos, but Jon has pointed me there (thanks!).
>
>
>> It is only in the last week or two that I became aware of these performance issues, and would like to make sure we
>> are protecting ourselves sufficiently against it given a specific 7uXX release.
>>
> With the release of Java 8, these issues should be fixed.
>
>
>
> Is there any chance that some of the more critical fixes might get backported to a future 7uXX release?
> If not, would it be worthwhile for someone from the community (and i am happy to volunteer) to become involved in
> backporting
> them to a specific 7uXX release? I am happy to help with that, if there's any chance that it would be entertained.
> If such a backport is already on the cards, so much the better, and we can wait for that to happen.
>
> thanks Albert, all!
> -- ramki
>
>
>> thanks, and sorry again for hijacking the webrev discussion for this....
>> -- ramki
>>
> Best,
> Albert
>
>>
>>
>> On Thu, Nov 7, 2013 at 1:48 PM, Vladimir Kozlov <vladimir.kozlov at oracle.com <mailto:vladimir.kozlov at oracle.com>>
>> wrote:
>>
>> On 11/7/13 1:37 PM, Albert Noll wrote:
>>
>> Hi,
>>
>>
>> On 11/07/2013 08:39 PM, Vladimir Kozlov wrote:
>>
>> On 11/7/13 11:04 AM, Igor Veresov wrote:
>>
>> I’d vote to put it under PrintCodeCache. And make the messages not
>> warnings, but just “compiler disabled/enabled”. What do you think?
>>
>>
>> Unfortunately there could be customer's tools which look for this
>> message. So changing it, at least now for jdk8, is not good. With
>> small codecache we will expect this message showing up. But with big
>> codecache it should not happen. I think we should keep it as warning
>> but throttle it when small codecache is used as Chris suggested.
>>
>> May be put it under combined check:
>>
>> if (PrintCodeCache || ReservedCodeCacheSize > X)
>>
>> Do we have a state now when we definitely will not compile any more?
>> Or we always making progress? I think it will be difficult to find
>> when it should be printed only once.
>>
>> With the current version (when sweeper is enabled) we should not reach a
>> state (unless the entire code cache is filled with OSR-methods or native
>> methods) where we disable compilation and never enable it.
>> As soon as we free memory from the code cache, we re-enable compilation.
>> The message will be printed very frequently, if the code cache is
>> significantly smaller than the application demands.
>>
>> We could solve the 'problem' also by adding code that prints the warning
>> only if compilation is
>> disabled for a certain time. The current patch (webrev.01) defines a
>> virtual time for the sweeper (we increment time counter by one every
>> time we call mark_active_nmethods), which we could use.
>>
>>
>> Or only print 10th (or whatever) message, first one must print.
>>
>> Thanks,
>> Vladimir
>>
>>
>>
>> Best,
>> Albert
>>
>> Thanks,
>> Vladimir
>>
>>
>> igor
>>
>> On Nov 7, 2013, at 3:24 AM, Albert Noll <albert.noll at oracle.com <mailto:albert.noll at oracle.com>>
>> wrote:
>>
>> Hi Chris,
>>
>> On 11/06/2013 03:18 AM, Chris Plummer wrote:
>>
>> BTW, one thing I forgot to mention is I now see a lot of messages
>> for the codecache filling up. For example:
>>
>> Java HotSpot(TM) Client VM warning: CodeCache is full. Compiler has
>> been disabled.
>> Java HotSpot(TM) Client VM warning: Try increasing the code cache
>> size using -XX:ReservedCodeCacheSize=
>> CodeCache: size=2700Kb used=2196Kb max_used=2196Kb free=503Kb
>>
>> With b111, I was only seeing one message. I suspect with b111, once
>> this message appeared compilation was never re-enabled so the
>> message never appeared again. In that case seeing in many times now
>> is actually a good indicator. However, it appears even when not
>> using -XX:+PrintCodeCache, and I can see this output being a
>> distraction for programs whose normal operation may involve
>> constraining the codecache and having it constantly filling up.
>> Perhaps this message should be off by default, or possibly only
>> appear once.
>>
>> You are right. The previous version just never re-enabled
>> compilation. I also agree that the
>> output is distracting. There are multiple ways to solve this issue.
>> I would go for a product -XX flag
>> which allows to turn this warning on/off. Would that be ok or do you
>> have a different solution in mind?
>>
>> Best,
>> Albert
>>
>> cheers,
>>
>> Chris
>>
>> On 11/5/13 5:59 PM, Chris Plummer wrote:
>>
>> Hi Albert,
>>
>> I applied your patch and got some new numbers. Performance is now
>> even better than it was with b110. See the chart I added to the bug.
>>
>> Nice work!
>>
>> Chris
>>
>> On 11/5/13 6:44 AM, Albert Noll wrote:
>>
>> Hi,
>>
>> could I get reviews for this small patch?
>>
>> bug: https://bugs.openjdk.java.net/browse/JDK-8027593
>> webrev: http://cr.openjdk.java.net/~anoll/8027593/webrev.00/
>> <http://cr.openjdk.java.net/%7Eanoll/8027593/webrev.00/>
>>
>> Problem: The implementation of the sweeper (8020151) causes a
>> performance regression for small code cache sizes. There are two
>> issues that cause this regression:
>> 1) NmethodSweepFraction is only adjusted according to the
>> ReservedCodecacheSize if TieredCompilation is enabled. As a
>> result, NmethodSweepFraction remains 16 (if TieredCompilation is
>> not used). This is way too large for small code cache sizes
>> (e.g., <5m).
>> 2) _request_mark_phase (sweeper.cpp) is initialized to false. As
>> a result, mark_active_nmethods() did not set _invocations and
>> _current, which results in not invoking the sweeper (calling
>> sweep_code_cache()) at all. When TieredCompilation is enabled
>> this was not an issue, since NmethodSweeper::notify() (which sets
>> _request_mark_phase) is called much more frequently.
>>
>> Solution: 1) Move setting of NmethodSweepFraction so that it is
>> always executed.
>> Solution: 2) Remove need_marking_phase(),
>> request_nmethod_marking(), and reset_nmetod_marking().
>> I think that these checks are not needed since
>> 8020151, since we do stack scanning of
>> active nmethods irrespective of the value of
>> what need_marking_phase() returns. Since
>> the patch removes need_marking_phase()
>> printing out the warning (line 327 in
>> sweeper.cpp) is incorrect, i.e., we continue
>> to invoke the sweeper. I removed the warning
>> and the associated code.
>>
>>
>> Also, I think that we can either remove -XX:MethodFlushing or
>> -XX:UseCodeCacheFlushing. Since 8020151, one of them is redundant
>> and can be removed. I am not quite sure if we should do that now
>> so it is not included in the patch.
>>
>> Testing
>> bug: https://bugs.openjdk.java.net/browse/JDK-8027593 also shows
>> a performance evaluation.
>>
>> Many thanks for looking at the patch.
>> Best,
>> Albert
>>
>>
>>
>>
>>
>>
>>
>
>
More information about the hotspot-compiler-dev
mailing list