RFR(XXS): 8244278: Excessive code cache flushes and sweeps

[Man Cao]

Hi,

[@Laurent]
> Thanks Man for your results.
> I will try your fix on jdk15 repo and run my Marlin tests & benchmark to
see if there are any gain in these cases.
You are welcome.
I have run DaCapo at JDK tip, with default JVM options. I didn't see any
noticeable difference in performance with and without my bugfix.
This is probably due to significantly reduced code cache flushes with a
large default ReservedCodeCacheSize (240MB) for +TieredCompilation.
I checked the logs for tradesoap for runs without my bugfix, to count the
number of completed flushes (NMethodSweeper::sweep_code_cache()):
~550 for runs with -XX:-TieredCompilation -XX:ReservedCodeCacheSize=40m on
JDK11
~35 for runs with default options on JDK tip
(+TieredCompilation, ReservedCodeCacheSize=240m)
The flushes are reduced by more than 15X with the default options.

[@Nils]
> Looking at sweeper.cpp I see something that looks wrong. The _last_sweep
counter is updated even if no sweep was done. In low code cache usage
scenarios that means will might never reach the threshold.
> Can you try it out and see if things improve?
The change makes sense to me. I can try it out together after resolving the
next issue.

> The sweeper should wake up
> regularly, but now it is only awakened when hitting the SweepAggressive
> threshold. This is wrong.
> I suggest holding of the fix until all the problems have been ironed out.
Could you elaborate what is the expected frequency to wake up the sweeper?
Should we increase the default value for StartAggressiveSweepingAt instead?

As you suggested before, currently the sweeper is awakened for every new
allocation in the code cache, after the usage is above 10%.
This is definitely too frequent that it hurts performance, especially for
the -XX:-TieredCompilation case.
We do find that turning off code cache flushing in JDK11
(-XX:-UseCodeCacheFlushing)
could significantly improve performance, by ~20% for an important
production workload configured with -XX:-TieredCompilation!
Thus, we strongly support keeping the default flushing frequency low.

-Man