RFR(M): 8244660: Code cache sweeper heuristics is broken

[Nils Eliasson]

Hi,

In a recent bug (JDK-8244278) it was discovered that the heuristics for 
StartAggressiveSweepingAt was wrong. Aggressive sweeping means that the 
sweeper is trying to start a sweep for every new allocation. With 
default settings before the fix, aggressive sweeping started at already 
at 10% full.

However with the fix applied - there was no sweeps at all. This is also 
wrong - the sweeper should trigger regularly. The problem is that the 
sweeper thread is sleeping and there were no code path in which it was 
awakened (except through StartAggressiveSweepingAt). So the counters 
counted, thresholds were reached, but the sweeper slept on.

Investigating this I encountered the next problem. The old heuristics 
had two different thresholds:

1) Number of safepoints with stack scans.
With the transition to handshakes from safepoints we can no longer rely 
on safepoints happening. I have removed all of that code.

2) number of bytes of nmethods that had been marked as zombie or not 
entrant (see NMethodSweeper::possibly_enable_sweeper). This is now the 
only regular way that sweeps are triggered. (The other two are 
aggressive sweeping and whitebox testing).

I changed the threshold test to be against a new flag - 
SweeperThreshold. This is because users with different sized code caches 
might want different thresholds. (Otherwise there would be no way to 
control the sweepers intensity).

The default is MIN2(1 * M, ReservedCodeCacheSize / 256));
At default tiered ReservedCodeCacheSize that is about 1M.
At a small code cache of 40M that is about 150k
The threshold is capped at 1M because even if you have an enormous code 
cache - you don't want to fragment it, and you probably don't want to 
commit more than needed.

This patch simplified the sweeper quite a bit. Since the sweeper only is 
awaken when there is actual job - no more complex heuristics is needed 
in the sweeper thread. This also has the benefit that the sweeper will 
always sleep when there is no job.

To be able to notify the sweeper thread from 
NMethodSweeper::possibly_enable_sweeper I had to use a different monitor 
than CodeCache_lock. I added a new monitor that replaces the 
CodeCache_lock for signaling. This monitor (CodeSweeper_lock) is only 
used for signaling the sweeper thread - and the lock is always released 
before doing anything else.

With this patch - JDK-8244278 can be applied on top to fix the 
aggressive sweeping, and the sweeper will continue to work.

This path applies on top of JDK-8244658 that removes dead code in the 
sweeper.

Bug: https://bugs.openjdk.java.net/browse/JDK-8244660
Webrev: http://cr.openjdk.java.net/~neliasso/8244660/webrev.01/


Please review,

Nils Eliasson