RFR: 8356000: C1/C2-only modes use 2 compiler threads on low CPU count machines [v4]

[Aleksey Shipilev]

> There is an unfortunate limitation with default tiered policy that we would have at least 2 threads on 1 CPU machine: 1 thread for C1, and 1 thread for C2.
> 
> But if we select C1-only or C2-only modes, we _also_ get 2 compiler threads, for which we have no good reason. These threads would just step on each other toes. The fix changes the behavior for 1..3 CPU hosts in C1/C2-only configurations, by using 1 thread instead of 2 threads. The change for 1 CPU config is what we really need. The change in 2..3 CPU configs is an additional effect, but I think it is still good not to use 100%/66% of the CPUs in those configurations as well.
> 
> 
> $ for I in `seq 1 8`; do build/linux-x86_64-server-release/images/jdk/bin/java \
>   -XX:-TieredCompilation -XX:ActiveProcessorCount=${I} \
>   -XX:+PrintFlagsFinal 2>&1 | grep "CICompilerCount "; done
> 
> # Before
>      intx CICompilerCount                          = 2
>      intx CICompilerCount                          = 2
>      intx CICompilerCount                          = 2
>      intx CICompilerCount                          = 3
>      intx CICompilerCount                          = 3
>      intx CICompilerCount                          = 3
>      intx CICompilerCount                          = 3
>      intx CICompilerCount                          = 4
> 
> # After
>      intx CICompilerCount                          = 1
>      intx CICompilerCount                          = 1
>      intx CICompilerCount                          = 1
>      intx CICompilerCount                          = 3
>      intx CICompilerCount                          = 3
>      intx CICompilerCount                          = 3
>      intx CICompilerCount                          = 3
>      intx CICompilerCount                          = 4
> 
> 
> It is a minor bug in `CompilationPolicy::initialize`, but it gets in the way studying Leyden in tight CPU scenarios. 
> 
> Additional testing:
>  - [x] New regression test passes with the fix, fails without it
>  - [x] GHA
>  - [x] Linux AArch64 server fastdebug, `all`

Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 10 additional commits since the last revision:

 - Revert buffer size change
 - Merge branch 'master' into JDK-8356000-c1-c2-compiler-count
 - Merge branch 'master' into JDK-8356000-c1-c2-compiler-count
 - Better test, patch amendments
 - Merge branch 'master' into JDK-8356000-c1-c2-compiler-count
 - Merge branch 'master' into JDK-8356000-c1-c2-compiler-count
 - Unnecessary arch limitation
 - Simplify test
 - Adjust test bound
 - Fix

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/24972/files
  - new: https://git.openjdk.org/jdk/pull/24972/files/f8519b46..c43b18a6

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=24972&range=03
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=24972&range=02-03

  Stats: 47592 lines in 764 files changed: 24205 ins; 14405 del; 8982 mod
  Patch: https://git.openjdk.org/jdk/pull/24972.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/24972/head:pull/24972

PR: https://git.openjdk.org/jdk/pull/24972