RFR: JDK-8312018: Improve reservation of class space and CDS

Wed Aug 9 14:08:35 UTC 2023

This PR rewrites memory reservation of class space and CDS to have ASLR and a much better chance of low-address placement.

------

Motivation:

The JVM attempts to place the class space at a location usable for unscaled or zero-based encoding.

It does so by:
1. piggybacking on Java heap placement
2. alternatively, attempt to map at HeapBaseMinAddress
3. (ppc and aarch64 only) attempt to map within the lower 32 GB with a "ladder approach"

This approach has drawbacks:

- We don't try particularly hard. (1) relies on the Java heap using CompressedOops *and* being reserved in low address ranges *and* the range above the heap being free. (2) depends on HeapBaseMinAddress being free. (3) is restricted to two platforms.

- Only (2) has even a chance to get us into unscaled (zero base zero shift) territory. (1) and (3) only give us zero-based non-zero-shift encoding.

- HeapBaseMinAddress is an odd choice. It is 2GB on all platforms, and that denies us half of the valuable low address range below 4G right away, further reducing the chance of ever going unscaled.

- It is only randomized for CDS=on, and even then with a very low value range.

- Importantly, we only try for zero-based encoding if CDS=off or CDS=dump. For CDS runtime, atm we need to set the encoding base to the start of the archive, and therefore will never run zero-based. The reason is that if the archive contains heap objects, those will come with pre-computed narrow Klass IDs, which will be based on the start address of the archive. At runtime, the archive can be placed anywhere (and since JDK-8296565, that placement is randomized), but wherever it is placed, the encoding base has to be.

- It reduces the chance of getting a zero-based java heap - this is because when attempting to place the heap, we leave a gap for the future class space. But a zero-based Java heap is more valuable than a zero-based class space - it makes sense to optimize for the former and take the latter if possible.

- It introduces an unnecessary dependency between heap reservation and class space reservation. That makes the code base brittle.

-------

The patch introduces a new API to reserve memory within an address range at a randomized location, while trying to be smart about it. The API is generic, and future planned uses of this API could include replacing the zero-based heap allocation and the zero-based reservation of Shenandoah Collection Sets, thereby allowing us to consolidate coding.

This PR complements @iklams current work that rewrites archive heap initialization at runtime. Once his work is in, we will be able to recalculate narrow Klass IDs for objects loaded from the archive, and that will allow us to reap the benefits of this patch for the CDS runtime case too.

-------

Example (linux amd64):

We start the JVM with a 30GB heap. 

In the stock JVM, the JVM will place the heap in the lower address ranges starting at 2G (0x8000_0000). But then it is unable to place the class space in lower regions too, so it placed it at 32 GB (0x8_0000_0000), and we don't have zero-based encoding (Narrow klass base: 0x0000000800000000). This scenario repeats for every iteration, so we will always use these two addresses (no ASLR): 

thomas at starfish $ ./images/jdk/bin/java -Xshare:off -Xmx30g -Xlog:gc+heap+exit -Xlog:gc+metaspace -version
[0.019s][info][gc,metaspace] Compressed class space mapped at: 0x0000000800000000-0x0000000840000000, reserved size: 1073741824
[0.019s][info][gc,metaspace] Narrow klass base: 0x0000000800000000, Narrow klass shift: 0, Narrow klass range: 0x40000000
...
[0,112s][info][gc,heap,exit]  garbage-first heap   total 1032192K, used 4915K [0x0000000080000000, 0x0000000800000000)

In the patched JDK, the heap will also be placed at 2G (0x8000_0000). But the class space will be nestled below that, at a random location lower than 2G (in this run, at 0x1f00_0000):

thomas at starfish $ ./images-new/jdk/bin/java -Xshare:off -Xmx30g -Xlog:gc+heap+exit -Xlog:gc+metaspace -version
[0.017s][info][gc,metaspace] Compressed class space mapped at: 0x000000001f000000-0x000000005f000000, reserved size: 1073741824
[0.017s][info][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 0, Narrow klass range: 0x5f000000
...
[0,107s][info][gc,heap,exit]  garbage-first heap   total 1032192K, used 5898K [0x0000000080000000, 0x0000000800000000)

And it will be randomized, see output from multiple runs - every time we end up with a zero-base, but the actual start address of the class space differs:

[0.019s][info][gc,metaspace] Compressed class space mapped at: 0x0000000025000000-0x0000000065000000, reserved size: 1073741824
[0.019s][info][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 0, Narrow klass range: 0x65000000
...
[0.017s][info][gc,metaspace] Compressed class space mapped at: 0x0000000035000000-0x0000000075000000, reserved size: 1073741824
[0.017s][info][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 0, Narrow klass range: 0x75000000
...
[0.018s][info][gc,metaspace] Compressed class space mapped at: 0x000000001f000000-0x000000005f000000, reserved size: 1073741824
[0.018s][info][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 0, Narrow klass range: 0x5f000000
...
[0.016s][info][gc,metaspace] Compressed class space mapped at: 0x000000001c000000-0x000000005c000000, reserved size: 1073741824
[0.016s][info][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 0, Narrow klass range: 0x5c000000

-------

Tests:

The patch comes with a full set of gtests that test the new API from all angles, all of which have been executed on many platforms.

GHAs are green, too.

-------------

Commit messages:
 - wip
 - wip
 - JDK-8312018-Improve-odds-for-zero-base-optimized-reservation-of-class-space

Changes: https://git.openjdk.org/jdk/pull/15041/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=15041&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8312018
  Stats: 776 lines in 16 files changed: 661 ins; 70 del; 45 mod
  Patch: https://git.openjdk.org/jdk/pull/15041.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/15041/head:pull/15041

PR: https://git.openjdk.org/jdk/pull/15041