RFR: JDK-8312018: Improve reservation of class space and CDS

[Thomas Stuefe]

On Wed, 26 Jul 2023 11:34:18 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> This PR rewrites memory reservation of class space and CDS to have ASLR and a much better chance of low-address placement.
> 
> ------
> 
> Motivation:
> 
> The JVM attempts to place the class space at a location usable for unscaled or zero-based encoding.
> 
> It does so by:
> 1. piggybacking on Java heap placement
> 2. alternatively, attempt to map at HeapBaseMinAddress
> 3. (ppc and aarch64 only) attempt to map within the lower 32 GB with a "ladder approach"
> 
> This approach has drawbacks:
> 
> - We don't try particularly hard. (1) relies on the Java heap using CompressedOops *and* being reserved in low address ranges *and* the range above the heap being free. (2) depends on HeapBaseMinAddress being free. (3) is restricted to two platforms.
> 
> - Only (2) has even a chance to get us into unscaled (zero base zero shift) territory. (1) and (3) only give us zero-based non-zero-shift encoding.
> 
> - HeapBaseMinAddress is an odd choice. It is 2GB on all platforms, and that denies us half of the valuable low address range below 4G right away, further reducing the chance of ever going unscaled.
> 
> - It is only randomized for CDS=on, and even then with a very low value range.
> 
> - Importantly, we only try for zero-based encoding if CDS=off or CDS=dump. For CDS runtime, atm we need to set the encoding base to the start of the archive, and therefore will never run zero-based. The reason is that if the archive contains heap objects, those will come with pre-computed narrow Klass IDs, which will be based on the start address of the archive. At runtime, the archive can be placed anywhere (and since JDK-8296565, that placement is randomized), but wherever it is placed, the encoding base has to be.
> 
> - It reduces the chance of getting a zero-based java heap - this is because when attempting to place the heap, we leave a gap for the future class space. But a zero-based Java heap is more valuable than a zero-based class space - it makes sense to optimize for the former and take the latter if possible.
> 
> - It introduces an unnecessary dependency between heap reservation and class space reservation. That makes the code base brittle.
> 
> -------
> 
> The patch introduces a new API to reserve memory within an address range at a randomized location, while trying to be smart about it. The API is generic, and future planned uses of this API could include replacing the zero-based heap allocation and the zero-based reservation of Shenandoah Collection Sets, thereby allowing us to consolidate coding.
> 
> This PR complements @ikl...

Attention @iklam

-------------

PR Comment: https://git.openjdk.org/jdk/pull/15041#issuecomment-1671367200