RFR: 8364202: CDS without G1 gives build error in slowdebug, asserts in fastdebug

[Thomas Stuefe]

On Tue, 29 Jul 2025 06:14:46 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> This patch fixes two problems:
> 
> - when building without INCLUDE_CDS_JAVA_HEAP (disabling G1 at configure, or building 32-bit), we get a linker error in slowdebug.
> - when running with CDS, but without INCLUDE_CDS_JAVA_HEAP, we initialize CompressedKlassPointers via the "lenient" route where we allow optimized encoding base choice. This can lead to zero-based encoding, in which case we should not attempt to setup a protection zone at the start of the encoding range.

NMT gtest failure on windows unrelated.

@tschatzl , @iklam , could you please review?

src/hotspot/share/cds/metaspaceShared.cpp line 1605:

> 1603: 
> 1604:       // Set up compressed Klass pointer encoding: the encoding range must
> 1605:       //  cover both archive and class space.

Reviewer Notes, changes in CDS explained:
- Line 1626: fix bug by only establishing a protection zone if the encoding base points to the start of the archive after setup (aka we don't end up running zero-based encoding)
- Lines 1619, 1623: We only have two cases: We either need to dictate the encoding (if CDS heap area exists), in which case the encoding base must be the start of the mappings. Or, we let the JVM choose freely, in which case we end up with base=zero if possible, base=mapping start otherwise. Assert both cases.
- Line 1606: When initializing `CompressedKlassPointers`, we should always feed in the mapping start for klass range start. This has no practical consequences, but it is cleaner and more consistent with the CDS=off path. I also added a comment to compressedKlass.hpp in that regard.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26523#issuecomment-3132551317
PR Comment: https://git.openjdk.org/jdk/pull/26523#issuecomment-3132552458
PR Review Comment: https://git.openjdk.org/jdk/pull/26523#discussion_r2239324348