RFR: 8330174: Protection zone for easier detection of accidental zero-nKlass use [v8]

[Thomas Stuefe]

> If we wrongly decode an nKlass of `0`, and the nKlass encoding base is not NULL (typical for most cases that run with CDS enabled), the resulting pointer points to the start of the Klass encoding range. That area is readable. If CDS is enabled, it will be at the start of the CDS metadata archive. If CDS is off, it is at the start of the class space. 
> 
> Now, both CDS and class space allocate a safety buffer at the start to prevent Klass structures from being located there. However, that memory is still readable, so we can read garbage data from that area. In the case of CDS, that area is just 16 bytes, after that come real data. Since Klass is large, most accesses will read beyond the 16-byte zone.
> 
> We should protect the first page in the narrow Klass encoding range to make analysis of errors like this easier. Especially in release builds where decode_not_null does not assert. We already use a similar technique in the heap, and most OSes protect the zero page for the same reason.
> 
> This patch does that. Now, decoding an `0` nKlass and then using the result `Klass` - calling virtual functions or accessing members - crashes right away.
> 
> Additionally, the patch provides a helpful output in the register/stack section, e.g:
> 
> 
> RDI=0x0000000800000000 points into nKlass protection zone
> 
> 
> 
> Testing: 
> - GHAs.
> - I tested the patch manually on x64 Linux for both CDS on, CDS off and zero-based encoding, CDS off and non-zero-based encoding.
> - I tested manually on Windows x64
> - I also prepared an automatic gtest, but that needs some preparatory work on the gtest suite first to work (see https://bugs.openjdk.org/browse/JDK-8348029)
> 
> -- Update 2024-01-22 -- 
> I added a jtreg test that is more thorough than a gtest (also scans the produced hs-err file)

Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:

  remove test coding

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/23190/files
  - new: https://git.openjdk.org/jdk/pull/23190/files/58fa6a1e..14641c77

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=23190&range=07
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=23190&range=06-07

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/23190.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/23190/head:pull/23190

PR: https://git.openjdk.org/jdk/pull/23190