RFR: 8340297: Use-after-free recognition for metaspace and class space [v3]

[Thomas Stuefe]

On Mon, 18 Aug 2025 06:34:25 GMT, Johan Sjölen <jsjolen at openjdk.org> wrote:

>> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 11 commits:
>> 
>>  - Merge branch 'openjdk:master' into JDK-8340297-Metaspace-API-for-checking-if-address-is-in-use
>>  - merge master
>>  - copyrights
>>  - fix big-endian problem on AIX
>>  - Update klass.cpp
>>  - Update metaspace.hpp
>>  - Update metaspace.hpp
>>  - Update metaspace.hpp
>>  - fix rebase error
>>  - fix mac build
>>  - ... and 1 more: https://git.openjdk.org/jdk/compare/57210af9...d9696c76
>
> src/hotspot/share/memory/metaspace.cpp line 1051:
> 
>> 1049: }
>> 1050: 
>> 1051: bool Metaspace::metadata_is_live(const Metadata* md, FailureHint* hint) {
> 
> Why can't `FailureHint` have an arm called 'no_failure' and just skip it as  an out-parameter, having the failure be the return value?

Hmm, I played around with that, but it is more awkward to use. 

- The hint parameter is optional. So if you are not interested in it, you can just call "klass_is_live(k, true)".
- With return code and hint squashed into one, it becomes more awkward to combine this call with another and still catch the hint, see shenandoaAssert. Something like

    if (klass != nullptr && !Metaspace::klass_is_live(klass, false, &hint)) {
       print hint
    }

becomes 

    if (klass != nullptr && ((Metaspace::FailureHint h = Metaspace::klass_is_live(klass, false)) != Metaspace::FailureHint::none ) {
       print hint
    }


or forces you to split the expression into multiple lines.

> src/hotspot/share/memory/metaspace.hpp line 162:
> 
>> 160:     unknown = 0,
>> 161:     // outside class space/metaspace
>> 162:     outside,                          // 1
> 
> I don't see why we need to write the value of each enum arm, can't our IDEs figure that out for us if necessary :-)?

Okay, removed. Originally I had "speaking" hint codes to aid debugging.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/25891#discussion_r2284154343
PR Review Comment: https://git.openjdk.org/jdk/pull/25891#discussion_r2284156735