RFR: 8356228: NMT does not record reserved memory base address correctly

[Afshin Zafari]

On Wed, 11 Jun 2025 11:10:06 GMT, Afshin Zafari <azafari at openjdk.org> wrote:

>>> Is it possible to capture this logic in a test, so we can always check for this behavior?
>> 
>> Work on it.
>
>> Where in the code do we release and check for the special case of `mtClassShared`?
> 
> In `virtualMemoryTracker.cpp, remove_released_region(adddress, size)` in an `if` block as this:
> 
> if (reserved_rgn->mem_tag() == mtClassShared) {
>     if (reserved_rgn->contain_region(addr, size)) {
>       // This is an unmapped CDS region, which is part of the reserved shared
>       // memory region.
>       // See special handling in VirtualMemoryTracker::add_reserved_region also.
>       return true;
>     }
> 
>     if (size > reserved_rgn->size()) {
>       // This is from release the whole region spanning from archive space to class space,
>       // so we release them altogether.
>       ReservedMemoryRegion class_rgn(addr + reserved_rgn->size(),
>                                      (size - reserved_rgn->size()));
>       ReservedMemoryRegion* cls_rgn = _reserved_regions->find(class_rgn);
>       assert(cls_rgn != nullptr, "Class space region  not recorded?");
>       assert(cls_rgn->mem_tag() == mtClass, "Must be class mem tag");
>       remove_released_region(reserved_rgn);
>       remove_released_region(cls_rgn);
>       return true;
>     }
>   }

> I'm not 100% sure I understand why we have to first reserve with `mtNone` and only then change it to `mtClassShared`. Why can't we reserve with `mtClassShared` right from the start?

The sequence is as follows:
_All the reserve/release operations here are from `os::` namespace and therefore will call NMT tracking API_
1- CDS requests for reserving `S` bytes at address `Base` with alignment `A` and memTag `M`.
2- Due to OS alignment limitations regarding `A`, new base `XBase` to be used with new size `XS`. Read X as extra here.
3- Do instead, reserve `XS` bytes at `XBase`.
4- If succeeds, memory would be like `<--s1---><----S----><----s2--->`, where `XS = s1 + S + s2` and `XBase + s1 == Base`  
5- Release extra parts `s1` and `s2`.
6- request at step 1 above is fulfilled now.

If at step 1, the memTag `M` is `mtClassShared` , the releases at step 5 will be ignored by NMT. Which means that there are regions in NMT lists that are actually released (at os level) but NMT takes them still as reserved. Therefore, further reservations by os at those regions (e.g., for stack) will intersect with existing NMT regions and the NMT assertions fail. So the memTag is to be `mtNone` to bypass/avoid the NMT ignore branch.

If we want to track the new memory region as belongs to CDS, we need to set the memTag of the region to `mtClassShared` with the corresponding `MemTracker` API.(as is done here in this PR)

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/25719#discussion_r2139917194