RFR: 8369393: NMT: poison the canaries of malloc header under ASAN build [v20]

[Thomas Stuefe]

On Mon, 10 Nov 2025 11:31:28 GMT, Afshin Zafari <azafari at openjdk.org> wrote:

>> src/hotspot/share/nmt/mallocHeader.hpp line 126:
>> 
>>> 124:   inline static OutTypeParam resolve_checked_impl(InTypeParam memblock);
>>> 125: 
>>> 126:   void asan_poison_self() {
>> 
>> Why this complex? Why not just poison/unpoison the whole header in one call? You could simplify quite a bit that way.
>
> The only reason is to avoid unpoisoning when accessing other fields as MemTag or MST_Marker, and poisoning back when access is completed. Now MemTag and mst_marker fields are free to be accessed without unpoisoning first.
> I can change the issue title and do the whole header poisoning, if it is preferred.

I see what you mean. 

Ugh. I forgot that we cannot selectively poison write access only but still allow read access, because otherwise poisoning the whole header would be much simpler and nicer. But unpoisoning every time one checks a header would be highly annoying.

In order to simplify, and not to pay too high a penalty on access if ASAN is enabled, I would consider:
- don't bother poisoning `_size`. This field is frequently accessed, as opposed to the `_canary` field. I think cases where we overwrite `_size` but *don't* overwrite either the byte before it (which should be guarded by ASAN anyway since it precedes the first valid byte) or the canary (which is now also poisoned and directly precedes the user payload) should be exceedingly rare. 
- I would just not bother checking NMT block integrity at all if ASAN is enabled. That should reduce the number of accesses to the `_canary` to zero, and since ASAN reports right away when the canary is hit, we will see it right away. Now you won't need unpoisoning on canary read.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/27685#discussion_r2510611771