RFR: JDK-8298908: Instrument Metaspace for ASan [v6]

[Justin King]

On Thu, 5 Jan 2023 16:29:13 GMT, Justin King <jcking at openjdk.org> wrote:

>> This change instruments Metaspace for ASan. Metaspace allocates memory using `mmap`/`munmap` which ASan is not aware of. Fortunately ASan supports applications [manually poisoning/unpoisoning memory](https://github.com/google/sanitizers/wiki/AddressSanitizerManualPoisoning). ASan is able to detect poisoned memory, similar to `use-after-free`, and will raise an error similarly called `use-after-poison`. This provides and extra layer of defense and confidence.
>> 
>> The header `sanitizers/address.h` defines macros for poisoning/unpoisoning memory regions. These macros can be used regardless of build mode. When ASan is not available, they are implemented using a NOOP approach which still compiles the arguments but does so such that they will be stripped out by the compiler due to being unreachable. This helps with maintenance.
>> 
>> This also has the added benefit of making [LSan](https://bugs.openjdk.org/browse/JDK-8298445) more accurate and deterministic, as LSan will not look for pointers to malloc memory in poisoned memory regions.
>> 
>> IMO the benefit of doing this greatly outweighs the cost.
>
> Justin King has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Use -fno-common as suggested by ASan docs
>   
>   Signed-off-by: Justin King <jcking at google.com>

Yeah, I was originally going for the most pedantic implementation. But I like your proposed approach and it strikes a happy middle ground.

> That explains (I think) another question I had. Which is how are overlapped ranges handled. Given four subsequent addresses a b c d, I should be able to poison [b,c) and then poison [a,d), right? Or do I have to take care to only pass non-overlapping mappings?

ASAN is fine with overlapping. It doesn't check the current state of poisoning/unpoisoning when doing the operations. Though there are variants that do such as __sanitizer_annotate_contiguous_container, but that is for instrumenting contiguous containers similar to std::vector. That will come at a later time for applicable Hotspot containers, but out of scope for this change and unrelated to Metaspace.

> Great. I'll take a look next week since I still have vacation.

No problem.

-------------

PR: https://git.openjdk.org/jdk/pull/11702