RFR: 8306561: Possible out of bounds access in print_pointer_information

[Thomas Stuefe]

On Thu, 26 Oct 2023 17:09:04 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> MallocTracker::print_pointer_information in src/hotspot/share/services/mallocTracker.cpp is called to check the highest pointer address of the reserved region. To do so it aligns the test pointer down to the next 8 Byte boundary and casts this address to class MallocHeader in order to use this classes eye-catcher member _canary for validation. Method looks_valid() dereferences _canary's content. _canary has an offset of 14 bytes relative to the class. Therefore it resides outside the reserved region for the highest pointer address, which causes a segmentation violation.
>> 
>> We would expect the same error also for other platforms than AIX as memory is read, which is not allocated. Interestingly, Linux seems to allow this access for 5 times 4K above the reserved region.
>> 
>> As a solution, looks_valid() should check _canary's address as being invalid, and return false immediately.
>
> Yes, let it blow up then.

> @tstuefe: I helped analyzing this problem by writing a plain c test pgm mmaping a page and trying to read beyound. On AIX as expected the very next byte after the requested region leads to a segmentation violation, but on linux (both flavours, linuxintel and linuxppc64) I was able to read exactly 20 KB beyond, before running into segmentation violation. This might be the reason, why the developer of print_pointer_information() was not aware that he creates code that could crash. Thomas, do you have an idea why linux (and maybe other platforms) map more memory as requested? It has nothing to do with memory pages. The additional memory does not end at the next memory page boundary, but exactly 20KB after the end of the requested region. Astonishing is that at the lower end of the region there is no extra memory accessible.

Plain bad luck and rare test execution.

Whether or not you can read over the end of an mmaped segment depends on whether there are VMAs mapped beyond that. Linux kernel clusters VMAs to keep VMA fragmentation down. So you may have adjacent mappings. This is subject to ASLR of course but I always see the VMAs pretty much clustered regardless of ASLR. And though this sounds like it should be random, you can have a pretty consistent order of VMA allocation across many VM runs, and therefore similarly looking process memory maps.

Just looking at /proc/pid/maps will be probably clearer to you than me explaining it :)

-------------

PR Comment: https://git.openjdk.org/jdk/pull/16381#issuecomment-1782772667