RFR: 8234930: Use MAP_JIT when allocating pages for code cache on macOS [v6]

[Thomas Stuefe]

On Fri, 4 Dec 2020 21:19:53 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

> > 
> > Your original chain of thought, if I understand you correctly, was like this:
> > 
> > * We want to provide MAP_JIT on a mapping. Apple tells us to.
> > * On reserve, we add it to the initial mmap call. Easy.
> > * But hotspot later - commit/uncommit - replaces that mapping again. With subsequent mmap calls. On those, MAP_FIXED is specified since the original mapping gets replaced.
> > * But on those secondary mmap calls we cannot add MAP_JIT. Because it cannot be combined with MAP_FIXED.
> > * Therefore we switch to: on commit, do mprotect(RW). On uncommit, madvise(MADV_FREE) + mprotect(NONE).
> > 
> > Am I right so far?
> 
> That's correct.
> 
> > My thought is this:
> > In existing code, the technique used to commit and uncommit memory is to switch the MAP_NORESERVE flag on the mapping, combined with a change in protection.
> 
> I read this as MAP_NORESERVE is supposed to be an attribute of the mapping. I don't think this is true. On Linux it specifies, can the system overcommit in statisfying this mapping (https://man7.org/linux/man-pages/man2/mmap.2.html) By default overcommit is enabled and MAP_NORESERVE is also a "noop" on Linux, i.e. it specifies the mode that is enabled for all mappings anyway.

Thats not true. On Linux, if you mmap without specifying MAP_NORESERVE, the mmap size will increase your process commit charge. What that means depends on the overcommit setting. The default setting is a heuristic one where you are allowed some overcharging but there is a limit. My experience is that this mode gives us about 150% of the overcharge limit, after that you get allocation failures.

> This also mean that you can drop MAP_NORESERVE flag from uncommit and it will work anyway, on Linux and macOS.
> 
> What definition of MAP_NORESERVE do you use?
> 
> > E.g. mmap(MAP_NORESERVE, PROT_NONE) to uncommit. No documentation says this releases memory. But from experience we see it work on Linux - the kernel takes the hint that the pages are not needed anymore.
> 
> It's not a hint. From https://man7.org/linux/man-pages/man2/mmap.2.html:
> 
> > If the memory region specified by addr and len
> > overlaps pages of any existing mapping(s), then the overlapped
> > part of the existing mapping(s) _will be discarded._
> 
> So one of uncommit's tasks is to release the memory.
> 
> > If MAP_NORESERVE is a noop on Mac, the only other effect those mmap calls have is the protection change: in uncommit, the mmap(MAP_NORESERVE|MAP_FIXED, PROT_NONE) would be equivalent to a mprotect(PROT_NONE).
> 
> mmap(MAP_NORESERVE, ...) is equal to mmap(...), but it does not mean that mmap(MAP_NORESERVE) is noop.
> 
> Also, after you do mprotect(NONE), you can do mprotect(RWX) and get back the original content. But after you do mmap(FIXED) the memory content is discarded, you cannot reverse this operation.

Oh you are right! I had this completely wrong.

The reclaim effect is not caused by the fact that we specify MAP_NORELEASE. That's incidental. It comes from us plain replacing the old mapping with a blank new one and thereby discarding the old mapping?

So MAP_NORESERVE may not matter but mmap is still needed. Strictly speaking probably not even PROT_NONE would matter to have a reclaim effect. You could just map the new mapping with full rights and the memory would still reclaimed and stay that way until next time you touch it. Then commit could be a no-op (this is how things work on AIX).

Okay, I think I get closer to understanding the problem: 
- on reserve we create mapping 1 with MAP_JIT
- on uncommit, we replace the mapping with mapping 2, discarding mapping 1. mapping 2 does not need MAP_JIT. If we never recommit thats fine.
- but then, on re-commit, we replace mapping 2 with mapping 3. mapping 3 again needs MAP_JIT. But since this needs MAP_FIXED, MAP_JIT will not work.

In other words, once a mapping with MAP_JIT is established, we must never replace it, since any subsequent mmap calls would need to be established with MAP_FIXED. Sigh.

My only remaining question is: is there really an observable difference between replacing the mapping with mmap and calling madvice(MADV_FREE)? And if there is, does it matter in practice?  I wonder if the perceived difference between madvise(MADV_FREE) and mmap() is just a display problem. Seems the kernel is lazy about reclaiming that memory - but that is fine and makes sense performance wise, it just throws off statistics. By that logic, using mmap() tries to second-guess what the kernel does could maybe be inferior to using madvise.

I found this: https://stackoverflow.com/questions/7718964/how-can-i-force-macos-to-release-madv-freed-pages. One remark recommends MADV_FREE_REUSABLE to deal with the display problem; could that be a solution (still aiming for using madvise() for all, not just executable, mappings, thereby removing the need to pass exec to commit/uncommit).

Cheers, Thomas

-------------

PR: https://git.openjdk.java.net/jdk/pull/294