RFR(M): 8152995: Solaris os::available_memory() doesn't do what we think it does

[Erik Österlund]

Looping in runtime.

Thanks,
/Erik

On 2016-04-06 16:09, Erik Österlund wrote:
> Hi,
>
> Bug: https://bugs.openjdk.java.net/browse/JDK-8152995
> CR: http://cr.openjdk.java.net/~eosterlund/8152995/webrev.00/
>
> On Solaris, the os::available_memory() function is currently 
> calculated with sysconf(_SC_AVPHYS_PAGES).
>
> Unfortunately this does not match intended semantics. The intended 
> semantics is to return how much memory can be allocated by mmap into 
> physical memory. But, what _SC_AVPHYS_PAGES does is to return how many 
> physical pages are available to be used by virtual memory as backing 
> storage on-demand once it is touched, without any connection 
> whatsoever to virtual memory.
>
> Even if we mmap to commit heap memory without NORESERVE, the 
> _SC_AVPHYS_PAGES metric does not change its value - at least not until 
> somebody actually touches the mmaped memory and it starts becoming 
> backed by actual physical memory. So the JVM can in theory commit the 
> whole physical memory, and _SC_AVPHYS_PAGES will still reply that all 
> that memory is still available given that it has not been touched yet.
>
> It is likely that this is related to random swap-related test 
> failures, where too many JVMs are created based on this metric. Even 
> if it is not, the os::available_memory() call is still broken in its 
> current state and should be fixed regardless.
>
> My proposed fix uses kstat to get the available memory that can be 
> mmapped (which actually relates to virtual memory). It then uses 
> swapctl() to find out the amount of free swap, subtracting that from 
> the kstat value, to make sure we do not count swap memory as being 
> available for grabbing, to mimick the current behaviour of other 
> platforms. The code iterates over the potentially many swap resources 
> and adds up the free swap memory.
>
> kstat gives us all memory that can be made available, including memory 
> already used by the OS for things like file caches, and swap memory. 
> When this value is 0, mmap will fail. That's why I calculate the 
> amount of swap and remove that, assuming it is okay to use memory that 
> isn't immediately available but can be made available, as long as it 
> does not involve paging to the swap memory.
>
> Testing:
> * JPRT
> * Made my own test program that can be found in the comments of the 
> BUG to report on memory values, so I could verify what is going on and 
> that when the new os::available_memory() becomes 0, is indeed when 
> paging to swap starts happening using vmstat.
>
> I need a sponsor to push this if anyone is interested.
>
> Thanks,
> /Erik