OperatingSystemMXBean unaware of container memory limits

[Bob Vandette]

> On Jul 11, 2019, at 1:18 PM, Andrew Azores <aazores at redhat.com> wrote:
> 
> Hi Bob (and all),
> 
> On Fri, 2019-06-21 at 08:50 -0400, Bob Vandette wrote:
>>> 
>> Fixing the existing OS Mbean is pretty straight forward.  Just have
>> each method call the new Metrics API, check for error returns
>> in case this API is not supported on this platform and if success,
>> return result, otherwise call existing logic.
>> 
> 
> I've had free time to circle back to this issue lately, so I've
> prepared a patch which modifies the OperatingSystemMXBean as you've
> described here. This seems to work perfectly well for
> getTotalPhysicalMemorySize, getFreePhysicalMemorySize, and
> getTotalSwapSpaceSize.


> getFreeSwapSpaceSize looks at a glance to be
> implementable by comparing Metrics' getMemoryAndSwapUsage,
> getMemoryUsage, and getMemoryAndSwap limit, but the implementation
> would only provide an approximation (and not necessarily accurately)
> since the amount of memory/swap used would likely change in between or
> due to subsequent calls to the Metrics API.

That one appears a bit tricky to get right.

Assuming memory and swap limits are set, returning getMemoryAndSwapLimit -  getMemoryLimit - getMemoryAndSwapUsage.
Since the limits are constant, there’s no issue with variability between the two calls.

However, if the Swap limit is unlimited then I’d revert to the original logic since the limit is
the hosts when swap is unlimited.

One problem I see if that in cgroups, memory.swappiness can alter or disable swapping.
I didn’t provide a Metrics API that returns this value or take this into account in
the result of getMemoryAndSwapLimit.

> 
> Also, I'm not sure that there are reasonable equivalents for all of the
> other metrics implemented in
> jdk.management/unix/classes/com/sun/management/internal/OperatingSystem
> Impl.java available from Metrics or elsewhere which make sense in a
> containerized environment (ex. getCommittedVirtualMemorySize).

We’d have to experiment with this but since this is actual committed memory for
the current process, I would expect the existing logic to be accurate.

> 
> So, before I proceed much further with this patch, does anyone have
> input on how these issues should be handled such that the
> OperatingSystemMXBean is self-consistent wrt its container awareness?
> 

Looking at the list of native methods in OperatingSystemImpl.java,
it looks like the ones that are process based can be handled by the existing
logic.  The APIs that are system wide need to be forwarded to the Metrics
API.  The only problematic one I see is getSystemCpuLoad0.  This one needs
some thought.   I don’t see a way of determining the total containers
Cpu load assuming there are multiple process running in a single container
and the fact that Cgroups only provides cpu usage (in time) and not load.


Fall Thru:

    private native long getCommittedVirtualMemorySize0();
    private native long getMaxFileDescriptorCount0();
    private native long getOpenFileDescriptorCount0();
    private native long getProcessCpuTime0();
    private native double getProcessCpuLoad0();

Metrics:

    private native long getFreePhysicalMemorySize0();
    private native long getFreeSwapSpaceSize0();
    private native long getTotalPhysicalMemorySize0();
    private native long getTotalSwapSpaceSize0()

???:

    private native double getSystemCpuLoad0();

Bob.

> -- 
> Andrew Azores
> Software Engineer, OpenJDK Team
> Red Hat
>