OperatingSystemMXBean unaware of container memory limits

Fri Jun 21 12:40:33 UTC 2019

I agree with you Mario.  When I originally designed the jdk.internal.platform.Metrics
API, I was considering providing both a Host and Container Metrics implementation.
The problem is that the primary goal of containers is to provide isolation (hiding access to
the host) and it would be very difficult to provide reliable forms of both metrics from within
a container.  The Linux kernel does leak some host metric information but not everything.
This is why I decided to focus on the container configuration and metric data.  If Host
metrics are of interest you can always just run Java outside of a container.

Bob.

> On Jun 21, 2019, at 8:27 AM, Mario Torre <neugens.limasoftware at gmail.com> wrote:
> 
> Hi Kirk,
> 
> I think I understand what you mean, however then the OS Bean should be consistent regarding CPU information as well.
> 
> I think I remember why this was fixed the way it is now was because of incorrect behavior during GC configuration, or something along those lines, so I guess we would need two APIs after all to give tooling a way to sneak into the actual hardware properties.
> 
> I would guess, however, that from the point of view of a containerised VM its OS is the container not the bare metal (or virtualized metal perhaps), so tooling would need to check specifically for a separate bean.
> 
> That could be indicated via a property in the OS Bean (if it’s not the case already).
> 
> Nevertheless, I think consistency in the OS Bean should be achieved.
> 
> Cheers,
> Mario 
> 
> On Fri 21. Jun 2019 at 13:23, Kirk Pepperdine <kirk.pepperdine at gmail.com <mailto:kirk.pepperdine at gmail.com>> wrote:
> Hi Mario,
> 
> I don’t believe the MBean returns the wrong information. Is it not that the calls by-pass the container? Would it not be more appropriate to add a container aware mean? From a tooling perspective it’s a mistake to completely seal the JVM away from the hardware as it makes certain diagnostics more difficult to perform.
> 
> Kind regards,
> Kirk
> 
> 
>> On Jun 21, 2019, at 6:06 AM, Mario Torre <neugens.limasoftware at gmail.com <mailto:neugens.limasoftware at gmail.com>> wrote:
>> 
>> The way I understood the bug report is a two fold approach, i.e. fix the os bean and *possibly* add a container one or extend it to add more data.
>> 
>> I agree with you and Andrew that the current OS Bean returns wrong information, this should be fixed in any case I think.
>> 
>> Bob, do you have some design ideas to share regarding the new interface? I think this may be an interesting project to pick up, even for students wanting to write a thesis, as it seems time is a limiting factor here for you to work on that?
>> 
>> Cheers,
>> Mario
>> 
>> On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <sgehwolf at redhat.com <mailto:sgehwolf at redhat.com>> wrote:
>> Hi Bob,
>> 
>> On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
>> > Hi Andrew,
>> > 
>> > I am aware of the limitations of the OperatingSystemMXBean and was
>> > hoping to address these limitations during the implementation of
>> > https://bugs.openjdk.java.net/browse/JDK-8199944 <https://bugs.openjdk.java.net/browse/JDK-8199944>.
>> > 
>> > It would be helpful if you feel this is important to add some votes
>> > to this issue.
>> 
>> It seems strange that the getAvailableProcessors() returns the
>> container limit, while the memory limits are for the physical host. If
>> anything, shouldn't they agree (both physical host or both container
>> limits)?
>> 
>> When I briefly looked into it initially it seems to be a side-effect of
>> what is being used by the JDK code implementation-wise. IIRC
>> getAvailableProcessors() uses a runtime call. Memory reporting has it's
>> own logic[1], thus not reporting the container limit.
>> 
>> This seems weird from a consistency perspective. Thoughts?
>> 
>> If I understand you correctly, you are arguing that container reporting
>> should go into its own MX bean. On the other hand, CPU reporting for
>> the OS MX bean is container aware already. That makes me believe we
>> should rather make this consistent before evaluating a new MX bean.
>> 
>> Thanks,
>> Severin
>> 
>> [1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365 <http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365>
>> 
>> 
>> > Bob.
>> > 
>> > 
>> > > On Jun 20, 2019, at 9:43 AM, Andrew Azores <aazores at redhat.com <mailto:aazores at redhat.com>>
>> > > wrote:
>> > > 
>> > > Hi all,
>> > > 
>> > > Apologies if this is not the most appropriate list, in which case
>> > > please direct me where to go.
>> > > 
>> > > I've noticed a surprising result from the
>> > > com.sun.management.OperatingSystemMXBean implementation when
>> > > running in
>> > > a containerized (specifically, using Docker on Linux) environment.
>> > > The
>> > > bean appears to be container-aware for processors, in that running
>> > > with
>> > > Docker option `--cpus 1.0` for example, on a multicore system, will
>> > > cause both java.lang.Runtime#availableProcessors and
>> > > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
>> > > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
>> > > return 1. However, the Docker option `--memory 100M` (or any other
>> > > limit value) is not reflected in the value returned by
>> > > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
>> > > ,
>> > > and instead the returned value is still the total physical memory
>> > > of
>> > > the host machine - of which only a small portion may actually be
>> > > available to the "Operating System" of the JVM. Similarly for the
>> > > methods regarding free physical memory, total swap, and free swap.
>> > > 
>> > > I have attached a patch which adds a small reproducer to the
>> > > existing
>> > > MemoryAwareness test.
>> > > 
>> > > This seems like a bug to me, since if the imposed container limit
>> > > on
>> > > processors as a resource is included as part of the "Operating
>> > > System"
>> > > resource reporting, then surely memory resources should be reported
>> > > the
>> > > same way. As I said, I found the current behaviour quite
>> > > surprising.
>> > > 
>> > > -- 
>> > > Andrew Azores
>> > > Software Engineer, OpenJDK Team
>> > > Red Hat
>> > > <jdk-osmxbean-container-memory-test-01.patch>
>> > 
>> > 
>> 
>> -- 
>> pgp key: http://subkeys.pgp.net/ <http://subkeys.pgp.net/> PGP Key ID: 80F240CF
>> Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF
>> 
>> Java Champion - Blog: http://neugens.wordpress.com <http://neugens.wordpress.com/> - Twitter: @neugens
>> Proud GNU Classpath developer: http://www.classpath.org/ <http://www.classpath.org/>
>> OpenJDK: http://openjdk.java.net/projects/caciocavallo/ <http://openjdk.java.net/projects/caciocavallo/>
>> 
>> Please, support open standards:
>> http://endsoftpatents.org/ <http://endsoftpatents.org/>
> 
> -- 
> pgp key: http://subkeys.pgp.net/ <http://subkeys.pgp.net/> PGP Key ID: 80F240CF
> Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF
> 
> Java Champion - Blog: http://neugens.wordpress.com <http://neugens.wordpress.com/> - Twitter: @neugens
> Proud GNU Classpath developer: http://www.classpath.org/ <http://www.classpath.org/>
> OpenJDK: http://openjdk.java.net/projects/caciocavallo/ <http://openjdk.java.net/projects/caciocavallo/>
> 
> Please, support open standards:
> http://endsoftpatents.org/ <http://endsoftpatents.org/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20190621/c5240040/attachment-0001.html>