OperatingSystemMXBean unaware of container memory limits

Fri Jun 21 12:27:43 UTC 2019

Hi Kirk,

I think I understand what you mean, however then the OS Bean should be
consistent regarding CPU information as well.

I think I remember why this was fixed the way it is now was because of
incorrect behavior during GC configuration, or something along those lines,
so I guess we would need two APIs after all to give tooling a way to sneak
into the actual hardware properties.

I would guess, however, that from the point of view of a containerised VM
its OS is the container not the bare metal (or virtualized metal perhaps),
so tooling would need to check specifically for a separate bean.

That could be indicated via a property in the OS Bean (if it’s not the case
already).

Nevertheless, I think consistency in the OS Bean should be achieved.

Cheers,
Mario

On Fri 21. Jun 2019 at 13:23, Kirk Pepperdine <kirk.pepperdine at gmail.com>
wrote:

> Hi Mario,
>
> I don’t believe the MBean returns the wrong information. Is it not that
> the calls by-pass the container? Would it not be more appropriate to add a
> container aware mean? From a tooling perspective it’s a mistake to
> completely seal the JVM away from the hardware as it makes certain
> diagnostics more difficult to perform.
>
> Kind regards,
> Kirk
>
>
> On Jun 21, 2019, at 6:06 AM, Mario Torre <neugens.limasoftware at gmail.com>
> wrote:
>
> The way I understood the bug report is a two fold approach, i.e. fix the
> os bean and *possibly* add a container one or extend it to add more data.
>
> I agree with you and Andrew that the current OS Bean returns wrong
> information, this should be fixed in any case I think.
>
> Bob, do you have some design ideas to share regarding the new interface? I
> think this may be an interesting project to pick up, even for students
> wanting to write a thesis, as it seems time is a limiting factor here for
> you to work on that?
>
> Cheers,
> Mario
>
> On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <sgehwolf at redhat.com> wrote:
>
>> Hi Bob,
>>
>> On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
>> > Hi Andrew,
>> >
>> > I am aware of the limitations of the OperatingSystemMXBean and was
>> > hoping to address these limitations during the implementation of
>> > https://bugs.openjdk.java.net/browse/JDK-8199944.
>> >
>> > It would be helpful if you feel this is important to add some votes
>> > to this issue.
>>
>> It seems strange that the getAvailableProcessors() returns the
>> container limit, while the memory limits are for the physical host. If
>> anything, shouldn't they agree (both physical host or both container
>> limits)?
>>
>> When I briefly looked into it initially it seems to be a side-effect of
>> what is being used by the JDK code implementation-wise. IIRC
>> getAvailableProcessors() uses a runtime call. Memory reporting has it's
>> own logic[1], thus not reporting the container limit.
>>
>> This seems weird from a consistency perspective. Thoughts?
>>
>> If I understand you correctly, you are arguing that container reporting
>> should go into its own MX bean. On the other hand, CPU reporting for
>> the OS MX bean is container aware already. That makes me believe we
>> should rather make this consistent before evaluating a new MX bean.
>>
>> Thanks,
>> Severin
>>
>> [1]
>> http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365
>>
>>
>> > Bob.
>> >
>> >
>> > > On Jun 20, 2019, at 9:43 AM, Andrew Azores <aazores at redhat.com>
>> > > wrote:
>> > >
>> > > Hi all,
>> > >
>> > > Apologies if this is not the most appropriate list, in which case
>> > > please direct me where to go.
>> > >
>> > > I've noticed a surprising result from the
>> > > com.sun.management.OperatingSystemMXBean implementation when
>> > > running in
>> > > a containerized (specifically, using Docker on Linux) environment.
>> > > The
>> > > bean appears to be container-aware for processors, in that running
>> > > with
>> > > Docker option `--cpus 1.0` for example, on a multicore system, will
>> > > cause both java.lang.Runtime#availableProcessors and
>> > > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
>> > > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
>> > > return 1. However, the Docker option `--memory 100M` (or any other
>> > > limit value) is not reflected in the value returned by
>> > > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
>> > > ,
>> > > and instead the returned value is still the total physical memory
>> > > of
>> > > the host machine - of which only a small portion may actually be
>> > > available to the "Operating System" of the JVM. Similarly for the
>> > > methods regarding free physical memory, total swap, and free swap.
>> > >
>> > > I have attached a patch which adds a small reproducer to the
>> > > existing
>> > > MemoryAwareness test.
>> > >
>> > > This seems like a bug to me, since if the imposed container limit
>> > > on
>> > > processors as a resource is included as part of the "Operating
>> > > System"
>> > > resource reporting, then surely memory resources should be reported
>> > > the
>> > > same way. As I said, I found the current behaviour quite
>> > > surprising.
>> > >
>> > > --
>> > > Andrew Azores
>> > > Software Engineer, OpenJDK Team
>> > > Red Hat
>> > > <jdk-osmxbean-container-memory-test-01.patch>
>> >
>> >
>>
>> --
> pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
> Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF
>
> Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
> Proud GNU Classpath developer: http://www.classpath.org/
> OpenJDK: http://openjdk.java.net/projects/caciocavallo/
>
> Please, support open standards:
> http://endsoftpatents.org/
>
>
> --
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20190621/6812425d/attachment.html>