[Proposal] Better systemd slice memory limit support for OpenJDK

[Severin Gehwolf]

On Thu, 2019-01-17 at 13:11 -0500, Bob Vandette wrote:
> Do you happen to know if this issue impacts Ubuntu 19.04?

I don't, sorry.

> Looking at our official OS support list, we’re only supporting Ubuntu 18.04 with JDK 12 which 
> is using kernel version 4.15.  Even if this gets bumped to 18.10, this is still only using 4.18.

I can confirm Ubuntu 18.04 doesn't seem to be affected with Kernel
4.15. RHEL-8 Beta, on the other hand, has 4.18 which is affected. I'm
also trying Ubuntu 18.10 now, which, in theory, should be affected.

> Go ahead and file a bug while we look into this issue a bit more.  I’d like to know the exact
> kernel cutoff and which distros are impacted.  If it’s wide-spread on distros getting released 
> this year, we should fix it in JDK 13.

OK. Here you go: https://bugs.openjdk.java.net/browse/JDK-8217338

Thanks,
Severin

> Bob.
> 
> 
> > On Jan 17, 2019, at 11:16 AM, Severin Gehwolf <sgehwolf at redhat.com>
> > wrote:
> > 
> > Hi Bob,
> > 
> > On Thu, 2019-01-17 at 10:59 -0500, Bob Vandette wrote:
> > > I checked a few systems I have access to and they all have
> > > use_hierarchy enabled.  When I set a memory
> > > limit the memory.stat hierarchical_memory_limit is identical to
> > > the memory.limit_in_bytes contents.
> > 
> > This seems to suggest they're running on older kernels. I know F29,
> > F28
> > are affected.
> > 
> > > Do you have any idea why the kernel behavior changed?
> > 
> > No, not really. It could be with the "unified control group
> > hierarchy"
> > work done in 3.16, but that's speculation:
> > https://lwn.net/Articles/601840/
> > 
> > Some more discussion with systemd/kernel folk is here:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1599387
> > 
> > >  Is this documented behavior?
> > 
> > Yes. Though, in the self-proclaimed very outdated document:
> > https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
> > 
> > Section 6, "Hierarchy support".
> > 
> > > I wouldn’t want to add a work-around for a transient kernel bug.
> > 
> > FWIW, cgroup-v1 is a legacy interface, so I doubt anything in that
> > code
> > has a great chance of getting "fixed". So far there is no sign of
> > evidence what the expected behaviour should be.
> > 
> > Thanks,
> > Severin
> > 
> > > Bob.
> > > 
> > > 
> > > > On Jan 17, 2019, at 9:57 AM, Severin Gehwolf <
> > > > sgehwolf at redhat.com> wrote:
> > > > 
> > > > Hi,
> > > > 
> > > > Current container awareness for OpenJDK seems to work for
> > > > systemd
> > > > slices too, on some systems. To be precise, this works for
> > > > older
> > > > Kernels e.g. 3.10. However, we've noticed that this breaks for
> > > > newer
> > > > Kernel versions[1] such as the one in F28, currently 4.19.14-
> > > > 200. If
> > > > the container support would also look at hierarchical memory
> > > > limits
> > > > exposed via memory.stat in the cgroup file system, it would
> > > > again
> > > > work[2]. A proof of concept implementation is here:
> > > > 
> > > > 
http://cr.openjdk.java.net/~sgehwolf/webrevs/container-systemd-slice-01/webrev/
> > > > 
> > > > This enhancement wouldn't change any existing container memory
> > > > limit
> > > > detection as it only kicks in when all other look-ups
> > > > determined that
> > > > there is no limit in place. I've verified this by running
> > > > Docker
> > > > container tests. The idea is to look for
> > > > hierarchical_memory_limit and
> > > > hierarchical_memsw_limit lines in the memory.stat file of the
> > > > cgroup
> > > > tree.
> > > > 
> > > > Would it be possible to consider such an enhancement upstream?
> > > > If so,
> > > > I'll file a bug and propose it for review.
> > > > 
> > > > This issue has been originally raised here:
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=1509371
> > > > 
> > > > Thanks,
> > > > Severin
> > > > 
> > > > [1] Java process gets killed by oom killer.
> > > >   See: 
> > > > http://cr.openjdk.java.net/~sgehwolf/webrevs/container-systemd-slice-01/before.txt
> > > > [2] Java process throws OutOfMemoryError as expected.
> > > >   See: 
> > > > http://cr.openjdk.java.net/~sgehwolf/webrevs/container-systemd-slice-01/after.txt
> > > > 
> > > 
> > > 
> 
>