[Proposal] Better systemd slice memory limit support for OpenJDK

[Bob Vandette]

Severin,

I’m looking at other Docker and Container issues for inclusion in JDK 13 and
I’m trying to figure out if we should do anything about the issue you reported.

If this problem is going away once the Linux distributions pick up Linux kernel
4.19 or greater, then I’d suggest we just wait.

It looks like support for Ubuntu 18.10 ends this July and Ubuntu 19.04 and Fedora 30 
are coming out next month (April).

Do you agree?

Bob.



> On Jan 17, 2019, at 2:24 PM, Severin Gehwolf <sgehwolf at redhat.com> wrote:
> 
> On Thu, 2019-01-17 at 13:11 -0500, Bob Vandette wrote:
>> Do you happen to know if this issue impacts Ubuntu 19.04?
> 
> I don't, sorry.
> 
>> Looking at our official OS support list, we’re only supporting Ubuntu 18.04 with JDK 12 which 
>> is using kernel version 4.15.  Even if this gets bumped to 18.10, this is still only using 4.18.
> 
> I can confirm Ubuntu 18.04 doesn't seem to be affected with Kernel
> 4.15. RHEL-8 Beta, on the other hand, has 4.18 which is affected. I'm
> also trying Ubuntu 18.10 now, which, in theory, should be affected.
> 
>> Go ahead and file a bug while we look into this issue a bit more.  I’d like to know the exact
>> kernel cutoff and which distros are impacted.  If it’s wide-spread on distros getting released 
>> this year, we should fix it in JDK 13.
> 
> OK. Here you go: https://bugs.openjdk.java.net/browse/JDK-8217338
> 
> Thanks,
> Severin
> 
>> Bob.
>> 
>> 
>>> On Jan 17, 2019, at 11:16 AM, Severin Gehwolf <sgehwolf at redhat.com>
>>> wrote:
>>> 
>>> Hi Bob,
>>> 
>>> On Thu, 2019-01-17 at 10:59 -0500, Bob Vandette wrote:
>>>> I checked a few systems I have access to and they all have
>>>> use_hierarchy enabled.  When I set a memory
>>>> limit the memory.stat hierarchical_memory_limit is identical to
>>>> the memory.limit_in_bytes contents.
>>> 
>>> This seems to suggest they're running on older kernels. I know F29,
>>> F28
>>> are affected.
>>> 
>>>> Do you have any idea why the kernel behavior changed?
>>> 
>>> No, not really. It could be with the "unified control group
>>> hierarchy"
>>> work done in 3.16, but that's speculation:
>>> https://lwn.net/Articles/601840/
>>> 
>>> Some more discussion with systemd/kernel folk is here:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1599387
>>> 
>>>> Is this documented behavior?
>>> 
>>> Yes. Though, in the self-proclaimed very outdated document:
>>> https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
>>> 
>>> Section 6, "Hierarchy support".
>>> 
>>>> I wouldn’t want to add a work-around for a transient kernel bug.
>>> 
>>> FWIW, cgroup-v1 is a legacy interface, so I doubt anything in that
>>> code
>>> has a great chance of getting "fixed". So far there is no sign of
>>> evidence what the expected behaviour should be.
>>> 
>>> Thanks,
>>> Severin
>>> 
>>>> Bob.
>>>> 
>>>> 
>>>>> On Jan 17, 2019, at 9:57 AM, Severin Gehwolf <
>>>>> sgehwolf at redhat.com> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> Current container awareness for OpenJDK seems to work for
>>>>> systemd
>>>>> slices too, on some systems. To be precise, this works for
>>>>> older
>>>>> Kernels e.g. 3.10. However, we've noticed that this breaks for
>>>>> newer
>>>>> Kernel versions[1] such as the one in F28, currently 4.19.14-
>>>>> 200. If
>>>>> the container support would also look at hierarchical memory
>>>>> limits
>>>>> exposed via memory.stat in the cgroup file system, it would
>>>>> again
>>>>> work[2]. A proof of concept implementation is here:
>>>>> 
>>>>> 
> http://cr.openjdk.java.net/~sgehwolf/webrevs/container-systemd-slice-01/webrev/
>>>>> 
>>>>> This enhancement wouldn't change any existing container memory
>>>>> limit
>>>>> detection as it only kicks in when all other look-ups
>>>>> determined that
>>>>> there is no limit in place. I've verified this by running
>>>>> Docker
>>>>> container tests. The idea is to look for
>>>>> hierarchical_memory_limit and
>>>>> hierarchical_memsw_limit lines in the memory.stat file of the
>>>>> cgroup
>>>>> tree.
>>>>> 
>>>>> Would it be possible to consider such an enhancement upstream?
>>>>> If so,
>>>>> I'll file a bug and propose it for review.
>>>>> 
>>>>> This issue has been originally raised here:
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1509371
>>>>> 
>>>>> Thanks,
>>>>> Severin
>>>>> 
>>>>> [1] Java process gets killed by oom killer.
>>>>>  See: 
>>>>> http://cr.openjdk.java.net/~sgehwolf/webrevs/container-systemd-slice-01/before.txt
>>>>> [2] Java process throws OutOfMemoryError as expected.
>>>>>  See: 
>>>>> http://cr.openjdk.java.net/~sgehwolf/webrevs/container-systemd-slice-01/after.txt
>>>>> 
>>>> 
>>>> 
>> 
>> 
>