Java 8 + Docker container - CMS collector leaves around instances that have no GC roots

[Jaikiran Pai]

On 04/12/18 9:27 AM, Jaikiran Pai wrote:
> After a bit more searching, we
> finally found these issues[1][2] that match exactly to what we are
> seeing (right down to the exact version of OS and docker and the nature
> of configuration). So it looks like it's a known issue with docker + the
> kernel version in use and apparently no known workaround (other than
> downgrading to a version of docker that doesn't hit this). There appears
> to be a commit[3] that has been done upstream but isn't yet released. We
> will evaluate how to either try and patch/test that change or figure out
> some other way (may be not set a --memory limit for now) to get past this.
>
> Thank you all again for the helpful replies.
>
> [1] https://github.com/opencontainers/runc/issues/1725
>
> [2] https://github.com/moby/moby/issues/37722
> [3]
> https://github.com/opencontainers/runc/commit/6a2c15596845f6ff5182e2022f38a65e5dfa88eb
>

An update - I found that docker-ce has committed a fix[1] for the kmem
usage issue. I upgraded to 18.09.1-beta2 of docker (which contains this
fix) on the exact same setup that we have been using to monitor this
issue. It's been almost 24 hours now and the docker stats command and
other related tools show that the memory usage is stable and well within
expected usage. On the older version of docker, for this duration of
run, we would already start seeing the usage rising over expected levels
and would die after a few days. We will continue to monitor this setup
for at least a week or more but at this point, I think this issue has
been narrowed down and sorted out with this fix in docker.

[1]
https://github.com/docker/docker-ce/commit/53e37f7583cd3d90a8069de4081ef9bebffb839f#diff-9f3e85b3e5dd57d45832939603ad323e

-Jaikiran