Disable Processor Counting based on Cgroups Container Support

[Severin Gehwolf]

Hi Bruno,

On Fri, 2023-05-12 at 10:18 -0700, Bruno Borges wrote:
> Hi folks,
> 
> Looking at the source code, I couldn't identify a way (on tip, or any
> previous release) to keep container support for memory limit, but
> disable it for CPU limit.

Why is that a use-case worth considering? Could you elaborate (see
below)?

> While I could achieve this by not setting a CPU limit in my container
> orchestration solution (e.g., Kubernetes), it wouldn't necessarily
> give me what I am looking for: I do want to set a CPU limit at the
> container level, but I want the JVM to count for all processors in
> the host/node.

This seems a contradiction to what you said above: Keep container
memory limit, but disable for CPU (=> host/node cpu values).

> While I can manually tweak this by modifying -
> XX:ActiveProcessorCount, this requires me to know upfront how many
> processors the Node/Host will have.

It's not clear to me why you'd want to have the node/host values for
CPU, yet container memory limits for the JVM *given that* you'd want to
verify the on-the-fly update of container resource limits. What's even
more strange is that you seem to be asking for host/node CPU values,
even though you have CPU container quotas in place. In that scenario
the only option is ActiveProcessorCount=$(nproc), IMO.

Anyway, in my experiments it behaves as expected as far as the
adjusting to changed resource config is concerned. The JVM dynamically
adjusts to the changed resource limits. At least as far as the
observable limits via the container detection code are concerned. I
cannot comment on the other aspects of the JVM reacting to those
changed values (a quick glance suggests some do handle those; some
probably not).

> Additionally, setting -XX:ActiveProcessorCount=-1 does not give me
> the behavior I was expecting either.

That will have no effect. With "-XX:+UseContainerSupport", it'll return
the container values.

> The reason I raise this is due to the new In-place Pod Resource
> Resizing feature in Kubernetes 1.27, that allows a pod to change the
> CPU limit without restarting the pod. See this article for
> details: https://medium.com/@karla.saur/trying-out-the-new-in-place-p
> od-resource-resizing-68a0b3c42b72

Got it. With a simple test program like this:

$ cat CPUsPoll.java 
public class CPUsPoll {
 public static void main(String[] args) throws Exception {
 Runtime rt = Runtime.getRuntime();
 while (true) {
 int cpus = rt.availableProcessors();
 System.out.println("activeProcessorCount = " + cpus);
 Thread.sleep(2000);
 }
 }
}

Run it in a container like this:
$ sudo podman run --rm -ti --cpu-quota=300000 --cpu-period=100000 -v $(pwd):/opt/classes:z -v $(pwd)/jdk:/opt/jdk:z fedora:37 /opt/jdk/bin/java -cp /opt/classes CPUsPoll
activeProcessorCount = 3
activeProcessorCount = 3
activeProcessorCount = 3
activeProcessorCount = 3
activeProcessorCount = 3
activeProcessorCount = 3
activeProcessorCount = 3
activeProcessorCount = 3
[...]

Then in another terminal, update the resources to say, 4 CPUs like so:

$ sudo podman update --cpu-period=100000 --cpu-quota=400000 fc435a1aea79
fc435a1aea796ad20cb523ba57a4c2eba499cd98c6feb57df659c712f663bee3

Back in the other terminal where the current CPU cores are being
printed I get:

activeProcessorCount = 3
[...]
activeProcessorCount = 4
activeProcessorCount = 4
activeProcessorCount = 4

Thus, I can update the resource config and container support will
figure it out.

> In an ideal world, the JVM would dynamically scale its thread pools,
> GC threads, and other components at JVM and Application level
> accordingly, but we are not there yet, I believe.

IMO, there are several angles to this problem:
   1. Container awareness picking up dynamic updates
   2. JVM actually adjusting to those updates
   3. Applications adjusting to those updates

Now the question is what you are looking for 1), 2) or 3) or something
else entirely? Note that most of the time it's application code which
needs to adjust to changed environment. Many assume those values remain
static.

Thanks,
Severin