Effect of setting CPU quota on Java performance

[David Holmes]

All "processors" are not created equal. Whether you bind to 4 or 8 
"cpus" it will matter a great deal which 4 or 8 you choose. It would 
probably be more instructive to keep the same set of CPUs and change the 
quota and then see if your performance follows expectations.

That said it will also depend a great deal on exactly how CPU quotas are 
implemented by the system and on how you are loading the system. If the 
quotas only kick in if the CPU's you have allocated to you are 
overloaded then you need to ensure your loading process runs on the same 
set of CPUs.

David

On 5/02/2018 3:48 PM, Ashutosh Mehra1 wrote:
> I have been trying to understand if setting CPU quota limit on a docker
> container, provided the "effective" CPUs are the same, has any impact on
> the application performance.
> As an example, if my app is running on 4 CPUs @ 100% quota, would I get
> same performance if my app is running on 8 CPUs at 50% quota? Note that
> "effective" CPUs is 4 in both cases.
> 
> Since OpenJDK early access builds for Java 10 have improved support for
> docker container (https://bugs.openjdk.java.net/browse/JDK-8146115), I
> decided to do some measurements using that build.
> I got the build OpenJDK build jdk-10-ea+40 from http://jdk.java.net/10/.
> This build by default has container support enabled.
> The system I used has 32 CPUs including 2 hyperthreads per core. I turned
> off hyperthreading for this experiment. That leaves me with 16 cores on 2
> sockets: 0-7 on 1 socket and 8-15 on 2nd socket.
> System details are:
> 
> # lsb_release -a
> No LSB modules are available.
> Distributor ID:          Ubuntu
> Description:             Ubuntu 16.04.2 LTS
> Release:                 16.04
> Codename:                xenial
> 
> # uname -r
> 4.4.0-103-generic
> 
> For measurements I used AcmeAir benchmark at
> https://github.com/sabkrish/acmeair/tree/microservice_changes.
> I ran the AcmeAir benchmark with the said build for following cases:
>   
> 1) Ran AcmeAir with JVM bound to 4 cpus (8-11) and no limit on quota. Lets
> call this 4CPU at 100.
> This used following JVM settings:
> CICompilerCount = 3, ParallelGCThreads = 4, ConcGCThreads = 1
>   
> 2) Case 8Cpu50Quota: Ran AcmeAir with JVM bound to 8 cpus (8-15) and 50%
> quota. Lets call this 8CPU at 50.
> In this case container support was enabled by default and it used
> following JVM settings:
> CICompilerCount = 3, ParallelGCThreads = 4, ConcGCThreads = 1
>   
> 3) Ran AcmeAir with JVM bound to 8 cpus (8-15) and 50% quota with
> -XX:-UseContainerSupport option to disable container support. Lets call
> this 8CPU at 50NoCS.
> This used following JVM settings:
> CICompilerCount = 4, ParallelGCThreads = 8, ConcGCThreads = 2
>   
> Load on the server was applied using JMeter which was running on same box
> but bound to 0-8 CPUs. I applied the load for few minutes to warm up the
> JVM before starting the final "measure" run.
> Throughput reported below is for the final "measure" run. All numbers
> reported below are an average of 10 iterations.
>   
> Throughput result:
> 4Cpu1 at 100  |  8Cpu at 50 |  8Cpu at 50NoCS
> 9621.5            |  6970.6      |  7252.1
>   
> I also measured Total compilation time (in seconds) and Total pause time
> (in seconds) for the duration of the server (which includes warm up
> phase):
>   
> Compilation time:
> 4Cpu1 at 100  |  8Cpu at 50  |  8Cpu at 50NoCS
> 79.8545          |  76.7041     |  100.085
>   
> GC Pasue time:
> 4Cpu1 at 100  |  8Cpu at 50  |  8Cpu at 50NoCS
> 1.829              |  1.886         |  1.927
>   
> I am quite surprised to see the drop in throughput between 4Cpu100Quota
> and 8Cpu50Quota case.
> Looking deeper into the results, I do notice that numbers for 8Cpu50Quota
> case were not very consistent, but in general I can say they are not
> matching 4Cpu100Quota case.
> I will be doing additional runs for this setup (and on different OS/kernel
> version) and increase warm up time for the JVM to see if that improves the
> consistency.
>   
> Meanwhile, couple of questions I wanted to put forward:
> 1) Has anyone else noticed this kind of difference in Java
> application/JVM performance when CPU quota is used?
> 2) What other open-source benchmarks are available that I can use to
> verify the behavior I am observing?
>   
> Any comments/feedback are welcome.
>   
> Thanks,
> Ashutosh Mehra
>