Review Request: 8004352: build-infra: Limit JOBS on large machines

[Volker Simonis]

Hi Erik,

I really like these changes! Thank you for addressing these problems.

Volker


On Fri, Feb 15, 2013 at 1:06 PM, Erik Joelsson <erik.joelsson at oracle.com> wrote:
> The current default for number of parallel build jobs is just equal to the
> number of cores in the system. While this works well on many machines, there
> have been several reports of this not working. I have been trying to come up
> with a better scheme for the default and the following is something that I
> think will do. It has been active in the build-infra forest for a while now.
>
> The complaints that were raised were usually concerning one of the
> following:
>
> * A big machine with very many cpus didn't scale well when using all of
> them, so there is no point in trying to, it just made the build less stable.
> Suggestion, introduce a hard cap.
> * A machine with lots of cpus but not as much memory would run out of
> memory. Suggestion, limit number of jobs on memory as well as cpus.
> * The build eats up all my resources, leaving my browser unusable.
> Suggestion, don't use everything unless asked for.
>
> So this is what I ended up with:
> 1. Take the min of num_cores and memory_size/1000
> 2. Cap at 16
> 3. If more than 4 cores, shave it to 90% rounded down to leave some room.
>
> The user can still override this in two ways. Either by using any of the
> configure arguments:
>
>   --with-num-cores        number of cores in the build system, e.g.
>                           --with-num-cores=8 [probed]
>   --with-memory-size      memory (in MB) available in the build system, e.g.
>                           --with-memory-size=1024 [probed]
>   --with-jobs             number of parallel jobs to let make run
> [calculated
>                           based on cores and memory]
>
> Or by setting JOBS=<number> on the make command line.
>
> Also not that this change will set the CONCURRENT_BUILD_JOBS for hotspot to
> the value of JOBS so that it too will be overridden by the above.
>
> http://cr.openjdk.java.net/~erikj/8004352/webrev.root.01/
>
> /Erik