build concurrency

David DeHaven david.dehaven at oracle.com
Tue Sep 15 21:28:30 UTC 2015


To determine the number of physical cores:

On Mac:
sysctl -n hw.physicalcpu_max

Or, if that's not available:
sysctl -n hw.ncpu


On Linux, you could parse /proc/cpuinfo (look at "physical id" and "cpu cores") using awk/grep/sed/sort/whatever

Or:
for cpudir in /sys/devices/system/cpu/cpu[0-9]*; do
  PHYS=`cat $cpudir/topology/physical_package_id`
  CORE=`cat $cpudir/topology/core_id`
  echo "$PHYS:$CORE"
done

On my quad core i7 (8 virtual processors) it outputs:
0:0
0:0
0:1
0:1
0:2
0:2
0:3
0:3

Ideally you'd build an array in the for loop and store the highest core id + 1 (to get core count). Then just add up the array entries as there should be one for each physical processor and you have the total number of physical cores.


No idea how to solve that on Windows or Solaris.

-DrD-

> Hi Erik,
> thanks for the explanation.
> 
> Regarding build times, the current heuristics scores  ok on my high-end machine (I get more or less same time as with JDK 8 build) - but with a lower spec machine (i.e. laptop with dual core intel i5) it gets much much worse - i.e. I used to be able to build in 7 minutes on my laptop (using ccache) - now build time is at least double that figure.
> 
> I know it's an hard problem to decide how many cores to use but there seem to be a pattern emerging:
> 
> * low-end machines get completely swamped by the build load
> * CPU bound tests run into troubles when reusing same concurrency settings, even on high-end hardware. Without playing with timeouts it's impossible to get a clean test sheet.
> * on relatively high-end HW, current build concurrency settings seem to be doing ok.
> 
> Realistically, I believe anything that uses more than n/2 virtual processors is going to face troubles sooner or later; the build might be ok since there's so much IO going on (reading/writing files) - but the more the build will become CPU intensive (and sjavac might help with that) the more current settings could become a bottleneck.
> 
> Maurizio
> 
> On 14/09/15 17:05, Erik Joelsson wrote:
>> Hello,
>> 
>> When I implemented the heuristic to choose a suitable default concurrency, I only ever worried about the build. I think having tests use the same concurrency setting must be a new feature? In any case, it seems like there is a case for reducing concurrency when running tests.
>> 
>> Another note. It at least used to be quite tricky to get correct information about cores vs hyperthreading from the OS. I know today we aren't even consistent with this across platforms. Perhaps we should revisit this heuristic and take hyperthreading into consideration too.
>> 
>> The current implemenation uses 100% of number of virtual cpus when 1 to 4 of them, then 90% at 5 to 16. After that it caps out at 16. (I might remember some detail wrong here)
>> 
>> /Erik
>> 
>> On 2015-09-14 04:10, Maurizio Cimadamore wrote:
>>> The information I posted was slightly incorrect, sorry - my machine has 8 cores (and 16 virtual processors) - so you see why choosing concurrency factor of 14 is particularly bad in this setup.
>>> 
>>> Maurizio
>>> 
>>> On 14/09/15 12:03, Maurizio Cimadamore wrote:
>>>> Hi,
>>>> I realized that the concurrency factor inferred by the JDK build might be too high; on a 16 core machine, concurrency is set to 14 - which then leads to absurd load averages (50-ish) when building/running tests. High load when building is not a big issue, but when running test this almost always turns into spurious failures due to timeouts. I know I can override the concurrency factor with --with-jobs - but I was curious as to why the default parameter is set to such aggressive value?
>>>> 
>>>> Thanks
>>>> Maurizio
>>> 
>> 
> 




More information about the build-dev mailing list