build concurrency

[Magnus Ihse Bursie]

On 2015-09-15 11:53, Maurizio Cimadamore wrote:
> Hi Erik,
> thanks for the explanation.
>
> Regarding build times, the current heuristics scores  ok on my 
> high-end machine (I get more or less same time as with JDK 8 build) - 
> but with a lower spec machine (i.e. laptop with dual core intel i5) it 
> gets much much worse - i.e. I used to be able to build in 7 minutes on 
> my laptop (using ccache) - now build time is at least double that figure.
>
> I know it's an hard problem to decide how many cores to use but there 
> seem to be a pattern emerging:

I'd like to expand that statement by saying that it's a hard problem to 
optimize the build for all kinds of platforms out of the box. We make no 
claims that the default configuration should be optimal for any specific 
setup. Hopefully, it will be "good enough" for most settings, but if you 
really need optimal performance, you will need to perform experiments on 
your specific setting, and choose number of jobs, ccache and precompiled 
headers that suits your particular hardware and type of build. Our 
internal build system at Oracle uses long configure lines to achieve 
(what we believe) are best performance for that setup, for instance.

That said, if we can make sane changes that improves performance for 
most users and degrades performance for none, or very few, out of the 
box, we will of couse try to do so.

/Magnus

>
> * low-end machines get completely swamped by the build load
> * CPU bound tests run into troubles when reusing same concurrency 
> settings, even on high-end hardware. Without playing with timeouts 
> it's impossible to get a clean test sheet.
> * on relatively high-end HW, current build concurrency settings seem 
> to be doing ok.
>
> Realistically, I believe anything that uses more than n/2 virtual 
> processors is going to face troubles sooner or later; the build might 
> be ok since there's so much IO going on (reading/writing files) - but 
> the more the build will become CPU intensive (and sjavac might help 
> with that) the more current settings could become a bottleneck.
>
> Maurizio
>
> On 14/09/15 17:05, Erik Joelsson wrote:
>> Hello,
>>
>> When I implemented the heuristic to choose a suitable default 
>> concurrency, I only ever worried about the build. I think having 
>> tests use the same concurrency setting must be a new feature? In any 
>> case, it seems like there is a case for reducing concurrency when 
>> running tests.
>>
>> Another note. It at least used to be quite tricky to get correct 
>> information about cores vs hyperthreading from the OS. I know today 
>> we aren't even consistent with this across platforms. Perhaps we 
>> should revisit this heuristic and take hyperthreading into 
>> consideration too.
>>
>> The current implemenation uses 100% of number of virtual cpus when 1 
>> to 4 of them, then 90% at 5 to 16. After that it caps out at 16. (I 
>> might remember some detail wrong here)
>>
>> /Erik
>>
>> On 2015-09-14 04:10, Maurizio Cimadamore wrote:
>>> The information I posted was slightly incorrect, sorry - my machine 
>>> has 8 cores (and 16 virtual processors) - so you see why choosing 
>>> concurrency factor of 14 is particularly bad in this setup.
>>>
>>> Maurizio
>>>
>>> On 14/09/15 12:03, Maurizio Cimadamore wrote:
>>>> Hi,
>>>> I realized that the concurrency factor inferred by the JDK build 
>>>> might be too high; on a 16 core machine, concurrency is set to 14 - 
>>>> which then leads to absurd load averages (50-ish) when 
>>>> building/running tests. High load when building is not a big issue, 
>>>> but when running test this almost always turns into spurious 
>>>> failures due to timeouts. I know I can override the concurrency 
>>>> factor with --with-jobs - but I was curious as to why the default 
>>>> parameter is set to such aggressive value?
>>>>
>>>> Thanks
>>>> Maurizio
>>>
>>
>