From sadhak001 at gmail.com  Wed Apr 19 07:42:28 2017
From: sadhak001 at gmail.com (Mani Sarkar)
Date: Wed, 19 Apr 2017 07:42:28 +0000
Subject: OpenJDK ARM build not using all the CPU cores
Message-ID: <CAGHtMWkCyVoSOwUp=Mxa=0NBrwwfnyrExnH38DoEnboej2kS8Q@mail.gmail.com>

Hi all,

Whilst build OpenJDK (latest version from
http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the build
system does not make use of all the CPU cores available on the machine even
though it detects them all.

(see snippets from the build log below)

Detects *4 cores*

checking for number of cores... 4


But scales down to *1 core* only

Build performance summary:
* Cores to use:   1
* Memory limit:   2021 MB
* ccache status:  installed and in use


We have use the --with-num-cores flag with configure but to no avail.

Has anyone else has noticed this, is this a known issue or expected
behaviour or any config that needs applying in order for this to make use
of all the cores.

Let me know if you need to see other aspects of the build log.

Thanks.

Cheers,
Mani
-- 
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*

From sadhak001 at gmail.com  Wed Apr 19 07:44:11 2017
From: sadhak001 at gmail.com (Mani Sarkar)
Date: Wed, 19 Apr 2017 07:44:11 +0000
Subject: Fwd: OpenJDK ARM build not using all the CPU cores
In-Reply-To: <CAGHtMWkCyVoSOwUp=Mxa=0NBrwwfnyrExnH38DoEnboej2kS8Q@mail.gmail.com>
References: <CAGHtMWkCyVoSOwUp=Mxa=0NBrwwfnyrExnH38DoEnboej2kS8Q@mail.gmail.com>
Message-ID: <CAGHtMWnKDahNhNb3AOUX3M=Uqppmu6Ra_yBYbAeR5kjamKrpAg@mail.gmail.com>

+adoption-discuss at openjdk.java.net <adoption-discuss at openjdk.java.net> to
the discussion

---------- Forwarded message ---------
From: Mani Sarkar <sadhak001 at gmail.com>
Date: Wed, 19 Apr 2017 at 08:42
Subject: OpenJDK ARM build not using all the CPU cores
To: build-dev at openjdk.java.net <build-dev at openjdk.java.net>,
build-infra-dev at openjdk.java.net <build-infra-dev at openjdk.java.net>


Hi all,

Whilst build OpenJDK (latest version from
http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the build
system does not make use of all the CPU cores available on the machine even
though it detects them all.

(see snippets from the build log below)

Detects *4 cores*

checking for number of cores... 4


But scales down to *1 core* only

Build performance summary:
* Cores to use:   1
* Memory limit:   2021 MB
* ccache status:  installed and in use


We have use the --with-num-cores flag with configure but to no avail.

Has anyone else has noticed this, is this a known issue or expected
behaviour or any config that needs applying in order for this to make use
of all the cores.

Let me know if you need to see other aspects of the build log.

Thanks.

Cheers,
Mani
-- 
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*
-- 
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*

From aph at redhat.com  Wed Apr 19 07:56:09 2017
From: aph at redhat.com (Andrew Haley)
Date: Wed, 19 Apr 2017 08:56:09 +0100
Subject: OpenJDK ARM build not using all the CPU cores
In-Reply-To: <CAGHtMWkCyVoSOwUp=Mxa=0NBrwwfnyrExnH38DoEnboej2kS8Q@mail.gmail.com>
References: <CAGHtMWkCyVoSOwUp=Mxa=0NBrwwfnyrExnH38DoEnboej2kS8Q@mail.gmail.com>
Message-ID: <d9fbcb4b-a081-e6f9-8484-d97c1fc613c9@redhat.com>

On 19/04/17 08:42, Mani Sarkar wrote:
> Whilst build OpenJDK (latest version from
> http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the build
> system does not make use of all the CPU cores available on the machine even
> though it detects them all.

It's because you have almost no memory installed.  If you want to
override the number of cores to use you can do so manually, but the
idea is to prevent you from running out of memory during the build.
It's deliberate, not a bug.

Andrew.


From sadhak001 at gmail.com  Wed Apr 19 08:07:48 2017
From: sadhak001 at gmail.com (Mani Sarkar)
Date: Wed, 19 Apr 2017 08:07:48 +0000
Subject: OpenJDK ARM build not using all the CPU cores
In-Reply-To: <d9fbcb4b-a081-e6f9-8484-d97c1fc613c9@redhat.com>
References: <CAGHtMWkCyVoSOwUp=Mxa=0NBrwwfnyrExnH38DoEnboej2kS8Q@mail.gmail.com>
 <d9fbcb4b-a081-e6f9-8484-d97c1fc613c9@redhat.com>
Message-ID: <CAGHtMW=L5g_ZoS7AghYnfLEMfGJUJ0U6aceUGAX2KcpxfBP-zw@mail.gmail.com>

Thanks Andrew for clarifying.

On Wed, 19 Apr 2017 at 08:56 Andrew Haley <aph at redhat.com> wrote:

> On 19/04/17 08:42, Mani Sarkar wrote:
> > Whilst build OpenJDK (latest version from
> > http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the
> build
> > system does not make use of all the CPU cores available on the machine
> even
> > though it detects them all.
>
> It's because you have almost no memory installed.  If you want to
> override the number of cores to use you can do so manually, but the
> idea is to prevent you from running out of memory during the build.
> It's deliberate, not a bug.
>
> Andrew.
>
> --
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*

From erik.joelsson at oracle.com  Wed Apr 19 09:26:46 2017
From: erik.joelsson at oracle.com (Erik Joelsson)
Date: Wed, 19 Apr 2017 11:26:46 +0200
Subject: OpenJDK ARM build not using all the CPU cores
In-Reply-To: <8AE7C8DA-8E55-4E62-83E3-FAEC67415063@oracle.com>
References: <CAGHtMWkCyVoSOwUp=Mxa=0NBrwwfnyrExnH38DoEnboej2kS8Q@mail.gmail.com>
 <d9fbcb4b-a081-e6f9-8484-d97c1fc613c9@redhat.com>
 <CAGHtMW=L5g_ZoS7AghYnfLEMfGJUJ0U6aceUGAX2KcpxfBP-zw@mail.gmail.com>
 <8AE7C8DA-8E55-4E62-83E3-FAEC67415063@oracle.com>
Message-ID: <1b0405d4-de42-589d-d79d-6c1f94539e3a@oracle.com>

If you want to do the override at configure time, the correct configure 
parameter is --with-jobs=4. The --with-num-cores isn't all that useful 
except for easier testing of the logic for calculating the default 
number of jobs (which is why I left it there).

The default limitation of jobs based on memory is rather conservative (1 
job per 1024MB IIRC). We don't like to assume that building is the only 
thing the system ever does. Also most of the build tasks aren't that 
memory hungry, but some are. I recommend you try manually increasing 
jobs until you find your optimal level of concurrency for the types of 
builds you do.

/Erik


On 2017-04-19 11:07, Ioi Lam wrote:
> If you really want, you can override the settings with
>
> make JOBS=4 images
>
> Of course, YMMV due to the low amount of memory.
>
> Ioi
>
>> Mani Sarkar <sadhak001 at gmail.com> ? 2017?4?19? ??4:07 ???
>>
>> Thanks Andrew for clarifying.
>>
>>> On Wed, 19 Apr 2017 at 08:56 Andrew Haley <aph at redhat.com> wrote:
>>>
>>>> On 19/04/17 08:42, Mani Sarkar wrote:
>>>> Whilst build OpenJDK (latest version from
>>>> http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the
>>> build
>>>> system does not make use of all the CPU cores available on the machine
>>> even
>>>> though it detects them all.
>>> It's because you have almost no memory installed.  If you want to
>>> override the number of cores to use you can do so manually, but the
>>> idea is to prevent you from running out of memory during the build.
>>> It's deliberate, not a bug.
>>>
>>> Andrew.
>>>
>>> --
>> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
>> <http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
>> (@adoptopenjdk & @adoptajsr programs)
>> *Meet-a-Project - *MutabilityDetector
>> <https://github.com/MutabilityDetector>*  |  **Bitbucket
>> <https://bitbucket.org/neomatrix369>* * |  **Github
>> <https://github.com/neomatrix369>* * |  **LinkedIn
>> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
>> *Come to Devoxx UK 2017:* http://www.devoxx.co.uk/
>>
>> *Don't chase success, rather aim for "Excellence", and success will come
>> chasing after you!*


From sadhak001 at gmail.com  Wed Apr 19 10:09:43 2017
From: sadhak001 at gmail.com (Mani Sarkar)
Date: Wed, 19 Apr 2017 10:09:43 +0000
Subject: OpenJDK ARM build not using all the CPU cores
In-Reply-To: <1b0405d4-de42-589d-d79d-6c1f94539e3a@oracle.com>
References: <CAGHtMWkCyVoSOwUp=Mxa=0NBrwwfnyrExnH38DoEnboej2kS8Q@mail.gmail.com>
 <d9fbcb4b-a081-e6f9-8484-d97c1fc613c9@redhat.com>
 <CAGHtMW=L5g_ZoS7AghYnfLEMfGJUJ0U6aceUGAX2KcpxfBP-zw@mail.gmail.com>
 <8AE7C8DA-8E55-4E62-83E3-FAEC67415063@oracle.com>
 <1b0405d4-de42-589d-d79d-6c1f94539e3a@oracle.com>
Message-ID: <CAGHtMW=+7+vRuRhvMVDQUO_iXB-UfFwet6xZh+GA8AFxab-UpQ@mail.gmail.com>

Thanks Erik, I have updated our scripts with this flag.

Cheers,
Mani

On Wed, 19 Apr 2017 at 10:29 Erik Joelsson <erik.joelsson at oracle.com> wrote:

> If you want to do the override at configure time, the correct configure
> parameter is --with-jobs=4. The --with-num-cores isn't all that useful
> except for easier testing of the logic for calculating the default
> number of jobs (which is why I left it there).
>
> The default limitation of jobs based on memory is rather conservative (1
> job per 1024MB IIRC). We don't like to assume that building is the only
> thing the system ever does. Also most of the build tasks aren't that
> memory hungry, but some are. I recommend you try manually increasing
> jobs until you find your optimal level of concurrency for the types of
> builds you do.
>
> /Erik
>
>
> On 2017-04-19 11:07, Ioi Lam wrote:
> > If you really want, you can override the settings with
> >
> > make JOBS=4 images
> >
> > Of course, YMMV due to the low amount of memory.
> >
> > Ioi
> >
> >> Mani Sarkar <sadhak001 at gmail.com> ? 2017?4?19? ??4:07 ???
> >>
> >> Thanks Andrew for clarifying.
> >>
> >>> On Wed, 19 Apr 2017 at 08:56 Andrew Haley <aph at redhat.com> wrote:
> >>>
> >>>> On 19/04/17 08:42, Mani Sarkar wrote:
> >>>> Whilst build OpenJDK (latest version from
> >>>> http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the
> >>> build
> >>>> system does not make use of all the CPU cores available on the machine
> >>> even
> >>>> though it detects them all.
> >>> It's because you have almost no memory installed.  If you want to
> >>> override the number of cores to use you can do so manually, but the
> >>> idea is to prevent you from running out of memory during the build.
> >>> It's deliberate, not a bug.
> >>>
> >>> Andrew.
> >>>
> >>> --
> >> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
> >> <http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC
> Advocate
> >> (@adoptopenjdk & @adoptajsr programs)
> >> *Meet-a-Project - *MutabilityDetector
> >> <https://github.com/MutabilityDetector>*  |  **Bitbucket
> >> <https://bitbucket.org/neomatrix369>* * |  **Github
> >> <https://github.com/neomatrix369>* * |  **LinkedIn
> >> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
> >> *Come to Devoxx UK 2017:* http://www.devoxx.co.uk/
> >>
> >> *Don't chase success, rather aim for "Excellence", and success will come
> >> chasing after you!*
>
> --
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*

From sadhak001 at gmail.com  Wed Apr 19 20:20:41 2017
From: sadhak001 at gmail.com (Mani Sarkar)
Date: Wed, 19 Apr 2017 20:20:41 +0000
Subject: OpenJDK ARM build not using all the CPU cores
In-Reply-To: <CAGHtMW=+7+vRuRhvMVDQUO_iXB-UfFwet6xZh+GA8AFxab-UpQ@mail.gmail.com>
References: <CAGHtMWkCyVoSOwUp=Mxa=0NBrwwfnyrExnH38DoEnboej2kS8Q@mail.gmail.com>
 <d9fbcb4b-a081-e6f9-8484-d97c1fc613c9@redhat.com>
 <CAGHtMW=L5g_ZoS7AghYnfLEMfGJUJ0U6aceUGAX2KcpxfBP-zw@mail.gmail.com>
 <8AE7C8DA-8E55-4E62-83E3-FAEC67415063@oracle.com>
 <1b0405d4-de42-589d-d79d-6c1f94539e3a@oracle.com>
 <CAGHtMW=+7+vRuRhvMVDQUO_iXB-UfFwet6xZh+GA8AFxab-UpQ@mail.gmail.com>
Message-ID: <CAGHtMWkBtmZRM1Tm7WDi7ZXHC9Up+3dbtWQiWqveBFwnDem25w@mail.gmail.com>

Thanks Erik we applied that flag and it has picked up the cores, we wll see
when the build finishes how much or how long it took to finish compared to
our previous 1-3 hour builds.

On Wed, 19 Apr 2017 at 11:09 Mani Sarkar <sadhak001 at gmail.com> wrote:

> Thanks Erik, I have updated our scripts with this flag.
>
> Cheers,
> Mani
>
> On Wed, 19 Apr 2017 at 10:29 Erik Joelsson <erik.joelsson at oracle.com>
> wrote:
>
>> If you want to do the override at configure time, the correct configure
>> parameter is --with-jobs=4. The --with-num-cores isn't all that useful
>> except for easier testing of the logic for calculating the default
>> number of jobs (which is why I left it there).
>>
>> The default limitation of jobs based on memory is rather conservative (1
>> job per 1024MB IIRC). We don't like to assume that building is the only
>> thing the system ever does. Also most of the build tasks aren't that
>> memory hungry, but some are. I recommend you try manually increasing
>> jobs until you find your optimal level of concurrency for the types of
>> builds you do.
>>
>> /Erik
>>
>>
>> On 2017-04-19 11:07, Ioi Lam wrote:
>> > If you really want, you can override the settings with
>> >
>> > make JOBS=4 images
>> >
>> > Of course, YMMV due to the low amount of memory.
>> >
>> > Ioi
>> >
>> >> Mani Sarkar <sadhak001 at gmail.com> ? 2017?4?19? ??4:07 ???
>> >>
>> >> Thanks Andrew for clarifying.
>> >>
>> >>> On Wed, 19 Apr 2017 at 08:56 Andrew Haley <aph at redhat.com> wrote:
>> >>>
>> >>>> On 19/04/17 08:42, Mani Sarkar wrote:
>> >>>> Whilst build OpenJDK (latest version from
>> >>>> http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the
>> >>> build
>> >>>> system does not make use of all the CPU cores available on the
>> machine
>> >>> even
>> >>>> though it detects them all.
>> >>> It's because you have almost no memory installed.  If you want to
>> >>> override the number of cores to use you can do so manually, but the
>> >>> idea is to prevent you from running out of memory during the build.
>> >>> It's deliberate, not a bug.
>> >>>
>> >>> Andrew.
>> >>>
>> >>> --
>> >> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
>> >> <http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC
>> Advocate
>> >> (@adoptopenjdk & @adoptajsr programs)
>> >> *Meet-a-Project - *MutabilityDetector
>> >> <https://github.com/MutabilityDetector>*  |  **Bitbucket
>> >> <https://bitbucket.org/neomatrix369>* * |  **Github
>> >> <https://github.com/neomatrix369>* * |  **LinkedIn
>> >> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
>> >> *Come to Devoxx UK 2017:* http://www.devoxx.co.uk/
>> >>
>> >> *Don't chase success, rather aim for "Excellence", and success will
>> come
>> >> chasing after you!*
>>
>> --
> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
> <http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
> (@adoptopenjdk & @adoptajsr programs)
> *Meet-a-Project - *MutabilityDetector
> <https://github.com/MutabilityDetector>*  |  **Bitbucket
> <https://bitbucket.org/neomatrix369>* * |  **Github
> <https://github.com/neomatrix369>* * |  **LinkedIn
> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
> *Come to Devoxx UK 2017:* http://www.devoxx.co.uk/
>
> *Don't chase success, rather aim for "Excellence", and success will come
> chasing after you!*
>
-- 
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*