RFR: race with nested repos in hgforest.sh

[David Holmes]

On 2/02/2013 9:06 AM, Chris Hegarty wrote:
>
> On 1 Feb 2013, at 22:37, David Holmes<david.holmes at oracle.com>  wrote:
>
>> Hi Chris,
>>
>> On 2/02/2013 12:40 AM, Chris Hegarty wrote:
>>> [ to build-dev and core-libs-dev, expect reviewer from either, but will
>>> integrate through jdk8/tl ]
>>>
>>> This issue is mainly of interest to Oracle engineers, but it effects the
>>> public hgforest script.
>>>
>>> When hgforest.sh is run with an addition argument to specify a closed
>>> server, there is a problem/race between the creation of the directories
>>> to hold nested repositories and the clone itself. These directories need
>>> to be created before the clone command is executed, otherwise it will
>>> fail, as below.
>>
>> I think I reported this myself just last week - probably internally to build or build-infra.
>
> Ah, I missed that.
>
>> The weird thing is that based on the list of repos it shouldn't be grabbing the closed one yet.
>
> In my remote/slow environment I end up with all clones running concurrently.

I can see that happening if the sleeps are too short. But what I noticed 
from the logging was that the closed repos were being processed before 
the open ones - which shouldn't happen.

>>
>>> The trivial fix is to back off these nested repos until their containing
>>> directory exists.
>>
>> That seems to assume/require that the nested repo will always be later in the list. They happen to be but ....
>
> No so. It will just wait for the process to clone the outer repo to start.

Ok. Some clone will make progress and so we should ultimately get to 
start the outer repo.

Thanks,
David

>>
>> Workaround is to run get_source.sh twice first without the "extra base" then with it.
>
> Good suggestion.
>
> -Chris
>
>>
>> David
>> -----
>>
>>>
>>> Webrev:
>>> http://cr.openjdk.java.net/~chegar/hgforest_nestedRepos/webrev/
>>>
>>> sh ./get_source.sh http://xxx.yyy.oracle.com |&  tee clone.log
>>> # Repositories: corba jaxp jaxws langtools jdk hotspot jdk/src/closed
>>> jdk/make/closed jdk/test/closed hotspot/make/closed hotspot/src/closed
>>> hotspot/test/closed deploy install sponsors pubs
>>>
>>> Waiting 5 secs before spawning next background command.
>>> Waiting 5 secs before spawning next background command.
>>> Waiting 5 secs before spawning next background command.
>>> jdk/src/closed: /java/devtools/sparc/mercurial/0.9.5/bin/python -u
>>> /java/devtools/sparc/mercurial/latest/bin/hg clone
>>> http://xxx.yyy.oracle.com/jdk8/tl/jdk/src/closed jdk/src/closed
>>> jdk/src/closed: abort: No such file or directory: jdk/src/closed
>>> jdk/make/closed: /java/devtools/sparc/mercurial/0.9.5/bin/python -u
>>> /java/devtools/sparc/mercurial/latest/bin/hg clone
>>> http://xxx.yyy.oracle.com/jdk8/tl/jdk/make/closed jdk/make/closed
>>> jdk/make/closed: abort: No such file or directory: jdk/make/closed
>>> Waiting 5 secs before spawning next background command.
>>> ....
>>>
>>>
>>> -Chris.