A parallel HttpClient sendAync question

[Weijun Wang]

> On Nov 13, 2018, at 11:41 AM, Chris Yin <xu.y.yin at oracle.com> wrote:
> 
>> 
>> On 13 Nov 2018, at 10:35 AM, Weijun Wang <weijun.wang at oracle.com> wrote:
>> 
>> I'm scanning a file and downloading links inside:
>> 
>> lines.flapMap(x -> Stream.ofNullable(findURIFrom(x)))
>>    .map(l -> download(c, l))
>>    .forEach(f -> f.join());
>> 
>> CompletableFuture<HttpResponse<Path>> download(HttpClient c, URI link) {
>>   return c.sendAsync(HttpRequest.newBuilder(link).build(),
>>           HttpResponse.BodyHandlers.ofFile(Path.of(link.getPath())));
>> }
>> 
>> However, it seems the download is one by one and not parallel. I guess maybe map() is lazy and each request is only send when forEach() is called and the next one is only sent after join().
>> 
>> I can only collect the jobs into a list and then call join() on CompletableFuture.allOf(list). Is there a simpler way?
> 
> Just guess your Stream (lines) is not parallel, so either use parallel stream or collect into list before join should work?
> 
> lines.parallel().flapMap(x -> Stream.ofNullable(findURIFrom(x)))
>    .map(l -> download(c, l))
>    .forEach(f -> f.join());

Does not work. Still downloading sequentially.

> 
> Or
> 
> lines.flapMap(x -> Stream.ofNullable(findURIFrom(x)))
>    .map(l -> download(c, l))
>    .collect(Collectors.toList())
>    .forEach(f -> f.join());
> 
> Regards,
> Chris
> 
>> 
>> Thanks
>> Max