And yet another cool project: fetching multiple URLs at once

[Sebastiaan Koppe via Digitalmars-d]

On Wednesday, 16 December 2015 at 15:54:09 UTC, Andrei 
Alexandrescu wrote:
> Right now we can nicely stream an URL as an input range. A 
> great extension would be to fetch several URLs at once. When 
> accessing r.front for that range, the user gets a pair of URL 
> and data chunk.
>
> Of course the point is to make all these fetches work 
> simultaneously. Inside, the range would e.g. use asynchronous 
> curl.
>
>
> Andrei

Whenever I want to do that I use vibe.d and my observe library 
(based on RX observables) and the code would look a bit like this:

urls
	.observe()
	.zip(localPaths)
	.fork(numParallel)
	.concatMap!((part)
	{
		return downloadFile(part[0],part[1]).retry(3)
	})
	.subscribe(
		(p){writeln("File %s downloaded to %s",p[0],p[1]);},
		(e){writeln("Error ",e);},
		(){writeln("Completed!");}
	);

Here observe() makes an observable out of any range; fork() uses 
vibe.d's Fibers to achieve concurrency; downloadFile() returns 
another observable that gets concatMap'ped; and retry does what 
you think it does. Also, errors get propagated without throws and 
catches.

I think what I am trying to say is that a lot of stuff is already 
available on code.dlang.org, just not in phobos. Which begs the 
question, should it be? And if it does, shouldn't you rather 
consider merging vibe.d with phobos? Or parts of it?