Curl support RFC

[Andrei Alexandrescu]

On 3/14/11 4:11 PM, Jonas Drewsen wrote:
> On 14/03/11 18.55, Andrei Alexandrescu wrote:
>> Assume the body of the loop does some time-consuming processing - like
>> e.g. writing to another HTTP stream. Then your network reads should not
>> wait for that processing. While the user code does something, you should
>> already have the next transfer in flight.
>>
>> Example: a utility that efficiently uses GET from one http source and
>> uses the data to POST it to an http target should be an efficient
>> few-liner. (FTP versions and mixed ones too.)
>>
>>
>> Andrei
>
> I get it. Any existing implementation that does this I can have a look at?

Unfortunately not at the moment. I wanted to define such a thing for 
std.stdio called byLineAsync and byChunkAsync but never got to it.

The basic idea is:

1. Define a new range type, e.g. AsyncHttpInputRange

2. Inside that range start a secondary thread that does the actual 
transfer and passes read buffers to the main thread by means of messages

3. See std.concurrency and the free chapter 
http://www.informit.com/articles/printerfriendly.aspx?p=1609144 for details

4. Control congestion (too many buffers in flight) with setMaxMailboxSize.

5. Make sure you have a little protocol that stops the secondary thread 
when the range is destroyed.


Andrei