Curl wrapper

[Andrei Alexandrescu]

On 5/18/11 6:07 AM, Jonas Drewsen wrote:
> Select will wait for data to be ready and ask curl to handle the data
> chunk. Curl in turn calls back to a registered callback handler with the
> data read. That handler fills the buffer provided by the user. If not
> enough data has been receive an new select is performed until the
> requested amount of data is read. Then the blocking method can return.

Perhaps this would be too complicated. In any case the core 
functionality must be paid top attention. And the core functionality is 
streaming.

Currently there are two proposed ways to stream data from an HTTP 
address: (a) by using the onReceive callback, and (b) by using 
byLine/byChunk. If either of these perform slower than the 
best-of-the-breed streaming using libcurl, we have failed.

The onReceive method is not particularly appealing because the client 
and libcurl block each other: the client is blocked while libcurl is 
waiting for data, and the client blocks libcurl while inside the 
callback. (Please correct me if I'm wrong.)

To make byLine/byChunk fast, the basic setup should include a hidden 
thread that does the download in separation from the client's thread. 
There should be K buffers allocated (K = 2 to e.g. 10), and a simple 
protocol for passing the buffers back and forth between the client 
thread and the hidden thread. That way, in the quiescent state, there is 
no memory allocation and either both client and libcurl are busy doing 
work, or one is much slower than the other, which waits.

The same mechanism should be used in byChunkAsync or byFileAsync.


Thanks,

Andrei