Curl support RFC

[Andrei Alexandrescu]

On 3/14/11 4:16 AM, Jonas Drewsen wrote:
> On 13/03/11 23.44, Andrei Alexandrescu wrote:
>> Sweet. As has been discussed, often the content is not text so you may
>> want to have content return ubyte[] and add a new property such as
>> "textContent" or "text".
>
> I've already changed it to void[] as done in the std.file module. Is
> ubyte[] better suited?

Yah, as per the ensuing discussion.

>> As discussed, properties may be better here than setXxx and getXxx. The
>> setReceiveCallback hook should take a ubyte[]. The
>> setReceiveHeaderCallback should take a const(char)[]. That way you won't
>> need to copy all headers, leaving safely that option to the client.
>
> I've already replaced the set/get methods with properties and renamed
> them. Hadn't thought of using const(char)[].. thanks for the hint.

A good general guideline: make sure that the user could easily and 
safely use a loop that reads a large http stream (with hooks and all) 
without allocating one item each pass through the loop.

>> Regarding a range interface, it would be great if you allowed e.g.
>>
>> foreach (line; Http.get("https://mail.google.com").byLine()) {
>> ...
>> }
>>
>> The data transfer should happen concurrently with the foreach code. The
>> type of line is char[] or const(char)[]. Similarly, there would be a
>> byChunk interface that transfers in ubyte[] chunks.
>>
>> Also we need a head() method for the corresponding command.
>>
>> Andrei
>
> That would be neat. What do you mean about concurrent data transfers
> with foreach?

Assume the body of the loop does some time-consuming processing - like 
e.g. writing to another HTTP stream. Then your network reads should not 
wait for that processing. While the user code does something, you should 
already have the next transfer in flight.

Example: a utility that efficiently uses GET from one http source and 
uses the data to POST it to an http target should be an efficient 
few-liner. (FTP versions and mixed ones too.)


Andrei