Curl support RFC

[Jonas Drewsen]

On 14/03/11 18.55, Andrei Alexandrescu wrote:
> On 3/14/11 4:16 AM, Jonas Drewsen wrote:
>> On 13/03/11 23.44, Andrei Alexandrescu wrote:
>>> Sweet. As has been discussed, often the content is not text so you may
>>> want to have content return ubyte[] and add a new property such as
>>> "textContent" or "text".
>>
>> I've already changed it to void[] as done in the std.file module. Is
>> ubyte[] better suited?
>
> Yah, as per the ensuing discussion.
>
>>> As discussed, properties may be better here than setXxx and getXxx. The
>>> setReceiveCallback hook should take a ubyte[]. The
>>> setReceiveHeaderCallback should take a const(char)[]. That way you won't
>>> need to copy all headers, leaving safely that option to the client.
>>
>> I've already replaced the set/get methods with properties and renamed
>> them. Hadn't thought of using const(char)[].. thanks for the hint.
>
> A good general guideline: make sure that the user could easily and
> safely use a loop that reads a large http stream (with hooks and all)
> without allocating one item each pass through the loop.

Makes sense. I'll keep that in mind.

>>> Regarding a range interface, it would be great if you allowed e.g.
>>>
>>> foreach (line; Http.get("https://mail.google.com").byLine()) {
>>> ...
>>> }
>>>
>>> The data transfer should happen concurrently with the foreach code. The
>>> type of line is char[] or const(char)[]. Similarly, there would be a
>>> byChunk interface that transfers in ubyte[] chunks.
>>>
>>> Also we need a head() method for the corresponding command.
>>>
>>> Andrei
>>
>> That would be neat. What do you mean about concurrent data transfers
>> with foreach?
>
> Assume the body of the loop does some time-consuming processing - like
> e.g. writing to another HTTP stream. Then your network reads should not
> wait for that processing. While the user code does something, you should
> already have the next transfer in flight.
>
> Example: a utility that efficiently uses GET from one http source and
> uses the data to POST it to an http target should be an efficient
> few-liner. (FTP versions and mixed ones too.)
>
>
> Andrei

I get it. Any existing implementation that does this I can have a look at?

/Jonas