Curl support RFC

[Jonas Drewsen]

On 13/03/11 23.44, Andrei Alexandrescu wrote:
> On 3/11/11 9:20 AM, Jonas Drewsen wrote:
>> Hi,
>>
>> So I've spent some time trying to wrap libcurl for D. There is a lot of
>> things that you can do with libcurl which I did not know so I'm starting
>> out small.
>>
>> For now I've created all the declarations for the latest public curl C
>> api. I have put that in the etc.c.curl module.
>
> Great! Could you please create a pull request for that?

Will do as soon as I've figured out howto create a pull request for a 
single file in a branch. Anyone knows how to do that on github? Or 
should I just create a pull request including the etc.curl wrapper as well?

>> On top of that I've created a more D like api as seen below. This is
>> located in the 'etc.curl' module. What you can see below currently works
>> but before proceeding further down this road I would like to get your
>> comments on it.
>>
>> //
>> // Simple HTTP GET with sane defaults
>> // provides the .content, .headers and .status
>> //
>> writeln( Http.get("http://www.google.com").content );
>
> Sweet. As has been discussed, often the content is not text so you may
> want to have content return ubyte[] and add a new property such as
> "textContent" or "text".

I've already changed it to void[] as done in the std.file module. Is 
ubyte[] better suited?

I'll add a text property as well.


>> //
>> // GET with custom data receiver delegates
>> //
>> Http http = new Http("http://www.google.dk");
>
> You'll probably need to justify the existence of a class hierarchy and
> what overridable methods there are. In particular, since you seem to
> offer hooks via delegates, probably classes wouldn't be needed at all.
> (FWIW I would've done the same; I wouldn't want to inherit just to
> intercept the headers etc.)
>
>> http.setReceiveHeaderCallback( (string key, string value) {
>> writeln(key ~ ":" ~ value);
>> } );
>> http.setReceiveCallback( (string data) { /* drop */ } );
>> http.perform;
>
> As discussed, properties may be better here than setXxx and getXxx. The
> setReceiveCallback hook should take a ubyte[]. The
> setReceiveHeaderCallback should take a const(char)[]. That way you won't
> need to copy all headers, leaving safely that option to the client.

I've already replaced the set/get methods with properties and renamed 
them. Hadn't thought of using const(char)[].. thanks for the hint.


>> //
>> // POST with some timouts
>> //
>> http.setUrl("http://www.testing.com/test.cgi");
>> http.setReceiveCallback( (string data) { writeln(data); } );
>> http.setConnectTimeout(1000);
>> http.setDataTimeout(1000);
>> http.setDnsTimeout(1000);
>> http.setPostData("The quick....");
>> http.perform;
>
> setPostData -> setTextPostData, and then changing everything to
> properties would make it something like textPostData. Or wait, there
> could be some overloading going on... Anyway, the basic idea is that
> generally get and post data could be raw bytes, and the user could elect
> to transfer strings instead.

I'll make sure both text and byte[]/void[] versions will be available.

>> //
>> // PUT with data sender delegate
>> //
>> string msg = "Hello world";
>> size_t len = msg.length; /* using chuncked transfer if omitted */
>>
>> http.setSendCallback( delegate size_t(char[] data) {
>> if (msg.empty) return 0;
>> auto l = msg.length;
>> data[0..l] = msg[0..$];
>> msg.length = 0;
>> return l;
>> },
>> HttpMethod.put, len );
>> http.perform;
>
> The callback would take ubyte[].

Already fixed.


>> //
>> // HTTPS
>> //
>> writeln(Http.get("https://mail.google.com").content);
>>
>> //
>> // FTP
>> //
>> writeln(Ftp.get("ftp://ftp.digitalmars.com/sieve.ds",
>> "./downloaded-file"));
>>
>>
>> // ... authenication, cookies, interface select, progress callback
>> // etc. is also implemented this way.
>>
>>
>> /Jonas
>
> This is all very encouraging. I think this API covers nicely a variety
> of needs. We need to make sure everything interacts well with threads,
> in particular that one can shut down a transfer (or the entire library)
> from a thread or callback and have the existing transfer(s) throw an
> exception immediately.

I'll have a look at it.


> Regarding a range interface, it would be great if you allowed e.g.
>
> foreach (line; Http.get("https://mail.google.com").byLine()) {
> ...
> }
>
> The data transfer should happen concurrently with the foreach code. The
> type of line is char[] or const(char)[]. Similarly, there would be a
> byChunk interface that transfers in ubyte[] chunks.
>
> Also we need a head() method for the corresponding command.
>
> Andrei

That would be neat. What do you mean about concurrent data transfers 
with foreach?


/Jonas