Curl support RFC

[Andrei Alexandrescu]

On 3/11/11 9:20 AM, Jonas Drewsen wrote:
> Hi,
>
> So I've spent some time trying to wrap libcurl for D. There is a lot of
> things that you can do with libcurl which I did not know so I'm starting
> out small.
>
> For now I've created all the declarations for the latest public curl C
> api. I have put that in the etc.c.curl module.

Great! Could you please create a pull request for that?

> On top of that I've created a more D like api as seen below. This is
> located in the 'etc.curl' module. What you can see below currently works
> but before proceeding further down this road I would like to get your
> comments on it.
>
> //
> // Simple HTTP GET with sane defaults
> // provides the .content, .headers and .status
> //
> writeln( Http.get("http://www.google.com").content );

Sweet. As has been discussed, often the content is not text so you may 
want to have content return ubyte[] and add a new property such as 
"textContent" or "text".

> //
> // GET with custom data receiver delegates
> //
> Http http = new Http("http://www.google.dk");

You'll probably need to justify the existence of a class hierarchy and 
what overridable methods there are. In particular, since you seem to 
offer hooks via delegates, probably classes wouldn't be needed at all. 
(FWIW I would've done the same; I wouldn't want to inherit just to 
intercept the headers etc.)

> http.setReceiveHeaderCallback( (string key, string value) {
> writeln(key ~ ":" ~ value);
> } );
> http.setReceiveCallback( (string data) { /* drop */ } );
> http.perform;

As discussed, properties may be better here than setXxx and getXxx. The 
setReceiveCallback hook should take a ubyte[]. The 
setReceiveHeaderCallback should take a const(char)[]. That way you won't 
need to copy all headers, leaving safely that option to the client.

> //
> // POST with some timouts
> //
> http.setUrl("http://www.testing.com/test.cgi");
> http.setReceiveCallback( (string data) { writeln(data); } );
> http.setConnectTimeout(1000);
> http.setDataTimeout(1000);
> http.setDnsTimeout(1000);
> http.setPostData("The quick....");
> http.perform;

setPostData -> setTextPostData, and then changing everything to 
properties would make it something like textPostData. Or wait, there 
could be some overloading going on... Anyway, the basic idea is that 
generally get and post data could be raw bytes, and the user could elect 
to transfer strings instead.

> //
> // PUT with data sender delegate
> //
> string msg = "Hello world";
> size_t len = msg.length; /* using chuncked transfer if omitted */
>
> http.setSendCallback( delegate size_t(char[] data) {
> if (msg.empty) return 0;
> auto l = msg.length;
> data[0..l] = msg[0..$];
> msg.length = 0;
> return l;
> },
> HttpMethod.put, len );
> http.perform;

The callback would take ubyte[].

> //
> // HTTPS
> //
> writeln(Http.get("https://mail.google.com").content);
>
> //
> // FTP
> //
> writeln(Ftp.get("ftp://ftp.digitalmars.com/sieve.ds",
> "./downloaded-file"));
>
>
> // ... authenication, cookies, interface select, progress callback
> // etc. is also implemented this way.
>
>
> /Jonas

This is all very encouraging. I think this API covers nicely a variety 
of needs. We need to make sure everything interacts well with threads, 
in particular that one can shut down a transfer (or the entire library) 
from a thread or callback and have the existing transfer(s) throw an 
exception immediately.

Regarding a range interface, it would be great if you allowed e.g.

foreach (line; Http.get("https://mail.google.com").byLine()) {
    ...
}

The data transfer should happen concurrently with the foreach code. The 
type of line is char[] or const(char)[]. Similarly, there would be a 
byChunk interface that transfers in ubyte[] chunks.

Also we need a head() method for the corresponding command.


Andrei