Curl support RFC

[Johannes Pfau]

Jonas Drewsen wrote:
>Hi,
>
>   So I've been working a bit on the etc.curl module. Currently most
> of 
>the HTTP functionality is done and some very simple Ftp.
>
>I would very much like to know if this has a chance of getting in
>phobos if I finish it with the current design. If not then it will be
>for my own project only and doesn't need as much documentation or all
>the features.
>
>https://github.com/jcd/phobos/tree/curl
>
>I do know that the error handling is currently not good enough... WIP.
>
>/Jonas
>
>
>On 11/03/11 16.20, Jonas Drewsen wrote:
>> Hi,
>>
>> So I've spent some time trying to wrap libcurl for D. There is a lot
>> of things that you can do with libcurl which I did not know so I'm
>> starting out small.
>>
>> For now I've created all the declarations for the latest public curl
>> C api. I have put that in the etc.c.curl module.
>>
>> On top of that I've created a more D like api as seen below. This is
>> located in the 'etc.curl' module. What you can see below currently
>> works but before proceeding further down this road I would like to
>> get your comments on it.
>>
>> //
>> // Simple HTTP GET with sane defaults
>> // provides the .content, .headers and .status
>> //
>> writeln( Http.get("http://www.google.com").content );
>>
>> //
>> // GET with custom data receiver delegates
>> //
>> Http http = new Http("http://www.google.dk");
>> http.setReceiveHeaderCallback( (string key, string value) {
>> writeln(key ~ ":" ~ value);
>> } );
>> http.setReceiveCallback( (string data) { /* drop */ } );
>> http.perform;
>>
>> //
>> // POST with some timouts
>> //
>> http.setUrl("http://www.testing.com/test.cgi");
>> http.setReceiveCallback( (string data) { writeln(data); } );
>> http.setConnectTimeout(1000);
>> http.setDataTimeout(1000);
>> http.setDnsTimeout(1000);
>> http.setPostData("The quick....");
>> http.perform;
>>
>> //
>> // PUT with data sender delegate
>> //
>> string msg = "Hello world";
>> size_t len = msg.length; /* using chuncked transfer if omitted */
>>
>> http.setSendCallback( delegate size_t(char[] data) {
>> if (msg.empty) return 0;
>> auto l = msg.length;
>> data[0..l] = msg[0..$];
>> msg.length = 0;
>> return l;
>> },
>> HttpMethod.put, len );
>> http.perform;
>>
>> //
>> // HTTPS
>> //
>> writeln(Http.get("https://mail.google.com").content);
>>
>> //
>> // FTP
>> //
>> writeln(Ftp.get("ftp://ftp.digitalmars.com/sieve.ds",
>> "./downloaded-file"));
>>
>>
>> // ... authenication, cookies, interface select, progress callback
>> // etc. is also implemented this way.
>>
>>
>> /Jonas
>
Hi,
I really like the API. A few comments:

You use the internal curl progress meter. According to the
documentation (It's a little hidden, look at CURLOPT_NOPROGRESS) the
progress meter is likely to removed in future curl versions. The
download progress should be easy to reimplement, although you'd have to
parse the Content-Length header. Upload shouldn't be to difficult either
(One problem: What does curl pass as ultotal/dltotal when chunked
encoding is used or the total size is not known?). Then we could also
use different delegates for upload/download.

The callback interface suits curl best and I actually like it, but how
will it interact with streams? As an example: If someone wrote a
stream/filter that decoded gzip for files it should be usable with
the http streams as well. But files/ filestreams have a pull
interface (no callbacks, stream.read() in a loop). So how could a gzip
stream be written without to much code duplication supporting files and
the http stuff?

Do you plan to add some kind of support for header parsing? I think
something like what the .net webclient uses
( http://msdn.microsoft.com/en-us/library/system.net.webclient(v=VS.100).aspx )
would be great. Especially the HeaderCollection supporting headers as
strings and as data types (for both parsing and formatting), but
without a class hierarchy for the headers, using templates instead.

I've written D parsers/formatters for almost all headers in
rfc2616 (1 or 2 might be missing) and for a few additional commonly
used headers (Content-Disposition, cookie headers). The parsers are
written with ragel and are to be used with curl (continuations must be
removed and the parsers always take 1 line of input, just as you get it
from curl). Right now only the client side is implemented (no parsers
for headers which can only be sent from client-->server ). However, I
need to add some more documentation to the parsers, need to do
some refactoring and I've got absolutely no time for that in the next 2
weeks ('abitur' final exams). But if you could wait 2 weeks or if
you wanted to do the refactoring yourself, I would be happy to
contribute that code.


-- 
Johannes Pfau
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20110314/3f8e8f80/attachment.pgp>