etc.curl: Formal review begin

Walter Bright newshound2 at digitalmars.com
Wed Aug 24 09:35:27 PDT 2011


On 8/24/2011 2:18 AM, Johannes Pfau wrote:
> Walter Bright wrote:
>> On 8/17/2011 4:12 PM, David Nadlinger wrote:
>>> the etc.curl module by Jonas Drewsen is at the front of the review
>>> queue.
>>
>> Thanks for doing this. I preface my remarks with saying I have never
>> done http: programming and know little about it.
>>
>> 1. The first, most obvious thing I'd like to do is present a URL and
>> get a string back that is the contents of that web page:
>>
>>      string contents = getURL("http://www.digitalmars.com");
>>
>
> That's the first example in the API docs:
>
> // Simple GET with connect timeout of 10 seconds
> Http.get("http://www.google.com").connectTimeout(dur!"seconds"(10)).toString();

Ok, but 1. there's no indication in the doc that that's what it does because 1. 
it is an incomplete fragment and 2. a simpler interface (like what I proposed) 
would likely be better.


> or, if you don't care about timeouts:
> string contents = Http.get("http://www.google.com").toString();

This example needs to be first and needs an explanation. Timeouts, error 
recovery, progress indicators, etc., should all be brought up later.



>> and contents[] should be the web page; what I'd see if I opened it in
>> a browser and selected "view source". From perusing the etc.curl doc I
>> have absolutely no idea how to do this.
>>
>> 2. I'd like to get the contents as an input range, which I can then
>> iterate like any other range.
>
> auto res = Http.getAsync("http://www.digitalmars.com");
> auto range = res.byChunk(100);
> writeln(range.front);
> range.popFront();
> writeln(range.front);
>
> There's only a byChunk for synchronous requests, but I think this first
> reads the complete Result into memory and then returns slices. (This
> is a limitation caused by curl's API, as curl uses callbacks)

Ok.

>> 3. A lot of web sites like twitter.com have an API. How do I access
>> that API from D?
>
> The twitter API is a REST API, so if you want to get information from
> twitter, you use a GET request with a special url:
>
> string xmlTimeLine =
> Http.get("http://twitter.com/statuses/user_timeline.xml?id=walterbright").toString();
>
> Posting is a little more complicated, as it requires authentication:
>
> ubyte[] result;
> Http twitter =
> Http("http://api.twitter.com/1/statuses/update.format?status=Hello%20World");
> twitter.setAuthentication("user", "password");
> twitter.onReceive = (ubyte[] data) { result ~= data; return
> data.length; };
> twitter.method = Http.Method.post;
> twitter.perform();
>
> (Status should be url encoded, Http.escape can be used for that)
>
> Not sure, if this still works though. Twitter is switching from HTTP
> basic authentication to OAuth. Also, this code is not tested ;-)

I'm glad to see this can be done. A complete example in the documentation would 
be very informative and useful.


>> 4. How do I download a file from ftp?
>>
>>     ubyte[] file =
>> downloadFile("ftp://www.digitalmars.com/whatever.zip");
>
> Ftp.get("ftp://www.digitalmars.com/whatever.zip","/tmp/downloaded-file");
>
> or
>
> ubyte[] file = Ftp.get("ftp://www.digitalmars.com/whatever.zip").bytes;

Great! Please add to the documentation.


I also think that etc.curl would benefit greatly from an article written about 
it showing how to use it to accomplish various fun and simple tasks.


More information about the Digitalmars-d mailing list