etc.curl: Formal review begin

Jonas Drewsen jdrewsen at nospam.com
Thu Aug 25 03:23:06 PDT 2011


On 24/08/11 18.35, Walter Bright wrote:
> On 8/24/2011 2:18 AM, Johannes Pfau wrote:
>> Walter Bright wrote:
>>> On 8/17/2011 4:12 PM, David Nadlinger wrote:
>>>> the etc.curl module by Jonas Drewsen is at the front of the review
>>>> queue.
>>>
>>> Thanks for doing this. I preface my remarks with saying I have never
>>> done http: programming and know little about it.
>>>
>>> 1. The first, most obvious thing I'd like to do is present a URL and
>>> get a string back that is the contents of that web page:
>>>
>>> string contents = getURL("http://www.digitalmars.com");
>>>
>>
>> That's the first example in the API docs:
>>
>> // Simple GET with connect timeout of 10 seconds
>> Http.get("http://www.google.com").connectTimeout(dur!"seconds"(10)).toString();
>>
>
> Ok, but 1. there's no indication in the doc that that's what it does
> because 1. it is an incomplete fragment and 2. a simpler interface (like
> what I proposed) would likely be better.
>
>
>> or, if you don't care about timeouts:
>> string contents = Http.get("http://www.google.com").toString();
>
> This example needs to be first and needs an explanation. Timeouts, error
> recovery, progress indicators, etc., should all be brought up later.
>
>
>
>>> and contents[] should be the web page; what I'd see if I opened it in
>>> a browser and selected "view source". From perusing the etc.curl doc I
>>> have absolutely no idea how to do this.
>>>
>>> 2. I'd like to get the contents as an input range, which I can then
>>> iterate like any other range.
>>
>> auto res = Http.getAsync("http://www.digitalmars.com");
>> auto range = res.byChunk(100);
>> writeln(range.front);
>> range.popFront();
>> writeln(range.front);
>>
>> There's only a byChunk for synchronous requests, but I think this first
>> reads the complete Result into memory and then returns slices. (This
>> is a limitation caused by curl's API, as curl uses callbacks)
>
> Ok.
>
>>> 3. A lot of web sites like twitter.com have an API. How do I access
>>> that API from D?
>>
>> The twitter API is a REST API, so if you want to get information from
>> twitter, you use a GET request with a special url:
>>
>> string xmlTimeLine =
>> Http.get("http://twitter.com/statuses/user_timeline.xml?id=walterbright").toString();
>>
>>
>> Posting is a little more complicated, as it requires authentication:
>>
>> ubyte[] result;
>> Http twitter =
>> Http("http://api.twitter.com/1/statuses/update.format?status=Hello%20World");
>>
>> twitter.setAuthentication("user", "password");
>> twitter.onReceive = (ubyte[] data) { result ~= data; return
>> data.length; };
>> twitter.method = Http.Method.post;
>> twitter.perform();
>>
>> (Status should be url encoded, Http.escape can be used for that)
>>
>> Not sure, if this still works though. Twitter is switching from HTTP
>> basic authentication to OAuth. Also, this code is not tested ;-)
>
> I'm glad to see this can be done. A complete example in the
> documentation would be very informative and useful.
>
>
>>> 4. How do I download a file from ftp?
>>>
>>> ubyte[] file =
>>> downloadFile("ftp://www.digitalmars.com/whatever.zip");
>>
>> Ftp.get("ftp://www.digitalmars.com/whatever.zip","/tmp/downloaded-file");
>>
>> or
>>
>> ubyte[] file = Ftp.get("ftp://www.digitalmars.com/whatever.zip").bytes;
>
> Great! Please add to the documentation.
>
>
> I also think that etc.curl would benefit greatly from an article written
> about it showing how to use it to accomplish various fun and simple tasks.

Actually I truely believe that much of the documentation you're missing 
is actually in the source code. As I've previously mentioned ddoc does 
not handle mixins very well.

In the etc.curl case it results in the Result and AsyncResult structs 
documentation is nearly missing. They are pretty important and 
especially the byChunk()/byLine() methods (mixed in) are not visible in 
the docs because ddoc cannot handle it.

I would really like to know how to handle this situation because it 
understandably confuses people and makes it impossible to figure out how 
to do stuff as you have experienced yourself.

I will see if I can find the time to put an article together though.

Thanks.
/Jonas


More information about the Digitalmars-d mailing list