Curl wrapper

[Jonas Drewsen]

On 18/05/11 10.09, Johannes Pfau wrote:
> jdrewsen wrote:
>> Please see comments below.
>>
>> Den 17-05-2011 16:42, Andrei Alexandrescu skrev:
>>> Thanks for the response! A few more answers and comments within
>>> (everything deleted counts as "sounds great").
>>>
>>> On 5/17/11 3:50 AM, Jonas Drewsen wrote:
>>>>> 14. Isn't the max redirect configurable via a parameter?
>>>>
>>>> Yes. It is called Http.followLocation from libcurls followLocation
>>>> option. I will rename it to maxRedirections for clearity.
>>>
>>> You mention this about many higher-level functions: "Maximum
>>> redirections are set to 10." Then I'm thinking, if it's an important
>>> parameter, make it a defaulted parameter for the function.
>>
>> I guess I should just remove that comment because other defaults has
>> been selected as well e.g. timeouts.
>>
>>>>> 16. Documentation should point to descriptions of the HTTP methods
>>>>> wrapped (e.g. "post", "del" etc).
>>>>
>>>> Do you mean point to the relevant RFC?
>>>
>>> Yah, or a good tutorial. (I didn't know DEL existed!)
>>
>> Well DEL is actually called DELETE in the HTTP RFC. But delete is a
>> reserved word i D so I used del() instead. delete() wouldn't work in
>> following code in think:
>>
>> with(auto http = ...) {
>> 	delete(...);
>> }
>>
>>>>> 22. byLine/byChunk should not expose a string or generally data
>>>>> that can be shared safely by the client. That's because it would
>>>>> need to create a new string with each iteration, which is very
>>>>> inefficient. Instead, they should expose char[]/ubyte[] just like
>>>>> File.byLine does, and reuse the buffer with each call. Essentially
>>>>> byLine/byChunk should be near-zero-overhead means to transfer and
>>>>> inspect arbitrarily large streams.
>>>>
>>>> byLine/byChunk is currenly only available for the async methods
>>>> which is implemented using message passing. This means that for
>>>> passing the message a copy has to be made because a message is
>>>> by-value or immutable data. Is there another way for me to pass the
>>>> message without doing a copy... some kind of move semantic?
>>>
>>> A library element is planned but not on the short list yet. You can
>>> work around that by internally doing a cast. As long as you observe
>>> the commitment that buffers are not accessed simultaneously by more
>>> than one thread you're well within defined behavior.
>>
>> ok nice to know.
>>
>>> One more suggestion - you may want to also provide classic boring
>>> synchronous read/write methods that take a user-provided buffer. I'm
>>> sure people could use such e.g. when they want to stream data inside
>>> their own threads, or even in the main thread if they don't mind
>>> blocking.
>>
>> This would mean hooking into libcurl and selecting on its sockets.
>> Totally doable.
> I'm not sure if it'd be that easy. Using select to implement a blocking
> api is possible, but select is only used to wait until data is ready,
> reading and writing the data is still done by curl. And as curl does
> not allow you to supply the data buffer, having user-provided buffers
> is afaik impossible.

_Doable_ - not easy :)

Select will wait for data to be ready and ask curl to handle the data 
chunk. Curl in turn calls back to a registered callback handler with the
data read. That handler fills the buffer provided by the user. If not 
enough data has been receive an new select is performed until the 
requested amount of data is read. Then the blocking method can return.

>> But I would rather stop here feature wise and wrap
>> this thing up. I would like to focus my efforts on a candidate for
>> std.net.event/reactor/proactor module.
>
> Have you already started working on std.net.event? Do you plan to base
> that on libev / libevent?

I'll finish the curl wrapper before starting on it.

I have had a look at libev/event before and they are very nice libs. But 
I think I'll have to implement it from scratch for two reasons:

1, AFAIK we cannot include code in std phobos that is not boost licensed 
or a license as liberal. libev for example requires you to distribute 
the license with your software.

2, It introduces a dependency to an external project in phobos. 
Currently no dependencies are present. This makes phobos very easy to 
install and use out of the box.

The optimal solution to these problems from my point of view would be to 
get that "much discussed" package tool in place soon (CPAN,apt alike).

Heck, now I think about it maybe I should do that before any std.net 
stuff. Let's see how the wind blows - many interesting projects :)

> I've been trying to get familiar with libev lately, so I have bindings
> for livev 3.9 / 4.04 (combined, version can be selected with version
> statements): http://dl.dropbox.com/u/24218791/d/src/libev.html
> (comments in the c header have been turned into doc comments. The
> binding is partially based on Leandro Lucarella bindings:
> http://git.llucax.com.ar/?p=software/ev.d.git)
>

Very nice. I've also stumbled upon some other async D libs out there 
that could be a good starting point.

It seems that Leandro has both bindings and a wrapper in place. And a 
wrapper is really nice to have to make it more D'ish.

/Jonas