Curl wrapper

Wed May 18 05:52:07 PDT 2011

Jonas Drewsen wrote:
>On 18/05/11 10.09, Johannes Pfau wrote:
>> jdrewsen wrote:
>>> Please see comments below.
>>>
>>> Den 17-05-2011 16:42, Andrei Alexandrescu skrev:
>>>> Thanks for the response! A few more answers and comments within
>>>> (everything deleted counts as "sounds great").
>>>>
>>>> On 5/17/11 3:50 AM, Jonas Drewsen wrote:
>>>>>> 14. Isn't the max redirect configurable via a parameter?
>>>>>
>>>>> Yes. It is called Http.followLocation from libcurls followLocation
>>>>> option. I will rename it to maxRedirections for clearity.
>>>>
>>>> You mention this about many higher-level functions: "Maximum
>>>> redirections are set to 10." Then I'm thinking, if it's an
>>>> important parameter, make it a defaulted parameter for the
>>>> function.
>>>
>>> I guess I should just remove that comment because other defaults has
>>> been selected as well e.g. timeouts.
>>>
>>>>>> 16. Documentation should point to descriptions of the HTTP
>>>>>> methods wrapped (e.g. "post", "del" etc).
>>>>>
>>>>> Do you mean point to the relevant RFC?
>>>>
>>>> Yah, or a good tutorial. (I didn't know DEL existed!)
>>>
>>> Well DEL is actually called DELETE in the HTTP RFC. But delete is a
>>> reserved word i D so I used del() instead. delete() wouldn't work in
>>> following code in think:
>>>
>>> with(auto http = ...) {
>>> 	delete(...);
>>> }
>>>
>>>>>> 22. byLine/byChunk should not expose a string or generally data
>>>>>> that can be shared safely by the client. That's because it would
>>>>>> need to create a new string with each iteration, which is very
>>>>>> inefficient. Instead, they should expose char[]/ubyte[] just like
>>>>>> File.byLine does, and reuse the buffer with each call.
>>>>>> Essentially byLine/byChunk should be near-zero-overhead means to
>>>>>> transfer and inspect arbitrarily large streams.
>>>>>
>>>>> byLine/byChunk is currenly only available for the async methods
>>>>> which is implemented using message passing. This means that for
>>>>> passing the message a copy has to be made because a message is
>>>>> by-value or immutable data. Is there another way for me to pass
>>>>> the message without doing a copy... some kind of move semantic?
>>>>
>>>> A library element is planned but not on the short list yet. You can
>>>> work around that by internally doing a cast. As long as you observe
>>>> the commitment that buffers are not accessed simultaneously by more
>>>> than one thread you're well within defined behavior.
>>>
>>> ok nice to know.
>>>
>>>> One more suggestion - you may want to also provide classic boring
>>>> synchronous read/write methods that take a user-provided buffer.
>>>> I'm sure people could use such e.g. when they want to stream data
>>>> inside their own threads, or even in the main thread if they don't
>>>> mind blocking.
>>>
>>> This would mean hooking into libcurl and selecting on its sockets.
>>> Totally doable.
>> I'm not sure if it'd be that easy. Using select to implement a
>> blocking api is possible, but select is only used to wait until data
>> is ready, reading and writing the data is still done by curl. And as
>> curl does not allow you to supply the data buffer, having
>> user-provided buffers is afaik impossible.
>
>_Doable_ - not easy :)
>
>Select will wait for data to be ready and ask curl to handle the data 
>chunk. Curl in turn calls back to a registered callback handler with
>the data read. That handler fills the buffer provided by the user. If
>not enough data has been receive an new select is performed until the 
>requested amount of data is read. Then the blocking method can return.

I think it's not necessary to do multiple calls if the first didn't
return enough data. Most read calls are only guaranteed to return a
maximum of data, less is always possible. It's a bigger problem if
the user supplied buffer is smaller than the curl buffer. And you
always end up copying data. So it might be better
to just return a reference to the buffer allocated by curl, or just
forget about this api, curl just doesn't work well this way.

>>> But I would rather stop here feature wise and wrap
>>> this thing up. I would like to focus my efforts on a candidate for
>>> std.net.event/reactor/proactor module.
>>
>> Have you already started working on std.net.event? Do you plan to
>> base that on libev / libevent?
>
>I'll finish the curl wrapper before starting on it.
>
>I have had a look at libev/event before and they are very nice libs.
>But I think I'll have to implement it from scratch for two reasons:
>
>1, AFAIK we cannot include code in std phobos that is not boost
>licensed or a license as liberal. libev for example requires you to
>distribute the license with your software.

That could indeed be a problem. I think libev/libevent even have the
same license. Probably it's really best to implement it from scratch,
but it will be quite some work. I'd still have a look at the libev
documentation though, it contains some details about OS quirks. And try
to avoid Linux AIO ;-)

>2, It introduces a dependency to an external project in phobos. 
>Currently no dependencies are present. This makes phobos very easy to 
>install and use out of the box.
>
>The optimal solution to these problems from my point of view would be
>to get that "much discussed" package tool in place soon (CPAN,apt
>alike).
>
>Heck, now I think about it maybe I should do that before any std.net 
>stuff. Let's see how the wind blows - many interesting projects :)
>
>> I've been trying to get familiar with libev lately, so I have
>> bindings for livev 3.9 / 4.04 (combined, version can be selected
>> with version statements):
>> http://dl.dropbox.com/u/24218791/d/src/libev.html (comments in the c
>> header have been turned into doc comments. The binding is partially
>> based on Leandro Lucarella bindings:
>> http://git.llucax.com.ar/?p=software/ev.d.git)
>>
>
>Very nice. I've also stumbled upon some other async D libs out there 
>that could be a good starting point.
>
>It seems that Leandro has both bindings and a wrapper in place. And a 
>wrapper is really nice to have to make it more D'ish.

Yes, I have a higher level wrapper for libev as well, but the new
'shared' stuff made it a lot more complicated to get that right.
>/Jonas
>
>
-- 
Johannes Pfau
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20110518/3028a192/attachment.pgp>