Curl wrapper

[jdrewsen]

Den 18-05-2011 14:52, Johannes Pfau skrev:
> Jonas Drewsen wrote:
>> On 18/05/11 10.09, Johannes Pfau wrote:
>>> jdrewsen wrote:
>>>> Please see comments below.
>>>>
>>>> Den 17-05-2011 16:42, Andrei Alexandrescu skrev:
>>>>> Thanks for the response! A few more answers and comments within
>>>>> (everything deleted counts as "sounds great").
>>>>>
>>>>> On 5/17/11 3:50 AM, Jonas Drewsen wrote:
>>>>>>> 14. Isn't the max redirect configurable via a parameter?
>>>>>>
>>>>>> Yes. It is called Http.followLocation from libcurls followLocation
>>>>>> option. I will rename it to maxRedirections for clearity.
>>>>>
>>>>> You mention this about many higher-level functions: "Maximum
>>>>> redirections are set to 10." Then I'm thinking, if it's an
>>>>> important parameter, make it a defaulted parameter for the
>>>>> function.
>>>>
>>>> I guess I should just remove that comment because other defaults has
>>>> been selected as well e.g. timeouts.
>>>>
>>>>>>> 16. Documentation should point to descriptions of the HTTP
>>>>>>> methods wrapped (e.g. "post", "del" etc).
>>>>>>
>>>>>> Do you mean point to the relevant RFC?
>>>>>
>>>>> Yah, or a good tutorial. (I didn't know DEL existed!)
>>>>
>>>> Well DEL is actually called DELETE in the HTTP RFC. But delete is a
>>>> reserved word i D so I used del() instead. delete() wouldn't work in
>>>> following code in think:
>>>>
>>>> with(auto http = ...) {
>>>> 	delete(...);
>>>> }
>>>>
>>>>>>> 22. byLine/byChunk should not expose a string or generally data
>>>>>>> that can be shared safely by the client. That's because it would
>>>>>>> need to create a new string with each iteration, which is very
>>>>>>> inefficient. Instead, they should expose char[]/ubyte[] just like
>>>>>>> File.byLine does, and reuse the buffer with each call.
>>>>>>> Essentially byLine/byChunk should be near-zero-overhead means to
>>>>>>> transfer and inspect arbitrarily large streams.
>>>>>>
>>>>>> byLine/byChunk is currenly only available for the async methods
>>>>>> which is implemented using message passing. This means that for
>>>>>> passing the message a copy has to be made because a message is
>>>>>> by-value or immutable data. Is there another way for me to pass
>>>>>> the message without doing a copy... some kind of move semantic?
>>>>>
>>>>> A library element is planned but not on the short list yet. You can
>>>>> work around that by internally doing a cast. As long as you observe
>>>>> the commitment that buffers are not accessed simultaneously by more
>>>>> than one thread you're well within defined behavior.
>>>>
>>>> ok nice to know.
>>>>
>>>>> One more suggestion - you may want to also provide classic boring
>>>>> synchronous read/write methods that take a user-provided buffer.
>>>>> I'm sure people could use such e.g. when they want to stream data
>>>>> inside their own threads, or even in the main thread if they don't
>>>>> mind blocking.
>>>>
>>>> This would mean hooking into libcurl and selecting on its sockets.
>>>> Totally doable.
>>> I'm not sure if it'd be that easy. Using select to implement a
>>> blocking api is possible, but select is only used to wait until data
>>> is ready, reading and writing the data is still done by curl. And as
>>> curl does not allow you to supply the data buffer, having
>>> user-provided buffers is afaik impossible.
>>
>> _Doable_ - not easy :)
>>
>> Select will wait for data to be ready and ask curl to handle the data
>> chunk. Curl in turn calls back to a registered callback handler with
>> the data read. That handler fills the buffer provided by the user. If
>> not enough data has been receive an new select is performed until the
>> requested amount of data is read. Then the blocking method can return.
>
> I think it's not necessary to do multiple calls if the first didn't
> return enough data. Most read calls are only guaranteed to return a
> maximum of data, less is always possible. It's a bigger problem if
> the user supplied buffer is smaller than the curl buffer. And you
> always end up copying data. So it might be better
> to just return a reference to the buffer allocated by curl, or just
> forget about this api, curl just doesn't work well this way.

Yeah. That's part of the reason why I'm not interested in implementing it.

>>>> But I would rather stop here feature wise and wrap
>>>> this thing up. I would like to focus my efforts on a candidate for
>>>> std.net.event/reactor/proactor module.
>>>
>>> Have you already started working on std.net.event? Do you plan to
>>> base that on libev / libevent?
>>
>> I'll finish the curl wrapper before starting on it.
>>
>> I have had a look at libev/event before and they are very nice libs.
>> But I think I'll have to implement it from scratch for two reasons:
>>
>> 1, AFAIK we cannot include code in std phobos that is not boost
>> licensed or a license as liberal. libev for example requires you to
>> distribute the license with your software.
>
> That could indeed be a problem. I think libev/libevent even have the
> same license. Probably it's really best to implement it from scratch,
> but it will be quite some work. I'd still have a look at the libev
> documentation though, it contains some details about OS quirks. And try
> to avoid Linux AIO ;-)
>> 2, It introduces a dependency to an external project in phobos.
>> Currently no dependencies are present. This makes phobos very easy to
>> install and use out of the box.
>>
>> The optimal solution to these problems from my point of view would be
>> to get that "much discussed" package tool in place soon (CPAN,apt
>> alike).
>>
>> Heck, now I think about it maybe I should do that before any std.net
>> stuff. Let's see how the wind blows - many interesting projects :)
>>
>>> I've been trying to get familiar with libev lately, so I have
>>> bindings for livev 3.9 / 4.04 (combined, version can be selected
>>> with version statements):
>>> http://dl.dropbox.com/u/24218791/d/src/libev.html (comments in the c
>>> header have been turned into doc comments. The binding is partially
>>> based on Leandro Lucarella bindings:
>>> http://git.llucax.com.ar/?p=software/ev.d.git)
>>>
>>
>> Very nice. I've also stumbled upon some other async D libs out there
>> that could be a good starting point.
>>
>> It seems that Leandro has both bindings and a wrapper in place. And a
>> wrapper is really nice to have to make it more D'ish.
>
> Yes, I have a higher level wrapper for libev as well, but the new
> 'shared' stuff made it a lot more complicated to get that right.
>> /Jonas
>>
>>