Curl support RFC

[Jonas Drewsen]

On 29/03/11 17.31, Johannes Pfau wrote:
> Jonas Drewsen wrote:
>>
>> This is a nice protocol parser. I would very much like it to be used
>> with the curl API but without it being a dependency. This is already
>> possible now using the onReceiveHeader callback and this would
>> decouple the two. At least until std.protocol.http is in phobos as
>> well - at that point convenience methods could be added :)
>>
>> /Jonas
>
> Thanks, I think I'll propose the parser for the new experimental
> namespace when it's available.

I'm looking forward to that.

> About the headersReceived callback: You're totally right, it can be
> done with the onReceiveHeader callback right now. But I think in the
> common case the user wants the headers in an key/value array. So if the
> user doesn't want to use the onReceiveHeader api, a headersReceived
> callback would probably be convenient. But, as said it's not necessary.

I'll put it on my todo and reconsider when I get to it :)

> Reading the curl documentation showed another small trap:
> CURLOPT_HEADERFUNCTION
> ------------------------------------------------------------
> It's important to note that the callback will be invoked for the
> headers of all responses received after initiating a request and not
> just the final response. This includes all responses which occur during
> authentication negotiation. If you need to operate on only the headers
> from the final response, you will need to collect headers in the
> callback yourself and use HTTP status lines, for example, to delimit
> response boundaries.
> ------------------------------------------------------------
>
> I think if we store the headers into an array, we should only store the
> headers of the final response. Another question is should all headers
> or only final headers trigger the onReceiveHeader callback? Passing
> only the final headers would require extra work, passing all headers
> should at least be documented.

Yeah... I've discovered this myself as well.

The current implementation does as libcurl does it an passes all headers 
not just for the final subrequest.

> Thinking of this more, this also means the _receiveHeaderCallback is
> not 100% correct, as it expects all lines after the first line to be
> header or empty lines, but it's possible that we get multiple statuslines.
> It still works, the regex doesn't match anything and the code
> ignores that line. But this way, the stored statusline will always be
> the first statusline, which isn't optimal. We'd also need to detect if a
> line is a statusline to reset the headers array if it's used. Seems
> like we have to think about this some more.

My local version already takes care of this. It was the wrong place for 
parsing status lines and headers anyway. It is now moved to the Http 
class where it should have been all the time.

I have implemented almost all of the features/changes suggested now. The 
last one I'm currently fighting is the support for "foreach" and async 
.byLine/.byChunk. I may have to make some changes in the current design 
to support this with the calling API that I would like to expose.

I wonder who could take the step and open a std.experimental package for 
submissions?

Thank you for the feedback!