Should "std.net.curl" be moved from Phobos to Deimos?

[H. S. Teoh]

On Tue, Nov 26, 2013 at 04:19:31PM -0800, Andrei Alexandrescu wrote:
> On 11/26/13 3:40 PM, Adam D. Ruppe wrote:
> >Again though, my position is that curl is ok and I use it myself, but
> >let's not assume we need the entire kitchen sink that curl offers,
> >especially given the fact that std.net.curl barely scratches it
> >anyway.  If someone does need one of curl's less common features, it
> >is still there for them, but for the 99% we can do with the
> >std.net.curl api, I'm sure we could make that happen in ~1000 lines
> >of curl replacement.
> 
> Care to make it interesting? I may have some funds for this. But we're
> looking for no less than a glorious 100% replacement.
[...]

This piqued my interest.

Please excuse another of my TL;DR stories / rants. Some time earlier
this year, I wanted to write a simple utility to (1) login to a website
with a preconfigured username/password; (2) navigate to a specific page
on the site; (3) extract a specific HTML table from that page, and (4)
parse and format the table into an ASCII-graphics text format. Sounds
like the perfect job for a quick-n-dirty D program, right? In the old
days, I'd hack up a Perl script to do this in a couple o' hours, but
this time, I thought, since I have D at my disposal, let's do this the
Right Way(tm).  Where in the past I'd just run wget or curl (the CLI
utility) to fetch the web pages, I thought it'd be much cleaner to do
the web interaction directly so that I didn't have to mess around with
parsing temporary cookie files and what-not just to keep track of my
login session.

So I scrolled through the Phobos docs, and lo and behold, there was
std.net.curl sitting right there, looking all pretty and beckoning to
me. I glanced over its API, and found that it was just the right fit for
what I needed to do. Or so I thought...  About 30 minutes into the
coding, I ran into a blocker: std.net.curl didn't let me specify
Content-Type of POST requests!!! The server simply refused to accept
anything that wasn't explicitly stated to be x/www-urlencoded, but
std.net.curl only allows text/plain or application/octet-stream!  I
simply couldn't login.  Thinking that I must've missed something
obvious, I spent the rest of the day (and probably several more hours
the next day) combing through the docs to find out how to set the
Content-Type to something else. Finally I resorted to reading the
std.net.curl source code, only to discover, to my dismay, that there was
NO way to do this unless I bypassed all the nice high-level syntactic
sugar, and got my hands down and dirty with low-level libcurl API calls
(or their thin wrappers thereof).

At that point, I threw up my hands. The original task was far too simple
for something this heavy-handed, so I backed off and said, it was nice
knowing you, std.net.curl, but I think we're better off going our
separate ways.

TL;DR: I would *love* to see a replacement for std.net.curl that isn't
so straitjacketed that I can't even do something so basic as logging on
to a website. I'm sure there are very good reasons why the current API
is the way it is, but the fact of the matter is that, from an end-user's
POV, its limitations were a deal-breaker.

So, I'm all for a D-only replacement of std.net.curl with a superior
API. Getting rid of the external dependence on libcurl would be an added
bonus.


T

-- 
"I'm running Windows '98." "Yes." "My computer isn't working now." "Yes, you already said that." -- User-Friendly