The end of curl (in phobos)

[Jonathan M Davis via Digitalmars-d]

On Sunday, May 08, 2016 11:33:07 Andrei Alexandrescu via Digitalmars-d wrote:
> On 5/8/16 11:05 AM, Jonathan M Davis via Digitalmars-d wrote:
> > On Sunday, May 08, 2016 02:44:48 Adam D. Ruppe via Digitalmars-d wrote:
> >> On Saturday, 7 May 2016 at 20:50:53 UTC, Jonas Drewsen wrote:
> >>> But std.net.curl supports not just HTTP but also FTP etc. so i
> >>> guess that won't suffice.
> >>
> >> We can always implement ftp too, it isn't that complicated of a
> >> protocol.
> >>
> >> Though, I suspect its users are a tiny minority and they might
> >> not mind depending on a separate curl package.
> >
> > An alternative would be to move std.net.curl into a dub package.
>
> That would still be a breaking change, is that correct?

Any replacement would be a breaking change, unless the intention is to keep
to std.net.curl's API while ripping out the actual curl usage, and that
seems unnecessarily restrictive to me and likely to end up with something
inferior to what we would otherwise get. IIRC, we've already had issues with
std.net.curl not quite doing what we want (though at least some of those
could likely be fixed if we have full control over what the underlying code
is doing).

But given that std.net.curl handles stuff like SSL/TLS, we _can't_ actually
replace all of its functionality - at least not without adding a dependency
on a different C library, since there's no way that it's sane to do the
crypto stuff ourselves without a crypto expert, and even then, we should
think twice about it. I could see implementing the SSL/TLS protocols
themselves but not the crypto they use. If we replace std.net.curl, we
likely should just provide the basic HTTP functionality, and leave the rest
to a dub package that we move std.net.curl to.

But even if we did manage to replace all of std.net.curl's functionality in
Phobos, I definitely think that we should put in dub. Whatever problems
we've had with it, they're definitely not bad enough for us to want to
eradicate it as D code, just remove it from Phobos. And if it were a dub
package, updating your code to use it should be pretty trivial. Once
std.net.curl was deprecated in Phobos, and it was put in dub as a separate
package, anyone who wanted to update would just change their import
statements to use the module name from the dub package and add it as a
dependency to their dub.json/dub.sdl file. So, as far as breaking changes
go, moving std.net.curl from Phobos to code.dlang.org would be very easy to
deal with.

> I'm unclear on what the reasons are for removing libcurl so I'd love to
> see them stated clearly. Walter's argumentation was vague - code that we
> don't control etc. There have been past reports of issues with libcurl
> on windows, have those not been permanently solved?
>
> I even see a plus: dealing with libcurl is a good exercise in eating our
> dogfood regarding "interfacing with C libraries is trivial" stance.
> Having to deal with it is a reflection of what other projects have to do
> on an ongoing basis.

Walter's main complaint seemed to be that he didn't like the idea of
depending on C libraries other than the C runtime in our standard library,
since it meant that there was code in our standard library that we did not
have control of, and it arguably looks bad to have to use C libraries in our
own standard library. But the big problem that I'm aware of has had to do
with the fact that we then have an external dependency that not everyone has
on their system. Even on Linux, on distros that separate out "dev" packages
so that you have to install one package to use a library and another to
build code using it, libcurl is not necessarily going to be there to be
built against (and it's definitely not on Windows normally). And that's
definitely caused problems - though including libcurl with the dmd installer
(which we didn't do initially) has reduced those problems.

Personally, I've run into issues with the tools repo due to its dependence
on std.net.curl, and while that may be related to issues on my system rather
than anything that's been done with Phobos itself, that doesn't exactly give
me warm, fuzzy feelings for std.net.curl.

In addition, Phobos is not tied to curl's release cycle, and if curl gets a
version bump on someone's system, and Phobos hasn't been updated to match,
they're going to have a problem. And if we updated to match the version
bump, and their distro hadn't, then we'd also have a problem. AFAIK, curl's
API has been fairly stable such that this hasn't been a problem, but it's
definitely a concern with C libraries in general, and it was pointed out at
dconf that it _is_ a problem with the sqlite bindings. So, even if we decide
to keep std.net.curl, I think that versioning issues alone should make us
very leery of including C libraries in Phobos - be they via bindings or
wrappers. I don't think that we want to be in a situation where someone has
to use a specific version of dmd and Phobos, because it's the one that
matches a C library on their system that Phobos depends on. And while that
might be manageable with one C library dependency, it likely wouldn't be
with multiple. Having such dependencies in dub packages should make it
easier to manage them.

Another thing to consider is portability. Just because a library is written
in C doesn't mean that it's going to work on all of the targets that we want
to hit with dmd, ldc, and gdc. We have control over that sort of thing if
the code is in D. We don't if it's in C. From the looks of it, curl is
available for the vast majority of platforms, so this likely won't be a
problem with curl, but it is a reason to think twice before considering
adding another dependency on a C library to Phobos.

If std.net.curl were not already in Phobos, I think that it's quite clear
that it would be a great fit for a dub package, and I would definitely vote
against its inclusion in Phobos. And at this point, I will likely do the
same for any C library. I think that those all belong on code.dlang.org and
not in Phobos.

- Jonathan M Davis