Downloading files over TLS

Fri Jun 26 10:12:09 UTC 2020

Downloading files over TLS. This seems that it's something that 
should be quite simple to do. My high level goals are 
cross-platform and easy distribution. I don't need anything fancy 
just a simple API like this:

download("https://url.com", "/local/file");

Because of these goals, I have a few requirements, which I don't 
think are unreasonable:

* Cross-platform (macOS, Linux, Windows, possible FreeBSD)

* Minimal runtime requirements (I don't mind a few compile time 
requirements). I also don't mind runtime requirements that are 
guaranteed to exist

* Ideally using the platform provided TLS implementation, or at 
least platform provided certificates (I've been bitten several 
times due to old certificates making request fail)

* On Linux I would really like to have a fully statically linked 
binary, this will make distribution easier

* Due to the previous requirement, I think using musl will be 
required. For most parts glibc works fine for static linking, but 
I think in the case of networking it does not work. The DNS 
lookup (or something like that) doesn't work with static linking

I think the main problem of all this is picking the right TLS 
implementation.

The obvious candidate is std.net.curl, but I'm going to explain 
why that is not ideal, at least not how it's currently 
implemented. std.net.curl relies on OpenSSL, which is deprecated 
on several platforms. On several platforms it's not the main, 
platform provided TLS implementation. As far as I know, it also 
provides its own certificates and doesn't rely on the ones 
provided by the platform.

I going to break down some alternatives per platform.

macOS:
* std.net.curl. Depends on libcurl and OpenSSL. OpenSSL has been 
deprecated on macOS for many years. In one wants to use OpenSSL 
anyway, one need to ship it with the binary (statically link it). 
For other problems, see above

* requests [1]. This is a potential better alternative than 
std.net.curl since it doesn't rely on libcurl. But, it still 
relies on OpenSSL, so it has the same problems as std.net.curl

* vibe.d [2]. Does not depend on libcurl, but does depend on 
OpenSSL. But, it also supports Botan, which is promising. I'll 
discuss Botan separately

* Hunt [3]. Same problem as the other ones, relies on OpenSSL

* Arsd [4]. Relies on OpenSSL

* SecureTransport [5]. This is an Apple specific library provided 
by the platform. It seems to more or less follow the same idea as 
OpenSSL, it's independent on the transport layer and works with 
BSD sockets. Initially this looks perfect, it's provided by the 
platform and uses certificates provided by the platform. The main 
problem is that it has been deprecated. It also only supports TLS 
up to version 1.2 and since it's deprecated, it's likely that it 
won't support any newer versions

* Network [6]. This is an Apple specific library provided by the 
platform. This is the recommend alternative to SecureTransport. 
The problem is that this is not just an alternative TLS 
implementation, it's a completely different alternative to BSD 
sockets. The API is completely different and will require some 
extra layers to to provide a cross-platform API. This means that 
I cannot use any of the existing library to just add TLS, it will 
be a completely different implementation, which might be ok. 
Another big problem is that it only available on macOS 10.14 and 
later. I have not decided yet if this is acceptable or not

* NSURLSession [7]. This is an  Apple specific Objective-C class, 
provided by the platform. It's higher level than the Network 
framework and is available on all versions of macOS. This will 
again require a some extra layers to hide the platform specific 
API. There's also a problem with LDC which has very limited 
support for Objective-C interoperability.

Linux:
As far as I know, OpenSSL (or some version of it, like LibreSSL) 
is what's provided by the platform. So I guess that has to be 
acceptable on Linux.

* std.net.curl. The main problem with std.net.curl on Linux is 
that it uses `dlopen` to load the TLS library. `dlopen` doesn't 
work for a statically linked executable. Side note, `dlopen` 
doesn't work on iOS (for other reasons) so it would be nice if 
this could be fixed

* requests. Seems to do that same as std.net.curl, uses `dlopen`

* For the other alternatives (mentioned in the macOS section), I 
haven't investigated if they use `dlopen` or not

Windows:
I don't know that much of this platform.

* std.net.curl and basically all other options already mentioned 
relies on OpenSSL, which is not provided by the platform

* SChannel. As far as I know, this the the platform provided 
implementation of TLS on Windows.

* Are there any high level APIs, like NSURLSession, on Windows 
that can be used to download files?

Botan:

This seems like a good candidate. It even exists a full D port of 
it [8].

* This is not used by any specific platform but it can use the 
platform provided certificates, which is a plus.

* The downside is that the code to access the platform provided 
certificates has not been ported to D (or it was added to a newer 
version of the C++ implementation).

* I think the the D port of Botan does not verify certificates, 
which is not great. It might be due to the previous point

* I also don't know if the D port is updated to match the latest 
version of the C++ implementation. By experience I know this can 
be a huge task, so I don't expect it

* Botan only supports up to TLS 1.2 (even the C++ 
implementation). Due to the previous point it's unclear if the D 
port of Botan will get support for TLS 1.3

I know there are other alternative TLS implementation, but I have 
not looked into detail on these.

A question regarding libcurl. I know that curl (the CLI tool) can 
take advantage of SecureTransport and SChannel as alternatives to 
OpenSSL. Due to std.net.curl containing some OpenSSL related 
code, is TLS handled completely outside of libcurl? So 
std.net.curl  can't easily take advantage of SecureTransport and 
SChannel the same way as curl does?

Is there anything obvious I'm missing or why does this seem so 
difficult? Do I have too high expectations and requirements?

[1] https://code.dlang.org/packages/requests
[2] https://code.dlang.org/packages/vibe-d
[3] https://code.dlang.org/packages/hunt
[4] https://code.dlang.org/packages/arsd-official%3Ahttp
[5] 
https://developer.apple.com/documentation/security/secure_transport
[6] https://developer.apple.com/documentation/network
[7] 
https://developer.apple.com/documentation/foundation/nsurlsession
[8] https://code.dlang.org/packages/botan

--
/Jacob Carlborg