D future ...

[Kelly Sommers via Digitalmars-d]

As an outsider to the D community but someone who has really 
wanted to love D the last few years I hope to shed some light 
from "outside the bubble" on why I haven't used D and why I use 
what I use and what I'm looking for. I hope this can be well 
received.

I write a lot of C and Go. But they aren't what I truly want in 
"systems" programming languages. I write high performance 
infrastructure code like databases or server software that 
handles millions of requests per second. One of my projects is an 
OSS HTTP server written in C that can serve 10 million 
requests/second.

Let's clarify a couple things about C and Go. C's 
advantages/disadvantages are:
1. Manual memory management. This could also be seen as a 
disadvantage but having the power to tailor memory access is an 
advantage for memory usage, allocation and access optimizafions. 
Let's face it, high perfowmcne code is all about memory access.
2. No GC pauses impacting request response times. Java, Go, .NET 
etc all have this problem.
3. Harder to write and write well, safely and securely.
4. Few concepts to learn.
5. Composability of libraries is poor. No package system, often 
not cross platform.
6. A lot of low level fundamental libraries are written in C like 
async I/O libraries or storage engine libraries.
7. Static compilation.

Go's advantages/disadvantages:
1. Static compilation.
2. No manual memory management.
3. Suffers from GC pauses.
4. Great composability of libraries. For example, you can easily 
"go get" a Redis protocol library, an ACID memory mapped 
persisted b+tree library and build a little database quite 
quickly.
5. Few concepts to learn.
6. Performance overhead when calling C libraries like storage 
engines.
7: The statement that Go is mostly used in scripting-like 
contexts isn't correct. It's very popular in the infrastructure 
space like databases and distributed systems implementations.
8. Lack of generics causes a lot of boilerplate code in 
comparison to other modern languages.

As an engineer who works on distributed systems in the scale of 
thousands of nodes I have to point out that GC'd languages need 
to be thought more as in the same category because of how they 
operate in production. You have to spend significant effort 
keeping these processses happy so that the GC's don't pause too 
much hurting response times. I argue all the time that GC'd 
languages are not systems languages. There's not enough control 
and you can't eliminate GC pauses completely and these GC's don't 
scale to modern physical hardware well.

But Go is popular in this category of space despite the 
GC/generics because of how composable infrastructure code is in 
the Go community. "go get" a Raft library, gossip library, 
storage library and you're off and running much faster than in 
other languages. However the overhead of calling C libraries and 
the GC impact performance. Go knows this weakness of the GC and 
has been working hard to eliminate GC pauses as much as possible. 
A great effort into sub-millisecond pauses with large heaps was 
recently achieved: 
https://github.com/golang/proposal/blob/master/design/17503-eliminate-rescan.md

What I really want is what C++ wanted to deliver but it doesn't. 
I want something better than writing C but with the same 
performance as C and the ability to interface with C without the 
performance loss and with easily composable libraries.

D in my opinion in some ways is close to these goals. It's 
simpler to understand and write than C++. It has the problem of 
being a GC'd language however and it's unclear to me if the GC in 
D is evolving like the Go GC is. I'm not happy about having to 
use a GC but if I have to I hope to see the one I'm investing on 
evolve because it most certainly impacts production systems very 
much.

The things I really want from D to really sway me would be the 
following (some already exist):
1. Evolve the GC like Go has.
2. No overhead calling C libraries.
3. Easily composable libraries.
4. Good IDE support.

#2 is really why Go has gained a lot of popularity but #1 is 
extremely important on how production systems behave. #2 is 
likely something Go cannot solve and why there's an opportunity 
for another language to jump in with better performance.

The programming community hasn't found a good modern systems 
language yet that aligns with these tradeoffs. I actually think D 
and Swift (no GC, ARC, no calling C overhead) are closest to 
having the potential for the correct tradeoffs to be systems 
languages.

Rust probably is aligned more than anyone with these goals at the 
moment but every time I try to learn rust my head hurts and it's 
not enjoyable.

I hope this sheds some light on why I really like D but as an 
outsider what I'm looking and need in the space I build software 
and why I think D could fill a gap others aren't yet.

Thank you for reading my thoughts.