Thoughts on Backward Compatibility

[H. S. Teoh]

On Fri, Feb 16, 2024 at 01:44:51AM +0000, Paul Backus via Digitalmars-d wrote:
[...]
> ### What can we learn from this for D?
> 
> First, that the success and popularity of a programming language is
> mostly determined by factors other than stability and backward
> compatibility (or lack thereof).

+100.  Over the past 5-10 years or so, I've been finding myself wishing
that D would introduce some breaking changes so that it could clean up
some of its dark, ugly corners that have been like flies in the ointment
for a long time.

At the same time, I'd had deprecations and breaking changes that are
really rather minor, but extremely frustrating, because:


> Second, that even without an edition bump, small-scale breaking
> changes with easy migration paths aren't a big deal.

The worst feeling is when you upgrade your compiler, and suddenly you
find yourself having to do major code surgery in order to make
previously-fine code work again.  Having an easy migration path for
breaking changes is very important.

I'd also add that the migration path should be *easy*: it shouldn't take
too much thought to upgrade the code, and should not involve tricky
decisions based on subtle semantic differences that require deep
understanding of the code to make the right choice.

The std.math.approxEqual deprecation is a major example that I keep
running into.  It's intended to be replaced by isClose, with all the
right intentions. But it was frustrating because (1) it didn't feel
necessary -- previous code worked fine even if there were some
pathological cases that weren't being handled correctly.  (2) The
deprecation message didn't give a clear migration path -- isClose has
different parameters with subtly different semantics from approxEqual,
and it wasn't obvious how you should replace calls to approxEqual with
equivalent calls to isClose.  There were also no easy defaults that you
could use that replicated the previous behaviour; you had to sit down
and think about each call, then look up the docs to be sure.  (3) The
choice of name felt like a serious blunder, even if it was made for all
the right reasons.

All of these added up to a very frustrating experience, even if the
intentions were right.

If we could have done this over again, I'd have proposed to keep the old
semantics of approxEqual, perhaps add another parameter that would use
the new semantics. And make sure the deprecation message is clear about
how exactly you go about deciding what to put in the new parameter.
I.e., for lazy authors not changing anything would let their code
continue to work as before; if they wanted to have the new semantics
they'd have to explicitly opt in.


> Third, that even with an edition bump, large-scale breaking changes
> that make migration difficult should probably be avoided.

Yes, large breaking changes are a no-no.  Unless my old code can
continue compiling as before, and I have to opt-in to the new stuff.
Editions would help with this, but it still depends on the execution.
There should always be a good migration path that doesn't require you to
rewrite 5-10 year old code that you no longer remember the details of
and can no longer confidently reimplement without spending
disproportionate amounts of time to re-learn the ins and outs of it.


> Fourth, that breaking changes should be used to give D programmers
> more of what they already like about D, not to take the D language in
> new directions.

TBH, @nogc, dip1000, @live, etc., feel a lot like D trying to go in
entirely new directions.  The fact that it's been years and still
practically nobody understands exactly how it works and what it does, is
not a good sign. And all this while things like `share` and static
initialization of AA's are stagnating.  Built-in AA's are one of my
major reasons for choosing D, and seeing it languish for years with
elementary features like static initialization not fixed is quite
disheartening.  Worse when it feels like D wants to move to newer
pastures when its current features are still half-done and has
problematic corner cases.  I.e., what I like about D is stagnating,
while new features that I have little interest in are being pushed on
me.


> To Walter, Atila, and the rest of D's leadership, I hope this post
> provides some helpful data points for you to take into account when
> designing D's language editions and planning future language changes.
> 
> To everyone else reading this, I'd like to leave you with one last
> question: what do **you** like about D? What strengths does D have, as
> a language, that you'd like to see become even stronger?
[...]

What I like about D:

- Meta-programming power.
   - CTFE should be improved.  By a lot.  It was a big disappointment
     that Stefan's newCTFE never materialized.  IMO we should be
     improving this story instead of trying to chase rainbows like ARC
     with @live and dip1000 and what-not.  We should make this so good
     that I'll never need to use an external codegen utility again. And
     it should not introduce crazy compile times. This is a primary D
     strength, its story should be maximally optimized.

   - The template story should be improved.  There should be a way of
     working with templates that cut down on needless bloat.  Lots of
     room for exploration here.  We shouldn't be confined by C++
     limitations here. This is one of D's primary strengths and where we
     can pioneer even more.  One area is improving IFTI to make it work
     for even more common cases.  Another is recognizing common patterns
     like chains of ranges, and optimizing symbol generation so that you
     don't end up with unreasonably huge symbols. Esp. when it's a
     one-of-a-kind UFCS chain (it's unlikely you're ever going to have
     exactly the same chain twice with exactly the same template
     arguments -- no point encoding every argument in the symbol, just
     an ID that gets incremented per instantiation is good enough).

   - Compile-time introspection and DbI. This is another huge D
     strength, and we should be working on streamlining it even more.
      - Clean up __traits(), make std.traits more sensible.
      - Fix things like scoping issues with static foreach.  Introduce
        local aliases so that static foreach doesn't need crazy hacks
        with {{...}} and temporary templates just for injecting new
        identifiers per iteration without running into multiple
        declaration errors.
      - Improve the syntax for retrieving members of some symbol.
        Something prettier than __traits(getAllMembers,...). This is a
        primary D strength, it should be dressed in better syntax than
        this.
      - Maybe first-class types to make the metaprogramming story even
        more powerful.

- GC.  Instead of bending over backwards trying to woo the @nogc crowd
  who are mass migrating to Rust anyway, what about introducing write
  barriers that would allow us existing D users to use a much more
  competitive GC algorithm?  Stop-the-world GC in 2024 shouldn't even be
  a thing anymore. We aren't in 1998 anymore.  D should embrace the GC,
  not sacrifice it for the sake of wooing a crowd that isn't likely to
  adopt D regardless.  Instead of trying to get away from the GC, what
  about making the GC experience better for existing D users?

- Built-in AA's.  It's been at least a decade.  Why is static
  initialization support still sketchy?

- Built-in unittests. The default experience should be top-of-the-line.
  We shouldn't need to import a dub package for something beyond the
  current dumb built-in test runner. Named unittests, the ability to
  select which tests to run, the ability to run all tests regardless of
  failure and show stats afterwards -- these are all basic
  functionalities that ought to work out-of-the-box.

- Automatic type & attribute inference. `auto` was revolutionary when I
  first joined D (C++ got it only years later).  We should improve type
  and attribute inference to the max (e.g., in default parameters for
  enums: there should be no need to repeat the enum name). Nobody like
  spelling out attribute soup, just like nobody likes spelling out
  explicit types when it's already obvious from context. The compiler
  should automate this to the max.  A little bit of breakage here IMO is
  acceptable as long as it gets us to an even better place.  Have
  negated attributes be a thing as well.  In fact, make attributes
  first-class citizens so that we can use DbI / metaprogramming to
  manipulate it.  This is what we should be focusing our efforts on
  instead of trying to woo an amorphous group of hypothetical potential
  users somewhere out there who aren't particularly likely to adopt D to
  begin with.


T

-- 
People tell me that I'm skeptical, but I don't believe them.