`ref T` should be a type!!

[Manu]

On Mon, Apr 1, 2019 at 4:10 AM Walter Bright via Digitalmars-d
<digitalmars-d at puremagic.com> wrote:
>
> On 3/31/2019 6:19 PM, Manu wrote:
> > Are you saying that it's impossible to design a solution where ref is
> > part of the type and that's less complex than C++?
>
> I couldn't think of a way. I could have simply copied the C++ design. But why
> copy a nightmare of kludges?
>
>
> > Is the complexity is that ecosystem inherent, or is it somewhat
> > informed by backwards-compatibility, and other C++ design constraints?
>
> There was no backwards compatibility issue when ref was designed. It's just a
> poor choice (like volatile).
>
>
> > The fact that I can write `struct Ref(T)` and that it propagates the
> > type system in a natural way suggests to me that the design is not
> > impossible.
>
> I have no idea why you want to do such things, but apparently you have a
> solution that works for you, so that's all good.
>
>
> >> Please show a use case. I can't think of one.
> > A use case for propagating ref in the type? You can't imagine why a
> > template argument might want to distinguish `int` and `ref int`?
>
> Indulge me. Why would it?

Propagating compile time information as types is the fundamental of D's meta.
Not being able to propagate ref's is really problematic. Not being
able to declare a local ref is really problematic.

void fun(ref int x)
{
  bool b = is(typeof(x) == ref); // nope! seems so obvious (you should
watch their faces when I show them how to do it)

  int i = 10;
  ref int j = i; // nope! again, watch their faces when I say they can't do this

  ref auto resolveLongExpression =
long.expression().that[0].we().dont_want_to.repeat;  // nope!
  foreach(i; things)
  {
    resolveLongExpression++; // repeated access to a deep expression

    // use a pointer you say? pointers change semantics; assignment
and increment mean something different now
    // if there's any meta involved, then pointer's semantic
distinction is something you're gonna have to write `static if()`
nonsense to detect and special-case!
  }

  // other kinds of expressions that we don't want to repeat, but may
want to carry the ref-ness
  auto ref x = returnsValue();
  ++x; // increment local
  auto ref y = returnsRef();
  ++y; // increment reference
}

alias Param = Parameters!(fun)[0];
void g(Param arg) // nope!
{
  ... lots of code
}

this is obviously much better:

bool isRef(alias fn, int argIndex, int i = 0)
{
  static if (i == __traits(getParameterStorageClasses, fn, argIndex).length)
    enum isRef = false;
  else
    enum isRef = __traits(getParameterStorageClasses, fn, argIndex)[i]
== "ref" ? true : isRef!(fn, argIndex, i + 1);
}

string myFun = "void g(" ~ (isRef!(fun, 0) ? "ref " : "") ~ "Param arg)\n" ~
"{\n" ~
"    lots;\n" ~
"    of;\n" ~
"    code;\n" ~
"}\n";
mixin (myFun);

Whole function (or any other construction that fits this pattern) is a
string because some storage class appears in the declaration.
Say goodbye to autocomplete, debugging, refactoring, syntax
highlighting, readability, and editability/maintainability.


Sorry, I don't have a real-world case at hand right now, I didn't
start the thread complaining this time; I'm just supporting someone
who is feeling the same feels.
I've learned to try and avoid this stuff in D, because D just can't
reasonably do it. But it's very annoying to avoid, and it's a major
weakness of D's meta, and it's very disappointing to new users when
they work this out.


> > I don't know how you've never found yourself static-if-ing on ref-ness
> > of stuff in every bit of functional meta you've ever written. Ref
> > creates an out-of-language (we have no language features to interact
> > with 'storage class') suite of conditions that I frequently have to
> > wrangle my way through.
> > We have strong language mechanisms to reason about the type system,
> > there is no language to reason about storage class that's not a kludge
> > of awkward hacks and ugly meta utilities.
>
> If I was faced with that, I'd step back and re-assess why the design required
> such machinations.

This is like a general argument against meta-programming. You don't
get to decide when this stuff comes up in the wild; it just does.
The worst cases are usually when automating bindings between systems;
that might be to foreign languages, scripting solutions,
tooling/editor interactions, or perhaps some high-level application
scaffolding.
One of D's biggest attractions is the ability to use powerful
in-language meta-programming features to solve binding and boilerplate
tasks that would traditionally require external tooling. When I use D,
I lean into that promise HARD. The reason to choose D is to solve
particular categories of problems I can't possibly solve in other
languages.
Ref (and other storage classes) bite me in the arse every single time.
Imagine how much worse this will be when 'scope' proliferates!


> It reminds me of when I peruse Phobos, I'm disturbed by the overly complex code
> there - something Andrei referred to in the "generality creep" thread.

I'm not commenting on phobos complexity (which is indeed hideous, and
I avoid phobos).
There are a lot of programming tasks; one programming task is writing
programs, but another overwhelmingly popular and most tedious and
painful task is developing and maintaining bindings and boilerplate
interaction to large established ecosystems.
Basically everyone I know who has been drawn to D are initially
attracted by the ability to automate boilerplate and terrible cruft
from external tooling and reducing build complexity. That's the stuff
that sends them packing in search of a brighter future.
A lot of meta is used to automate boilerplate and binding machinery.

If a user is 'just writing programs', then C++ is just fine. They're
attracted to D by the promise of simplifying or addressing the
mountain of code that C++ _can't_ express. Boilerplate meta is one of
the most common tasks that stokes initial interest in D. To dismiss
that, is to misunderstand D's value proposition to a huge audience.
This is certainly the case where D is discussed in the project I'm
work on right now, and has been the case since I ever started
discussing D with colleagues. It's the most immediate and painful
problem they see an obvious solution for. You should value that
use-case if you want to attract users.

Everything else D has to offer is a nice bonus that these people would
appreciate in time, but they need to be drawn in by successfully
solving the problems immediately before them that D appears to be
well-placed to solve.
I have watched so many cases of people trying out some such solution,
and then they encounter ref, I start to explain to them what they need
to do, they are surprised, then they're angry at me for wasting their
time, and then they lose interest.

Storage class is really, *really* bad. The terribleness of it in terms
of the impression it leaves on people is really important. I've been
trying to tell you this for so long, I don't think there's anything
else I can do.


> If C++ does it better, I don't see how. I find the implementation code of the
> STL to be horrifying in its complexity. I don't know how to write such code
> without thinking "something has gone terribly wrong".

It's not about STL. I hope I've made that clear.
But that said, you have invented the most powerful meta-programming
language out there, you've gotta own that, and you can't be upset when
people try to apply it to solve specific problems it's uniquely
well-fit for.
I agree there's a HUGE danger of abusing D's meta-programming
features, I constantly have to restrain myself, but the kind of work I
refer to is not of that kind.


> If I may, I once talked to a Ferrari mechanic. He said that most engines inside,
> where customers didn't see the innards, were sloppily made. But that the Ferrari
> engine was a thing of beauty inside and a joy to work on. Ironically, Ferraris
> are also known for cheap, poorly done cab interiors. An unusual focus for a car
> company :-) but one that appeals to my inner engineer. The engine in my Dodge
> was built the same way, it's blueprinted with all top quality parts inside it,
> but the outside is plain jane, no chrome parts, no dress-up, etc.

That's a nice thought, but that's not important to this particular
class of problem.
What I'm talking about here is attracting users who are able to
successfully solve their long-standing and terribly painful problems.
Cases where storage class comes up as a problem is almost always of
that category. Apparently I seem to suffer this category of task 10x
more than the average guy.

I've tried to make this point before, and I'll try again;
Most programmers are not 'great' programmers, they're just
programmers, they're employees of businesses, at 5pm they go home and
hang out with the kids and shag the wife.
They're not too worried about building a ferrari engine; they mostly
just wanna get paid, they have a job to do, and they almost invariably
have an established ecosystem of existing cruft they interact with.
There is a high probability they have this rats-next of
hand-maintained boilerplate and tooling that they HATE maintaining,
and that's the angle I've continually had the most success gaining
interest from new users.

Our goal is to attract these people, and ideally, improve their
ability to do a good job more quickly. We want them to feel more
productive and make less mistakes, but we can't help them if they
never come to the party!

D's key advantage out-of-the-gate as I have seen it for so many years,
is that it's *theoretically* possible to build bridges between
existing ecosystems where we can take a foothold against 40 years of
establishment.
This is the land of boilerplate and connective meta. I've written a lot of it.
They won't come to us if they need to swim the crocodile infested
river. We must build a bridge to them and make crossing the river
unreasonably simple and appear like the obvious thing to do.

Storage class hurts more than anything else towards this end.

Anyway... I'm going to bed. I'll complain about it again next year.