DIP69 - Implement scope for escape proof references

[Manu via Digitalmars-d]

On 7 December 2014 at 08:15, Walter Bright via Digitalmars-d
<digitalmars-d at puremagic.com> wrote:
> On 12/6/2014 1:50 AM, Manu via Digitalmars-d wrote:
>>>
>>> I didn't fully understand Manu's issue, but it was about 'ref' not being
>>> inferred by template type deduction. I didn't understand why 'auto ref'
>>> did
>>> not work for him. I got the impression that he was trying to program in D
>>> the same way he'd do things in C++, and that's where the trouble came in.
>>
>> NO!!
>> I barely program C++ at all! I basically write C code with 'enum' and
>> 'class'. NEVER 'template', and very rarely 'virtual'.
>> Your impression is dead wrong.
>
>
> I apologize for misunderstanding you.
>
>
>> It's exactly as Marc says, we have the best tools for dealing with
>> types of any language I know, and practically none for dealing with
>> 'storage class'.
>> In my use of meta in D, 'ref' is the single greatest cause of
>> complexity, code bloat, duplication, and text mixins. I've been
>> banging on about this for years!
>>
>> I've been over it so many times.
>> I'll start over if there is actually some possibility I can convince
>> you? Is there?
>
>
> I know there's no easy way to derive a storage class from an expression. The
> difficulty in my understanding is why this is a great cause of problems for
> you in particular (and by implication not for others). There's something
> about the way you write code that's different.

Perhaps it's the tasks I typically perform with meta?
I'm a fairly conservative user of templates, but one place where they
shine, and I'm always very tempted to use them, is the task of
serialisation, and cross-language bindings.
Those tasks typically involve mountains of boilerplate, and D is the
only language expressive enough to start to really automate that
mechanical mess.
I've done extensive work bindings to C/C++, Lua, and C#. They're all
subtly different tasks, but share a lot in common, and the major
characteristic which reveals my problems with things like ref, and
auto ref, is that these things aren't open to interpretation. They're
not cases where the compiler can make decisions for you (ie, auto ref
fails), and the ABI + API are strict (ie, ref must match correctly).

There's a lot more more to it, but perhaps I'm among few who have had
such extensive need to engage in these tasks?


>> I've lost faith that I am able to have any meaningful impact on issues
>> that matter to me, and I'm fairly sure at this point that my
>> compounded resentment and frustration actually discredit my cause, and
>> certainly, my quality of debate.
>
>
> This is incorrect, you had enormous influence over the vector type in D! And
> I wish you had more.

I appreciate that. That was uncontroversial though; I didn't need to
spend months or years trying to justify my claims on that issue. I
feel like that was a known item that was just somewhere slightly down
the list, and I was able to bring it forward.
I was never in a position where I had to argue against Andrei to sell that one.

I have std.simd sitting here, and I really want to finish it, but I
still don't have the tools to do so.
I need, at least, forceinline to complete it, but that one *is*
controversial - we've talked about this for years.

GDC and LDC both have a forceinline, so I could theoretically support
those compilers, but then I can't practically make use of them without
some sort of attribute aliasing system, otherwise I need to triplicate
the code for each compiler, just to insert a different (compiler
specific) forceinline attribute name. It'd be really great if we
agreed on just one.

There is also a practical problem with GCC (perhaps it's an
incompatbility with my std.simd design), where I need to change the
sse-level between functions.
There is a GCC attribute to do this ('target'), but it would rely on,
at least, a few subtle tweaks. In this case, I need to be able to feed
a template argument to a UDA, which doesn't work because UDA
declarations seem to be parsed prior to knowledge of functions
template args. There may be a further problem with GCC though which I
haven't been able to prove yet though.

The reason for this is that an important design goal for my work was
to be able to semi-automatically generate multiple code paths for
different SSE versions, which can be selected at runtime based on
available hardware.
It's a particularly awkward issue in C/C++, usually requiring you to
have multiple modules which are each built with different compile
flags, and then do some magic while linking to resolve the runtime
selection problem.

Anyway, I have been able to use SIMD directly in my own software with
the support we have, but I haven't been able to complete the library
that I want to produce.
I need a little more language support. I've reached out a few times,
but there hasn't been too much interest in the past. I really do wanna
finish it one of these days though!


>> There has been a couple of instances where the situation has been
>> appropriate that I've tried to make use of auto ref, but in each case,
>> the semantics have never been what I want.
>
>
> I don't really know what you want. Well, perhaps a better statement is
> 'why', not what.

Because a function is a function. It's a fundamental and ultimately
*simple* element of a language, with simple and reliable in behaviour;
you write a function, compiler emits some code with a matching symbol
name. There is perhaps nothing more simple or fundamental to the
language.

A template is not a function. It may *generate* a function, but now
we've added many extra details which require careful consideration and
handling, like where the code is, or isn't, emitted? How many
permutations are there? What about static/dynamic libraries? Calling
with various, or arbitrary types, always required careful
consideration, more so than an explicit type.
Templates are a powerful (often dangerous) **tool**, which should only
be applied deliberately and very carefully.
Obviously auto-ref gives hard answers to some of those questions, but
it is still a template, and that instantly makes it much more complex.

Making a function that has nothing to do with templates into a
template is not what I want... it couldn't be further from what I ever
wanted!

The ABI is different; there are 2 functions now (or zero, until one is
called). Thich means taking the address of a function with auto-ref is
a complex issue.
ref-ness isn't a decision I ever want the compiler to make for me. I
either want ref, or I don't, and I will state that to the compiler
explicitly.
This is especially true when interacting with other languages; while D
remains in relative infancy, I think that is an overwhelmingly common
situation when a project is modestly large.

In the situation where templates are involved, it would be nice to be
able to make that explicit statement that some type is ref or not at
the point of template instantiation, and the resolution should work
according to the well-defined rules that we are all familiar with.
Within a complex template, if logic should be performed, we have
powerful tools to do that already; nobody ever complains about
Unqual!, or PointerTarget!... these things are very easy to read and
understand. ref should fit right in there with the others, is(x ==
ref), UnRef!T, etc.
Instead, it's practically orthogonal to the language. Wrangling
anything involving storage classes is really, really tedious, and
almost always results in extensive code duplication, or text mixins
when duplication just get too much.

This issue really surprised me at the time, because the whole thing
was in response to a topic that I consistently raised and pushed.
Who was the customer welcoming that solution? I could never work it
out; there was nobody else that seemed particularly invested in the
problem, except the usual suite of language enthusiasts that took it
as an interesting intellectual problem to discuss. The person who
cared about that issue the most (afaict) was completely unsatisfied
with the solution. I said at the time that I would have preferred that
no action was taken, than to compound, and further set in stone, an
issue that I was already extremely critical of.


Don't double down on this mistake with scope!


>> ref is a bad design. C++'s design isn't fantastic, and I appreciate
>> that D made effort to improve on it, but we need to recognise when the
>> experiment was a failure. D's design is terrible; it's basically
>> orthogonal to the rest of the language. It's created way more
>> complicated edge cases for me than C++ references ever have. Anyone
>> who says otherwise obviously hasn't really used it much!
>> Don't double down on that mistake with scope.
>
>
> Why does your code need to care so much about to ref or not to ref? That's
> the central point here, I think.

That same logic could ask why I want to have 'short', or 'int' in some
places, but not others. There's no absolute answer. As a software
engineer, and particularly, a native software engineer, it's a
fundamental part of the type system (sorry, not part of the type
system!) that I must have control over.

Why offer 'ref' if you don't intend people to use it?
I put to you, why do you hate 'ref' so much? Remove it from the
language if people aren't meant to use it.

ref is just a pointer with some semantics removed (pointer
re-assignment, indexing). It's use facilitates some forms of generic
code which would fail otherwise, and also reduces some possibilities
for end-user misuse or mistakes when using pointers (ie, user may
intend an assignment, but unsuspectingly re-assign a pointer; result
looks the same, but exposes bug somewhere else).

It's also used as an optimisation where I require control over the
ref-ness of things, but don't want to retrofit the code with a sea of
'*' and '&' operators.

It's also used when interfacing C++, where it appears in API's extensively.

I'm sure there are many more common and useful cases where ref is a good tool.


>> My intended reply to this post was to ask you to justify making it a
>> storage class, and why the design fails as a type constructor?
>> Can we explore that direction to it's point of failure?
>>
>> As a storage class, it runs the risk of doubling out existing bloat
>> caused by ref. As a type constructor, I see no disadvantages.
>> It even addresses some of the awkward problems right at the face of
>> storage classes, like how/where the attributes actually apply. Type
>> constructors use parens; ie, const(T), and scope(T) would make that
>> matter a lot  more clear.
>>
>> Apart from the storage class issue, it looks okay, but it gives me the
>> feeling that it kinda stops short.
>> Marc's proposal addressed more issues. I feel this proposal will
>> result in more edge cases than Marc's proposal.
>> The major edge case that I imagine is that since scope return values
>> can't be assigned to scope local's, that will result in some awkward
>> meta, requiring yet more special cases. I think Mark's proposal may be
>> a lot more relaxed in that way.
>
>
> The disadvantages of making it a type qualifier are:
>
> 1. far more complexity. Type constructors interact with everything, often in
> unanticipated ways. We spent *years* working out issues with the 'const'
> type qualifier, and are still doing so. Kenji just fixed another one.

What you describe is is the opposite of 'complexity'. Uniform
behaviour is what people expect from a language.
I understand complexity may arise as a result of interaction with
other features, but that's a worthwhile issue to explore if you ask
me. That's complexity that can actually be addressed, rather than
pushed to the side.
Edge cases which are orthogonal to the rest of the language (ie, ref),
are a terrible idea. You don't buffer against complexity by creating a
whole new class of complexity, which we have no effective tools to
manage or mitigate.


> 2. we are never going to get users to use 'scope' qualifiers pervasively.
> It's been a long struggle to get 'const' used.

I don't think scope will be as important as const. I think we should
*try it*, to experimentally see how it plays out.
That said, scope should be able to be inferred quite effectively in
many most cases. Again, I think we will only be able to measure the
effectiveness of this when we try it.


> 3. we added 'inout' as a type qualifier to avoid code duplication engendered
> by 'const'. It hurts my brain to even think about how that might interact
> with 'scope' qualifiers.

inout was an interesting idea (with a terrible name!). I'm still not
sure if I think it was a good idea or not, but I have found it very
useful.

What's the issue? I don't quite see how inout and scope overlap (no
differently than const or immutable?).


> Yes, I agree unequivocably, that 'scope' as a type qualifier is more
> expressive and more powerful than as a storage class. Multiple inheritance
> is also more expressive and more powerful than single inheritance. But many
> times, more power perhaps isn't better than redoing the program design to
> use something simpler and less complex.

I don't think that's a reasonable comparison. Multiple inheritance is
(I think quite well agreed) a wildly unpopular and super-complex
disaster.

Comparing scope-as-a-storage-class or scope-as-a-type-constructor is
nothing like comparing multiple inheritence and interfaces...
interfaces aren't orthogonal to the language for a start! And they're
well understood, and precedented in other languages.

Don't introduce red herrings which incite unrelated, yet strong
emotional response.

My argument is that scope as storage class is MORE COMPLEX in that it
is orthogonal to the language, than scope as type constructor. I think
users will also find it extremely unintuitive, when they realise that
all the usual tools for interacting with types in the language are
unavailable in this special case.
I think scope will also prove to be more popular than ref, so, while
you don't hear so much about how much of a disaster ref is, you'll
start to hear all the exact same problems arise when people are trying
to use scope, because it's a far more interesting type qualifier (and
should probably be the default).

...I can't wait for 'auto scope'! ;)