DIP69 - Implement scope for escape proof references

[Walter Bright via Digitalmars-d]

On 12/13/2014 4:44 PM, Manu via Digitalmars-d wrote:
> On 13 December 2014 at 15:11, Walter Bright via Digitalmars-d
> <digitalmars-d at puremagic.com> wrote:
>> On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote:
>>>
>>> I did just give some examples, I'll repeat; auto ref fails when the
>>> function is extern.
>>
>>
>> Don't make it extern, then.
>
> Do you think I just interact with other languages for fun or something?

I'm not trying to insult you - but why need 'auto ref' for extern functions? The 
extern functions are ref or not-ref, and declare the interface to match. There's 
no point to auto ref there.


>> If there's source to the function, it'll often be inlined which will remove
>> the indirection.
>
> Not if it's extern (applicable to ref, not auto-ref), or wrapped, or
> if I ever want to capture a function pointer.
> It's also semantically different; I can change the caller's value.

Again, if it is extern it has a fixed definition of ref or not. The D side 
doesn't decide if it's ref or not, it just follows whatever the extern 
declaration is.

This is why I do not understand why you are running into this issue everywhere.


>>> What do you get when you take a pointer of a function with an auto-ref
>>> arg? ...an error, because it's not actually a function!
>>> So in a context where I'm dealing with functions and function pointers
>>> (very, very frequent), I suddenly have something that's not a function
>>> find it's way into my meta and I have to special-case the hell out of
>>> it.
>>
>>
>> Why are function pointers and ints going to the same argument of a function?
>> I thought you weren't using templates?
>
> I'm confused. I'm talking about capturing function pointers. Not about
> function arguments.
> Capturing pointers of functions with auto-ref args (aka, template
> functions) is a serious nuisance, and impossible outside the site of
> instantiation.

WHY are you doing this? I have no idea what coding pattern would need to pass 
functions by reference.


>> I wonder what is the need for the code that you are writing.
>
> General function adaptation. There are lots of reasons to wrap
> functions. Adaptation to existing or external API's is the most common
> in my experience.

Are those API's all templated? My guess is no. If they're not, you don't need 
templates or auto-ref to interface to them.


> Functions are what programs are! I really struggle to understand why
> you have such trouble sympathising with my view here.

Because you don't provide any examples of why you need this stuff. You say "I 
need this feature because I'm doing X", but I have no idea what X is or why you 
need X.

I.e. you present and advocate for particular solutions, but not the problems.


> Languages generate code, which is packaged into blocks we call 'functions'...
> and they require to adhere to strict ABI requirements. That is
> programming in a nutshell. Surely it's reasonable to want to retain
> complete control over this most primitive and basic of tasks.
> One important aspect of that control is where the code is generated;
> is it 'here', or 'there'?

Why is that important?


> And that's a critical distinction between functions and templates.

You can control that with function templates, too. The compiler won't 
instantiate a template locally if the import instantiates it.


> Look at LuaD. Lots of awkward cases have emerged there. There are many
> more to come too; the known bug list are mostly nasty issues of this
> nature that I've been putting off.
> I can't offer any insight into my commercial code sadly, and I no
> longer have access to it :/
>
> Situations like this appear frequently:
> https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R178
>
> And something like this tends to appear as one aspect of the solution:
> https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R68
> 'struct Ref(T)' leads to its own problems though, in that it's a
> localised concept. No external code anywhere understands it as a
> 'ref', so if 3rd party code has any special treatment for ref, that
> code now fails.
>
> Then more magic like this:
> https://github.com/JakobOvrum/LuaD/pull/76/files#diff-ec8c532aeca798240de4d70ee639fc16R90
> Since we need to recognise 'ref' and substitute it for our magic
> 'struct Ref(T)'.
>
>
> I've probably spent more hours wrangling D meta of this sort than most
> people. This sort of thing always happens.
> If Ref!T is the tool that resolves the situation, then it's clear
> demonstration that ref should be part of the type.

I'll take a look at your references in a moment.


>>> When I'm writing a function that's not a template, I intend, and
>>> expect, to write a function that's _not a template_.
>>> Templates and functions are different things. I think it's a massive
>>> mistake to have created a way to write a template that looks nothing
>>> like a template.
>>
>>
>> A function template is a function that takes both compile-time args and
>> run-time args. C++ tried to make them completely different animals, when
>> they are not. Don't think "template", think "compile-time argument to a
>> function". I convinced Andrei to never mention "template" in his D book, as
>> it brings forth all kinds of connotations and expectations that impede
>> understanding what D templates actually are. I think the result was
>> successful.
>
> You can spin it however you like, but it's exactly the same thing.
> They are completely different animals. A function template is not a
> function at all until it's instantiated. When it's instantiated, it
> becomes a function, or even, one of a suite of functions. And it's not
> known to the author where the function is.
> (Note: I define 'function' in this context to mean 'some code that is emitted')

That's the C++ view of templates, which is outdated and unnecessary.


> This one:many relationship between definition and code creates some
> sorts of problems that aren't present with functions, which relate 1:1
> with their codegen.
> It is possible to take a pointer to a function. It is not possible to
> take a pointer to a template, unless your code exists at the callsite
> where the parameters for instantiation are known.

It's not possible to take a pointer to a function unless that function exists, 
either.


> This makes certain things more complicated.

I don't see a problem with taking the address of an instantiated template function.


> I reject that 'a template is a function with compile time args'.
> Template functions have rather different characteristics, which result
> in special consideration, and some restrictions on usage.
> It's a cute idea, and something that might sound nice in a book... but
> it's not the reality.

I've found templates to be far more useful when viewed that way.


> 'functions' are a much simpler, more fundamental concept, and also one
> that is easily portable between languages.

It's always going to be true that in order to interface D with X, you'll have to 
use a D subset that X understands. BTW, D can interface to C++ template functions!


> I also don't have any idea why it exists! What's it for?

So you can make a template work with both lvalues and rvalues - and implement 
perfect forwarding.

> You love to make people justify exactly what things are for;

It's not a question of loving it, it's a question of practicality.


> I think I said that clearly: "What do you get when you take a pointer
> of a function with an auto-ref
> arg? ...an error, because it's not actually a function!"

What's baffling to me is why even declare a function pointer parameter as 
auto-ref? I don't use floating point to index strings, either, that doesn't mean 
floating point is a failed concept.


> WRT to mixing ref and value parameters; I'm wrapping/adapting existing
> api's. I have no control over the code/api that exists. If that api
> uses ref, for whatever reason that it makes sense to do so, I need to
> handle the case.
> There's no 'pattern', only a mechanical process.

If the existing code uses ref, use ref on the D side. If it does not, do not use 
ref on the D side. What's the problem?


> That's not a tree, that's a graph.

Sure. Do you want a scope design that precludes graphs? And even with trees, are 
you sure you want a design where nobody but the root can point to anything in 
the tree?

How would you do a symbol table? You couldn't ever return a pointer to the found 
symbol if it was all scoped.


> Tree elements only have one reference; their parent. Management of
> trees is very simple.

Again, what about the contents of the tree?


>> That would fail if scope were transitive.
> I'm not sure I see why...
> The situation is: a whole transitive scope graph is received as scope,
> how does that inhibit my accessing a resource contained in the graph?
> It inhibits my _escaping_ said resource... as it should. Right?

How would you manage a symbol table lookup, and then store a pointer to the 
found symbol in the AST?


> I was once arguing for: int^ rcInt;
> I think I'm going back in that direction. That's what other languages
> do. And scope would create additional opportunity here; it would allow
> implicit casting of T^ -> T*.

scope actually does that. (int^ rcInt; isn't transitive, either.)


> Perhaps scope and RC are different things?
> Solve scope purely for RC, and you've broken scope for other purposes
> that it's useful; that is, inhibiting escaping of data.

RC is the canonical use of scope. The beauty is the compiler does not have to 
recognize RC as special. It becomes just ordinary library code.


> struct Wrap
> {
>    OpaqueThing *ptr;
>
>    this() {}
>    ~this() {}  // <- constructors that twiddle RC effectively.
>    this(this) {}
>
>    this() scope {}
>    ~this() scope {}  // <- Overloadable for 'scope', in which case,
> don't twiddle the RC!
>    this(this) scope {}
>
>    // lots of operations on OpaqueThing wrapped up here...
> }
>
> void f(scope Wrap x) <-- RC twiddling is effectively elided here due
> to constructor overloads
> {
> }

First off, get rid of the scope this overloads. Second, add an 'alias this' to 
implement a default conversion of Wrap to OpaqueThing*. Third, declare f as:

     void f(scope ref OpaqueThing* x);


> scope just needs to say "I will not let this memory escape". Then it
> finally allows us to safely put data on the stack, something we can't
> do in D today. I don't think scope should aim to do anything more than
> that.

That's EXACTLY what this proposal is!


> If scope can be useful to RC, that's great! But from this, it's
> starting to look to me like scope might be partially useful to RC, but
> scope and RC aren't exactly the same thing.

Wheels are cars aren't the same thing, either, but cars need wheels.


> RC seems to require something like head-scope in order to not place
> awkward restrictions on usage of RC objects. That said, full-scope
> isn't without use cases; it allows us to safely use the stack,
> potentially eliminating much GC load.

This proposal IS head-scope.