RFC: Last words on Primary Type Syntax aka. `ref` return function pointer type sytnax

[Quirin Schroll]

I have to apologize to Walter in particular for not posting this 
earlier. Since the meeting where we discussed the question how to 
handle ambiguities related to keywords that have a meaning on 
their own (“lone”) and when given arguments in parentheses and 
possible solutions to it.

For those interested, the ambiguity arises from a type like `ref 
int function()` (one that requires parentheses in many contexts) 
when one wants to use it after such a keyword. A motivating 
example:
```d
static int x;
scope (ref int function()) fp = ref () => x;
```
The above is obviously intended to be a declaration of a scope 
variable, but it could also parse as a scope guard, since by 
design, scope guards take a sequence of tokens with balanced 
parentheses as argument to allow for adding new scope guards.

A comprehensive table. If anyone finds another keyword, let me 
know.

| Keyword       | Lone                   | With arguments         
  |
|---------------|------------------------|-------------------------|
| `scope`       | [scope attribute][1]   | [scope guard][2]       
  |
| `align`       | default [alignment][3] | set [alignment][3]     
  |
| `extern`      | [extern attribute][4]  | [linkage attribute][5] 
  |
| `deprecated`  | [warn on use][6]       | [warn w/ msg on 
use][6] |

| Construct        | Technically applicable to | Useful to apply 
to        |
|------------------|---------------------------|---------------------------|
| scope attribute  | any variable              | locals, 
parameters        |
| scope guard      | statement                 | statement        
          |
| alignment        | any symbol but parameters | 
global/member/local vars† |
| extern attribute | any symbol                | globals          
          |
| linkage          | any symbol or FP/DG type‡ | any symbol or 
FP/DG type  |
| deprecated       | any symbol                | any global 
public symbol  |

(† Alignment is not semantically applied to local variables when 
their type is inferred, which is a bug. It also cannot be applied 
to function parameters.)
(‡ *FP/DG type* stands for *function pointer or delegate type.*)

## The solution proposed in the DIP

### Officially

Do nothing for `align`, `extern`, and `deprecated`. They’re 
greedy as of now and stay like that; that is, if an opening 
parenthesis follows, it’s the argument list to the with-arguments 
construct, thus it’s absolutely never the lone construct.

The `scope` keyword means the `scope` attribute unless: In a 
statement block, if `scope` is the first token of a statement, 
it’s a scope guard if and only if:
    * It follows the pattern `scope` `(` *Token* `)`, where 
*Token* is a single token;
    * or it follows the pattern `scope` `(` *Tokens* `)` `{`, 
where *Tokens* is a sequence of tokens with parentheses balanced.

For reference, the condition for `scope` in the current form of 
the language is: “The `scope` keyword means the `scope` attribute 
unless: In a statement block, if `scope` is the first token of a 
statement.”

### Informally

#### `align`, `extern`, and `deprecated`

If you’re in a hurry, use an attribute block if the attribute is 
used outside of a statement scope. It may not be pretty, but it 
gets the job done.

Specific alternatives that should work in statement and 
declaration scope:

**`align`** ― Use `align(default)`. If you care about alignment, 
you know `align(default)` exists.

**`extern`** ― Add the linkage in scope: `extern extern(D)`, 
which is probably even informative to readers. If you need 
`extern` variables, you also deal with linkage, so this isn’t 
news.

**`deprecated`** ― Add a message.

#### `scope`

Among the goals of the DIP is to enable programmers to declare 
`scope` variables of FP/DG type in the initially presented, 
straightforward manner with no strings attached.

Being statements, scope guards are naturally limited to statement 
blocks, so special-casing `scope` everywhere (e.g. global scope, 
function parameter lists) to fix statements in particular doesn’t 
feel right to me. The DIP retains existing scope guards and even 
allows for potential future scope guards of now two distinct 
forms:
* Single-token scope guards work exactly like the existing ones.
* Multi-token scope guards (should any ever be proposed) will 
have to govern a block statement.

 From the sacrifice in the second bullet point, we gain that the 
following works as intended. The comment describes the *current* 
parsing.
```d
scope (ref int function()) fp = …; // ill-formed scope guard
// distinct from
scope ref int function() fp = …; // reference variable declaration
```

Timon Gehr in particular has stated interest in putting 
clarifying parentheses around function pointer and delegate types 
that don’t strictly need the parentheses:
```d
scope (int function()) fp = …; // ill-formed scope guard
// same as
scope int function() fp = …; // variable declaration
```

If the type of the variable being declared is expressed a single 
token, e.g. `int` or `FP`, `scope (int)` and `scope (FP)` parse 
as (ill-formed) scope guards. That means, you actually can’t put 
parentheses around a type and get a type in this specific 
context. It’s the only exception to this core principle, and it 
has to exist to retain existing scope guards. Not many will 
notice, for why would you put parentheses around a single-token 
type anyway? On the other hand, double parentheses are absolutely 
guaranteed to work: With the proposed changes, `scope 
((`*Tokens*`))` will never parse as anything but lone `scope` and 
a type. This isn’t relevant for people writing code directly in a 
text editor, but it is relevant for code generation via string 
mixins. The need for generating a `scope` variable can come up 
where the type of that variable cannot be inferred and might be 
single-token or require parentheses. In that case, `"scope ((" ~ 
typeAsString ~ ")) varName"` works for both `typeAsString  = 
"int"` and `typeAsString = "ref int function()"`.

Of course, another way to avoid introducing a scope guard is 
ensuring `scope` isn’t the first token. If other storage classes 
apply, or could be applied without harm, they can be prepended. 
That isn’t possible or pretty in common cases.

## What Walter takes issue with

He didn’t like that the four constructs in question needed *all* 
different approaches. This might have been due to my lack of 
English speaking skills (I’m German) and a good bit of 
nervousness, because it’s not exactly true.

He’s right insofar as it would be bad *if* the four constructs 
actually needed lots of different approaches. The details before 
should have made clear that a one-size-fits-all approach works to 
a large degree: An attribute block solves 3 out of 7 cases: lone 
`align`, `extern`, and `deprecated` at declaration scope. The 
remaining 4 cases are lone `scope`, `align`, `extern`, and 
`deprecated` at statement scope.

If you only consider what’s meaningful, that removes two cases: 
`extern` and `deprecated` at statement scope, i.e. we’re at 3 out 
of 5.

The approach of the DIP for `scope` in statement scope is to make 
it work if you do what’s intuitively clear. Since `scope` is also 
a function parameter attribute, there’s an argument to be had 
that `scope (ref int function())` and `scope (int 
function())`―which are both valid as parameter 
declarations―should both be valid as local variable declarations. 
Of course, that cannot be guaranteed.

The only somewhat sad case is lone `align` in statement scope. 
Since this only affects code that’s yet to be written, it’s not 
clear if lone `align` will be used much, given how much clearer 
`align(default)` is. There’s a good chance lone `align` will be 
deprecated, even.

## What Walter proposed in the meeting

Walter’s idea was to handle those ambiguities by the following 
rule: If what follows one of the four keywords inside parentheses 
pareses as a type and that type *requires* parentheses, then it’s 
the lone construct; otherwise parse as the with-arguments 
construct.

### The upside

It’s a uniform approach and it works. Should I have missed a 
keyword, the likelihood it works for it as well is near 100%.

### The downsides

In short:
1. In some contexts, no type requires parentheses.
2. There are types that require parentheses in some context, but 
not in another context, but some other type does require 
parentheses there.
3. Even from a technical standpoint, determining if a type 
requires parentheses in a specific context isn’t difficult as of 
now.
4. That might be subject to change, though.
5. What types require parentheses isn’t necessarily clear to a 
programmer. Even when it is, one might *want* to use parentheses 
even if they’re not necessary (cf. T. Gehr) for one’s own clarity 
or style.

Details:
1. Those contexts are where nothing but a type is expected (e.g. 
the argument to `cast` or `typeid`) or where parsing a type is 
prioritized (e.g. a template argument). Those are of little 
concern.
2. Those types are FP/DG types with linkage, e.g. `extern(C) int 
function()`. They don’t need parentheses in parameter lists and 
similar contexts, but require parentheses when used e.g. as 
function return types or variable types. See 
[DIP1049 § Linkage][7] for details.
3. The DMD frontend functionality for determining if a type needs 
parentheses would be completely new. It goes somewhat against the 
quest to keep the compiler simple. With this DIP alone, it’s a 
simple check if the type starts with `extern` or `ref`.
4. This is related in particular to tuple DIPs that use 
parentheses to denote tuples. Those make `(T, U)` a type that 
requires parentheses, where a lookahead of arbitrary length is 
needed to find the comma.
5. See below.

If superfluous parentheses proliferate in some styles in some 
contexts, programmers could assume they’re required from 
experience. An example for an assumption of that kind that I had 
for ~10 years: D didn’t require `auto` and `ref` to be adjacent 
until last 2.110, but I thought it did, and it actually did in 
one context. Linters might suggest them where they change 
behavior because they interact with the token preceding them. 
That is, it complicates writing a good linter. The DIP mentions 
`(const int)` as an example of what’s newly allowed, and that of 
course extends to `(const Ptr)` which, if `Ptr` is an alias to a 
type with indirections (pointer, slice, function pointer, 
delegate, class, …), makes sense to use with `scope`. Thus, 
`scope (const Ptr)` might be the preferred style of some 
programmer or linter, but since `const Ptr` doesn’t require 
parentheses, it would render the construct an ill-formed scope 
guard. Of course one *can* use `scope const(Ptr)` or `scope const 
Ptr` or even `const scope Ptr`, but the whole point of this is 
that one might not want to. This is similar to east-const vs. 
west-const in C++, where if you prefer east-const, it’s annoying 
that it’s not available for pointer types:
  * `int const*` west-const available;
  * `const int*` east-const available;
  * `int* const` west-const required, thus
  * there’s no way to express the latter using east-const style.

A more relevant example would be an array or slice of function 
pointers. There’s a stylistic urge to write `(void function())[]` 
instead of `void function()[]`, but the proposed parsing rule 
locks one in: With `scope`, it’s `scope void function()[]` 
without `ref` and `scope (ref void function())[]` or `scope (ref 
void function()[])`, but not `scope ((ref void function())[])` 
with `ref`. The last one is excluded because, given the inner 
parentheses, the outer ones aren’t required anymore.

## Last notes

I implemented the DIP as submitted with one exception that’s 
noted in the DIP: Explicit linkage not supported for template 
lambda, except for alias. Anyone can play around with the 
implementation. I haven’t had time to update it to incorporate 
the latest changes. There are probably a few bugs.

I have made no attempt to implement Walter’s suggestion. 
Something like that requires time I don’t have right now.

---

It’s a long post, there are probably errors and things I could’ve 
explained or phrased better. Sorry in advance. Any sort of 
comment is welcome.

[1]: https://dlang.org/spec/attribute.html#scope
[2]: https://dlang.org/spec/statement.html#scope-guard-statement
[3]: https://dlang.org/spec/attribute.html#align
[4]: https://dlang.org/spec/declaration.html#extern
[5]: https://dlang.org/spec/attribute.html#LinkageAttribute
[6]: https://dlang.org/spec/attribute.html#deprecated
[7]: 
https://github.com/dlang/DIPs/blob/master/DIPs/DIP1049.md#linkage