Request for Comment assert(__ctfe)

[Petar]

On Tuesday, 7 April 2020 at 17:33:03 UTC, Johannes Pfau wrote:
>>> On Sunday, 5 April 2020 at 18:25:10 UTC, Petar Kirov 
>>> [ZombineDev] wrote:
>>>>
>>>> Another, more radical and interesting option is to not 
>>>> perform codegen unless it's actually necessary. Instead of 
>>>> blacklisting `@ctfe` or `assert(__ctfe)` functions, have a 
>>>> whitelist of what functions are allowed to codegen-ed.
>>>>
>>>> [...]
>
> How would such a lazy compilation model work without explicit 
> 'export' animations?

In my post [1] I said that the `export` storage class is the key 
to making this work. The way I envision this is similar to a 
copying garbage collector:
1. [MARK] Starting from the set of ~GC~ CG roots [2] the compiler 
traverses the static call graph [3] and marks any callees as 
`export` (i.e. the `export` inference step in my proposal). All 
functions determined to have the `export` attribute are added to 
the code generation queue
2. [COPY] Do code generation by poping function from the queue 
until it becomes empty.
3. (Optional) produce a report of all functions that were not 
added to the queue.

[1]: 
https://forum.dlang.org/post/xbllqrpvflazfpowizwj@forum.dlang.org
[2]: CG (call graph) roots: all functions marked explicitly as 
`export`, or implicitly under rules B) - F) in my post [1] (e.g. 
`main()`, `unittest`s, virtual functions, etc.)
[3]: After statements inserted from `static`, `version` and 
`mixin` declarations are resolved, the set of all functions that 
may be called from a given function. Even if a function is called 
conditionally at runtime, it's still part of the static call 
graph.

> If I wanted to build a proprietary library libfoo.so and 
> provide only .di files with stripped function bodies, all 
> functions have to be compiled into the .so file. But how does 
> the compiler know the result is going to be a shared library in 
> a separate compilation model, where the compiler produces only 
> an object file?

To produce the libfoo.{so,dll,dylib}, I assume that you have a 
whole package of D modules that you want to compile.

---

(The case where libfoo is produced from a single D module is the 
trivial base case: only functions determined to be `export` will 
be part of the object file, from which libfoo will be produced.)

Also, as in C++ (AFAIU), templates can't be part of your dynamic 
library ABI, only specific template instances can be. If type 
templates participate in the function signature (the parameter 
types, or return type) of an `export`-ed function, then those 
type templates are inferred to be `export` as well (and all of 
their members). I suspect we can do better here, but it's easier 
to start pessimistically to relax the rules later when we have a 
better formal model an empirical experience.

---

So, finally, how would we produce a dynamic library?

1. In our implementation files (*.d) we would mark as `export` 
all public functions that we want to be part of our dynamic 
library API/ABI.

2. When the compiler produces *.di header files it would include 
the declarations of only public, protected or package symbols 
that are determined to be export.

3. When the object files are produced all `export` functions will 
be compiled (including private ones, i.e. private symbols, called 
from public exported ones).

4. The rest of the symbols that are not determined to be `export` 
are discarded.

---

All in all, what we're left with is:
1. A strict compilation mode in which only symbols determined to 
be `export` are included in libraries. This way we have a much 
more precise definition of ABI.
2. A lazy compilation mode, based on (dmd -i) in which the 
compiler automatically imports and compiles functions as needed, 
starting from `main()`. In a package-wise compilation (not 
separate compilation), regular free functions are treated like 
template functions - they're not compiled unless used.

Actually 1. and 2. are exactly the same compilation strategy, 
just very different use cases ;)

> I guess such a lazy compilation model might be quite 
> interesting, but as it is a quite radical idea, it needs lots 
> of thinking, proper specification and testing first. (Testing 
> as you'll probably have to emit everything as weak symbols 
> then, and you probably want to test how linkers deal with that 
> if you make excessive use of it)

I don't think we should rely that much on linkers for object 
file-level GC. The compiler shouldn't produce functions that are 
not determined to be `export` in the first place.


Some examples:

--------------------
module app;

string generateMixin(T)(string b)
{
     assert(__ctfe);
     return T.stringof ~ " " ~ b ~ ";";
}

void main() @nogc
{
     mixin(generateMixin!string("b"));
}
--------------------

`dmd -c app.d` should produce an object file with a single 
(empty) function - `main()`;

---------------------------
module foo;

private void privateFunc() {}

public void publicTemplate(T)()
{
     privateFunc();
}
---------------------------

Q: How would this work?
A: Uner my proposal, the compiler should produce an **empty** 
object file.

What could work is the following:

---------------------------
module foo;

private void privateFunc() {}

public void publicTemplate(T)()
{
     privateFunc();
}

// the obvious way:
export void publicFunc() { publicTemplate!int(); }

// the syntax sugar way 1a:
export alias publicFunc = publicTemplate!int;

// the syntax sugar way 1b:
export
{
     alias publicFunc = publicTemplate!int;
     alias publicFunc = publicTemplate!string;
}

// the syntax sugar way 2:
export
{
     publicTemplate!int;
     publicTemplate!string;
}

// Note: the syntax sugar is not part of my proposal,
// as it's not important for now.
---------------------------

Or:

---------------------------
module foo;

private void privateFunc() {}

public void publicTemplate(T)()
{
     privateFunc();
}

----
module bar;

import foo;

export void actuallyPublicInterfaceFunction()
{
     publicTemplate!int();
}

---

dmd -i bar.d

---------------------------

In this way we tell the compiler to compile the exported function 
in `bar` as well as any other functions that they may call.