Second Draft: Coroutines

[Mai Lapyst]

On Friday, 24 January 2025 at 06:16:27 UTC, Richard (Rikki) 
Andrew Cattermole wrote:
> "If the compiler generates an error that a normal function 
> would not have, the error is guaranteed to not be a false 
> positive when considering a multithreaded context of a 
> coroutine."

With error you mean an exception? As there are compiler errors 
(as in the compiler refuses to compile something), and execptions 
(i.e. `throw X`). Just makeing sure we're on the same page. If 
so, then I get what you are meaning and should ofc be the case, 
as is not really different as non-multithreaded non-coroutine 
code: any exception thrown shouldn't be a false-positive as long 
as the logic guarding it is not flawed in any form.

>> Like `tag`: there should be no situation where an outside 
>> entitiy should control the state of the coroutine, not even in 
>> as a part of a library or do I miss something?
>
> You may wish to complete a coroutine early.
>
> Nothing bad should happen if you do this.
>
> If it does, that is likely a compiler bug, or the user did 
> something nasty.

Hmmm, thats indeed a reason for changing `tag`; you wouldn't need 
a cancelation token as the tag is this cancelation token to some 
extend. On that note, we could add a third negative value to 
indicate an coroutine was canceled from an external source or one 
could generally specify that any negative value means canceled 
and libraries can "encode" their own errorcodes into this...

>>> Then `yield` would be a keyword, which in turn breaks code 
>>> which is known to exist.
>> 
>> Which is the same with `await`; I honestly like the way rust 
>> solved it: any Future (rust's equivalent to a coroutine type), 
>> has implicitly the `.await` method, so instead of writing 
>> `await X`, you have `X.await`. This dosn't break exisiting 
>> code as `.await` is still perfectly fine an method invocation. 
>> When we're here to reduce breaking code as much as possible, I 
>> strongly would go with the `.await` way instead of adding a 
>> new keyword.
>
> I don't expect code breakage.
>
> Its a new declaration so I'd be calling for this to only be 
> available in a new edition.

Sadly it will; take for example my own little attempt to build a 
somewhat async framework ontop of fibers: 
https://github.com/Bithero-Agency/ninox.d-async/blob/f5e94af440d09df33f1d0f19557628735b04cf43/source/ninox/async/futures.d#L42-L44 it declares a function `await` for futures; if `await` will become a general keyword, it will have the same problems as if `yield` becomes one: all places where `await` was an identifier before become invalid.

> Worse case scenario we simply won't parse it in a function that 
> isn't a coroutine.

Which could be done also with `yield` tbh. I dont see why `await` 
is allowed to break code and `yield` is not. We could easily make 
both only available in coroutines / `@async` functions.

> I am struggling to see how the waker/poll API from Rust is not 
> a more complicated mechanism for describing a dependency for 
> when to continue.

It's easier, as it describes how an coroutine should be woken up 
by the executor, a dependency system is IMO more complicated 
because you need to differentiate between dependencies whereas 
Wakers serve only one purpose: wakeup a coroutine / Future that 
was pending before to be re-polled / executed.

---

I've read a second time through your DIP and also took a look at 
your implementation and have some more questions:

> opConstructCo

You use this in the DIP to showcase how an coroutine would be 
created, but it's left unclear if this is part of the DIP or not. 
Which is weird because without it the translation
```d
ListenSocket ls = ListenSocket.create((Socket socket) {
	...
});
```
to
```d
ListenSocket ls = ListenSocket.create(
	InstantiableCoroutine!(__generatedName.ReturnType, 
__generatedName.Parameters)
		.opConstructCo!__generatedName);
);
```
would not be possible as the compiler would not know that 
opConstructCo should be invoked here.

Which also has another problem: how do one differentiate between 
asyncronous closures and non-asyncronous closures? Because you 
clearly intend here to use the closure passed to 
`ListenSocket.create` as an coroutine, but it lacks any indicator 
that it is one. Imho it should be written like this:
```d
ListenSocket ls = ListenSocket.create((Socket socket) @async {
	...
});
```

> GenericCoroutine

Whats this type anyway? I understand that `COState` is the state 
of the coroutine, aka the `__generatedName` struct which is 
passed in as a generic parameter and I think the 
`execute(COState)(...)` function is ment to be called through a 
type erased version of it that is somehow generated from each 
COState encountered. But what is `GenericCoroutine` itself? Is it 
your "Task" object that holds not only the state but also the 
type erased version of the execute function for the executor?

> Function calls

I also find no information in the DIP on how function calls 
itself are transformed. What the transformation of a function 
looks like is clear, but what about calling them in a non-async 
function? I would argue that this should be possible and have an 
type that reflects that they're a coroutine as well as the 
returntype, similar to rust's `Future<T>`. This would also proof 
that coroutines are zero-overhead, which I would really like them 
to be in D.

> ```d
> struct AnotherCo {
>     int result() @safe @waitrequired {
>         return 2;
>     }
> }
> 
> int myCo() @async {
>     AnotherCo co = ...;
>     // await co;
>     int v = co.result;
>     return 0;
> }
> ```

How is `AnotherCo` here a coroutine that can be `await`ed on? 
With my current understanding of your proposal, only functions 
and methods are transformed, which means that `AnotherCo.result` 
would be the coroutine, not it's whole parent struct.