Second Draft: Coroutines

[Mai Lapyst]

On Thursday, 23 January 2025 at 23:09:42 UTC, Richard (Rikki) 
Andrew Cattermole wrote:
>
> On 24/01/2025 10:17 AM, Sebastiaan Koppe wrote:
>> On Thursday, 23 January 2025 at 20:37:59 UTC, Richard (Rikki) 
>> Andrew Cattermole wrote:
>>>
>>> On 24/01/2025 9:12 AM, Sebastiaan Koppe wrote:
>>>> Upon yielding a coroutine, say a socket read, you'll want to 
>>>> park the coroutine until the socket read has completed. This 
>>>> requires a signal on completion of the async operation to 
>>>> the execution context to resume the coroutine.
>>>
>>> Right, I handle this as part of my scheduler and worker pool.
>>>
>>> The language has no knowledge, nor need to know any of this 
>>> which is why it is not in the DIP.
>> 
>> Without having a notion on how this might work I can't 
>> reasonably comment on this DIP.
>> 
>>> How scheduling works, can only lead to confusion if it is 
>>> described in a language only proposal (I've had Walter attach 
>>> on to such descriptions in the past and was not helpful).
>> 
>> You don't need to describe how scheduling works, just the 
>> mechanism by which a scheduler gets notified when a coroutine 
>> is ready for resumption.
>> 
>> Rust has a Waker, C++ has the await_suspend function, etc.
>
> Are you wanting this snippet?
>
> ```d
> // if any dependents unblock them and schedule their execution.
> void onComplete(GenericCoroutine);
>
> // Depender depends upon dependency, when dependency has value 
> or completes unblock depender.
> // May need to handle dependency for scheduling.
> void seeDependency(GenericCoroutine dependency, 
> GenericCoroutine depender);
>
> // Reschedule coroutine for execution
> void reschedule(GenericCoroutine);
>
> void execute(COState)(GenericCoroutine us, COState* coState) {
>     if (coState.tag >= 0) {
>         coState.execute();
>
>         coState.waitingOnCoroutine.match{
>             (:None) {};
>
>             (GenericCoroutine dependency) {
>                 seeDependency(dependency, us);
>             };
>
>             // Others? Future's ext.
>         };
>     }
>
>     if (coState.tag < 0)
>         onComplete(us);
>     else
>         reschedule(us);
> }
> ```
>
> Where ``COState`` is the generated struct as per Description -> 
> State heading.
>
> Where ``GenericCoroutine`` is the parent struct to ``Future`` 
> as described by the DIP, that is not templated.
>
> Due to this depending on sumtypes I can't put it in as-is.
>
> Every library will do this a bit differently, but it does give 
> the general idea of it. For example you could return the 
> dependency and have it immediately executed rather than let the 
> scheduler handle it.

First off: nice work on the proposal here; I really like it. 
Would love to try it once it's in an beta stage as it's quite 
promising.
As an individual that implemented their own userspace eventloop 
via fibers, I would love to have another utility in my belt to 
use in the implementation.

The only thing I had a hard time figuring out what you ment by 
"If it causes an error, this error is guaranteed to be wrong in a 
multi-threaded application of it.";
What I think is that you mean that any exception created / 
captured by the coroutine is guranteed to be indeed an execption 
and should be threaded as such.
Correct me if I'm wrong.

Another thing is the visibility of the members of the created 
struct; shouldn't some of them be read-only (aka const for anyone 
outside) or completly be private?
Like `tag`: there should be no situation where an outside entitiy 
should control the state of the coroutine, not even in as a part 
of a library or do I miss something?

> Then `yield` would be a keyword, which in turn breaks code 
> which is known to exist.

Which is the same with `await`; I honestly like the way rust 
solved it: any Future (rust's equivalent to a coroutine type), 
has implicitly the `.await` method, so instead of writing `await 
X`, you have `X.await`. This dosn't break exisiting code as 
`.await` is still perfectly fine an method invocation. When we're 
here to reduce breaking code as much as possible, I strongly 
would go with the `.await` way instead of adding a new keyword.

For yield the only thing I can think of is to introduce a way 
like `Fiber.yield`, maybe `Coro.yield` that gets picked up by any 
dlang edition that understands coroutine and gets rewritten into 
a proper yield while older versions would see a reference to an 
function / field, which can be provided to these editions as a 
symbol with `static assert(false, "...")` to inform them about 
the inproper usage; but that would have the same problems as 
there could well be already such a construct... But if we're 
using an attribute, I like the `@yield` from Quirin's post a lot 
more (and `__yield` seems very clumpsy to me).

> Rust has a Waker, ...
> ...
> ```
> coState.waitingOnCoroutine.match{
>     (:None) {};
>     (GenericCoroutine dependency) {
>         seeDependency(dependency, us);
>     };
>     // Others? Future's ext.
> };
> ```

The waker design seems much more flexible than a dependency 
system. For example, with wakers one could implement asyncronous 
IO by using epoll and invoking the waker when there's date 
available. I'm a bit confused on how that would look in your 
proposal. Sure your executor uses a match on a sumtype to 
determine what's it waiting on, but how does one "register" a 
custom dependency type? Granted, the compiler can scan the code 
and pickup any type thats been waited on as a dependency, but how 
does a executor know how to handle it? Currently, the type must 
be known beforehand from the executor, thus meaning that the 
executor and the IO library must be developed as one, instead of 
being two seperate things that only share a common protocol 
between them. And even when having compiler support for sumtypes, 
when the sumtype is dynamically created, there will be times 
where the sumtypes does not contain all types the executor can 
process, ending up with unreachable branches which could lead to 
compiler warnings or even errors that are cryptic.

While I agree that we should have a notion on how coroutines can 
be put to sleep until an certain event took place, I think 
dependencies aren't a great solution to that. As mentioned would 
a waker API be better suited for this task as it lets executor 
and IO be their own thing instead of trying to forcefully combine 
it into one.