First Draft: Coroutines

[Sebastiaan Koppe]

On Tuesday, 6 August 2024 at 02:20:52 UTC, Richard (Rikki) Andrew 
Cattermole wrote:
> On 06/08/2024 3:50 AM, Sebastiaan Koppe wrote:
>> On Monday, 5 August 2024 at 01:36:31 UTC, Richard (Rikki) 
>> Andrew Cattermole wrote:
>>> Coroutines is a stack machine transformed representation for 
>>> a function. It enables storing the state of a function 
>>> externally to the stack to allow for high throughput event 
>>> handling.
>> 
>> Thank you for spearheading this. Having coroutines in the 
>> language would be a big step forward, and hopefully pave the 
>> way for more non-blocking programming in D.
>> 
>> There are a few points I would like to bring up early though.
>> 
>> - I see no mention of C++'s coroutines. I think it would be 
>> good to learn from their design and implementation.
>>
>> - As you might know I am a big proponent of C++'s 
>> Senders/Receivers (a.k.a. P2300). One awesome integration they 
>> have is that Senders can be awaited by coroutines, and 
>> coroutines can await Senders. This allows for users to pick 
>> their preference, e.g. use convenient coroutines but suffer 
>> some allocation costs, and use Senders for more performant 
>> sections if need be. See
>> https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2300r7.html#design-awaitables-are-senders and https://ericniebler.com/2024/02/04/what-are-senders-good-for-anyway/
>
> My conclusion about P2300 is that it is all library code. This 
> DIP would allow you to implement it.

https://github.com/symmetryinvestments/concurrency already 
implements a fairly large chunk of it.

I suppose what you mean is that this DIP would allow me to 
integrate coroutines with it.

>> - I don't understand the choice for `@async`. Sometimes 
>> resembling the word "coroutine" would be better in my opinion.
>
> It was ``@co`` for a long time.
>
> I had a number of people request ``async`` instead.
>
>> - In `while(Future!string readLine = socket.readUntil("\n"))` 
>> I suspect it to do an explicit await. How does that work?
>
> When a coroutine is acquired, it'll yield automatically, it is 
> implicit.

What do you mean with 'acquired'?

>> - I find the use of `Future` a bit confusing. Futures 
>> generally carry too much synchronisation overhead with them 
>> and I doubt coroutines need a full Future implementation 
>> anyway. Perhaps call it a Task like they do in C++?
>
> This DIP does not propose library code. You are free to write 
> whatever library code you want and have it construct itself 
> from the language representation of a coroutine.

I fail to see what it then _does_ provide.

> This is a key design goal as there is a good chance we'll want 
> different solutions for different tasks.

It ultimately needs to integrate with the language coroutine 
support, so I don't understand how it can be completely separate. 
And if it can't, then the language needs to provide some 
low-level interface to it, which, granted, libraries can extend 
on.

>> - The use of `@isasync` is a bit too cute for me. It also 
>> hides the fact sometime is a coroutine.
>
> I want to be placing a lot more emphasis on UDA's like 
> ``@Route``, what we do right now with reflection is far too 
> costly in terms of how to register symbols.

I think it distracts from the proposal. Likely `@Route` will be 
implemented in a library anyway.

> But yes, I do want to hide that it is a coroutine. The average 
> person shouldn't have to care if its asynchronous of 
> synchronous, it does not enable them to archive business goals 
> faster.

I very much like the low-level/high-level proposed approach to 
Phobos3. I very much do not want the language to hide coroutine 
details, and instead want to be in full control, when and if a 
coroutine is called/yielded to.

>> - Instead of allowing a coroutine in a function that is not a 
>> coroutine itself - and injecting a blocking call - I would 
>> require explicit call to evaluate the coroutine instead. No 
>> magic.
>
> And that is library code ;)
>
> But yes, I did try to explain that you did not need language 
> integration for this. In fact the prime sieve example show 
> cases exactly what you suggest!

I do not understand what it does - and how! It is unclear to me 
how `&generate` is turned into a `InstantiableCoroutine!(int)`, 
how `makeInstance` works (or what it even does) and what 
`ch.block` does.

I can guess of course, but that explanation needs to be part of 
the DIP.