Second Draft: Coroutines

[Richard (Rikki) Andrew Cattermole]

Before reading all this, I have something I want to make clear about 
coroutines that probably should be said earlier contextually to it.

There will be people who are not happy with our library design and 
implementation. It will NOT matter what choices we will make, we cannot 
make everyone happy if we limit the language feature to one solution.

This group includes me, due to -betterC (and some other misc concerns).

Alternatively, which is what I've gone with, we can just not do that. We 
can make it work for any library. Then people can do their own thing or 
pick someone elses. This is a strength of D not a weakness.

And the best part? It is not more complicated. It is not more work to 
implement, if anything it is a subset of what you would need to have 
instead. Nor does it give a worse user experience. It is a different 
design with better tradeoffs for us, that is all.

On 15/01/2025 10:10 PM, Atila Neves wrote:
>> If you want me to change it, I need a lot more feedback including:
>>
>> - How you are interpreting it
>> - What questions you had after reading it
>> - what did you expect it to contain
> 
> Sure. I think the feedback would be quite long, though.

I cannot see a problem with it and I've given evidence that I have good 
reason to not, so a statement like "I don't understand it" is not 
helpful if the goal is to see changes.

So yes please, give me more information that I can take action on!

It may be a good idea to ask Mike for help, this kind of feedback is 
something he is good at (considering his job).

> I wonder if it would be better to have a coroutine library first; I know 
> that it would be a lot more clumsy to use than it would be with language 
> support. But maybe having a library prove itself useful first would be 
> the way forward.

Been there done that.

https://github.com/Project-Sidero/eventloop/tree/master/source/sidero/eventloop/coroutine

It is absolutely hell to write them by hand.

Ask Adam all about how much hell it was to do in C# before they added 
async/await, yes they wrote the library before the language feature and 
were stuck with some less than desirable choices (at least in terms of D 
they are anyway).

This also happens to be why my library is so much hell to write the 
coroutines for currently, because you are effectively replacing the 
language and that is intentional.

>> Other languages can tie the feature to a specific library, which will 
>> not work for us.
> 
> Why is that?

Well for one thing, I am not putting experimental code into Phobos let 
alone druntime for an event loop. This needs to work outside of it.

I need to be able to modify my existing event loop that is designed for 
coroutines in -betterC, to use it. Otherwise I will not be able to find 
any problems it may have or other tunings that will give a better user 
experience once we turn it on.

Plus I see no reason to start tieing this language feature to a specific 
library. We do not box. But we do use templates. We love templates. We 
love generating types and symbols to do this kind of thing. It is both a 
well loved aspect of D, and a very well understood one. Lean into it, 
not against it.

>> Consider why we cannot: a coroutine language feature is tied to its 
>> representation in library, which is tied to is eventloop, which is 
>> tied to sockets, windowing, pipes, processes, thread pool ext.
> 
> How is this different in other languages?

As far as I'm aware it is not, but you do have to acknowledge it to 
understand the decision on this front.

>> None of which can be in druntime, has to be Phobos.
> 
> Why is that?

It is an absolute massive project.

With a ton of platform and runtime specific things.

Trust me, it does not belong in druntime.

You cannot convince me that it is the right place.

>> Also coroutines are used in both generative and event handling basis, 
>> they are not the same library wise. Tieing it to just one is going to 
>> be hell for someone. Most likely me as I'm responsible for the user 
>> experience.
> 
> Again, how is this different in other languages?

C++ has a massive proposal to handle generative data handling side of 
things. It includes scheduler support, (note that this proposal does not 
need the language to be aware of such things).

I cannot find the paper in question, otherwise I would link it.

In other languages like C# they do not use their support for generating 
data (multiple value returns), the focus is upon event handling.

They are sadly different use cases and are going to result in different 
libraries.

Rust literally ties the language to POSIX specific event loop function 
calls, that end up requiring them to use undocumented API's on Windows 
to make work.

At some point you gotta admit, having the compiler produce a state 
struct with a handle method with everything a library needs to work with 
the language feature looks quite simple in comparison ;)

Building up the state machine and extraction of information such as what 
is returned, how it completes with what types (including exceptions 
ext.) happens in all languages. But they tend to go a step further and 
start messing around with library code, this doesn't, nor would it be to 
our advantage.

>>> Why |@async return| instead of |yield|?
>>
>> Then ``yield`` would be a keyword, which in turn breaks code which is 
>> known to exist.
> 
> C++ got around that with `co_yield`.
> 
>> There is no benefit to doing this. But we _could_ do it.
> 
> Familiarity would be a benefit.

To C++ that has had them for only a couple of years.

 From my perspective, C++ has an ugly solution to the problem, that need 
not exist in terms of syntax.

Now compare it to what I proposed:

- Uses an attribute that exists for the same concept, but in a different 
place in grammar.
- It would still be read in a way that is understood control flow wise, 
even if you did not understand coroutines.
- Does not risk breaking code.

To me this is a much better solution that fits D, rather than blindingly 
copying another language with very different needs in terms of syntax 
than we have.

We don't need to copy C++, nor do we have the same baggage as C++ so our 
choices can be different on this, so why should we?

>> All language attributes are in the grammar, there is nothing special 
>> going on there.
>>
>> https://dlang.org/spec/grammar.html#attributes
> 
> For historical reasons, yes. I'm aware one can't attach an attribute to 
> `return` otherwise right now, but wherever they already work I would 
> argue that `core.attributes` is the way to go.

Which has to be imported.

I argue similarly, but there are target audience and language awareness 
to what I recommend.

``@async`` is special, it is used to trigger a head line language 
feature with a very large target audience. Therefore it goes in language.

All of the library attributes in the DIP that the average developer 
doesn't need to know exists, they are in ``core.attributes``.

Plus, coroutines really need to have support for slicing and dicing at 
the parser level. I worked really hard to make that possible for Walter 
due to his issues with ``opApply``. It took weeks of back and forth with 
Adam, for me to come up with the second draft. Just so I could make it 
easier on Walter, but at the same time prevent any of the very large 
number of issues that have happened to Adam in C# when working with 
teams, they can get nasty.

In a DIP this size, there is a lot of contextual information that 
shouldn't be in it. This is a great example of it.