A problem with generators

[bearophile]

D currently has two ways to define a generator, the first is the older one with opApply (And opApplyReverse):
int opApply(int delegate(ref Type [, ...]) dg);


The syntax of opApply is terrible: hard to remember, bug-prone, noisy, intrusive. And this syntax doesn't even gain performance, because dmd is not very good at inlining here.
Several times in the past I have said that a very good syntax for this is the Python one:

def foo():
  yield 5

It's really easy to use and remember, not bug prone. This possible syntax is about 75 times better than the current opApply:

yield(int) foo() {
  yield 5;
}


That's sugar for something like:

struct foo {
    int opApply(int delegate(ref int) dg) {
        int result;
        int r = 5;
        result = dg(r);
        if (result)
            goto END;
      END:
        return result;
    }
}


Python syntax was good enough that C# has copied it.


The other way to create a struct/class generator in D2 is with the Range protocol, with the methods that ask for the next item, etc. I have seen that such generators sometimes are harder to write, because you have to manage the state yourself, but they can be more efficient and they are more powerful.

Both ways are useful, because they are useful for different situations. They are one the inverted-out version of the other.


The implementations of generators in C# and Python (and D, but not Lua) suffer of a problem, that increases the computational complexity of the code when there is a nested generator.

And both Python and C# have found similar solutions that improve both the syntax and the performance (both languages have not yet implemented it).

Python version:
http://python.org/dev/peps/pep-0380/

With that this code:

def foo():
  for x in some_iterable:
    yield x


Becomes:

def foo():
  yield from some_iterable


In C# it can be called "yield foreach":

http://kirillosenkov.blogspot.com/2007/10/yield-foreach.html

http://connect.microsoft.com/VisualStudio/feedback/details/256934/yield-return-to-also-yield-collections

http://blogs.msdn.com/wesdyer/archive/2007/03/23/all-about-iterators.aspx


As D2 will start coming out of its embryo state, it too will probably have to fece this problem with iterators. In the meantime I hope the opApply syntax will be replaced by something better (the yield I've shown).

Bye,
bearophile