Any guide to stream (string) processing in D? | Re: Any python-like generator patterns in D?

[Samuel Lampa]

On 02/21/2013 04:29 PM, bearophile wrote:
> Samuel Lampa:
>
>> What I still remain looking for is a concise guide on how to do 
>> stream (string) processing in D, in a (hopefully) as simple and 
>> elegant way as possible? [2] Any tips?
>
> Are you willing to explain what you mean, or to show Python links/code?

Sure. Please find some sample python code, where I create a pipeline of 
generator objects that when executed, loads only one item at a time (one 
line in this case). The last expression ("for line ... print ...") will 
drive the execution of the whole chain of generators, yielding another 
line through the chain for every line that is iterated over in the for 
loop, keeping memory usage to a minimum.

# ---- PYTHON CODE STARTS HERE ----

# Define some functions that return generator objects

def generate_lines(filename):
     for line in open(filename):
         yield line.rstrip("\n")

def generate_uppercase_lines(input_gen_obj):
     for line in input_gen_obj:
         yield line.upper()

def generate_lines_for_output(input_gen_obj):
     for line in input_gen_obj:
         yield "Line: " + line


# Chain together the generator object to produce a "pipeline":

gen_lines = generate_lines("infile.txt")
gen_uppercase_lines = generate_uppercase_lines(gen_lines)
gen_lines_for_output = generate_lines_for_output(gen_uppercase_lines)


# Do something with the last generator object in the pipeline,
# in order to drive the whole pipeline, one line at a time.

for line in gen_lines_for_output:
     print line

# ---- PYTHON CODE ENDS HERE ----


Also, python has an even more compact syntax for creating generator objects:

[new generator obj] = ([function to do something]([item]) for [item] in 
[other generator obj])

Writing the above code with this syntax would be something like this:


# ---- PYTHON CODE STARTS HERE ----

# Simultaneously create the generator objects, and the pipeline
gen_lines = (line.rstrip("\n") for line in open("infile.txt"))
gen_uppercase_lines = (line.upper() for line in gen_lines)
gen_final_lines = ("Line: " + line for line in gen_lines)

# Drive the pipeline, one line at a time
for line in gen_final_lines:
     print line

# ---- PYTHON CODE ENDS HERE ----

Hope this clarifies.

Otherwise, the main go-to tutorial on python generators is the slides by 
David Beazley:

http://www.dabeaz.com/generators/
http://www.dabeaz.com/generators/Generators.pdf

Cheers
// Samuel