Using lazy code to process large files

[Daniel Kozak via Digitalmars-d-learn]

using http://dlang.org/phobos/std_utf.html#byCodeUnit could help

On Wed, Aug 2, 2017 at 2:59 PM, Martin Drašar via Digitalmars-d-learn <
digitalmars-d-learn at puremagic.com> wrote:

> Dne 2.8.2017 v 14:45 Steven Schveighoffer via Digitalmars-d-learn
> napsal(a):
>
> > The problem is that you are 2 ranges deep when you apply splitter. The
> > result of the map is a range of ranges.
> >
> > Then when you apply stringStripleft, you are applying to the map result,
> > not the splitter result.
> >
> > What you need is to bury the action on each string into the map:
> >
> > .map!(a => a.splitter(",").map!(stringStripLeft).join(","))
> >
> > The internal map is because stripLeft doesn't take a range of strings
> > (the result of splitter), it takes a range of dchar (which is each
> > element of splitter). So you use map to apply the function to every
> > element.
> >
> > Disclaimer: I haven't tested to see this works, but I think it should.
> >
> > Note that I have forwarded your call to join, even though this actually
> > is not lazy, it builds a string out of it (and actually probably a
> > dstring). Use joiner to do it truly lazily.
> >
> > I will also note that the result is not going to look like what you
> > think, as outputting a range looks like this: [element, element,
> > element, ...]
> >
> > You could potentially output like this:
> >
> > output.write(result.joiner("\n"));
> >
> > Which I think will work. Again, no testing.
> >
> > I wouldn't expect good performance from this, as there is auto-decoding
> > all over the place.
> >
> > -Steve
>
> Thanks Steven for the explanation. Just to clarify - what would be
> needed to avoid auto-decoding in this case? Process it all as an arrays,
> using byChunk to read it, etc?
>
> @kdevel: Thank you for your solution as well.
>
> Martin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d-learn/attachments/20170802/52124825/attachment.html>