stream interfaces - with ranges

[Mehrdad]

On Thursday, 17 May 2012 at 14:02:09 UTC, Steven Schveighoffer 
wrote:
> 2. I realized, buffering input stream of type T is actually an 
> input range of type T[].

The trouble is, why a slice? Why not an std.array.Array? Why not 
some other data source?
(Check/egg problem....)




Another problem I've noticed is the following:


Say you're tokenizing some input range, and it happens to just be 
a huge, gigantic string.

It *should* be possible to turn it into tokens with slices 
referring to the ORIGINAL string, which is VERY efficient because 
it doesn't require *any* heap allocations whatsoever. (You just 
tokenize with opApply() as you go, without every requiring a heap 
allocation...)

However, this is *only* possible if you don't use the concept of 
an input range!

Since you can't slice an input range, you'd be forced to use the 
front() and popFront() properties. But, as soon as you do that, 
you're gonna have to store the data somewhere... so your 
next-best option is to append it to some new gigantic array 
(instead of a bunch of small arrays, which require a lot of heap 
allocations), but even then, it's not as efficient as possible, 
because there's O(n) extra memory involved -- which defeats the 
whole purpose of working on small chunks at a time with no heap 
allocations.
(If you're going to do that, after all, you might as well read 
the entire thing into a giant string at the beginning, and work 
with an array anyway, discarding the whole idea of a range while 
doing your tokenization.)


Any ideas on how to solve this problem?