std.stream replacement

[H. S. Teoh]

On Fri, Mar 08, 2013 at 09:30:30PM -0500, Jonathan M Davis wrote:
> On Saturday, March 09, 2013 01:59:33 Stewart Gordon wrote:
> > On 07/03/2013 12:07, Steven Schveighoffer wrote:
> > <snip>
> > 
> > > I don't really understand the need to make ranges into streams.
> > 
> > <snip>
> > 
> > Ask Walter - from what I recall it was his idea to have range-based
> > file I/O to replace std.stream.
> > 
> > Thikning about it now, a range-based interface might be good for
> > reading files of certain kinds, but isn't suited to general file
> > I/O.
> 
> In general, ranges should work just fine for I/O as long as they have
> an efficient implementation which underneathbuffers (and preferably
> makes them forward ranges). Aside from how its implemented internally,
> there's no real difference between operating on a range over a file
> and any other range. The trick is making it efficient internally.
> Doing something like reading a character at a time from a file every
> time that popFront is called would be horrible, but with buffering, it
> should be just fine. Now, you're not going to get a random-access
> range that way, but it should work fine as a forward range, and
> std.mmfile will probably give you want you want if an RA range is what
> you really need (and that, we have already).
[...]

I think the new std.stream should have a low-level stream API based on
reading & simultaneously advancing by n bytes. This is still the most
efficient approach for low-level file I/O.

On top of this core, we can provide range-based APIs which are backed by
buffers implemented using the stream API. Conceptually, it could be
something like this:

	module std.stream;

	struct FileStream {
		File _impl;
		...

		// Low-level stream API
		void read(T)(ref T[] buffer, size_t n);
		bool eof();
	}

	struct BufferedStream(T, SrcStream) {
		SrcStream impl;
		T[]    buffer;
		size_t readPos;

		enum BufSize = ...; // some suitable value

		this() {
			buffer.length = BufSize;
		}

		// Range API
		T front() { return buffer[readPos]; }
		bool empty() {
			return impl.eof && readPos >= buffer.length;
		}
		void popFront() {
			if (++readPos >= buffer.length) {
				// Load next chunk of file into buffer
				impl.read(buffer, BufSize);
				readPos = 0;
			}
		}
	}

Suitable adaptor functions/structs/etc. can be used for automatically
converting between streams and range APIs via BufferedStream, etc..

As for making ranges into streams: it could be useful for transparently
substituting, say, a string buffer for file input for generic code that
operates on streams. I'm not sure if ranges are the right thing to use
here, though; if all you have is an input stream, then generic code that
uses BufferedStream on top that would be horribly inefficient. It may
make more sense to require an array.

Another approach could be to extend the idea of a range, to have, for
lack of a better term, a StreamRange or something of the sort, that
provides a read() method (or maybe more suitably named, like
copyFrontN() or something along those lines) that is equivalent to
copying .front and calling popFront n times. But we already have trouble
taming the current variety of ranges, so I'm not sure if this is a good
idea or not.  Jonathan probably will hate the idea of introducing yet
another range type to the mix. :)


T

-- 
"How are you doing?" "Doing what?"