Streaming library

[Daniel Gibson]

Andrei Alexandrescu schrieb:
> On 10/11/2010 07:49 PM, Daniel Gibson wrote:
>> Andrei Alexandrescu schrieb:
>>> Agreed. Maybe this is a good time to sart making a requirements list
>>> for streams. What are the essential features/feature groups?
>>>
>>> Andrei
>>
>> Maybe something like the following (I hope it's not too extensive):
>>
>> * Input- Output- and InputAndOutput- Streams
>> - having InputStream and OutputStream as an interface like in the old
>> design may be a good idea
>> - implementing the standard operations that are mostly independent from
>> the data source/sink
>> like read/write for basic types, strings, ... in mixin templates is
>> probably elegant to create
>> streams that are both Input and Output (one mixin that implements most
>> of InputStream and
>> one that implements most of OutputStream)
> 
> So far so good. I will point out, however, that the classic read/write 
> routines are not all that good. For example if you want to implement a 
> line-buffered stream on top of a block-buffered stream you'll be forced 
> to write inefficient code.

So what's a possible alternative to the classic read/write routines?

> 
> Also, a requirement that I think is essential is separation between 
> formatting and transport. std.stream does not have that. At the top 
> level there are two types of transport: text and binary. On top of that 
> lie various formatters.
> 

Ok, one should differ between text and binary streams. I was mostly focused on binary streams 
(because that's what I use).
So there might be a hierarchy like
* Input/Output-Stream (interface for all those read/write operations)
   - BinaryStream // abstract class implementing writeInt() etc using write(void* buf, size_t len)
     * BinarySocketStream
     * BinaryFileStream
     * ...
   - TextStream // abstract class implementing writeInt() etc using something like to!string and 
write(char[])
     * TextFileStream
     * ...

(This for both Input- and Outputstreams)

>> * Two kinds of streams:
>> 1. basic streams: reading/writing from/to:
>> * network (socket)
>> * files
>> * just memory (currently MemoryStream)
>> * Arrays/Ranges?
>> * ...
>> 2. streams wrapping other streams:
>> * for buffering - buffer input/output/both
>> - with the possibility to peek?
>> * to modify data when it's read/written (e.g. change endianess -
>> important for networking!)
>> * custom streams.. e.g. could parse/create CSV (comma seperated values)
>> data or similar
> 
> Would these be streams be different in their interface?

No. I just wanted to point out that it must be possible (and should be easy) to wrap streams.

> 
>> * Also there are different types of streams: seekable, resettable (a
>> network stream is neither), ...
> 
> Agreed. Question: is there a file system that offers resettable but not 
> seekable files? I'm thinking of collapsing the two together.

As mentioned before in other branches of this thread: Probably no file system, but maybe archive 
files (zip, ...)

> 
>> * functionality/methods needed/desirable:
>> - low level access
>> * void read(void *buf, size_t len) // read *exactly* len bytes into buf
>> * void write(void *buf, size_t len) // write *exactly* len bytes from
>> buf to stream
>> - convenient methods to read/write basic types in binary (!) from/to 
>> stream
> 
> Again, binary vs. text is a capability of the stream. For example, a tty 
> can never transport binary data - programs like gzip refuse to write 
> binary data to a terminal. (Then of course a binary stream can always 
> accommodate text data.)
> 
>> * <type> read<Type>() (like int readInt()) or T read(T)() (like int
>> read!int())
> 
> Templates will be difficult for a class hierarchy.

Ok.
Another issue, as you mentioned line based streams: of course read(T)()/write(T)() would be quite 
messy on them (endless static if(is(T==int)) { ... } else static if(is(T==float)) {...} etc).

(here were a lot of templated methods)
>> * void writeString(char[] str) // same for wchar and dchar
>> - could write str into the stream with its length (as ushort xor uint
>> xor ulong,
>> _not_ size_t!) prepended
>> * char[] readString() // same for wchar and dchar
>> - read length of the string and then the string itself that will be
>> returned
> 
> Many of these capabilities involve template methods. Is a template-based 
> approach preferable to a straight class hierarchy? I tend to think that 
> in the case of streams, classic hierarchies are most adequate.

Ok agreed.
I forgot that templated methods can't be overridden.

As writeArray(T)() etc is hardly possible, at least for strings there should be write(char[]) 
write(dchar[]) and write(wchar[]) (maybe with some const-stuff added). These should just write the 
string to the stream, without its length.
(Analogous for read(...))

> 
>> - all that array/string/low level stuff but reading *at most* len (or
>> array.length) values
>> and returning the amount actually read ( readUpTo() ?)
>> * useful e.g. for parsing http (you don't know how long the header is 
>> etc)
>> * the same for write? don't see much use for that though..
>>
>> - some way to determine whether the stream
>> * is at its definite end (eof on file, socket closed or something like
>> that)
>> * currently empty (for input stream) - just doing a read() would block ?
>>
>> - Output streams need flush()
>> - for Input streams skip(size_t noBytes) or even skip(T)(size_t noElems)
>> may be
>> handy to just throw away data we're not interested in without having it
>> copied around - especially for non-seekable streams (network..)
> 
> OK, that's a good start. Let's toss this back and forth a few times and 
> see what sticks.
> 
> 
> Andrei


Cheers,
- Daniel