std.stream replacement

[Stewart Gordon]

On 10/03/2013 15:48, Marco Leise wrote:
<snip>
> For most binary formats you need to deal with endianness for
> short/int/long

Endian conversion is really part of decoding the data, rather than of 
reading the file.  As such, it should be a layer over the raw file I/O 
API/implementation.

And probably as often as not, you want to read in or write out a struct 
that includes some multi-byte numerical values, e.g. an image file 
header which has width, height, colour type, bit depth, possibly a few 
other parameters such as compression or interlacing, and not all of 
which will be integers of the same size.  ISTM the most efficient way to 
do this is to read the block of bytes from the file, and then do the 
byte-order conversions in the file-format-specific code.

> and blocks of either fixed size or with two versions (e.g. a revised
> extended bitmap header)or alltogether dynamic size.

Yes, that's exactly why we have in std.stream a method that reads a 
number of bytes specified at runtime, and why it is a fundamental part 
of any stream API that is designed to work on binary files.

> Some formats may also reading the
> last bytes first, like ID3 tags in MP3s.

Do you mean ID3 data is stored backwards in MP3 files?  Still, that's 
half the reason that file streams tend to be seekable.

> And then there are compressed formats with data types of < 8 bits or
> dynamic bit allocations.

But:
- it's a very specialised application
- I would expect most compressed file formats to still have byte-level 
structure
- implementing this would be complicated given bit-order considerations 
and the way that the OS (and possibly even the hardware) manipulates files

As such, this should be implemented as a layer over the raw stream API.

> It's all obvious, but I had a feeling your use cases are too
> restricted.
<snip>

The cases I've covered are the cases that seem to me to be what should 
be covered by a general-purpose stream API.

Stewart.