Streaming library

[Andrei Alexandrescu]

On 10/14/10 21:22 CDT, Rainer Deyke wrote:
> On 10/14/2010 15:49, Andrei Alexandrescu wrote:
>> Good point. Perhaps indeed it's best to only deal with bytes and
>> characters at transport level.
>
> Make that just bytes.
>
> Characters data must be encoded into bytes before it is written and
> decoded before it is read.  The low-level OS functions only deal with
> bytes, not characters.

I'm not so sure about that. For example, some code in std.stdio is 
dedicated to supporting fwide():

http://www.opengroup.org/onlinepubs/000095399/functions/fwide.html

As far as I understand, a wide stream is essentially an UCS-2 (or 
UTF-16? Not sure) stream that is impossible to abstract away as a stream 
of bytes.

I see Windows' commitment to fwide is... odd:

http://msdn.microsoft.com/en-us/library/aa985619%28VS.80%29.aspx

The ultimate question is whether we want to support that (as well as 
other dedicated text streams) or not.

> Text encoding is a complicated process - consider different unicode
> encodings, different non-unicode encodings, byte order markers, and
> Windows versus Unix line endings.  Furthermore, it is often useful to
> wedge an additional translation layer between the low-level (binary)
> stream and the high-level text encoding layer, such as an encryption or
> compression layer.
>
> Writing characters directly to streams made sense in the pre-Unicode
> world where there was a one-to-one correspondence between characters and
> bytes.  In a modern world, text encoding is an important service that
> deserves its own standalone module.

I'd say quite the opposite. Since now encodings are embedded all the way 
down at the low level (per fwide above), we can't pretend it's all bytes 
down there and leave characters to upper layers. There _are_ transports 
that deal with characters directly.

So the $1M question is, do we support text transports or not?

- fwide streams

- files for which isatty() returns true 
(http://www.opengroup.org/onlinepubs/009695399/functions/isatty.html)

- email protocol and probably other Internet protocols

- others?

If we don't support text at the transport level, things can still made 
to work but in a more fragile manner: upper-level protocols will need to 
_know_ that although the API accepts any ubyte[], in fact the results 
would be weird and malfunctioning if the wrong things are being passed. 
A text-based transport would clarify at the type level that a text 
stream accepts only UTF-encoded characters.

I think either way is not a catastrophe. We can make it work.


Andrei