[phobos] Fwd: [Issue 4025] New: Making network with the std.stdio.File interface

[Andrei Alexandrescu]

On 04/08/2010 01:23 PM, Steve Schveighoffer wrote:
>> The network socket is not a range, it's a File, and File does have
>> primitives such as rawWrite and rawRead, which we can add to and
>> improve.
>>
>> File offers ranges, but you're not required to use them.
>
> That's not what I read from Walter's comment...  He indicated that
> something like an e.g. zip library should take a range as input.
> This implies that all streams are shoehorned into range form.

If the zip library works with ranges, we can use it for transparently 
handling in-memory zip manipulation and also zip file manipulation.

>> Makes sense. I'm just a bit worried about stdio's poor buffering
>> interface. It only offers setvbuf(), which is quite opaque.
>
> The only reason to use FILE * as the underlying implementation is to
> be compatible with C's (f)printf.  It makes sense that you only need
> that compatibility for printing to a standard handle.  I think we can
> probably come up with an abstraction layer that uses FILE* only when
> dealing with standard handles.

It's more than printf. There are several I/O routines in stdio, and all 
use FILE* for both input and output. If a D application mixes calls to C 
APIs that do I/O with stdin, stdout, and stderr, we need to take a 
stance on what should happen.

> In that case, we are no longer limited to FILE*'s capabilities for
> other I/O types (e.g. sockets, IPC).

I think File does not need to be inextricably linked to FILE*.

>> The element type of the range would be ubyte[]. The number of bytes
>> transferred at a step should be settable via another primitive.
>> So:
>>
>> struct SeekableBufRange { // Range primitives @property bool
>> empty();
>>
>> @property ubyte[] front(); void popFront(); // Extra primitives
>> @property size_t bufsize(); @property void bufsize(size_t);
>> @property ulong  position(); @property void position(ulong); }
>>
>> How's that sound? This is one range that File could expose
>> directly.
>
> Horrible.  You are replacing a single function (rawRead) with all
> these functions:
>
> empty() front() popFront() bufsize() bufsize(size_t)

I don't think that accurately represents what's going on. rawRead does 
need a fair amount of paraphernalia to work. For example:

// Consume input using rawRead
auto buffer = new ubyte[1024];
size_t read;
while ((read = input.rawRead(buffer).length) > 0) {
    auto usable = buffer[0 .. read];
    ... use usable ...
}

Not that elegant. Compare and contrast with:

// Consume input using a range
foreach (buffer; input.byChunk(1024)) {
     ... use buffer ...
}

// Consume input straight from a range
input.bufsize = 1024;
foreach (buffer; input) {
     ... use buffer ...
}

> That doesn't even cover the awkwardness of how the code now has to
> read N bytes (a single line with rawRead):

I think "awkwardness" doesn't describe it.

> // read N bytes
> source.bufsize = N;
> auto data = source.front();
> source.popFront();

I think it's more often to want to consume stuff in a stream manner, as 
opposed to attempting to read some isolated bits. Ranges are optimized 
for the former.

> And it also doesn't cover how you now have to tack on these functions
> to standard memory ranges.  Or how the stream-based range has to
> handle awkward situations where someone might call front several
> times before calling popFront (not possible with rawRead).  Or how
> you have no control over the inevitable buffering scheme required to
> support such awkwardness.

We need to figure out all this stuff together, but so far I'm not at all 
convinced that seekable ranges are awkward.


Andrei