[phobos] Fwd: [Issue 4025] New: Making network with the std.stdio.File interface

[Steve Schveighoffer]

Forwarding back to phobos mailing list, because my mail client messed up, I accidentally made this a private conversation.



----- Original Message ----
> From: Andrei Alexandrescu <andrei at erdani.com>
> 
> On 04/08/2010 07:14 AM, Steve Schveighoffer wrote:
> > I agree that a range 
> > of such packets is possible, but the problem is,
> > how does one build such 
> > a range?  If all you have as an interface to
> > a network socket is a 
> > range, how do you make *that* range spit out
> > what you need?  In 
> > other words, you want a D abstraction to the
> > syscalls, I think we all 
> > agree on that.  I don't think a range
> > abstraction is good enough 
> > for all purposes.
>
> The network socket is not a range, it's a File, and 
> File does have primitives such as rawWrite and rawRead, which we can add to and 
> improve.
> 
> File offers ranges, but you're not required to use 
> them.

That's not what I read from Walter's comment...  He indicated that
something like an e.g. zip library should take a range as input.  This
implies that all streams are shoehorned into range form.

Using File is more like what I thought it should be.  If this is the case, then I think we can have a workable solution.

> > If you follow through the logic of how such a system would 
> > be built,
> > I think the best abstraction is a layer that abstracts out 
> > the
> > read/write functionality (unbuffered), and then build 
> > ranges/buffered
> > i/o on top of that.  The abstraction can be 
> > compile-time, we don't
> > need to do interfaces here.
> 
> Makes sense. 
> I'm just a bit worried about stdio's poor buffering interface. It only offers 
> setvbuf(), which is quite opaque.

The only reason to use FILE * as the underlying implementation is to be compatible with C's (f)printf.  It makes sense that you only need that compatibility for printing to a standard handle.  I think we can probably come up with an abstraction layer that uses FILE* only when dealing with standard handles.

In that case, we are no longer limited to FILE*'s capabilities for other I/O types (e.g. sockets, IPC).

> > A simple answer here would be: "A x range of type T"  where 
> > you
> > substitute x for input, forward, etc. and T for the type returned 
> > by
> > front.  And I'm not talking about the zip library, I'm talking 
> > about
> > the generic file/network stream.  It can't know that it's 
> > being used
> > by a zip function.
> 
> I'd speculate that the zip file 
> interface would need a seekable range - a range that is forward, but can be 
> positioned with an extra primitive seek().
> 
> The element type of the range 
> would be ubyte[]. The number of bytes transferred at a step should be settable 
> via another primitive. So:
> 
> struct SeekableBufRange {
>   // Range primitives
>  @property bool empty();
>     
>   @property ubyte[] front();
>     void popFront();
>    // Extra primitives
>   @property size_t bufsize();
>   @property void bufsize(size_t);
>   @property ulong  position();
>   @property void position(ulong);
> }
> 
> How's 
> that sound? This is one range that File could expose directly.

Horrible.  You are replacing a single function (rawRead) with all these functions:

empty()
front()
popFront()
bufsize()
bufsize(size_t)

That doesn't even cover the awkwardness of how the code now has to read N bytes (a single line with rawRead):

// read N bytes
source.bufsize = N;
auto data = source.front();
source.popFront();

And it also doesn't cover how you now have to tack on these functions to standard memory ranges.  Or how the stream-based range has to handle awkward situations where someone might call front several times before calling popFront (not possible with rawRead).  Or how you have no control over the inevitable buffering scheme required to support such awkwardness.

-Steve