[phobos] Improvement of stream

[Shin Fujishiro]

Thus I think we need a buffering layer that exposes a randomly
accessible array to upper layers.  ByLine() can be easily and
efficiently implemented with the following primitives defined:

Buffer
{
    // The entire buffer.
    ubyte[] buffer();

    // Slice of buffer() where data is available.
    ubyte[] available();

    // Moves the beginning of available() by n in buffer().
    void bump(sizediff_t n);

    // Reads next blob from a source.
    bool fetch();
}

Yes, cstdio-esque rawRead() is no good for high-level ByLine.  What
high-level I/O entities want is:  A randomly accessible buffer. Device
handles may expose block-oriented streaming primitives, but they
must be made "partially random accessible" by the buffering layer.


Shin

Andrei Alexandrescu <andrei at erdani.com> wrote:
> This is a good point. Allow me to expand it a bit and say that we need a 
> collection of solid use cases that we want to support. Without them, we 
> come up with a design that might not work well with certain use patterns.
> 
> Example of a requirement: "Given a block-oriented stream, define a 
> line-oriented range on top of it."
> 
> Do we need such a thing? Probably. Can it be implemented efficiently 
> with the rawRead(ubyte[]) primitive? NO.
> 
> struct ByLine(BlockRange) {
>      private BlockRange _input;
>      private char[] store;
>      ...
>      void popFront() {
>          // read one line from _input
>      }
> }
> 
> The problem is that the line reader must sometimes _append_ data to its 
> buffer (in situations when it has read a long line that doesn't fit in 
> one buffer). But the input stream does not support appending.
>
> BTW this has been a long source of irritation with C's stdio: the only 
> ways to read a line has been by reading one character at a time with 
> fgetc() (pig slow) or by using fgets() (unsafe) or fgetsn() (inefficient 
> for long lines) or by using getline() (nonportable, see 
> http://www.gnu.org/s/libc/manual/html_node/Line-Input.html).
> 
> Moral of the story? We need to have a number of well-defined scenarios 
> in mind when defining a streaming interface.
> 
> 
> Andrei
> 
> On 9/23/10 8:37 CDT, SHOO wrote:
> > Hmm... Does it mean to have to relay three classes to do I/O processing?
> >
> > auto handle = FileHandle("file");
> > scope (exit) handle.close();
> > auto buf = MemoryBuffer();
> > auto range = byLine(range);
> >
> > I think it is slightly complicatedly. What is the reason why it must
> > come to look like it?
> >
> > BTW, I don't know well what buffers must do. What is the requirement of
> > buffers?
> >
> > (2010/09/21 14:05), Shin Fujishiro wrote:
> >> SHOO<zan77137 at nifty.com> wrote:
> >>> I think that there are two problems about I/O operation.
> >>> - Location of buffering layers.
> >>> - Direction of seeking.
> >>>
> >> ...snip...
> >>>
> >>> It is necessary for the concept to be divided in two at least to realize
> >>> them. (Handles and Ranges) Or more(+ Port or Stream).
> >>> The opening difficult item appears when I think about this.
> >>
> >> How about putting a buffering layer between the two you said? Not
> >> only it just solves the who-does-buffering problem, but also opens a
> >> bit of freedom in the lowermost I/O device layer.
> >>
> >>
> >> Shin
> > _______________________________________________
> > phobos mailing list
> > phobos at puremagic.com
> > http://lists.puremagic.com/mailman/listinfo/phobos
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos


Shin