[phobos] Improvement of stream

[Andrei Alexandrescu]

Does the currently proposed streaming (on digitalmars.d) work for such 
needs?

Andrei

On 9/27/10 1:34 AM, Shin Fujishiro wrote:
> Thus I think we need a buffering layer that exposes a randomly
> accessible array to upper layers.  ByLine() can be easily and
> efficiently implemented with the following primitives defined:
>
> Buffer
> {
>      // The entire buffer.
>      ubyte[] buffer();
>
>      // Slice of buffer() where data is available.
>      ubyte[] available();
>
>      // Moves the beginning of available() by n in buffer().
>      void bump(sizediff_t n);
>
>      // Reads next blob from a source.
>      bool fetch();
> }
>
> Yes, cstdio-esque rawRead() is no good for high-level ByLine.  What
> high-level I/O entities want is:  A randomly accessible buffer. Device
> handles may expose block-oriented streaming primitives, but they
> must be made "partially random accessible" by the buffering layer.
>
>
> Shin
>
> Andrei Alexandrescu<andrei at erdani.com>  wrote:
>> This is a good point. Allow me to expand it a bit and say that we need a
>> collection of solid use cases that we want to support. Without them, we
>> come up with a design that might not work well with certain use patterns.
>>
>> Example of a requirement: "Given a block-oriented stream, define a
>> line-oriented range on top of it."
>>
>> Do we need such a thing? Probably. Can it be implemented efficiently
>> with the rawRead(ubyte[]) primitive? NO.
>>
>> struct ByLine(BlockRange) {
>>       private BlockRange _input;
>>       private char[] store;
>>       ...
>>       void popFront() {
>>           // read one line from _input
>>       }
>> }
>>
>> The problem is that the line reader must sometimes _append_ data to its
>> buffer (in situations when it has read a long line that doesn't fit in
>> one buffer). But the input stream does not support appending.
>>
>> BTW this has been a long source of irritation with C's stdio: the only
>> ways to read a line has been by reading one character at a time with
>> fgetc() (pig slow) or by using fgets() (unsafe) or fgetsn() (inefficient
>> for long lines) or by using getline() (nonportable, see
>> http://www.gnu.org/s/libc/manual/html_node/Line-Input.html).
>>
>> Moral of the story? We need to have a number of well-defined scenarios
>> in mind when defining a streaming interface.
>>
>>
>> Andrei
>>
>> On 9/23/10 8:37 CDT, SHOO wrote:
>>> Hmm... Does it mean to have to relay three classes to do I/O processing?
>>>
>>> auto handle = FileHandle("file");
>>> scope (exit) handle.close();
>>> auto buf = MemoryBuffer();
>>> auto range = byLine(range);
>>>
>>> I think it is slightly complicatedly. What is the reason why it must
>>> come to look like it?
>>>
>>> BTW, I don't know well what buffers must do. What is the requirement of
>>> buffers?
>>>
>>> (2010/09/21 14:05), Shin Fujishiro wrote:
>>>> SHOO<zan77137 at nifty.com>  wrote:
>>>>> I think that there are two problems about I/O operation.
>>>>> - Location of buffering layers.
>>>>> - Direction of seeking.
>>>>>
>>>> ...snip...
>>>>>
>>>>> It is necessary for the concept to be divided in two at least to realize
>>>>> them. (Handles and Ranges) Or more(+ Port or Stream).
>>>>> The opening difficult item appears when I think about this.
>>>>
>>>> How about putting a buffering layer between the two you said? Not
>>>> only it just solves the who-does-buffering problem, but also opens a
>>>> bit of freedom in the lowermost I/O device layer.
>>>>
>>>>
>>>> Shin
>>> _______________________________________________
>>> phobos mailing list
>>> phobos at puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/phobos
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
>
>
> Shin
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos