Streaming library

[Denis Koroskin]

On Thu, 14 Oct 2010 06:36:17 +0400, Andrei Alexandrescu  
<SeeWebsiteForEmail at erdani.org> wrote:

> On 10/13/10 21:20 CDT, Denis Koroskin wrote:
>> On Thu, 14 Oct 2010 03:47:12 +0400, Andrei Alexandrescu
>> <SeeWebsiteForEmail at erdani.org> wrote:
>>
>>> On 10/13/2010 06:23 PM, Denis Koroskin wrote:
>>>> On Thu, 14 Oct 2010 03:06:30 +0400, Andrei Alexandrescu
>>>> <SeeWebsiteForEmail at erdani.org> wrote:
>>>>> Gnu offers two specialized routines:
>>>>> http://www.gnu.org/s/libc/manual/html_node/Line-Input.html. It is  
>>>>> many
>>>>> times more efficient than anything that can be done in client code
>>>>> using the stdio API. I'm thinking along those lines.
>>>>>
>>>>
>>>> I can easily implement similar interface on top of chunked read:
>>>> ubyte[] readLine(ubyte[] lineBuffer); or bool readLine(ref ubyte[]
>>>> lineBuffer);
>>>
>>> You can't.
>>>
>>>> I've quickly looked through an implementation, too, and it's still
>>>> filling a buffer first, and then copying character byte-by-byte to the
>>>> output string (making realloc when needed) until a delimiter is found.
>>>> It is exactly as efficient as implemented externally.
>>>
>>> Except you don't have an interface to copy byte by byte. Oops...
>>>
>>>> It does the same
>>>> amount of copying and memory allocations. "Many times more efficient"  
>>>> is
>>>> just an overestimation.
>>>
>>> It's not. I measured because it was important in an application I was
>>> working on. It's shocking how some seemingly minor changes can make a
>>> big difference in throughput.
>>>
>>>> BTW, did you see my message about std.concurrency?
>>>
>>> Yes, but I'll need to leave the bulk of it to Sean. Thanks.
>>>
>>>
>>> Andrei
>>
>> Okay. Now give me your best and tell me mine is slower (sorry for a lack
>> of comments):
> [snip]
>
> If you're satisfied with this, then my point has been lost in the  
> midstream. I was saying it's impossible to implement a line reader on  
> top of a read(ubyte[]) interface without extra buffering and copying.  
> You provided a careful implementation that at the end of the day  
> inevitably does the extra buffering and copying.
>
>
> Andrei

You must have missed it somehow, but I did say many times that buffering  
needs to be done externally (e.g. in byLine and BufferedStream).

Can we outline basic Stream interface now so that we could move on?