Streaming library

[Andrei Alexandrescu]

On 10/13/10 21:20 CDT, Denis Koroskin wrote:
> On Thu, 14 Oct 2010 03:47:12 +0400, Andrei Alexandrescu
> <SeeWebsiteForEmail at erdani.org> wrote:
>
>> On 10/13/2010 06:23 PM, Denis Koroskin wrote:
>>> On Thu, 14 Oct 2010 03:06:30 +0400, Andrei Alexandrescu
>>> <SeeWebsiteForEmail at erdani.org> wrote:
>>>> Gnu offers two specialized routines:
>>>> http://www.gnu.org/s/libc/manual/html_node/Line-Input.html. It is many
>>>> times more efficient than anything that can be done in client code
>>>> using the stdio API. I'm thinking along those lines.
>>>>
>>>
>>> I can easily implement similar interface on top of chunked read:
>>> ubyte[] readLine(ubyte[] lineBuffer); or bool readLine(ref ubyte[]
>>> lineBuffer);
>>
>> You can't.
>>
>>> I've quickly looked through an implementation, too, and it's still
>>> filling a buffer first, and then copying character byte-by-byte to the
>>> output string (making realloc when needed) until a delimiter is found.
>>> It is exactly as efficient as implemented externally.
>>
>> Except you don't have an interface to copy byte by byte. Oops...
>>
>>> It does the same
>>> amount of copying and memory allocations. "Many times more efficient" is
>>> just an overestimation.
>>
>> It's not. I measured because it was important in an application I was
>> working on. It's shocking how some seemingly minor changes can make a
>> big difference in throughput.
>>
>>> BTW, did you see my message about std.concurrency?
>>
>> Yes, but I'll need to leave the bulk of it to Sean. Thanks.
>>
>>
>> Andrei
>
> Okay. Now give me your best and tell me mine is slower (sorry for a lack
> of comments):
[snip]

If you're satisfied with this, then my point has been lost in the 
midstream. I was saying it's impossible to implement a line reader on 
top of a read(ubyte[]) interface without extra buffering and copying. 
You provided a careful implementation that at the end of the day 
inevitably does the extra buffering and copying.


Andrei