Streaming transport interfaces: input

[Andrei Alexandrescu]

On 10/14/10 14:00 CDT, Denis Koroskin wrote:
> On Thu, 14 Oct 2010 22:22:07 +0400, Andrei Alexandrescu
> <SeeWebsiteForEmail at erdani.org> wrote:
>
>> On 10/14/10 12:56 CDT, Denis Koroskin wrote:
>>> appendDelim *requires* buffering for to be implemented. No OS provides
>>> an API to read from a file (be it pipe, socket, whatever) to read up to
>>> some abstract delimiter. It *always* reads in blocks.
>>
>> Clear. What may be not so clear is that read(ubyte[] buf) ALSO
>> requires buffering. Disk I/O comes in fixed buffer sizes (sometimes
>> aligned at 512 bytes or whatever), so ANY protocol that allows the
>> user to set the maximum bytes to read will require buffering and
>> copying. So how is appendDelim worse than read?
>>
>>> As such, if you
>>> need to read until a delimeter, you need to fetch block to some internal
>>> buffer, MANUALLY search through it and THEN copy to output string.
>>
>> And there's no way for the client to efficiently do that.
>>
>>> I've
>>> implemented that on top of chunked read interface, and it was 5% faster
>>> than getline()/getdelim() that GNU libc provides (despite you claming it
>>> to be "many times faster"). It's not.
>>
>> Please post your code.
>>
>
> http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=119248

I meant the baseline.

>>> Buffering requires and additional level of data copying, and this is bad
>>> for fast I/O.
>>
>> Agreed. But then you define routines that also requires buffering. How
>> do you reconcile your own requirement with your own interface?
>>
>
> My interface doesn't require any additional copying. You only copy when
> you need to buffer something, but in general you don't. My Stream
> interface is simply a thin portable layer on top of OS. See the code
> above for simple implementation that is built on top of fopen/fread (I
> used open/read initially but it gave 0 improvement so I went back to
> fopen/fread because GNU libc line-input uses them, too, so that would be
> the most fair comparison). It can't be any more efficient than that.
>
>>> If you need fast I/O or must pull that out of the stream
>>> interface. Otherwise chunked read will be less efficient due to
>>> additional copies to and from buffers.
>>>
>>> On the contrary line-based reading can be implemented on top of the
>>> chunked read without sacrificing a tiny bit of efficiency.
>>
>> Except for extra copying.
>>
>> appendDelim implementation:
>>
>> 1. Low-level read in internal buffers
>>
>> 2. Search for delimiter (assume found for simplicity)
>>
>> 3. Resize user buffer
>>
>> 4. Copy
>>
>> That's one copy, with the necessary corner cases when the delimiter
>> isn't found yet etc. (which increase copying ONLY if the buffer is
>> actually moved when reallocated).
>>
>> The implementation in your message on 10/13/2010 21:20 CDT:
>>
>> 1. Low-level read in internal buffers
>>
>> 2. Copy from internal buffers into the internal buffer provided by
>> your ByLine implementation
>>
>> 3. Copy from the internal buffer of ByLine into the user-supplied buffer
>>
>> That's two copies. Agreed?
>>
>>
>> Andrei
>
> I'm not sure what message are you talking about (first or second one).
> Second one
> (http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=119248)
> makes a chunked read to internal buffer (if not filled yet), then
> searches for a delimiter and then copies to a user-provided buffer.
> That's one copy in most cases. And that's what GNU libc does, too.

Your function calls fopen() and does not disable buffering by means of 
setvbuf(). By default fopen() opens in buffered mode. Does the existence 
of that buffer entail an extra copy?

Andrei