[RFC] I/O and Buffer Range

[Dmitry Olshansky]

31-Dec-2013 22:46, Joseph Cassman пишет:
> On Tuesday, 31 December 2013 at 09:04:58 UTC, Dmitry Olshansky wrote:
>> 31-Dec-2013 05:53, Joseph Cassman пишет:
>>> On Sunday, 29 December 2013 at 22:02:57 UTC, Dmitry Olshansky wrote:

>> I'm thinking there might be a way to bridge the new range type with
>> ForwardRange but not directly as defined at the moment.
>>
>> A possibility I consider is to separate a Buffer object (not a range),
>> and let it be shared among views - light-weight buffer-ranges. Then if
>> we imagine that these light-weight buffer-ranges are working as marks
>> (i.e. they pin down the buffer) in the current proposal then they
>> could be forward ranges.

I've created a fork where I've implemented just that.
As a bonus I also tweaked stream primitives so it now works with pipes 
or whatever input stdin happens to be.

Links stay the same:
Docs: http://blackwhale.github.io/datapicked/dpick.buffer.traits.html
Code: 
https://github.com/blackwhale/datapicked/tree/fwd-buffer-range/dpick/buffer

The description has largely simplified and the primitive count reduced.

1. A buffer range is a forward range. It has reference semantics.
2. A copy produced by _save_ is an independent view of the underlying 
buffer (or window).
3. No bytes can be discarded that are seen in some existing view. Thus 
each reference pins its position in the buffer.
4. 3 new primitives are:
    Range slice(BufferRange r);
Returns a slice of a window between the current range position and r. It 
must be a random access range.

    ptrdiff_t tell(BufferRange r);
Returns a difference in positions in the window of current range and r. 
Note that unlike slice(r).length this can be both positive and negative.

    bool seek(ptrdiff_t ofs);
Reset buffer state to an offset from the current position. Return 
indicates success of the operation. It may fail if there is not enough 
data, or (if ofs is negative) that this portion of data was already 
discarded.

5. Lookahead and lookbehind are a extra primitives that were left intact 
for the moment. Where applicable a range may provide lookahead:

Range lookahead(); //as much as available in the window
Range lookahead(size_t n); // either n exactly or nothing if not

And lookbehind:

Range lookbehind(); //as much as available in the window
Range lookbehind(size_t n); //either n exactly or nothing if not

These should probably be tested as separate traits.

>> input-source <--> buffer range <--> parser/consumer
>>
>> Meaning that if we can mix and match parsers with buffer ranges, and
>> buffer ranges with input sources we had grown something powerful indeed.
>
> Being able to wrap an already-in-use range object with the buffer
> interface as you do in the sample code
> (https://github.com/blackwhale/datapicked/blob/master/dgrep.d) is good
> for composability. Also allows for existing functionality in
> std.algorithm to be reused as-is.

It was more about wrapping an array but it's got to integrate well with 
what we have. I could imagine a use case for buffering an input range.
Then I think a buffer range of anything other then bytes would be in order.

> I think the new range type could also be added directly to some new, or
> perhaps retrofitted into existing, code to add the new functionality
> without sacrificing performance. In that way the internal payload
> already used to get the data (say by the input range) could be reused
> without having to allocate new memory to support the buffer API.
>
> As one idea of using a buffer range from the start, a function template
> by(T) (where T is ubyte, char, wchar, or dchar) could be added to
> std.stdio.

IMHO C run-time I/O has no use in D. The amount of work spent on 
special-casing the non-locking primitives of each C run-time,
repeating legacy mistakes (like text mode, codepages and locales) and 
stumbling on portability problems (getc is a macro we can't have) would 
have been better spent elsewhere - designing our own I/O framework.

I've put together up something pretty simple and fast for buffer range 
directly on native I/O:
https://github.com/blackwhale/datapicked/blob/fwd-buffer-range/dpick/buffer/stream.d

It needs a bit better error messages then naked enforce, and a bit of 
tweaks to memory management. It does runs circles around existing 
std.stdio already.

> It would return a buffer range object providing more
> functionality than byChunk or byLine while adding access to the entire
> stream of data in a file in a contiguous and yet efficient manner.

Drop 'efficient' if we talk interfacing with C run-time. Otherwise, yes, 
absolutely.

> Seems
> to help with the issues faced in processing file data mentioned in
> previous comments in this thread.


-- 
Dmitry Olshansky