[RFC] I/O and Buffer Range

Jason White 54f9byee3t32 at gmail.com
Mon Jan 6 23:59:59 PST 2014


On Monday, 6 January 2014 at 10:26:27 UTC, Dmitry Olshansky wrote:
> Ok, now I see. In my eye though serialization completely hides 
> raw stream write.
>
> So:
> struct SomeStream{
>     void write(const(ubyte)[] data...);
> }
>
> struct Serializer(Stream){
>     void write(T)(T value); //calls stream.write inside of it
> private:
>     Stream stream;
> }

I was thinking it should also have "alias stream this;", but 
maybe that's not the best thing to do for a serializer.

I concede, I've s/(read|write)Data/\1/g on

     https://github.com/jasonwhite/io/blob/master/src/io/file.d

and it now works on Windows with useful exception messages.

>> Shouldn't buffers/arrays provide a stream interface in 
>> addition to
>> buffer-specific operations?
>
> I think it may be best not to. Buffer builds on top of 
> unbuffered stream. If there is a need to do large reads it may 
> as well use naked stream and not worry about extra copying 
> happening in the buffer layer.
>
> I need to think on this. Seeing as lookahead + seek could be 
> labeled as read even though it's not.
>
>> I don't see why it would conflict with a
>> range interface. As I understand it, ranges act on a single 
>> element at a
>> time while streams act on multiple elements at a time. For 
>> ArrayBuffer
>> in datapicked, a stream-style read is just lookahead(n) and 
>> cur += n.
>> What capabilities are lost?
>
> In short - lookahead is slicing, read would be copying.
> For me prime capability of an array is slicing that is dirt 
> cheap O(1). On the other hand stream interface is all about 
> copying bytes to the user provided array.
>
> In this setting it means that if you want to wrap array as 
> stream, then it must follow generic stream interface. The 
> latter cannot and should not think of slicing and the like. 
> Then while wrapping it in some adapter up the chain it's no 
> longer seen as array (because adapter is generic too and is 
> made for streams). This is what I call capability loss.
>
>> If buffers/arrays provide a stream interface, then they can be 
>> used by
>> code that doesn't directly need the buffering capabilities but 
>> would
>> still benefit from them.
>
> See above - it would be better if the code was written for 
> ranges not streams. Then e.g. slicing of buffer range on top of 
> array works just as cheap as it was for arrays. And zero copies 
> are made (=performance).

Okay, I see. I'm just concerned about composability. I'll have to 
think more about how it's affected.

(BTW, you can probably simplify lookahead/lookbehind with 
look(ptrdiff_t n) where the sign of n indicates ahead/behind.)

> Actually these objects do just fine, since OS does the locking 
> (or makes sure of something equivalent). If your stream is TLS 
> there is no need for extra locking at all (no interleaving of 
> I/O calls is possible) regardless of its kind.
>
> Shared instances would need locking as 2 threads may request 
> some operation, and as OS locks only on per sys-call basis 
> something cruel may happen in the code that deals with 
> buffering etc.

Oh yeah, you're right.

As a side note: I would love to get a kick-ass I/O stream package 
into Phobos. It could replace std.stream as well as std.stdio. 
Stuff like serializers and lexers would be more robust and easier 
to write.


More information about the Digitalmars-d mailing list