streaming redux

[Dmitry Olshansky]

On 31.12.2010 1:14, Steven Schveighoffer wrote:
> On Thu, 30 Dec 2010 16:49:15 -0500, Dmitry Olshansky 
> <dmitry.olsh at gmail.com> wrote:
>
>> [snip]
>>> -----
>>> Question: Should we allow read to return an empty slice even if 
>>> atEnd is false? If we do, we allow non-blocking streams with burst 
>>> transfer. However, naive client code on non-blocking streams will be 
>>> inefficient because it would essentially implement busy-waiting.
>>>
>>> Why not return an integer so different situations could be 
>>> designated?  It's how the system call read works so you can tell no 
>>> data was read but that's because it's a non-blocking stream.
>>>
>>> I realize it's sexy to return the data again so it can be used 
>>> immediately, but in practice it's more useful to return an integer. 
>>> For example, if you want to fill a buffer, you need a loop anyways 
>>> (there's no guarantee that the first read will fill the buffer), and 
>>> at that point, you are just going to use the length member of the 
>>> return value to advance your loop.
>>>
>>> I'd say, return -1 if a non-blocking stream returns no data, 0 on 
>>> EOF, positive on data read, and throw an exception on error.
>>>
>> Maybe it's only me but I would prefer non-blocking IO not mixed with 
>> blocking in such a way. Imagine function that takes an 
>> UnbufferedInputTransport, how should it indicate that it expects only 
>> a non-blocking IO capable transport? Or the other way around. 
>> Checking return codes hardly helps anything, and it means supporting 
>> both types everywhere, which is a source of all kind of  weird problems.
>>  From my (somewhat limited) experience, code paths for blocking and 
>> non-blocking IO are quite different, the latter are performed by 
>> *special* asynchronous calls which are supported by all modern OSes 
>> for things like files/sockets.
>>
>> Then my position would be:
>> 1) All read/write methods are *blocking*, returning empty slices on EOF.
>> 2) Transport that supports asynchronous IO should implement extended 
>> interfaces like
>> interface AsyncInputTransport: UnbufferedInputTransport{
>>      void asyncRead(ubyte[] buffer, void delegate(ubyte[] data) 
>> callback=null);
>> }
>> interface AsyncOutputTransport: UnbufferedOutputTransport{
>>      void asyncWrite(ubyte[] buffer, void delegate(ubyte[] data) 
>> callback=null);
>> }
>> Where callback (if not null) is called with a slice of buffer 
>> containing actual read/written bytes on IO completion.
>> Any calls to read/asyncRead while there is asynchronous IO operation 
>> going on should throw, of course.
>
> On Linux, you set the file descriptor to blocking or non-blocking, and 
> read(fd) returns errno=EWOULDBLOCK when no data is available.  How 
> does this fit into your scheme?  I.e. if you call read() on a 
> AsyncInputTransport, what does it do when it gets this error?
The only general thing I can think of would be to suspend thread 
(core.Thread.yield), then re-attempt. There may be better platform 
specific ways (there is for GetOverlappedResult with wait flag in Win32).
>
> It's quite possible that there is some API I'm unaware of for doing 
> non-blocking and blocking I/O interleaved, but this has been my 
> experience.
>
> -Steve
You are right, I was referring to Win32 API, but I should have revisited 
that part in API docs. Just checked - indeed you should specify the 
intent in CreateFile operation.  So if asynchronous IO is chosen, then 
blocking IO could only be emulated as suggested above.

-- 
Dmitry Olshansky