[RFC] I/O and Buffer Range

[Joseph Cassman]

On Tuesday, 31 December 2013 at 01:51:23 UTC, Brad Anderson wrote:
> On Sunday, 29 December 2013 at 22:02:57 UTC, Dmitry Olshansky 
> wrote:
>> Proposal
>
> Having never written any parser I'm not really qualified to 
> seriously give comments or review it but it all looks very nice 
> to me.
>
> Speaking as just an end user of these things whenever I use 
> ranges over files or from, say, std.net.curl the byLine/byChunk 
> interface always feels terribly awkward to use which often 
> leads to me just giving up and loading the entire file/resource 
> into an array. It's the boundaries that I stumble over. byLine 
> never fits when I want to extract something multiline but 
> byChunk doesn't fit because I often if what I'm searching for 
> lands on the boundary I'll miss it.
>
> Being able to just do a matchAll() on a file, std.net.curl, 
> etc. without sacrificing performance and memory would be such a 
> massive gain for usability.
>
> Just a simple example of where I couldn't figure out how to 
> utilize either byLine or byChunk without adding some clunky 
> homegrown buffering solution. This is code that scrapes website 
> titles from the pages of URLs in IRC messages.
>
> [...]
>
> If I'm understanding your proposal correctly that get(url) 
> could be replaced with a hypothetical std.net.curl "buffer 
> range" and that could be passed directly to matchFirst. It 
> would only take up, at most, the size of the buffer in memory 
> (which could grow if the capture grows to be larger than the 
> buffer) and wouldn't read the unneeded portion of the resource 
> at all. That would be such a huge win for everyone so I'm very 
> excited about this proposal. It addresses all of my current 
> problems.
>
> [...]

I find the same to be the case with API's used to process files. 
It would be a real win to have a simple, efficient, and possibly 
non-blocking range interface over a file able to access a single 
byte or character at a time.

I remember a discussion on this forum that spoke positively about 
the async and await combination of keywords in .NET 4.5. If 
asynchronous and synchronous functionality could be combined with 
a flexible buffer in a UFCS call chain I think that would hit a 
sweet spot for code composability and understandability.

Joseph