[RFC] I/O and Buffer Range

[Dmitry Olshansky]

31-Dec-2013 05:51, Brad Anderson пишет:
> On Sunday, 29 December 2013 at 22:02:57 UTC, Dmitry Olshansky wrote:
>> Proposal
>
> Having never written any parser I'm not really qualified to seriously
> give comments or review it but it all looks very nice to me.
>
> Speaking as just an end user of these things whenever I use ranges over
> files or from, say, std.net.curl the byLine/byChunk interface always
> feels terribly awkward to use which often leads to me just giving up and
> loading the entire file/resource into an array. It's the boundaries that
> I stumble over. byLine never fits when I want to extract something
> multiline but byChunk doesn't fit because I often if what I'm searching
> for lands on the boundary I'll miss it.

Exactly, the situation is simply not good enough. I can assure you that 
on the side of parser writers it's even less appealing.

>
> Being able to just do a matchAll() on a file, std.net.curl, etc. without
> sacrificing performance and memory would be such a massive gain for
> usability.

.. and performance ;)

>
> Just a simple example of where I couldn't figure out how to utilize
> either byLine or byChunk without adding some clunky homegrown buffering
> solution. This is code that scrapes website titles from the pages of
> URLs in IRC messages.
[snip]
>
> I really, really didn't want to use that std.net.curl.get().  It causes
> all sorts of problems if someone links to a huge resource.

*Nods*

> I just could
> not figure out how to utilize byLine (the title regex capture can be
> multiline) or byChunk cleanly. Code elegance (a lot of it due to Jakob
> Ovrum's help in IRC) was really a goal here as this is just a toy so I
> went with get() for the time being but it's always sad to sacrifice
> elegance for performance. I certainly didn't want to add some elaborate
> evergrowing buffer in the middle of this otherwise clean UFCS chain (and
> I'm not even sure how to incrementally regex search the growing buffer
> or if that's even possible).

I thought to provide something like that, incremental match that takes 
pieces of data slice by slice, having to mess with the not-yet-matched 
kind of object. But it was solving the wrong problem. And it shows that 
backtracking engines simply can't work like that, they would want to go 
back to the prior pieces.

>
> If I'm understanding your proposal correctly that get(url) could be
> replaced with a hypothetical std.net.curl "buffer range" and that could
> be passed directly to matchFirst. It would only take up, at most, the
> size of the buffer in memory (which could grow if the capture grows to
> be larger than the buffer) and wouldn't read the unneeded portion of the
> resource at all. That would be such a huge win for everyone so I'm very
> excited about this proposal. It addresses all of my current problems.

That's indeed what the proposal is all about. Glad it makes sense :)

>
>
> P.S. I love std.regex more and more every day. It made that
> entitiesToUni function so easy to implement: http://dpaste.dzfl.pl/688f2e7d

Aye, replace with functor rox!

-- 
Dmitry Olshansky