std.d.lexer requirements

[Piotr Szturmaj]

Michel Fortin wrote:
> The next issue, which I haven's seen discussed here is that for a parser
> to be efficient it should operate on buffers. You can make it work with
> arbitrary ranges, but if you don't have a buffer you can slice when you
> need to preserve a string, you're going to have to build the string
> character by character, which is not efficient at all. But then you can
> only really return slices if the underlying representation is the same
> as the output representation, and unless your API has a templated output
> type, you're going to special case a lot of things.

Instead of returning whole strings, you may provide another sub-range 
for the string itself. I wrote JSON parser using this approach 
(https://github.com/pszturmaj/json-streaming-parser) and thanks to that 
it is possible to parse json without a single heap allocation. This 
could be also used in XML parser.