std.d.lexer requirements

[Walter Bright]

On 8/2/2012 4:52 AM, deadalnix wrote:
> Le 02/08/2012 09:30, Walter Bright a écrit :
>> On 8/1/2012 11:49 PM, Jacob Carlborg wrote:
>>> On 2012-08-02 02:10, Walter Bright wrote:
>>>
>>>> 1. It should accept as input an input range of UTF8. I feel it is a
>>>> mistake to templatize it for UTF16 and UTF32. Anyone desiring to feed it
>>>> UTF16 should use an 'adapter' range to convert the input to UTF8. (This
>>>> is what component programming is all about.)
>>>
>>> I'm no expert on ranges but won't that prevent slicing? Slicing is one
>>> of the
>>> main reasons for why the Tango XML parser is so amazingly fast.
>>>
>>
>> You don't want to use slicing on the lexer. The reason is that your
>> slices will be spread all over memory, as source files can be huge, and
>> all that memory will be retained and never released. What you want is a
>> compact representation after lexing. Compactness also helps a lot with
>> memory caching.
>>
>
> Token are not kept in memory. You usually consume them for other processing and
> throw them away.
>
> It isn't an issue.

The tokens are not kept, correct. But the identifier strings, and the string 
literals, are kept, and if they are slices into the input buffer, then 
everything I said applies.