std.d.lexer requirements

[Dmitry Olshansky]

On 04-Aug-12 14:02, Christophe Travert wrote:
> Jonathan M Davis , dans le message (digitalmars.D:174191), a écrit :
>> On Thursday, August 02, 2012 11:08:23 Walter Bright wrote:
>>> The tokens are not kept, correct. But the identifier strings, and the string
>>> literals, are kept, and if they are slices into the input buffer, then
>>> everything I said applies.
>>
>> String literals often _can't_ be slices unless you leave them in their
>> original state rather than giving the version that they translate to (e.g.
>> leaving \© in the string rather than replacing it with its actual,
>> unicode value). And since you're not going to be able to create the literal
>> using whatever type the range is unless it's a string of some variety, that
>> means that the literals often can't be slices, which - depending on the
>> implementation - would make it so that that they can't _ever_ be slices.
>>
>> Identifiers are a different story, since they don't have to be translated at
>> all, but regardless of whether keeping a slice would be better than creating a
>> new string, the identifier table will be far superior, since then you only need
>> one copy of each identifier. So, it ultimately doesn't make sense to use slices
>> in either case even without considering issues like them being spread across
>> memory.
>>
>> The only place that I'd expect a slice in a token is in the string which
>> represents the text which was lexed, and that won't normally be kept around.
>>
>> - Jonathan M Davis
>
> I thought it was not the lexer's job to process litterals. Just split
> the input in tokens, and provide minimal info: TokenType, line and col
> along with the representation from the input. That's enough for a syntax
> highlighting tools for example. Otherwise you'll end up doing complex
> interpretation and the lexer will not be that efficient. Litteral
> interpretation can be done in a second step. Do you think doing litteral
> interpretation separately when you need it would be less efficient?
>
Most likely - since you re-read the same memory twice to do it.

-- 
Dmitry Olshansky