Request for comments: std.d.lexer

[deadalnix]

On Sunday, 27 January 2013 at 10:55:49 UTC, Brian Schott wrote:
> On Sunday, 27 January 2013 at 10:32:39 UTC, deadalnix wrote:
>> Very happy to see that !
>>
>> Some remarks :
>> - Many parameters should be compile time parameters. Instead 
>> of runtime.
>
> I decided not to do this because the lexer actually calls 
> itself while parsing token strings. If they were compile-time 
> parameters, the compiler would likely generate a lot more code.
>

Have you measured ?

>> - You need to provide a way to specify haw textual 
>> representation of the token (ie value) is set. The best way to 
>> do it IMO is an alias parameter that return a string when 
>> called with a string (then the user can choose to keep the 
>> value from original string, create a copy, always get the same 
>> copy with the same string, etc . . .).
>
> The lexer does not operate on slices of its input. It would be 
> possible to special-case for this in the future.
>

Even without special casing for slice of the input, generating a 
new string or reusing an existing one is already an important 
thing. Depending of the processing that come after, it may be of 
great importance.