struct vs class for a simple token in my d lexer

[Dmitry Olshansky]

On 14.05.2012 21:16, Tobias Pankrath wrote:
>
>> struct Token{
>> uint col, line;
>> uint flags;//indicated info about token, serves as both type tag and
>> flag set;
>> //indicates proper type once token was cooked (like "31.415926" ->
>> 3.145926e1) i.e. values are calculated
>> union {
>> string chars;
>> float f_val;
>> double d_val;
>> uint uint_val;
>> long long_val;
>> ulnog ulong_val;
>> //... anything else you may need (8 bytes are plenty)
>> }//even then you may use up to 12bytes
>> //total size == 24 or 20
>> };
>
> I wouldn't convert the string representation to a value unless needed.
> Even the parser does not need to know.

Thx for snipping out the gist of it :)
See last sentence, flexibility is the king:
<<<
Where:
     Each raw token at start has chars == slice of characters in text 
(or if not UTF-8 source = copy of source). Except for keywords and 
operators.
     Cooking is a process of calculating constant values and such (say 
populating symbols table will putting symbol id into token instead of 
leaving string slice). Do it on the fly or after the whole source - let 
the user choose.
 >>>

-- 
Dmitry Olshansky