DCT: D compiler as a collection of libraries

[Roman D. Boiko]

On Friday, 11 May 2012 at 09:36:28 UTC, Jacob Carlborg wrote:
> On 2012-05-11 11:22, Roman D. Boiko wrote:
>> Locations will possible to calculate
>> both taking into account special token sequences (e.g., #line 3
>> "ab/c.d"), or discarding them.
>
> Aha, clever. As long as I can get out the information I'm happy 
> :) How about adding properties for this in the token struct?
There is a method for that in Lexer interface, for me it looks 
like it belongth there and not to token. Version accepting token 
and producing a pair of start/end Locations will be added.

>>>>> * Does it convert numerical literals and similar to their 
>>>>> actual values
>>>> It is planned to add a post-processor for that as part of 
>>>> parser,
>>>> please see README.md for some more details.
>>>
>>> Isn't that a job for the lexer?
>> That might be done in lexer for efficiency reasons (to avoid 
>> lexing
>> token value again). But separating this into a dedicated 
>> post-processing
>> phase leads to a much cleaner design (IMO), also suitable for 
>> uses when
>> such values are not needed.
>
> That might be the case. But I don't think it belongs in the 
> parser.
I will provide example code and a dedicated post later to 
illustrate my point.

>> Also I don't think that performance would be
>> improved given the ratio of number of literals to total number 
>> of tokens
>> and the need to store additional information per token if it 
>> is done in
>> lexer. I will elaborate on that later.
>
> Ok, fair enough. Perhaps this could be a property in the Token 
> struct as well. In that case I would suggest renaming "value" 
> to lexeme/spelling/representation, or something like that, and 
> then name the new property "value".
I was going to rename value, but couldn't find a nice term. 
Thanks for your suggestions!
As for the property with strongly typed literal value, currently 
I plan to put it into AST.