DCT: D compiler as a collection of libraries

[Ary Manzana]

On 5/12/12 12:17 PM, Roman D. Boiko wrote:
> On Saturday, 12 May 2012 at 03:32:20 UTC, Ary Manzana wrote:
>> As deadalnix says, I think you are over-complicating things.
>>
>> I mean, to store the column and line information it's just:
>>
>> if (isNewLine(c)) {
>> line++;
>> column = 0;
>> } else {
>> column++;
>> }
>>
>> (I think you need to add that to the SourceRange class. Then copy line
>> and column to token on the Lexer#lex() method)
>>
>> Do you really think it's that costly in terms of performance?
>>
>> I think you are wasting much more memory and performance by storing
>> all the tokens in the lexer.
>>
>> Imagine I want to implement a simple syntax highlighter: just
>> highlight keywords. How can I tell DCT to *not* store all tokens
>> because I need each one in turn? And since I'll be highlighting in the
>> editor I will need column and line information. That means I'll have
>> to do that O(log(n)) operation for every token.
>>
>> So you see, for the simplest use case of a lexer the performance of
>> DCT is awful.
>>
>> Now imagine I want to build an AST. Again, I consume the tokens one by
>> one, probably peeking in some cases. If I want to store line and
>> column information I just copy them to the AST. You say the tokens are
>> discarded but their data is not, and that's why their data is usually
>> copied.
>
> Would it be possible for you to fork my code and tweak it for
> comparison? You will definitely discover more problems this way, and
> such feedback would really help me. That doesn't seem to be inadequately
> much work to do.
>
> Any other volunteers for this?

Sure, I'll do it and provide some benchmarks. Thanks for creating the issue.