DCT: D compiler as a collection of libraries

[Roman D. Boiko]

On Tuesday, 15 May 2012 at 19:27:26 UTC, Timon Gehr wrote:
> On 05/14/2012 05:00 PM, Roman D. Boiko wrote:
>> Currently I think about making token a class instead of struct.
>> ...
>>
>> Could anybody suggest other pros and cons? Which option would 
>> you choose?
>
> Just use a circular buffer of value-type tokens. There is 
> absolutely no excuse for slow parsing.
I'd like to be able to do efficient token lookup based on its 
start index in the file (e.g., for autocompletion and calculating 
column/line numbers on demand). I also plan to have ability to 
update lexer, parser and semantic analysis results as the user or 
tool makes edits (incrementally, not by restarting), and to keep 
history of changes as long as it is needed.

To achieve this, I will use Token[N][] for storing tokens, and an 
immutable binary tree for efficient lookup and preserving history.

I don't worry much about keeping all tokens in memory, because 
information they contain is useful and needed for most scenarios. 
If, on the contrary, information is no longer used, it will be 
garbage collected.

This is a significant redesign of current implementation. It is 
based on feedback from this thread and completely satisfies my 
design goals. It is also going to be efficient.

Alpha release for community review is planned on May 21, and 
design documentation in 10 days after that.