DCT: D compiler as a collection of libraries

[deadalnix]

Le 15/05/2012 21:59, Roman D. Boiko a écrit :
> On Tuesday, 15 May 2012 at 19:27:26 UTC, Timon Gehr wrote:
>> On 05/14/2012 05:00 PM, Roman D. Boiko wrote:
>>> Currently I think about making token a class instead of struct.
>>> ...
>>>
>>> Could anybody suggest other pros and cons? Which option would you
>>> choose?
>>
>> Just use a circular buffer of value-type tokens. There is absolutely
>> no excuse for slow parsing.
> I'd like to be able to do efficient token lookup based on its start
> index in the file (e.g., for autocompletion and calculating column/line
> numbers on demand). I also plan to have ability to update lexer, parser
> and semantic analysis results as the user or tool makes edits
> (incrementally, not by restarting), and to keep history of changes as
> long as it is needed.
>
> To achieve this, I will use Token[N][] for storing tokens, and an
> immutable binary tree for efficient lookup and preserving history.
>
> I don't worry much about keeping all tokens in memory, because
> information they contain is useful and needed for most scenarios. If, on
> the contrary, information is no longer used, it will be garbage collected.
>
> This is a significant redesign of current implementation. It is based on
> feedback from this thread and completely satisfies my design goals. It
> is also going to be efficient.
>
> Alpha release for community review is planned on May 21, and design
> documentation in 10 days after that.
>

The only information that is usefull over time is the position. This is 
the one that you want to calculate lazily.

In other terms, what you want to keep in memory have no interest.

A circular buffer as Timon propose is a better idea.