Request for comments: std.d.lexer

[Walter Bright]

On 2/1/2013 3:22 AM, Dmitry Olshansky wrote:
> 01-Feb-2013 15:05, Walter Bright пишет:
>> On 1/30/2013 8:44 AM, Dmitry Olshansky wrote:
>>> In allocation scheme I proposed that ID could be a 32bit offset into
>>> the unique
>>> identifiers chunk.
>>
>> That only works if you know in advance the max size the chunk can ever
>> be and preallocate it. Otherwise, you have no guarantee that the next
>> allocated chunk will be within 32 bits of address of the previous chunks.
>>
>
> Well I supposed it's exactly one reallocatable block. Then token have an offset
> that doesn't care if the block was reallocated.
>
> Or rather the reallocating just RESERVE virtual RAM for it (say 1G), and COMMIT
> it page by page when you need to grow it. Once lexing is done, shrink virtual
> region to the actual used size to free up address space (e.g. if we are on 32bits).
>
> AS for 32bit limit that gives 4Gb maximum of the cumulative length of all unique
> identifier names is more then enough by any standard. I haven't seen a 4G
> codebase not to speak of identifiers alone that even if we count all the
> repetitions separately.

Your technique can work, provided the number of identifiers isn't large enough 
that memory fragmentation will prevent being able to reallocate the buffer to a 
larger size.