Request for comments: std.d.lexer

[jerro]

On Monday, 28 January 2013 at 11:00:09 UTC, Dmitry Olshansky 
wrote:
> 28-Jan-2013 14:39, Jacob Carlborg пишет:
>> On 2013-01-28 11:14, Walter Bright wrote:
>>
>>> Well, the source buffer can be large, and will span a lot of 
>>> memory
>>> cache lines, making accessing those slices slow.
>>
>> Would it be better to .dup the slices? Won't that be just as 
>> slow as
>> using the appender?
>>
>
> It would be better to compactly layout these one by one in a 
> separate buffer skipping all of the extra slack in the source 
> file(s).

Would it also make sense to do small string optimization? On 64 
bit, you could use the memory used by the string field itself to 
store strings of length up to 15 characters.