Request for comments: std.d.lexer

[Dmitry Olshansky]

28-Jan-2013 18:08, jerro пишет:
> On Monday, 28 January 2013 at 11:00:09 UTC, Dmitry Olshansky wrote:
>> 28-Jan-2013 14:39, Jacob Carlborg пишет:
>>> On 2013-01-28 11:14, Walter Bright wrote:
>>>
>>>> Well, the source buffer can be large, and will span a lot of memory
>>>> cache lines, making accessing those slices slow.
>>>
>>> Would it be better to .dup the slices? Won't that be just as slow as
>>> using the appender?
>>>
>>
>> It would be better to compactly layout these one by one in a separate
>> buffer skipping all of the extra slack in the source file(s).
>
> Would it also make sense to do small string optimization? On 64 bit, you
> could use the memory used by the string field itself to store strings of
> length up to 15 characters.

It very well could be iff slice is a field in the Token struct.

-- 
Dmitry Olshansky