Lexer in D

[Dmitry Olshansky]

03-Mar-2013 18:28, Namespace пишет:
>> I'd repeat that I think it makes no sense to separately treat isType.
>> In any case there is way more to types then built-in ones and is the
>> job of parser (to assume types) and semantic step (to type-check and
>> infer).
> Yes I've understood, but currently I want it so.
>
>> Another thing is to run -profile without inline to understand the
>> structure of time spent per each subroutine better.
> I did this and I refactored a lot of my code.
> Now I get 170 - 180 msecs and this is my current trace.log:
> http://dpaste.1azy.net/b94b19ff
>
> But currently I have no more ideas how I could gain more performance.
> Maybe I should disable the compiler while looping through the text?
> Or maybe I should allocate more space for the Token array (e.g.
> toks.length = 100;)?
> I hope that you or anyone other have further ideas.
> But anyway, thanks for the help! :)

Simple - don't use array append and don't produce and array.
Just produce a lazy forward range that is iterated.

At the very least use Appender!(Token[]) it ought to be much faster.

-- 
Dmitry Olshansky