Request for comments: std.d.lexer

[dennis luehring]

Am 31.01.2013 13:14, schrieb Jacob Carlborg:
> On 2013-01-27 10:51, Brian Schott wrote:
>> I'm writing a D lexer for possible inclusion in Phobos.
>>
>> DDOC:
>> http://hackerpilot.github.com/experimental/std_lexer/phobos/lexer.html
>> Code:
>> https://github.com/Hackerpilot/Dscanner/blob/range-based-lexer/std/d/lexer.d
>>
>>
>> It's currently able to correctly syntax highlight all of Phobos, but
>> does a fairly bad job at rejecting or notifying users/callers about
>> invalid input.
>>
>> I'd like to hear arguments on the various ways to handle errors in the
>> lexer. In a compiler it would be useful to throw an exception on finding
>> something like a string literal that doesn't stop before EOF, but a text
>> editor or IDE would probably want to be a bit more lenient. Maybe having
>> it run-time (or compile-time configurable) like std.csv would be the
>> best option here.
>>
>> I'm interested in ideas on the API design and other high-level issues at
>> the moment. I don't consider this ready for inclusion. (The current
>> module being reviewed for inclusion in Phobos is the new std.uni.)
>
> Just thinking out loud here. Would it be possible to lex a file in
> parallel? Cutting it in half (or similar) and lex both pieces
> simultaneously in parallel.

why not only the symbols at the border needs to be "connected" then

the question is: how many blocks(threads) are ok for 1,2,3,8 cores - can 
also depend on the speed of the filesystem