Request for comments: std.d.lexer

[Philippe Sigaud]

On Sun, Jan 27, 2013 at 11:42 AM, Brian Schott <briancschott at gmail.com> wrote:
> On Sunday, 27 January 2013 at 10:17:48 UTC, Philippe Sigaud wrote:
>>
>> * Having a range interface is good. Any reason why you made byToken a
>> class and not a struct? Most (like, 99%) of range in Phobos are
>> structs. Do you need reference semantics?
>
>
> It implements the InputRange interface from std.range so that users have a
> choice of using template constraints or the OO model in their code.

Hmm. You're the first person I see pushing this. Personally, I'd go
for a struct, if only because I don't need reference semantics.

>> * Also, is there a way to keep comments? Any code wanting the modify
>> the code might need them.
>> (edit: Ah, I see it: IterationStyle.IncludeComments)
>>
>> * I'd distinguish between standard comments and documentation
>> comments. These are different beasts, to my eyes.
>
>
> The standard at http://dlang.org/lex.html doesn't differentiate between
> them. It's trivial to write a function that checks if a token starts with
> "///", "/**", or "/++" while iterating over the tokens.

Yes but, the standard lexer was done with DMD in mind, and DMD has a
different code path for generating comments.

It's your project, sure, but I'd appreciate tokens differentiating
between the many comments.
Oh, and recognizing some inner Ddoc tokens, like ---- delimiters for
documentation code blocks. That way, code-in-doc could use the lexer
also.

Pretty please?


>> * I see Token has a startIndex member. Any reason not to have a
>> endIndex member? Or can and end index always be deduced from
>> startIndex and value.length?
>
>
> That's the idea.

Does it work for UTF-16 and UTF-32 strings?