std.d.lexer: pre-voting review / discussion

Johannes Pfau nospam at example.com
Wed Sep 11 11:43:51 PDT 2013


Am Wed, 11 Sep 2013 17:01:58 +0200
schrieb "Dicebot" <public at dicebot.lv>:

> std.d.lexer is standard module for lexing D code, written by 
> Brian Schott
> 

Question / Minor issue:

As we already have a range based interface I'd love to have partial
lexing / parsing, especially for IDEs.

Say I have this source code:
--------------------------------------------
1: module a;
2: 
3: void test(int a)
4: {
5:     [...]
6: }
7: 
8: void test2()
9: [...]
--------------------------------------------

Then I first do a full parse pass over the source. Now line 5 is being
edited. I know from the full parse that line 5 is part of a
FunctionDeclaration which starts at line 3 and ends at line 6. Now I'd
like to re-parse only that part:

--------------------------------------------
FunctionDeclaration decl = document.getDeclByLine(5);
decl.reparse(/*start_line=*/ 3, docBuffer);
--------------------------------------------

I think these are the two critical points related to this for the
proposed std.lexer:

* How can I tell the lexer to start lexing at line/character n? Of
  course the input could be sliced, but then line number and position
  information in the Token struct is wrong.
* I guess std.lexer slices the original input? This could make things
  difficult if the file buffer is edited in place. But this can
  probably be dealt with outside of std.lexer. (By reallocating all
  .value members)


(And once this is working, an example in the docs would be great)

But to be honest I'm not sure how important this really is. I think it
should help for more responsive IDEs but maybe parsing is not a
bottleneck at all?


More information about the Digitalmars-d mailing list