Pegged: Syntax Highlighting

[Andrej Mitrovic]

On 3/27/12, Philippe Sigaud <philippe.sigaud at gmail.com> wrote:
> Nice one. Care to explain how you did it?

Sure. Currently the "editor" is just a viewer (can't edit text
ironically :p), and is a port of one of the lessons of Neatpad
(http://www.catch22.net/tuts/neatpad). It's win32-specific and later
lessons cover very platform-specific unicode stuff so I haven't really
bothered with the rest of the tutorial.

What I have is one large char[]/wchar[] buffer, I store indices to
newlines within this buffer and when I need to lex a certain line I
just pass a slice into DDMD's lexer based on the position of the
newlines. I then store the beginning of each token and its type (e.g.
{ index 5, TOK.TOKImport }) as an array for that specific line. It's
easy to paint a line this way.

But I do have a couple of issues. One is that I have no way to figure
out where empty spaces are and not just spaces within string literals.
The DDMD API only exposes the beginning of each token and not its
length. And the lexer doesn't tokenize empty spaces between real
tokens. So with a string like this:
import foo;

The space between 'import' and 'foo' ends up being treated as
'TOK.TOKImport'. It's not a big issue when I only have foreground
coloring (empty space won't be drawn), but when I have background
coloring I end up with this:
http://i.imgur.com/0wUcR.png

The other issue is that DDMD explicitly takes a char[] and not just
any input range. The WinAPI text-drawing APIs require UTF16 arrays
(the unicode-aware functions anyway), so I end up having to store two
buffers, one UTF8 and one UTF16.

I'm sure these issues can be fixed in DDMD though. With that being
said the paint routine only takes about ~150 microseconds to finish
which is pretty neat.

Anyway if you have a win32 box you can clone:
https://github.com/AndrejMitrovic/DNeatpad
Then run:
DNeatpad\WindowsAPI\build.bat
DNeatpad\ddmd\build.bat
DNeatpad\textview\build.bat

That last one builds the "neatpad" folder as well. Anyway I was just
doing this for fun I have no intention on writing text editors. :)