What library functionality would you most like to see in D?

[Johann MacDonagh]

On 7/31/2011 5:57 AM, Jacob Carlborg wrote:
>> * Lexing and parsing:
>>
>> Standard facilities for these tasks could be very useful. Perhaps D
>> could get its own dlex and dyacc or some such tools. Personally, I
>> prefer sticking to LL(1), but LALR is generally more convenient and
>> flexible, and thus I'd suggest something YACC/ANTLR-like.
>>
>> (I know this doesn't have much to do with Phobos per se, but I figured
>> I'd mention it.)
>
> I think someone is working on this.

I've started on a port of DMD's lexer (not really a port ;) ):

https://github.com/jmacdonagh/phobos/compare/master...std.lang.d.lexer

Basically, you give it some string (string, wstring, or dstring), and it 
gives you a range of tokens back. The token has the type, a slice of the 
input that corresponds to the token, line / column, and a value (e.g. an 
integer constant).

Some features I'm planning:

1. Support D1 and D2.
2. Warnings and errors returned in the tokens. For example, if you use 
an octal constant for D2 code, it will correctly return an integer 
constant token with some kind of warning flag set and a message. In 
terms of errors, if the lexer hits "0xz012", it will return an error 
token for the slice "0xz" and then start lexing an integer constant 
"012". No exceptions, easy peasy.
3. CTFEable. Although I'll probably have to wait till the next DMD release.
4. Support any kind of character range. Not sure if people want to lex 
something that's not a string/wstring/dstring.

I'm glad this was brought up. I remember Walter's post last year asking 
for this module, but the conversation seemed to kill the idea. I started 
on this just for the fun of it, but then doubted whether Phobos wanted 
it. I feel that a hand written lexer / parser is going to be faster than 
something generated, but maybe I'm old fashioned.

Anyway, Jim, if you want to do this I can move on to something else. If 
you want, I can continue on. I didn't see a branch in your repo so I'm 
not sure what you've done.