Looking for champion - std.lang.d.lex

[Walter Bright]

Nick Sabalausky wrote:
> "Walter Bright" <newshound2 at digitalmars.com> wrote in message 
>> Does Goldie have (like Spirit) a set of canned routines for things like 
>> numeric literals?
> 
> No, but such things can easily be provided in the docs for simple 
> copy-paste. For instance:
> 
> DecimalLiteral = {Number} ({Number} | '_')*
> 
> HexLiteral = '0' [xX] ({Number} | [ABCDEFabcdef_])+
> 
> Identifier = ('_' | {Letter}) ('_' | {AlphaNumeric})*
> 
> {StringChar} = {Printable} - ["]
> StringLiteral = '"' ({StringChar} | '\' {Printable})* '"'
> 
> All one would need to do to use those is copy-paste them into their grammar 
> definition. Some sort of import mechanism could certainly be added though, 
> to allow for selective import of pre-defined things like that.

In the regexp code, I provided special regexes for email addresses and URLs. 
Those are hard to get right, so it's a large convenience to provide them.

Also, many literals can be fairly complex, and evaluating them can produce 
errors (such as integer overflow in the numeric literals). Having canned ones 
makes it much quicker for a user to get going.

I'm guessing that a numeric literal is returned as a string. Is this string 
allocated on the heap? If so, it's a performance problem. Storage allocation 
costs figure large when trying to lex millions of lines.


> There are many pre-defined character sets though (and others can be 
> manually-created, of course): 
> http://www.devincook.com/goldparser/doc/grammars/character-sets.htm
> 
>> Can the D version of Goldie be turned into one file?
> 
> Assuming just the library and not the included tools (many of which could be 
> provided as part of the library, though), and not counting files generated 
> for the static-style, then yes, but it would probably be a bit long.

Long files aren't a problem. That's why we have .di files! I worry more about 
clutter.