Formal review of std.lexer

[Andrej Mitrovic]

On 2/21/14, Joseph Cassman <jc7919 at outlook.com> wrote:
> 1. StringCache is a custom hash table. It looks like it's primary
> role is to reduce some sort of duplication.

It's used for string interning.

> 3. I tried to understand the reason for and usage of the
> "extraFields" parameter in "TokenStructure" but couldn't figure
> it out. Could some more explanation of its intent and usage be
> provided?

You could used it to inject a toString method. Here's what I did when I used it:

alias Token = TokenStructure!(TokID,
q{
    /// Better string representation
    void toString(scope void delegate(const(char)[]) sink) const
    {
        import std.conv;
        import %s;

        sink("(");
        sink(this.line.to!string);
        sink(":");
        sink(this.column.to!string);
        sink(")");
        sink(": ");
        sink(this.text ? this.text : tokToString(this.type));
    }
}.format(__MODULE__)
);

Side-note: Note how I've had to inject an import statement to my own
module because the string is mixed-in at the declaration site of a
module which I can't modify, where IIRC tokToString didn't exist
because it's located in *my* module. It's interesting how you can use
this feature in D, IOW:

-----
module mymodule;
import other_module;
void foo() { }
mixin_elsewhere!("import mymodule; foo();");
-----

-----
module other_module;
void mixin_elsewhere(string str)()
{
    mixin(str);
}
-----

P.S. the DLexer links here: https://github.com/Hackerpilot/lexer-demo/

I've used it to learn how to use DLexer, here's my more-documented
version of the above if it helps anyone, which also takes the D
parser's whitespace function and uses that in order to track column
numbers, it injects a toString like I've mentioned, and adds some
static asserts for sanity checks:

https://github.com/AndrejMitrovic/lexer-demo