std.d.lexer requirements

[Dmitry Olshansky]

On 04-Aug-12 15:48, Jonathan M Davis wrote:
> On Saturday, August 04, 2012 15:32:22 Dmitry Olshansky wrote:
>> I see it as a compile-time policy, that will fit nicely and solve both
>> issues. Just provide a templates with a few hooks, and add a Noop policy
>> that does nothing.
>
> It's starting to look like figuring out what should and shouldn't be
> configurable and how to handle it is going to be the largest problem in the
> lexer...
>

Let's add some meat to my post.
I've seen it mostly as follows:

//user defines mixin template that is mixed in inside lexer
template MyConfig()
{
	enum identifierTable = true; // means there would be calls to 
table.insert on each identifier
	enum countLines = true; //adds line, column properties to the lexer/Tokens

	//statically bound callbacks, inside one can use say:
	// skip() - to skip a char (popFront)
	// get() - to read next char (via popFront, front)
	// line, col - as readonly properties
	// (skip & get do the counting if enabled)
	
	bool onError()
	{
		skip(); //the most dumb recovery, just skip a char
		return true; //go on with tokenizing, false - stop prematurely
	}

	...
}

usage:


{
	auto my_supa_table = ...; //some kind of container (should a  set on 
strings and support .insert("blah"); )

	auto dlex = Lexer!(MyConfig)(table);
	auto all_tokens = array(dlex(joiner(stdin.byChunk(4096))));

	//or if we had no interest in table but only tokens:
	auto noop = Lexer!(NoopLex)();
	...		
}

-- 
Dmitry Olshansky