Tokenizing D at compile time?

[Timon Gehr]

On 08/26/2011 03:08 AM, dsimcha wrote:
> I'm working on a parallel array ops implementation for
> std.parallel_algorithm. (For the latest work in progress see
> https://github.com/dsimcha/parallel_algorithm/blob/master/parallel_algorithm.d
> ).
>
> To make it (somewhat) pretty, I need to be able to tokenize a single
> statement worth of D source code at compile time. Right now, the syntax
> requires manual tokenization:
>
> mixin(parallelArrayOp(
> "lhs[]", "=", "op1[]", "*", "op2[]", "/", "op3[]"
> ));

That is not real tokenization, can you go with

"lhs","[","]",","=",...

?

>
> where lhs, op1, op2, op3 are arrays.
>
> I'd like it to be something like:
>
> mixin(parallelArrayOp(
> "lhs[] = op1[] * op2[] / op3[]"
> ));
>
> Does anyone have/is there any easy way to write a compile-time D tokenizer?

I have written an almost complete tokenizer in D. Making it compile-time 
should be rather trivial, I will give it a try. (it will also convert 
embedded numerals to the correct type etc.)

If you don't need a complete tokenizer: What are the tokenizer features 
you need?