Latest string_token Code

[bearophile]

Ben Hanson:
> Will the optimiser create as fast code if you use indexes compared to pointers?

Do you mean the dmd optimizer or the llvm one?
LLVM is able to digest array indexes and pointers about equally efficiently. In one important case dmd seems to produce slower code with pointers compared to arrays :-)
In some situations it's better to use pointers because they allow a richer semantics, but in many situations arrays give better-looking (and a bit safer) code.
If you have some free time you can create an array-based version and compare their performance. Otherwise if your original C++ code uses pointers then maybe it's better to keep them, to avoid translation bugs. You need lot of care to translate code between two different languages, you need a strong rigour.

Another small thing I've seen in your code:

> template regex(CharT)
> {
> struct BasicStringToken
> {
...
> };
> }

D struct definitions don't need an ending semicolon, so it's better to not add it in D code.

Bye,
bearophile