Latest string_token Code

[Andrei Alexandrescu]

On 06/22/2010 08:13 AM, Ben Hanson wrote:
> Here's the latest with naming convention (hopefully) followed. I've implemented my
> own squeeze() function and used sizeof in the memmove calls.

I suggest you to look into using the range primitives (empty, front, 
back, popFront, and popBack) with strings of any width. Your code 
assumes that all characters have the same width and therefore will 
behave erratically on UTF-8 and UTF-16 encodings.

In the particular case of squeeze(), you may want to use uniq instead, 
which works on any forward range and will therefore decode characters 
properly:

http://www.digitalmars.com/d/2.0/phobos/std_algorithm.html#uniq


Andrei