Splitting a string on multiple tokens

[Dmitry Olshansky]

On 11-Oct-12 06:40, ixid wrote:
> On Wednesday, 10 October 2012 at 02:21:05 UTC, jerro wrote:
>> On Wednesday, 10 October 2012 at 00:18:17 UTC, ixid wrote:
>>> Is there an effective way of splitting a string with a set of tokens?
>>> Splitter feels rather limited and multiple passes gives you an array
>>> of arrays of strings rather than an array of strings. I'm not sure if
>>> I'm missing an obvious application of library methods or if this is
>>> absent.
>>
>> You can use std.regex.splitter like this:
>>
>> auto r = regex(`,| |(--)`);
>> auto str = "string we,want--to,split";
>> writeln(splitter(str, r)); //will pring ["string", "we", "want", "to",
>> "split"]
>
> Thank you, though that removes the tokens and being varied those would
> be messy to replace. Is there a way that lets you cut on tokens and keep
> those tokens at the ends of the statements they cause to get cut? This
> seem like basic parsing features that are absent.
>
Well I guess something along these lines:
auto r = regex(`(?<=,| |(--))`);
auto str = "string we,want--to,split";
writeln(splitter(str, r));
//will print: ["string ", "we,", "want--", "to,", "split"]

And just in case - splitter doesn't copy anything, it just slices the 
original array.

If you meant to use this for tight loops like in a compiler then you 
really need something handcrafted for optimal speed.

-- 
Dmitry Olshansky