splitter trouble

[John Colvin via Digitalmars-d-learn]

On Sunday, 30 October 2016 at 23:57:11 UTC, Ali Çehreli wrote:
> While working on a solution for Alfred Newman's thread, I came 
> up with the following interim solution, which compiled but 
> failed:
>
> auto parse(R, S)(R range, S separators) {
>     import std.algorithm : splitter, filter, canFind;
>     import std.range : empty;
>
>     static bool pred(E, S)(E e, S s) {
>         return s.canFind(e);
>     }
>
>     return range.splitter!pred(separators).filter!(token => 
> !token.empty);
> }
>
> unittest {
>     import std.algorithm : equal;
>     import std.string : format;
>     auto parsed = parse("_My   input.string", " _,.");
>     assert(parsed.equal([ "My", "input", "string" ]), 
> format("%s", parsed));
> }
>
> void main() {
> }
>
> The unit test fails and prints
>
> ["put", "ing"]
>
> not the expected
>
> ["My", "input", "string"].
>
> How is that happening? Am I unintentionally hitting a weird 
> overload of splitter?
>
> Ali

As usual, auto-decoding has plumbed the sewage line straight in 
to the drinking water...

Splitter needs to know how far to skip when it hits a match. 
Normally speaking - for the pred(r.front, s) overload that you're 
using here - the answer to that question is always 1. Except in 
the case of narrow strings, where it's whatever the encoded 
length of the separator is in the encoding of the source range 
(in this case utf-8), in order to skip e.g. a big dchar.* But in 
your case, your separator is more than one character, but you 
only want to skip forward one, because your separator isn't 
really a separator.

* see 
https://github.com/dlang/phobos/blob/d6572c2a44d69f449bfe2b07461b2f0a1d6503f9/std/algorithm/iteration.d#L3710

Basically, what you're doing isn't going to work. A separator is 
considered to be a separator, i.e. something to be skipped over 
and twisting the definition causes problems.

This will work, but I can't see any way to make it @nogc:

auto parse(R, S)(R range, S separators) {
     import std.algorithm : splitter, filter, canFind;
     import std.range : save, empty;

     return range
         .splitter!(e => separators.save.canFind(e))
         .filter!(token => !token.empty);
}