suggestion of type: ustring

[spir]

On 03/20/2011 05:12 PM, Jesse Phillips wrote:
> ZY Zhou Wrote:
>
>>> It would be prohibitively expensive to be constantly validating strings.
>>
>> No, it would be much much cheaper, since there are only 2 cases the validating is
>> needed
>>
>> 1) when you convert char[] to ustring, in this case, the validating is necessary
>> 2) when you use split on ustring. but since ustring is guaranteed to be valid, the
>> validating only need to check 2 bytes of data (start and end), much cheaper than
>> validating the entire string.
>>
>> after that, all the other functions will no longer need to worry about invalid
>> utf8 string, as long as the parameter is ustring, no validating is needed.
>
> Honestly, so far the only time I had problems processing utf has been when someone stuck a stupid BOM[1] at the beginning of the file.
>
> Question, what is so hard about inserting validity checks[2] into your code just as you have described? This way you don't have to put them in contracts of all your functions.
>
> 1. http://en.wikipedia.org/wiki/Byte_order_mark
> 2. http://digitalmars.com/d/2.0/phobos/std_utf.html#validate

Anyway, std.utf.validate just tries to *decode*! (see function source below) 
(decode itself throws an exception when stepping on invalid utf)
So, it would more as efficient to just decode (which in D means the same as 
converting to dstring) at start, and work with strings of code points all along 
your process. In addition to validating only once, further operations can be 
much faster each time you need to operate at the level of code points.

void validate(S)(in S s) if (isSomeString!S)
{
     immutable len = s.length;
     for (size_t i = 0; i < len; )
     {
         decode(s, i);
     }
}

Denis
-- 
_________________
vita es estrany
spir.wikidot.com