Handling invalid UTF sequences

[Brad Anderson]

On Thursday, 20 March 2014 at 22:51:27 UTC, monarch_dodra wrote:
> On Thursday, 20 March 2014 at 22:39:47 UTC, Walter Bright wrote:
>> Currently we do it by throwing a UTFException. This has 
>> problems:
>>
>> 1. about anything that deals with UTF cannot be made nothrow
>>
>> 2. turns innocuous errors into major problems, such as DOS 
>> attack vectors
>> http://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences
>>
>> One option to fix this is to treat invalid sequences as:
>>
>> 1. the .init value (0xFF for UTF8, 0xFFFF for UTF16 and UTF32)
>>
>> 2. U+FFFD
>>
>> I kinda like option 1.
>>
>> What do you think?
>
> I had thought of this before, and had an idea along the lines 
> of:
> 1. strings "inside" the program are always valid.
> 2. encountering invalid strings "inside" the program  is an 
> Error.
> 3. strings from the "outside" world must be validated before 
> use.
>
> The advantage is *more* than just a nothrow guarantee, but also 
> a performance guarantee in release. And it *is* a pretty sane 
> approach to the problem:
> - User data: validate before use.
> - Internal data: if its bad, your program is in a failure state.
>

I'm a fan of this approach but Timon pointed out when I wrote 
about it once that it's rather trivial to get an invalid string 
through slicing mid-code point so now I'm not so sure. I think 
I'm still in favor of it because you've obviously got a logic 
error if that happens so your program isn't correct anyway (it's 
not a matter of bad user input).