The Case Against Autodecode

Vladimir Panteleev via Digitalmars-d digitalmars-d at puremagic.com
Thu May 12 13:24:23 PDT 2016


On Thursday, 12 May 2016 at 20:15:45 UTC, Walter Bright wrote:
> On 5/12/2016 9:29 AM, Andrei Alexandrescu wrote:
> > I am as unclear about the problems of autodecoding as I am
> about the necessity
> > to remove curl. Whenever I ask I hear some arguments that
> work well emotionally
> > but are scant on reason and engineering. Maybe it's time to
> rehash them? I just
> > did so about curl, no solid argument seemed to come together.
> I'd be curious of
> > a crisp list of grievances about autodecoding. -- Andrei
>
> Here are some that are not matters of opinion.
>
> 1. Ranges of characters do not autodecode, but arrays of 
> characters do. This is a glaring inconsistency.
>
> 2. Every time one wants an algorithm to work with both strings 
> and ranges, you wind up special casing the strings to defeat 
> the autodecoding, or to decode the ranges. Having to constantly 
> special case it makes for more special cases when plugging 
> together components. These issues often escape detection when 
> unittesting because it is convenient to unittest only with 
> arrays.
>
> 3. Wrapping an array in a struct with an alias this to an array 
> turns off autodecoding, another special case.
>
> 4. Autodecoding is slow and has no place in high speed string 
> processing.
>
> 5. Very few algorithms require decoding.
>
> 6. Autodecoding has two choices when encountering invalid code 
> units - throw or produce an error dchar. Currently, it throws, 
> meaning no algorithms using autodecode can be made nothrow.
>
> 7. Autodecode cannot be used with unicode path/filenames, 
> because it is legal (at least on Linux) to have invalid UTF-8 
> as filenames. It turns out in the wild that pure Unicode is not 
> universal - there's lots of dirty Unicode that should remain 
> unmolested, and autocode does not play with that.
>
> 8. In my work with UTF-8 streams, dealing with autodecode has 
> caused me considerably extra work every time. A convenient 
> timesaver it ain't.
>
> 9. Autodecode cannot be turned off, i.e. it isn't practical to 
> avoid importing std.array one way or another, and then 
> autodecode is there.
>
> 10. Autodecoded arrays cannot be RandomAccessRanges, losing a 
> key benefit of being arrays in the first place.
>
> 11. Indexing an array produces different results than 
> autodecoding, another glaring special case.

12. The result of autodecoding, a range of Unicode code points, 
is rarely actually useful, and code that relies on autodecoding 
is rarely actually, universally correct. Graphemes are 
occasionally useful for a subset of scripts, and a subset of that 
subset has all graphemes mapped to single code points, but this 
only applies to some scripts/languages.

In the majority of cases, autodecoding provides only the illusion 
of correctness.



More information about the Digitalmars-d mailing list