Can std.json handle Unicode?

[Jacob Carlborg]

On 2013-03-23 21:08, Jonathan M Davis wrote:

> Curious. According to this page ( http://www.aivosto.com/vbtips/control-characters.html ) both space and delete are ASCII control characters (though
> neither std.ascii nor C's iscntrl deem space to be a control character), but
> neither of them are control characters according to recent Unicode standards.
> This section on DEL
>
> http://www.aivosto.com/vbtips/control-characters.html#DEL
>
> seems to say that DEL should basically be ignored. It seems to think that NUL
> should be treated the same way (and basically complains that languages like C
> ever treated it as a terminator).
>
> If I look at the RFC for json ( http://www.rfc-editor.org/rfc/rfc4627.txt ),
> it specifically lists control characters as being U+0000 through U+001F, which
> does _not_ include DEL or _any_ Unicode-specific control character. So, using
> either std.ascii or std.uni's isControl would be wrong. It specifically needs
> to check whether a character is < 32 when checking for control characters.
>
> And the grammar rule for string is
>
>           string = quotation-mark *char quotation-mark
>
>           char = unescaped /
>                  escape (
>                      %x22 /          ; "    quotation mark  U+0022
>                      %x5C /          ; \    reverse solidus U+005C
>                      %x2F /          ; /    solidus         U+002F
>                      %x62 /          ; b    backspace       U+0008
>                      %x66 /          ; f    form feed       U+000C
>                      %x6E /          ; n    line feed       U+000A
>                      %x72 /          ; r    carriage return U+000D
>                      %x74 /          ; t    tab             U+0009
>                      %x75 4HEXDIG )  ; uXXXX                U+XXXX
>
>           escape = %x5C              ; \
>
>           quotation-mark = %x22      ; "
>
>           unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
>
> So, it looks like the only characters that should be considered valid inside
> the double-quotes of a string which aren't escaped are / (which indicates the
> beginning of an escape sequence), and the characters listed in unescaped. So,
> in decimal, that would be 32 and 33, 35 - 91, and everything 93 and greater
> (up to 10FFFF). DEL is 127, so it should be considered valid.
>
> So, if std.json is using isControl, my guess is that whoever wrote that was
> not careful enough with the grammar (though it's easy enough to assume that
> everyone means the same thing by control characters), and I'd be concerned
> that std.json is not handling this set of grammar rules correctly with more
> characters than just DEL.

I see. Yes, one could think that "control character" would mean the same 
thing in every situation for a given encoding.

-- 
/Jacob Carlborg