std.data.json formal review

[via Digitalmars-d]

On Monday, 17 August 2015 at 22:21:50 UTC, Andrei Alexandrescu 
wrote:
> * stdx.data.json.generator: I think the API for converting 
> in-memory JSON values to strings needs to be redone, as follows:
>
> - JSONValue should offer a byToken range, which offers the 
> contents of the value one token at a time. For example, "[ 1, 
> 2, 3 ]" offers the '[' token followed by three numeric tokens 
> with the respective values followed by the ']' token.

For iterating tree-like structures, a callback-based seems nicer, 
because it can naturally use the stack for storing its state. (I 
assume std.concurrency.Generator is too heavy-weight for this 
case.)

>
> - On top of byToken it's immediate to implement a method (say 
> toJSON or toString) that accepts an output range of characters 
> and formatting options.

If there really needs to be a range, `joiner` and `copy` should 
do the job.

>
> - On top of the method above with output range, implementing a 
> toString overload that returns a string for convenience is a 
> two-liner. However, it shouldn't return a "string"; Phobos APIs 
> should avoid "hardcoding" the string type. Instead, it should 
> return a user-chosen string type (including reference counting 
> strings).

`to!string`, for compatibility with std.conv.

>
> - While at it make prettyfication a flag in the options, not 
> its own part of the function name.

(That's already done.)

>
> * stdx.data.json.lexer:
>
> - I assume the idea was to accept ranges of integrals to mean 
> "there's some raw input from a file". This seems to be a bit 
> overdone, e.g. there's no need to accept signed integers or 
> 64-bit integers. I suggest just going with the three character 
> types.
>
> - I see tokenization accepts input ranges. This forces the 
> tokenizer to store its own copy of things, which is no doubt 
> the business of appenderFactory. Here the departure of the 
> current approach from what I think should become canonical 
> Phobos APIs deepens for multiple reasons. First, 
> appenderFactory does allow customization of the append 
> operation (nice) but that's not enough to allow the user to 
> customize the lifetime of the created strings, which is usually 
> reflected in the string type itself. So the lexing method 
> should be parameterized by the string type used. (By default 
> string (as is now) should be fine.) Therefore instead of 
> customizing the append method just customize the string type 
> used in the token.
>
> - The lexer should internally take optimization opportunities, 
> e.g. if the string type is "string" and the lexed type is also 
> "string", great, just use slices of the input instead of 
> appending them to the tokens.
>
> - As a consequence the JSONToken type also needs to be 
> parameterized by the type of its string that holds the payload. 
> I understand this is a complication compared to the current 
> approach, but I don't see an out. In the grand scheme of things 
> it seems a necessary evil: tokens may or may not need a means 
> to manage lifetime of their payload, and that's determined by 
> the type of the payload. Hopefully simplifications in other 
> areas of the API would offset this.

I've never seen JSON encoded in anything other than UTF-8. Is it 
really necessary to complicate everything for such an infrequent 
niche case?

>
> - At token level there should be no number parsing. Just store 
> the payload with the token and leave it for later. Very often 
> numbers are converted without there being a need, and the 
> process is costly. This also nicely sidesteps the entire matter 
> of bigints, floating point etc. at this level.
>
> - Also, at token level strings should be stored with escapes 
> unresolved. If the user wants a string with the escapes 
> resolved, a lazy range does it.

This was already suggested, and it looks like a good idea, though 
there was an objection because of possible performance costs. The 
other objection, that it requires an allocation, is no longer 
valid if sliceable input is used.

>
> - Validating UTF is tricky; I've seen some discussion in this 
> thread about it. On the face of it JSON only accepts valid UTF 
> characters. As such, a modularity-based argument is to pipe UTF 
> validation before tokenization. (We need a lazy UTF validator 
> and sanitizer stat!) An efficiency-based argument is to do 
> validation during tokenization. I'm inclining in favor of 
> modularization, which allows us to focus on one thing at a time 
> and do it well, instead of duplicationg validation everywhere. 
> Note that it's easy to write routines that do JSON tokenization 
> and leave UTF validation for later, so there's a lot of 
> flexibility in composing validation with JSONization.

Well, in an ideal world, there should be no difference in 
performance between manually combined tokenization/validation, 
and composed ranges. We should practice what we preach here.

> * stdx.data.json.parser:
>
> - FWIW I think the whole thing with accommodating BigInt etc. 
> is an exaggeration. Just stick with long and double.

Or, as above, leave it to the end user and provide a `to(T)` 
method that can support built-in types and `BigInt` alike.