RFC: std.json sucessor

[Sönke Ludwig via Digitalmars-d]

Am 22.08.2014 20:08, schrieb Walter Bright:
> On 8/21/2014 3:35 PM, Sönke Ludwig wrote:
>> Destroy away! ;)
>
> Thanks for taking this on! This is valuable work. On to destruction!
>
> I'm looking at:
>
> http://s-ludwig.github.io/std_data_json/stdx/data/json/lexer/lexJSON.html
>
> I anticipate this will be used a LOT and in very high speed demanding
> applications. With that in mind,
>
>
> 1. There's no mention of what will happen if it is passed malformed JSON
> strings. I presume an exception is thrown. Exceptions are both slow and
> consume GC memory. I suggest an alternative would be to emit an "Error"
> token instead; this would be much like how the UTF decoding algorithms
> emit a "replacement char" for invalid UTF sequences.

The latest version now features a LexOptions.noThrow option which causes 
an error token to be emitted instead. After popping the error token, the 
range is always empty.

>
> 2. The escape sequenced strings presumably consume GC memory. This will
> be a problem for high performance code. I suggest either leaving them
> undecoded in the token stream, and letting higher level code decide what
> to do about them, or provide a hook that the user can override with his
> own allocation scheme.

The problem is that it really depends on the use case and on the type of 
input stream which approach is more efficient (storing the escaped 
version of a string might require *two* allocations if the input range 
cannot be sliced and if the decoded string is then requested by the 
parser). My current idea therefore is to simply make this configurable, too.

Enabling the use of custom allocators should be easily possible as an 
add-on functionality later on. At least my suggestion would be to wait 
with this until we have a finished std.allocator module.