Performance of std.json

[Jonathan M Davis via Digitalmars-d]

On Mon, 02 Jun 2014 00:18:18 +0000
David Soria Parra via Digitalmars-d <digitalmars-d at puremagic.com> wrote:

> Hi,
>
> I have recently had to deal with large amounts of JSON data in D.
> While doing that I've found that std.json is remarkable slow in
> comparison to other languages standard json implementation. I've
> create a small and simple benchmark parsing a local copy of a
> github API call
> "https://api.github.com/repos/D-Programming-Language/dmd/pulls"
> and parsing it 100% times and writing the title to stdout.
>
> My results as follows:
>    ./d-test > /dev/null  3.54s user 0.02s system 99% cpu 3.560
> total
>    ./hs-test > /dev/null  0.02s user 0.00s system 93% cpu 0.023
> total
>    python test.py > /dev/null  0.77s user 0.02s system 99% cpu
> 0.792 total
>
> The concrete implementations (sorry for my terrible haskell
> implementation) can be found here:
>
>     https://github.com/dsp/D-Json-Tests/
>
> This is comapring D's std.json vs Haskells Data.Aeson and python
> standard library json. I am a bit concerned with the current
> state of our JSON parser given that a lot of applications these
> day use JSON. I personally consider a high speed implementation
> of JSON a critical part of a standard library.
>
> Would it make sense to start thinking about using ujson4c as an
> external library, or maybe come up with a better implementation.
> I know Orvid has something and might add some analysis as to why
> std.json is slow. Any ideas or pointers as to how to start with
> that?

It's my understanding that the current design of std.json is considered
to be poor, but I don't haven't used it, so I don't know any the
details. But if it's as slow as you're finding to be the case, then I
think that that supports the idea that it needs a redesign. The
question then is what a new std.json should look like and who would do
it. And that pretty much comes down to an interested and motivated
developer coming up with and implementing a new design and then
proposing it here. And until someone takes up that torch, we'll be
stuck with what we have. Certainly, there's no fundamental reason why
we can't have a lightening fast std.json. With ranges and slices,
parsing in D in general should be faster than C/C++ (and definitely
faster than Haskell of python), and if it isn't, that indicates that
the implementation (if not the whole design) of that code needs to be
redone.

I know that vibe.d uses its own json implementation, but I don't know
how much of that is part of its public API and how much of that is
simply used internally: http://vibed.org

- Jonathan M Davis