Why is Json parsing slower in multiple threads?

[Steven Schveighoffer]

On 6/20/23 5:31 AM, Alexandre Bourquelot wrote:

> Thanks in advance, this has been annoying me for a couple of days and I 
> have no idea what might be the problem. Strangely enough I also have the 
> same problem when using `vibe-d` json library for parsing.

The issue, undoubtedly, is memory allocation. Your JSON parsers (both 
std.json and vibe-d) allocate an AA for each object, and parse the 
entire string into a DOM structure. The D GC has a single global lock to 
allocate memory -- even memory that might be on a free list. So the 
threads are all bottlenecked on waiting their turn for the lock.

Depending on what you want to do, like others here, I'd recommend a 
stream-based json parser. And then you also don't have to split it into 
lines.

If the goal is to build a huge representation of all the data, then 
there's not much else to be done, unless you want to pre-allocate. But 
you may end up having to drive that yourself.

I can possibly recommend, in addition to what others have mentioned, my 
jsoniopipe library.

-Steve