Serialization woes

[Robert Jacques]

On Tue, 17 Apr 2012 22:32:33 -0500, Andrej Mitrovic <andrej.mitrovich at gmail.com> wrote:

> On 4/18/12, Andrej Mitrovic <andrej.mitrovich at gmail.com> wrote:
>> On 4/17/12, Jacob Carlborg <doob at me.com> wrote:
>>> Feel free to use my serialization library and see what it can handle:
>>> https://github.com/jacob-carlborg/orange
>
> Simple test-case:
> http://pastebin.com/dcnm4nmA
>
> That's around 15k keys. It takes 31 seconds to serialize. And my
> structures are full of these hashes, so it would likely take an
> eternity to serialize my structures with Orange.
>
> Here's msgpack-d:
> http://pastebin.com/wsRQPiDJ
>
> 6.3 milliseconds to serialize. But it takes 32 milliseconds to
> unserialize, which is 5x slower. On big data structures this
> difference becomes quite noticeable.

Just to clarify, is

When orange was first posted, I noticed and reported some performance issues. I don't know if they've been fixed. I don't have time tonight to download/install msgpack-d, but here are some results for your benchmark on my own serializers:

JSON serialize: 0.331831 s
JSON deserialize: 0.243893 s

Binary serialize: 1.84782 s
Binary deserialize: 0.198327 s

On my old Intel Core2 Duo T7500 2.2 GHz Windows 7.

What I suspect is happening is two fold. First, XML is much more verbose than message pack. Second, both orange and my routines use an intermediate representation, which I don't recall message pack using. IRs do give you more power, but they also reduce performance, since they introduce a a second copy of the data.