Serialization woes

Andrej Mitrovic andrej.mitrovich at gmail.com
Tue Apr 17 19:56:46 PDT 2012


On 4/17/12, Jacob Carlborg <doob at me.com> wrote:
> Feel free to use my serialization library and see what it can handle:
> https://github.com/jacob-carlborg/orange

I was happy with serialization (using verious serialization libs) for
a while until I started using larger and more complex data structures.

I didn't have success with orange or ae's json or msgpack-d. While
trying to save a large hash-based data structure AE runs out of
memory, but I can't blame it since it's very simplistic in nature and
tries to generate a giant string.

Orange keeps consuming memory and spinning a single core. I've stopped
waiting after 5 minutes and over 2 Gigs of used memory. It seems to
work ok for small data structures though.

On the other hand, msgpack-d can save the structure in only 2 seconds
(sweet) and creates a 30 meg file, but somehow it takes over 80
seconds to unserialize it. :/

On a sample project it takes me 11 seconds to reparse 9000 xml files
and then do string processing to manually populate the fields of this
structure. So that's 70 seconds faster than using msgpack just
unserialize pre-parsed data. The point of using serialization for me
was to speed up the loading of the app (so I wouldn't lose 11 seconds
reparsing XML files over and over again). But that goes out the water
if I have to wait over a minute instead.

I really don't understand how msgpack is that much slower
unserializing data. The structure itself has a lot of hashes that lead
to other structures, e.g.:

struct Foo
{
    Bar[string];
    Boo[string]
}

struct Bar
{
   Doo[string];
}

etc etc.. maybe hashes are to blame? Who knows..

Are there any other serialization libs out there I could try? Maybe Thrift?


More information about the Digitalmars-d mailing list