serialization library
Paulo Herrera
pauloh81 at yahoo.ca
Sat Jan 6 02:07:20 PST 2007
On Thu, 09 Nov 2006 02:06:21 +0100, Bill Baxter
<dnewsgroup at billbaxter.com> wrote:
> Christian Kamm wrote:
>> Based on initial work from Tom S and clayasaurus, I've written this
>> serialization library. If hope something like this doesn't already
>> exist!
>
> Great!
>
>> http://www.math.tu-berlin.de/~kamm/d/serialization.zip
>> Currently, it only provides binary file io through the Serializer
>> class. It can
>> - write/read almost (hopefully) every type through a call to
>> Serializer.describe
>> - track class references and pointers by default
>> - serialize classes and structs through a templated 'describe' member
>> function
>> - write derived classes from base class reference*
>> - read derived classes into base class reference*
>> - serialize not default constructible classes*
>> (* for this to work, the class needs to be registered with the archive
>> type)
>> It has far less features than boost::serialization but is already in a
>> very usable state: FreeUniverse, a D game based on the Arc library,
>> uses it for writing and loading savegames as well as other persistant
>> state information.
>
> I'm using Boost::serialization but I'm not at all happy with it. But
> the things that I don't like mostly have to do with versioning, which it
> looks like you don't support anyway.
>
>> What it does not do/is missing:
>> - exception safety / multithread safety
>> - out-of-class/struct serialization methods (is it possible to check
>> whether a specific overload exists at compile time?)
>
> I could be mistaken but I think this is that ADL / Koenig Lookup
> territory that Walter doesn't want go into.
>
>> - static arrays need to be serialized with describe_staticarray (static
>> arrays can't be inout, so the general-purpose template method doesn't
>> work... is there a way around the problem?)
>> - things I forgot right now
>
> Endian issues?
>
>> Documentation is still rather sparse. This short example shows the
>> basic usage
>
>
> Just a wish list item, but I'd prefer an actual "file format" library as
> opposed to a serialization library. Maybe a file format library would
> build on top of the serialization library, but anyway, the key
> difference is that a serialization lib aims to turn *particular* data
> structures into a binary format that can be losslessly loaded back into
> the same data structure later.
>
> But that is not the way people design generic file formats, like say the
> Photoshop file format. Things like that need to be very extensible and
> shouldn't be tied to particular data structures. I think that's where
> boost::serialization gets into trouble. Once you start talking about
> versioning, you're no longer talking about one specific data structure.
>
> For instance Boost::serialization lacks a way to ignore blocks or skip
> chunks of data that are not recognized or obsolete. You actually have
> to load the obsolete thing into the proper (possibly obsolete) data
> structure and then delete the unnecessary thing you just created. This
> is not good from the forwards/backwards compatibility view. Old code
> simply cannot read the file (even if it understands the majority of the
> chunks that matter), and new code is forced to maintain old data
> structures just for the purpose of loading up obsolete data and throwing
> it away.
>
> How do you fix it? Very simple really. Just store the file as a series
> of chunks with fixed length headers, and each header contains the length
> of the data in that chunk. If you get a chunk header with a tag you
> don't understand, just ignore it. A particular chunk can have
> sub-chunks too. I think it's similar in many ways to a grammar
> definition:
>
> file:
> header chunklist
>
> chunklist:
> chunk
> chunk chunklist
>
> header:
> typeIndicator versionNumber DataEndianness
>
> chunk:
> chunkHeader data
>
> chunkHeader:
> chunkType DataLength
>
> data:
> // Here's where you list all the types of data known to you
>
> Or something like that.
> I'd like a library that helps me read and write my data in that sort of
> data-structure independent format.
>
> --bb
Take a look at the HDF file format that is used to serialize huge amounts
of scientific data.
It implements a format that is very similar to the one you described.
http://www.hdfgroup.org/
Paulo
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
More information about the Digitalmars-d-announce
mailing list