MessagePack for D released

[Masahiro Nakagawa]

On Mon, 26 Apr 2010 01:29:07 +0900, Michel Fortin  
<michel.fortin at michelf.com> wrote:

> Looks well done. There's one thing I'd suggest though. I'm pretty sure  
> you could make it even faster by skipping the mp_Object intermediary  
> representation and using templates. I know it's possible since I've done  
> it for a surprisingly similar serialization library I'm working on.
>
> The trick is to reuse the same pattern in the unpacker as you're already  
> using in the packer. For instance, the packer has this function:
>
>     ref Packer pack(T)(in T value) if (is(Unqual!T == long))
>
> so the unpacker could have this function (just changed 'in' by 'out'):
>
>     ref Unpacker unpack(T)(out T value) if (is(Unqual!T == long))
>
> My library works by unserializing everything directly a the right place  
> in a data structure while it parses the stream. Looks like this:
>
> 	MyStruct original;
> 	Archiver archiver;
> 	archiver.encode(original);
> 	immutable(byte)[] data = archiver.outout;
>
> 	MyStruct copy;
> 	Unarchiver unarchiver;
> 	unarchiver.input = data
> 	unarchiver.decode(copy);
>
> This is unlike mp_Object which is in itself an intermediary  
> representation that sits between the serialized data and the data  
> structure you actually want to rebuild. I still have something similar  
> to mp_Object as a convenience for types that prefer to implement a  
> custom unserialization process in an order not dictated by the input  
> stream, but this is less efficient:
>
> 	void decode(ref KeyUnarchiver archive) {
> 		archive.decode("var1", var1);
> 		archive.decode("var2", var2);
> 	}
Yeah, I know your approach(Protocol Buffers supports similar approach  
using schema).
Current implementation purposes separating deserialization and conversion.
D has Tuple and Variadic Templates, so I can rewrite execute method  
relatively easily if need.

> What I'm trying to put to work now is a way to deal with multiple  
> references to the same object.
MessagePack doesn't consider multiple references to same object.
This support slows down serialization / deserialization and makes stream  
deserializer difficult.
This problem poses little problem for product use.

> I'd also like a nice way to deal with Variant, but I'm under the  
> impression this won't be possible without adding serialization support  
> directly into Variant, or into TypeInfo.
I agree this point. This problem might be solved if D has ADL(but I don't  
like ADL).

> Masahiro, sorry: this started as a useful commentary on your  
> unserializer's approach and I ended up instead promoting what I am  
> doing. Your library seems targeted at making a MessagePack serializer,  
> with an emphasis on having a simple and portable serialization format,  
> which is great when you want to communicate in this format.
Your understanding is correct. MessagePack is designed at efficient, small,
and fast serialization(thus not perfect). I think this simplicity meets  
Phobos.

> But on my
> side, I care more about being able of recreating object graphs and  
> reinstantiating objects of the correct class when unserializing.
Perfect serialization needs environment support.
However, such serialization is very difficult if support.
(Cocoa, Java, and etc. Ruby's marshal is slow)
I don't know the library based perfect serialization.

> That does not seem possible with your library, and MessagePack doesn't  
> support this so it doesn't seem likely it can be added easily, am I  
> right?
You are right.
The author of MessagePack considers supporting class and reference  
extension.
But his conclusion is "There is no little merit in supporting".