Request for review - std.serialization (orange)

[Jacob Carlborg]

On 2013-03-30 21:02, Jesse Phillips wrote:
> Hello Jacob,
>
> These comments are based on looking into adding Protocol Buffer as an
> archive. First some details on the PB format.
> https://developers.google.com/protocol-buffers/docs/overview
>
> 1) It is a binary format

That shouldn't be a problem. Preferably it should support some kind of 
identity map and be able to deserialize fields in any order.

> 2) Not all D types can be serialized

Any data format that supports some kind of key-value mapping should be 
able to serialize all D types. Although, possibly in a format that is 
not idiomatic for that data format. XML doesn't have any types and the 
XML archive can serialize any D type.

> 3) Serialization is done by message (struct) and not by primitives

I'm not sure I understand this.

> 4) It defines options which can affect (de)serializing.

While Orange doesn't support the options Protocol Buffer seems to use 
directly, it should be possible by customizing the serialization of a 
type. See:

https://github.com/jacob-carlborg/orange/wiki/Custom-Serialization
https://github.com/jacob-carlborg/orange/wiki/Non-Intrusive-Serialization

Alternatively they are useful enough to have direct support in the 
serializer.

> I am looking at using Serializer to drive (de)serialization even if that
> meant just jamming it in there where Orange could only read PB data it
> has written. Keep in mind I'm not saying these are requirements or that
> I know what I'm talking about, only my thoughts.

That should be possible. I've been working a binary archive that tries 
to be as fast as possible, breaking rules to the left and right, doesn't 
conform to the implicit contract between the serializer and archive and 
so on.

> My first thought was at a minimum I could just use a function which does
> the complete (de)serialization of the type. Which would be great since
> the pbcompiler I'm using/modifying already does this.
>
> Because of the way custom serialization I'm stopped by point 3. I didn't
> realize that at first so I also looked at implementing an Archive. What
> I notice here is
>
> * Information is lost, specifically the attributes (more important with
> UDA).

Do you want UDA's passed to the archive for a give type or field? I 
don't know how easy that would be to implement. It would probably 
require a template method in the archive, which I would like to avoid, 
since it wouldn't be possible to use via an interface.

> * I am required to implement conversions I have no implementation for.

Just implement an empty method for any method you don't have use for. If 
it needs to return a value, you can most of return typeof(return).init.

> This leaves me concluding that I'd need to implement my own Serializer,
> which seems to me I'm practically reimplementing most of Orange to use
> Orange with PB.

That doesn't sound good.

> Does having Orange support things like PB make sense?

I think so.

> I think some work could be done for the Archive API as it doesn't feel
> like D2.

It started for D1.

> Maybe we could look at locking down custom Archive/Serializer
> classes while the internals are worked out (would mean XML
> (de)serialization is available in Phobos).

-- 
/Jacob Carlborg