[GSoC] Improved FlatBuffers and/or Protobuf Support ~ Binary Serialization

[Ahmet Sait]

Hi,
I've been thinking about working on binary serialization as my 
potential GSoC
project. It's originally one of the older entries in GSoC ideas 
page [0]. I
think D is pretty much cut out for this kind of task and 
serialization is a
topic I'm rather interested in so hopefully it will be a great 
candidate.

# About me:
My name is Ahmet Sait Koçak, currently studying Computer Science 
in Turkey. I
first met with D way back in high school which in itself is an 
interesting
story.

I've been introduced to programming in first year in high school 
with C for
some sort of competition. I then proceeded to learn C# myself and 
continued
coding as a hobby, having lots of fun. One of my biggest projects 
was LF2 IDE
[1] - a modding tool for the game LF2. It didn't take long for me 
to fall in
love with open source since that project made use of several OSS 
libraries
itself which led me to using version control, embracing git and 
open sourcing
nearly anything I code on GitHub from there on.

Fast forward 4 years I was hitting walls trying to do low level 
stuff in C#
and the fact that bytecode compiled languages being too easy to 
reverse
engineer was hindering my motivation to do anything commercial 
with them. I
never liked C++ but gave it another try telling myself "come on 
it's not that
bad" but failed miserably, there had to be a better way. Besides, 
I was
already crafting my dream language in my head.

One day, I sat in front of my computer and thought "I bet there 
is a language
called D". It was once in a life time magical moment reading 
through the home
page and seeing how it is the same language if I were to create 
one (static
reflection, native compiled, GC...). My first project in D was 
IDL [2] - it
made it possible for LF2 IDE to hot reload modded data files into 
the game's
memory, it was amazing working with slices for the first time. 
I'm a D user
and an evangelist ever since.

# Overview:
- Improving & updating D implementation of flatbuffers and/or 
protobuf
- Contributing the D support to the upstream repositories
- Better documentation & samples
- Benchmarking and making sure D rocks

# Key Points:
- Meta-programming (DbI, CTFE, mixin...)
I plan to make D meta features shine in this library.

   - It should be possible to parse schema and output mixable D 
code at
     compile time
   const schema = `message Person
   {
       required string name = 1;
       required int32 id = 2;
   }`;
   mixin(fromProtoSchema(schema));

   - There should be no need for a schema definition, a custom 
type annotated
     with UDAs should be enough
   struct Person
   {
       @protoID(1) string name;
       @protoID(2) int age;
   }
   serialize(Person("Walter", 42), stdout);

- Simple things should be simple
It should be dead simple to do basic stuff:
   auto obj = deserialize!SomeType(stdin);
   serialize(obj, stdout);

- Complex things should be possible
The library should be flexible and extensible without modification

- Support for library and tool based usage
It should be usable as a library without any additional setup but 
also usable
as a schema compiler.

- Support for common Phobos types
Nullable, tuples, std.datetime, std.complex, std.bigint, 
containers...

Existing work:
https://github.com/huntlabs/flatbuffers
https://github.com/dcarp/protobuf-d
https://github.com/msoucy/dproto

I'm personally not happy with any of the existing libraries but 
they will
likely be a valuable resource regardless.

Questions:
- How much work would be ideal for GSoC? Should I be working on 
flatbuffers
   only or protobuf too? (Seems like flatbuffers need more love)
- Should I tackle the std.serialization [3] idea?
- Any other serialization related suggestions?
- Anything I'm missing?

I'm still not entirely sure about my project (probably gonna 
write a few
proposals) so if you have other suggestions do not hesitate. All 
kinds of
constructive feedback is welcome!


[0] 
https://wiki.dlang.org/GSOC_2018_Ideas#FlatBuffers_Support_and.2For_Improved_Protocol_Buffer_Support
[1] https://github.com/ahmetsait/LF2.IDE
[2] https://github.com/ahmetsait/IDL
[3] https://wiki.dlang.org/GSOC_2019_Ideas#std.serialization