Finalizing D2

[Andrei Alexandrescu]

Denis Koroskin wrote:
> On Sat, 23 May 2009 22:20:14 +0400, Andrei Alexandrescu 
> <SeeWebsiteForEmail at erdani.org> wrote:
> 
>>>> * std.socket, std.socketstream: We need a real networking library.
>>>   what would it do on top of what that does?
>>
>> I haven't studied it, but Walter said he doesn't like it and I trust 
>> him. Anyhow, we'd need to create at least full range integration and 
>> support for protocols such as http, ftp, ssh, and imap. Today's 
>> languages load a webpage in one line, and that's great so we need to 
>> do that. It's even better to be able to process the webpage while it's 
>> loading (concurrency!), so we want to do that as well.
>>
>>
>> Andrei
> 
> We wrote a networking library with a unique modern flexible design.
> 
> It was initially written in C++, but I'm slowly porting it to D2 (it's 
> already usable and I wrote a few applications with it by now).
> 
> If anyone is interested, I may contribute it to Phobos. Its design 
> overview (very short one) is attached for those who are interested.
> 

Sounds great! The doc comes off as binary, so I'm pasting it below for 
others' convenience.

Andrei


There are two main concepts in "net" (which is a name of the library):

- Link
	Link is an establish connection between two computers. It is a very 
simple interface that has two important methods - void 
send(const(void)[] data) and void disconnect().

- Driver
	Driver is something that creates Links and transfer data between them.

Each driver has a set orthogonal properties (some guaranties that they 
provide):

- Reliable Boundary
	Indicates that packet boundary is guaranteed by the driver.
     This means, if packet with size N is sent and recieved on the other 
side,
     it is recieved as one packet with size N, neither splitted into 
several portions nor
	merged with other packets. Stream protocols like TCP often doesn't 
support reliable
	packet boundaries, datagrams like UDP often does support this feature.

- Reliable Content
	Indicates that packet consistency is guaranteed by the driver.
     This means, sent data is recieved without corruption of content. 
All corrupted data
     will be filtered out by the driver that supports this feature.

- Reliable Order
	Indicates that packet order is guaranteed by the driver.
     This means, packets will never change their order is driver supports
     this feature.
	
- Reliable Delivery
	Indicates that packet delivery is guaranteed by the driver.
     This means, while connection is still alive, any data sent is
     recieved on the other side (maybe after some time, but will be).
	
etc.

These are called driver capabilities. If a driver doesn't have some 
property which is important for your application (for example, content 
reliability, or packet order), you can create a proxy-driver that will 
externally add missing feature. This is one of the main ideas behind 
Drivers: they should be easily "decorable" (compoundable). Other example 
is, if your driver doesn't compress data automatically, you may easily 
wrap it with some driver that supports data compression.

Networking library provides a set of cross-platform proxies that provide 
any of the required features. Here is an incomplete list of implemented 
Driver Proxies (in addition to Proxies that fulfil requirements above - 
consistency, order, etc)

  - FastCompression Driver
		Compresses traffic before sending it over network

  - Local Driver
		A kind of "loop back" driver. "Sends" data within address space of the 
single application (no data copying ever occurs)

  - Signature Driver
		"Signs" every outgoing packet and filters out packets with wrong 
signature

  - Statistics Driver
		Gathers statistics on transferred data (number of lost packets, 
out-of-order packets, damaged packets, bytes sent/received, etc)

  - Timeout driver
		Automatically disconnects when a specified timeout is reached
		
Future work:
  - Encryption Driver
		

Why was it important for us? We develop games for embedded devices 
(think of game consoles, pocket pcs, phones etc). Some of them have very 
primitive hardware and software. For example, some of them don't 
implement BSD Sockets, have no TCP or UDP driver (*very* common case) 
etc. This is why our networking library doesn't rely on any of these 
features, although they are used when available. All that is needed is a 
simple ability to transfer data in *any* way. Everything is else 
configurable externally by our library. For example, our library 
provides cross-platform implementation of TCP over UDP.

You decide what features you create driver with depending on your needs. 
For example, when developing turn-based strategy, it is not very 
important to have ultra-low traffic, and ease of development is of more 
importance. In this case you may request all of the features and 
simplify your code dramatically.

Sometimes you need to connect over some specific protocol, such as TCP 
or UDP (for example, access a web-page over HTTP). In this case, you 
request some concrete driver implementation.

Drivers are created using factory methods like the following:

Driver createDriver(uint requiredDriverCaps, ...);
Driver wrapDriver(Driver hostDriver, uint requiredDriverCaps, ...);
Driver createTcpServerDriver(bool async, ushort listeningPort, ...);
Driver createTcpClientDriver(bool async, ushort defaultDestinationPort, 
...);
Driver createUdpDriver(bool async, ushort listeningPort, ushort 
defaultDestinationPort, ...);

etc

Packet Processing

Whenever a driver receives new packet, it unwinds it (some proxy drivers 
may add additional data to packets - checksum, packet index, etc - or 
completely modify it - encryption, compression, etc) and passes to the 
corresponding link. If a programmer wants to handle packets that come 
from links, he subscribes to them:

	// Using a listener
	link.addIncomingPacketProcessor(this);
	// Using a delegate
	link.addIncomingPacketProcessor(&someMethod);

These callbacks may be invoked either in main thread (synchronous, 
during implicit driver.processIncomingPackets() call) or in other thread 
(asynchronous) - behavior is specified during Driver intiialization.

Creating a new link is as simple as:

	auto link = driver.createLinkTo(host, port);

A new valid link is always returned even though the connection may not 
be establish immediately (non-blocking operation). You can start using 
it (sending data etc) without waiting until connection fully establishs. 
Notification callback will be invoked if connection fails.

Some drivers may emit new Links. Whenever new connection is received, a 
link is created and passed to Listeneres. You subscribe to Driver events 
the same way you do it with Links - driver.addIncomingLinksProcessor(...);

All the operations are inherently asynchronous (non-blocking). For 
example, there is no method link.receive() that waits until a link 
receives any packet (although it's often very handy, but mostly for 
prototyping). Operations like this implemented using helpers.

This is a core functionality, the one that "drives" the development. 
Everything else (HTTP/FTP/SSH connections, etc) needs to be built on top 
of the core functionality in a cross-platform manner.

Other cool feature that is implemented using our library is remote 
procedure call (which is very helpful, not only for debugging). In 
short, you may remotely invoke almost any method on any object with any 
arguments and get result back. D compile-time reflection capabilities 
will significantly simplify porting this.