Doost status update

[Marcin Kuszczak]

Dear D community!

I would like to give you some information about status of doost library, 
as it got few substantial updates recently.

*NEW MODULES*
*doost.storage.Storage/FileStorage*
I want to start from this one as this module as it introduces concept 
which is used in other modules. Basically Storage is continuous array of 
elements of same type. String and array of ints are also kinds of storage.
The difference is that Storage can be much bigger than computer memory. 
Storage can be also mapped to real physical devices such as File, 
Network connection etc. In fact it is very similar concept to Stream, 
but it has additional requirement that all elements of Storage are of 
the same type. Storage have narrower set of operations than arrays. 
Supported operations are:
- checking if element with given index is in storage or outside of 
stream of elements - eos
- consuming first or few first elements from storage - get
- peeking into any element in storage (only forward) - peek
- it is also possible to operate directly on 'working window' - frame

You can move across storage in one direction and operate on 'working 
window'. First element of storage have always index of '0'. Below simple 
visualization of storage:
sink -----------|O--working window--O|----------source
O - optional buffers for input and output

It might look complicated in a first look but, it's not in fact. Using 
this module it is possible to write all operations on arrays in a such a 
way that it will be possible to use any other Storage than string and 
have it read data from e.g. file.

In storage module there are definitions of above operations for strings, 
so you can use:
char[] text;
text.put("blabla");
char c= text.get()
if (text.eos(10)) throw new Exception("Error");

When using in you program you can template it in such a way that it will 
work nevertheless user will put as Storage string or e.g. FileStorage, 
which will read data from file.
When using for strings there should be no performance penalty as all 
functions should be inlined Unfortunately DMD does not optimize 
functions which takes arguments by ref. (See: bugzilla issue: 2008).
There is one concrete Storage implemented for files in package
doost.storage.

This module is already quite important in my library, but I still think 
it needs some more conceptual work.

*doost.text.Scanner*
Module Scanner is intended to be used for parsing texts. It consists from
set of functions, which uses internally only one operation: scan. Every
function scan returnes struct Matcher, which keeps information about
results of scanning: status of scanning, error message, matched position,
index of match (in case of matching against sets of sequences or sets of
characters).

position - position in input
repeatno - how many times repeat scanning; for uint.max functions scan 
for more than zero matches
All below functions works also with instances of Storages (see above).

Following template functions are available:
Matcher scan(input, CharClass, position, repeatno)  //CharClass -> 
basically checks if input from 'position' begins with one of characters 
defined in CharClass
Matcher scan(input, SequenceSet, position, repeatno) //SequenceSet -> 
same as CharClass, but for sequences
Matcher scan(input, RegExp, position, repeatno)
Matcher scan(input, array, position, repeatno)
Matcher scan(input, element_of_array, position, repeatno)

For below functions condition can be any of types for scan  + Matcher;
uint skip(ref T input, S condition, uint repeatno=1) // returns number 
of skipped characters
StorageType!(T) munch(T, S)(ref T input, S condition, uint repeatno=1) 
// consumes characters/sequences/etc. returning them from function

Additional high level functions:
Matcher scanEscapeSequence - scans for escape sequence
Matcher scanString - scans for string
Matcher scanRegion - scans for region; region is delimited by matching 
characters e.g. [5,5,4]
Matcher scanNumber - scans for number; needs some more work
Matcher scanDate - scans for number; needs some more work

Scanner uses internally among others CharClass by Jascha Wetzel from Tango
ported to Phobos. Module needs some additional work to make it feature
complete.

*doost.text.Escaper*
This is quite simple module, which should be used for 
escaping/unescaping strings. It should work for every escape sequence 
defined by D. It has only two functions:

escape(input)
unescape(input)

Escaper module should work also for Storage (see above).

*doost.util.serializer*
This is automatic serialization library. Currently it has two compile 
time 'backends': TextArchive and JsonArchive.
Some of most important features:
- can work with arrays and as well with storages
- highly customizable
- thread/exception safe
- tracking of references (classes/pointers)
- user defined types versioning
- importing of old classes
- user defined constructors
- user defined loaders/dumpers
- different archive classes

There is quite big suite of unit tests for serialization. You can see
serializer features in usage there. Serializer works also with storages
(see above).

*doost.api.Common/doost.core*
- doost.api is meant as something similar to tango.groups; Common is for 
commonly used definitions/imports etc.
- doost.core: Any (not many changes here), Traits (templates mostly 
taken from Phobos/Tango)


*UPDATED MODULES*
*doost.util.DUnit (doost.DUnit)*
This module was improved to make tests visualization in console more
readable and usefull. Currently it shows nice table with all necessary
informations. You can set width of this table, you can say if you want to
get timing or just result of unit test, you can also disable one or more
tests. It is possible to define groups of unittests. There are printed some
useful statistics at the end of table. Traces for tests are shown only in
case of failures. 

*doost.util.config.ProgramOptions*
Program Options now internally uses serializer to store information. It
means that every type you will use to configure your program can be
automatically stored as text. I think that this module needs some redesign.
It is especially visible after introducing serializer, which obsolete few
features and force to change implementation. Currently few unit tests (not
most important) doesn't pass, but making them work needs rethinking
architecture and redesign. I will keep it for later :-)

*WORK IN PROGRESS*
Currently I am working on database package. Idea is to make operation on 
database (almost) completely type-safe. I want to achieve this using Sql 
Object pattern. Thanks to this it will be possible to send such an 
object to database layer and get recordset as result.

*HOW CAN YOU HELP (if you feel it is worthy)*
- comment about design - especially about interfaces and use-cases.
- prepare patches and attach them as tickets
- documentation: it is rather sparse. I will be happy to get some help here
- adding unit tests - fixing bugs
- adding support for Tango. In this I would hope for some help from 
Tango team especially with Storage concept.
- improving build system. Currently I am using CodeBlocks, but its far from
perfect solution. There is a need for better/more universal builder. Right
now it should be dsss, later maybe something other.
- making Sqlite database from DDBI project working in latest revision 
:-) It's terribly broken right now.

Few modules of doost have dependencies on ddbi and std2, so you will need to
play a little bit with your compiler configuration. I used ddbi revision
62, as trunk version is broken for Sqlite. I successfully run doost
libraries on Tango with Tangobos (I had to add string aliases to
object.di), but doost itself doesn't contain any dependencies on Tango.
CodeBlocks project files might not work in your environment, so your
probably will have to adjust them to.

If you think, you can help with some tasks regarding library please let me
know e.g. by e-mail.

-- 
Regards
Marcin Kuszczak (Aarti_pl)
-------------------------------------
Ask me why I believe in Jesus - http://www.zapytajmnie.com (en/pl)
Doost (port of few Boost libraries) - http://www.dsource.org/projects/doost/
-------------------------------------