Bit subscriptions, RAM columnwise layouts

[bearophile]

This is the home page of the Wirbel language:
http://mathias-kettner.de/wirbel.html

This page shows something interesting:
http://mathias-kettner.de/wirbel_int_bitwise.html

>From the page:
<<
Subscription operators
Wirbel lets you use an integer a little bit like a list of boolean values with the fixed length of 32 or 64. You can used the square brackets to directly address the bits:

a = 0
a[0] = true
a[2] = false
print(a)

This will output 5 since this first bit (value 1) and the third bit (value 4) have been switched on. You can also query arbitrary bits:

if a[4]:
    print("Bit 4 is on -> a contains a 16")

Splices [slices] won't currently work but may follow in a later release of Wirbel. 
>>

For D the following is an alternative syntax seems a little more explicit:
int a = 156;
a.bits[0] = true;
a.bits[0..5] = 0;

Instead of:
int a = 156;
a[0] = true;
a[0..5] = 0;


Wirbel also has built-in bit sets, a bit as Pascal, but they are just 64-bit long:
http://mathias-kettner.de/wirbel_bitsets.html
They support the usual set operators (almost like the Python sets).

--------------------

DataDraw is a very interesting library for C, it can be used as a RAM-database:
http://datadraw.sourceforge.net/

This is a PDF that explains lot of things:
http://datadraw.sourceforge.net/manual.pdf

(Beside the things I talk about here it has few other features, like optional persistence on HD and undo).

The authors show that it can often have performance higher than C/C++ code. Such performance comes mostly from two things (it's explained at pages 25-27 of the PDF):
- On 64-bit CPUs/OSes when possible it uses 32-bit indexes instead of 64 bit pointers (when data is a lot, I think it automatically uses 64 bit indexes), this usually reduce the total amount of memory used, and this interacts well with caches.
- By default it uses parallel arrays, that is if you create an array of struct-like things, it doesn't actually keep them as an array of structs, but as many arrays of single items. So the programmer doesn't specify how the system lays and uses memory, and usually lets the system do by itself. There is a also syntax to say that some or all fields of such records have to actually be stored closely, this is useful for the less common situations when you need to process more than a field at a time). This often leads to better usage of the cache, and in the end speeds up several programs.

So today the caches have changed a little how optimal code has to be written in C. The authors say that a downside of C is that it forces the programmer to do such memory layout manually, and today this may lead to less performance.

In this regard I think it's like the difference from classes and structs in D: while in D structs are PODs, where the fields are where you want them to be, D classes order their fields as they like.
So I think it can be conceived some way in D to allow the compiler to perform such memory layouts as DataDraw does (as I've said DataDraw also allows a more manual memory layout, see the cache_together directive of page 25-26 of the PDF).

Note that today such shifting from record-wise layouts to column-wise ones is happening in true databases too, especially ones that are used for data mining, for performance reasons, because they interact better with caches, even the CPU caches too. They have even invented hybrid things, like PAX, look especially the figure4 at page 4 here:
http://www.cs.wisc.edu/multifacet/papers/vldb01_pax.pdf

Bye,
bearophile