Neat project: add pointer capability to std.bitmanip.bitfields

[Vladimir Panteleev via Digitalmars-d]

On Thursday, 3 December 2015 at 00:31:46 UTC, Andrei Alexandrescu 
wrote:
> Nice. What I'd say is that at the end of the day there's 
> documentation. -- Andrei

Just to provide a bit of perspective...

Although memory corruption may not seem so scary in short-lived 
programs where it can be trivially reproduced with the same 
input, it can be an absolute nightmare when it occurs in 
long-running server processes, for developers, sysadmins and 
end-users alike.

A long time ago, a network service I wrote started having severe 
memory corruption issues. It went from crashing about once a day 
to once every few hours, and every time it crashed it pissed off 
a dozen users or more. Great ire and vitriol[1] was expressed 
towards the service, and everything I tried only seemed to make 
the situation worse.

It took months of studying and playing with D GC internals (incl. 
unsucessful attempts to use the GC's own debugging code and 
additional debugging GC proxies - see Diamond), and finally after 
three nights of replay debugging a virtual machine which recorded 
a crash on my home PC, and trying to infer meaning from memory 
dumps of GC control structures, I've tracked down the bug. The 
final result was the addition of InvalidMemoryOperationError to 
Druntime.

So, this is why I am on a war campaign against anything that 
might result in memory corruption in D. Another recent example 
was the controversy over readln (yes, std.stdio.File.readln, one 
of the basic operations in any programming language) corrupting 
memory: sure, this patch will make D programs no longer crash and 
burn in weird ways, but it will also make readln slower! Think of 
the benchmarks! Thankfully a solution was found which was both 
safe and acceptably fast.

You said that "at the end of the day there's documentation". I 
would argue that at least in this case, it may not be enough. 
Consider, for example, a hypothetical user type "Pack", which 
takes a tuple/struct and automatically arranges the fields into a 
struct such that space is used optimally (e.g. all bools are 
clumped together into a bitfield, enums are only given as much 
bits as their .max needs, etc.). Pack only needs to know one 
property of each field: how many bits it really needs, and as 
such, it might elect to be agnostic of what a pointer is. If it 
uses std.bitmanip.bitfields as its backend, it will happily pack 
a pointer at its user's request, and the user will never see the 
pointer warning in Pack's documentation. And yes, although it's 
easy to point the finger at users and say "ha ha, it's your own 
fault, you did not RTFM", I think we should strive for better 
than that.

I was recently close to having a repeat of the memory corruption 
situation (with the same service, too) due to the struct 
alignment issue. Luckily there was only one week of strife before 
I narrowed down the bug. Memory corruption may only manifest in 
certain conditions (e.g. release mode, after updating/switching 
compilers), and rollback/bisect are not always viable or useful 
options. Even if the issue I filed had been fixed at the time, it 
would not have helped in that particular case, since, well, D is 
unsafe by default, and it is situations such as these that make 
me occasionally glance in Rust's direction.

[1]: 
http://dump.thecybershadow.net/41a97cdab29f9cd56340ce8a5163d6f8/rage.txt