GC conservatism -- again

[Ulrik Mikaelsson]

> I have posted about this problem several times. Never got any replies.
> I think memory management is D's "elephant in the room" in this regard.
I'm sorry to agree. I recently brought up this general question in a
thread called "Mixing GC and non-GC in D." in both druntime and here.
Only got one answer each in both druntime and this group. The problem
basically, from all I've read about GC:s in general and the D GC in
particular, is the problem of non-deterministic collection, both when
and more importantly IF the object will ever be collected. This makes
the GC unsuitable for "expensive" resources, such as file descriptors,
or very big chunks of memory.

The problem I've encountered in D, is that support for complementary
predictable allocation schemes which could alleviate some of the
problems (i.e. reference-counting and tree-allocation), is quite weak.
By weak, I mean undocumented and no supporting framework from the
stdlib. In a perfect world, this could even work hand-in-hand with the
GC, such that references to refcounted-object could be voided from the
destruction of the reference-counted object.

I understand these schemes are possible by some manual allocator and
GC-root trickery, but this is really something I, as a fairly new
d-programmer, expect from the stdlib/runtime of a language like D.
Implementing this for guarding memory "correctly" requires quite a lot
of understanding of destructors and the GC, that productive developers
should not need to know. For refcounting (which is the scheme I've had
time to really think about), I propose:
 - an interface "IRefCounted", for any resource that should be refcounted.
 - a ready implementation for memory allocation "RefCounted", which
allocates from a GC-scanned but non-GC-controlled pool
 - for D2 (not possible for D1 due to missing struct-dtor:s) a
C++-style "SmartRef" template-struct, which encapsulates a reference
to a typed IRefCounted, automatically releasing on reference-change or
struct-destruction.
 - Documentation on how to use the "RefCounted" implementation, as
well as implementing other resource-guards (I.E. for opening/closing
FD:s)

Is this a discussion that should be held in Phobos/Tango, druntime, or
on this list?

> 1) Diamond, the memory debugger that helps trace leaks to their source. I
> recently added a feature long overdue - show the exact shortest path of
> pointers / heap objects that cause a specified object to remain uncollected.
Great! I've been unsuccessfully looking for such a thing for D, good
to know it exists.

> 2) data.d - a module to allow simple handling of large amounts of plain
> data. The gist is that a simple proxy object keeps a pointer to a malloc'd
> region, which is free'd when the proxy object is collected. Works like a
> charm - practically no leaks (the proxy object is tiny), and keeps the
> managed heap small so collections run quickly.
Considering the "don't count on collection"-principle I mentioned
above, is this approach safe, especially if other parts of the program
grows large in address-space?