Keeping a list of instances and garbage-collection

[bearophile]

grauzone:
> First, I doubt this actually works. [...] To actually hide the pointer from the GC, you could XOR the size_t value with a constant.<

This is may be a stupid idea: Can't the OP just allocate with std.c.stdlib.malloc a block of void* pointers (plus keep an int length too), fill them with the object references and and then cast one of them back to object reference when necessary? Objects of such class can keep a similarly C-heap pointer to the cell of the block that contains its reference, and set it to null when they are removed.
It's not a general solution yet and it looks a bit messy.

Weak references may just need to be added to Phobos/Tango GC, if not present.

Bye,
bearophile