std.container.Array/RefCounted(T) leaking memory?

[Jonathan M Davis]

On Wednesday, January 12, 2011 15:29:51 %u wrote:
> Sorry to bump this up, but is RefCounted(T) really leaking, or am I missing
> something? I would like to use this in my program, and I'm curious as to
> why no one responded, since if it's actually leaking, it would be an
> important issue.

There probably aren't all that many people who saw your post, and out of those 
who did, there are probably very few - if any - who have actually done much with 
RefCounted. It's fairly new.

There's at least one major bug on Array at the moment ( 
http://d.puremagic.com/issues/show_bug.cgi?id=4942 ). There are also several 
bugs having to do with destructors at the moment which could be causing you 
problems.

Now, even assuming that you're not seeing any problem with a destructor-related 
bug and that you're not hitting a known bug with Array, there are three things 
that you need to be aware of which would likely show high memory usage 
regardless:

1. Array uses an array internally, and there is some caching that goes on with 
regards to arrays that has to do with appending. This means that if you're 
dealing with large arrays, you could have several which haven't been garbage 
collected yet simply because they're cached. Steven Schveighoffer has talked 
about it in several posts, and he has done some work to improve the situation, 
but I'm not sure that any of it has been in a release yet.

2. The garbage collector does not currently run in its own thread. IIUC, it only 
gets run when you try and allocate memory. So, if you allocate a bunch of 
memory, and then you never try and allocate memory again, no memory will be 
collected, regardless of whether it's currently being used or not.

3. As I understand it, the current garbage collector _never_ gives memory back 
to the OS. It will reclaim memory that you're not referencing any longer so that 
it doesn't necessarily need to go grab more memory from the OS when you try and 
allocate something, but once the garbage collector has gotten a block of memory 
from the OS, it doesn't give it back. So, currently you will _never_ see the 
memory usage of a D program go down, unless you're explicitly using malloc and 
free instead of GC-allocated memory.

So, if you really want to be testing for leaks, you probably should testing for 
short-lived small arrays in a loop with lots of iterations or something similar. 
Testing a couple of large arrays will almost certainly mean that the memory 
usage will be high and that it won't drop. On the other hand, if you keep 
creating small arrays in a loop where they have no references outside the loop 
and could be collected after they go out of scope, then you have lots of arrays 
for the garbage collector to collect, and if it fails to properly collect them, 
_then_ you'll see the memory usage continue to rise and show that there's a 
leak. But at this point, using a few large arrays is almost certainly going to 
look like it's leaking.

I'm sure that at some point D's garbage collector will improve so that these 
issues don't exist or at least or definitely diminished, but fixing the GC is not 
exactly at the top of the TODO list, so it's not currently as performant as 
would be nice. Once more important stuff has been fixed though, it'll get its 
turn.

- Jonathan M Davis