How to reuse the space of RAM of an array of strings?

H. S. Teoh hsteoh at quickfur.ath.cx
Fri Dec 21 21:28:14 UTC 2018


On Fri, Dec 21, 2018 at 08:41:10PM +0000, Giovanni Di Maria via Digitalmars-d-learn wrote:
[...]
> After to have inserted dynamically 5_000_000 strings to array, for six
> times, i measure the RAM with an utility (for example Wise Memory
> Optimizer), deleting the array every time.
> 
> Every time I destroy the array, but the RAM is always less.
> In particular:
> 
> - at beginning the FREE RAM is 1564 MB;
> - after first loop the FREE RAM IS 1480 MB
> - after second loop the FREE RAM IS 1415 MB
> - after third loop the FREE RAM IS 1402 MB
> - after forth loop the FREE RAM IS 1338 MB
> - after fifth loop the FREE RAM IS 1280 MB
> - after sixth loop the FREE RAM IS 1200 MB
> - at end the FREE RAM returns to 1564 MB
> 
> I want to reuse the dynamic array.
[...]

You need to understand that generally speaking, the GC does not return
memory to the OS; it retains that memory for future use, e.g., when you
allocate a new object.  Also, not all objects may be freed immediately
if the GC can still find references to it somewhere, including on the
stack.  And if you're in a 32-bit environment, there's a higher chance
you might run into the problem of false pointers (non-pointer values
that look like a valid pointer into an allocated object, causing the GC
to think the object is still live when it's actually already dead).
Furthermore, when allocating large numbers of objects the GC may create
additional internal data structures to keep track of said objects, and
may continue to retain these structures after the objects have been
collected.

If you want deterministic deallocation, your best bet is to use
malloc/free instead of the GC (that means you'll need to avoid
allocating operators like ~, ~=, array literals, and so on). Using @nogc
may help here.

A more reliable test is if you let your loop run forever, and see if the
memory usage eventually plateaus. It should stop increasing at a certain
point once you've reached steady state; if not, that would be stronger
proof of memory leakage.

Furthermore, if you want to reuse existing memory for the array, you
should just write to it directly instead of deallocating / reallocating.
I.e., instead of doing this:

	foreach (i; 0 .. 1_000_000)
	{
		auto arr = new int[1_000_000];
		... // do stuff with arr
		arr = null; // or arr.destroy or whatever
	}

do this:

	auto arr = new int[1_000_000]; // only allocate once
	foreach (i; 0 .. 1_000_000)
	{
		... // overwrite any previous contents of arr with new data
		... // do stuff with arr
	}

For your array of strings, you probably want to use a string buffer /
pool somewhere, i.e., a contiguous block of memory that you allocate
once, and take slices of as you populate your string array, instead of
allocating once per array element.  This will give you better control
over your memory usage, and also reduce GC load (so collection cycles
will generally be faster, since there's less stuff that needs to be
marked and collected).


T

-- 
Give a man a fish, and he eats once. Teach a man to fish, and he will sit forever.


More information about the Digitalmars-d-learn mailing list