How to free memory allocated via double[][] using dmd-2.0.12?

[Markus Dittrich]

BCS Wrote:

> Markus Dittrich wrote:
> > Hi,
> > 
> > For a data processing application I need to read a large number
> > of data sets from disk. Due to their size, they have to be read and 
> > processed sequentially, i.e. in pseudocode
> > 
> > int main()
> > {
> >     while (some condition)
> >     {
> >          double[][] myLargeDataset = read_data();
> >          process_data(myLargeDataset);
> >          // free all memory here otherwise next cycle will 
> >          // run out of memory
> >      }
> >     
> >    return 0;
> > }
> > 
> > Now, the "problem" is the fact that each single data-set saturates
> > system memory and hence I need to make sure that all memory
> > is freed after each process_data step is complete. Unfortunately,
> > using dmd-2.012 I have not been able to achieve this. Whatever
> > I do (including nothing, i.e., letting the GC do its job), the resulting 
> > binary keeps accumulating memory and crashing shortly after). 
> > I've tried deleting the array, setting the array lengths to 0, and manually
> > forcing the GC to collect, to no avail. Hence, is there something I am 
> > doing terribly wrong or is this a bug in dmd?
> > 
> > Thanks much,
> > Markus
> 
> One "hack" would be to have read_data() allocate a big buffer and then 
> slice the parts of the double[][] out of it. This would have the 
> advantage that you can just keep track of the buffer and on the next 
> pass just reuse it in it's entirety, you never have to delete it.
> 
> double[][] read_data()
> {
> 	static byte[] buff;
> 	if(buff.prt is null) buff = new byte[huge];
> 
> 	byte left = buff;
> 
> 	T[] Alloca(T)(int i)
> 	{
> 		T[] ret = (cast(*T)left.prt)[0..i];
> 		buff = buff[i*T.sizeof..$];
> 		return ret;
> 	}
> 
> 	/// code uses Alloca!(double) and Alloca!(double[]) for
> 	/// allocations. Don't use .length or ~=
> 
> 	
> }

Thanks much for you response! I could certainly role my own buffer
management. Unfortunately, the "real" app is more complicated 
than the "proof of concept" code I posted and doing so would require a bit
more work. After all, the main reason for using D for this type of thing
was the fact that I didn't want to deal with manual memory management ;)