Allocating large 2D array in D

[monarch_dodra]

On Monday, 4 February 2013 at 15:23:20 UTC, Sparsh Mittal wrote:
> I am allocating 2d array as:
>
> double[gridSize][gridSize] gridInfo;
>
> which works for small dimension, but for large dimension, 16Mb 
> limit comes.
>
> Would you please tell me how do allocate a large 2d array 
> (which has to be done as a dynamic array)? It is a square grid 
> and dimensions are already known.
>
> I also searched internet but could not find an answer. Thanks.

If all (but last of) the dimensions are known at compile time, 
then you can dynamically allocate an array of fixed sized arrays:

//----
enum size_t gridSize = 4_000;
enum size_t total   = gridSize * gridSize;
static assert (total == 16_000_000); //16 million doubles total
static assert (total * double.sizeof == 128_000_000); //126 Megs 
allocated

void main()
{
     double[gridSize][] gridInfo = new 
double[gridSize][](gridSize);
}
//----

This will give you a dense array: Eg: all the data is contiguous 
in memory.

You can also use the "convenient 2D notation":

//----
import std.stdio;
enum gridSize = 4;
enum total   = gridSize * gridSize;

void main()
{
     double[][] gridInfo = new double[][](gridSize, gridSize);
}
//----

DO NOTE though that in this case, you have a (probably compact 
but) jagged array. What this means is that instead of having one 
single solid block of data, you have a top level array, that 
references a second level of arrays.

EG: It is (should be) equivalent to:
//----
enum size_t gridSize = 4_000;
enum size_t total   = gridSize * gridSize;

void main()
{
     double[][] gridInfo  = new double[][](gridSize); //Create an 
array of empty array;

     //Initialize gridInfo. Scope block to avoid polution
     {
         double[]   megaBlock = new double[](total); //Create all 
the data
         foreach (i ; 0 .. gridSize) //Iterate on each gridInfo 
entry
         {
             size_t offset = i * gridSize;
             gridInfo[i] = megaBlock[offset .. offset + 
gridSize];//Connect the arrays;
         }
     }
}
//----

It's not a big deal, but indexing *might* be a little slower with 
this scheme.