Growing multidimensional dynamic arrays

[KillerSponge]

Hi all,

This seems like something that should be possible: how do I grow 
multidimensional arrays?

I want something like this:

struct X{ ... };
X*[][] listOfLists;
foreach ( x ; otherListOfX ) {
    if ( newListForArbitraryReason ) {
       listOfLists ~= new X*[];
    }

    listOfLists[$] ~= x;
}

Now, this doesn't compile, because I _have_ to give a size to new 
X*[](arbitrary_number), and the listOfLists[$] ~= x; line never 
works (hangs at runtime).

So, how would I go about doing this? My apologies if this is 
something really obvious.