how to handle very large array?

[Era Scarecrow]

On Thursday, 10 February 2022 at 01:43:54 UTC, H. S. Teoh wrote:
> On Thu, Feb 10, 2022 at 01:32:00AM +0000, MichaelBi via 
> Digitalmars-d-learn wrote:
>> thanks, very helpful! i am using a assocArray now...
>
> Are you sure that's what you need?

  Depends. if you do say TYPE[long/int] then you have effectively 
a sparse array, if there's a few entries (*Say a hundred million 
or something*) it will probably be fine.

  Depending on what you're storing, say if it's a few bits per 
entry you can probably use bitarray to store differing values. 
The 10^12 would take up.... 119Gb? That won't work. Wonder how 
25Gb was calculated.

  Though data of that size sounds more like a database. So maybe 
making index-access that does file access to store to media might 
be better, where it swaps a single 1Mb block (*or some power^2 
size*) doing read/writes. Again if the sparseness/density isn't 
heavy you could then get away using zram drive and leaving the 
allocating/compression to the OS so it all remains in memory 
(*Though that's not going to be a universal workaround and only 
works on linux*), or doing compression using zlib on blocks of 
data and swapping them in/out handled by the array-like object, 
but that i'm more iffy on.

  If the array is just to hold say results of a formula, you could 
then instead do a range with index support to generate the 
particular value and uses very little space (*depending on the 
minimum size of data needed to generate it*) though it may be 
slower than direct memory access.