Maximum array size

[Dave]

Ivan Kazmenko wrote:
> Hi again!
> 
> me:
>>> What is the 16M limit on the array length for?
>>> I wonder what's the 'reason' on that limit.
> 
> Lionello Lunesu:
>> I've put the same limit in my own array template I use for C++ projects. 
>>  My reasoning was that you shouldn't be using a dynamic array for such 
>> huge memory blocks because of the potential costly resize. The limit 
>> helped me catch some bugs where array were being used and caused a 
>> significant performance bottleneck. I rewrote those parts with a simple 
>> malloc/free or VirtualAlloc. In D there might be the added complexities 
>> of the garbage collector. Remember that it keeps scanning the allocated 
>> memory for pointers. For these huge blocks it's better to use some other 
>> memory allocation scheme.
> 
> I definitely won't want to resize such a huge array, at least not too often. My
> array is static. For resizing, I usually use the technique of doubling the
> requested index when it exceeds the current limit.
> 
> If my too-many-M array is scanned for pointers, which I do not store there, do I
> have a way of forbidding it?
> 
> Dave:
>> Why the need for such a large static array? Can't a dynamic array be used in
>> your code instead?
> 
> I could use dynamic array, yes, but I don't see any benefit since its size is
> fixed. Please explain why it could be better.
> 
> The current problem which requires such a big array is related to prime numbers
> - it's a program for a programming contest at http://recmath.org/contest/ (you
> know, various contests are the best way to learn something new, if you have
> enough free time, of course). I use a sort of Eratosphenes sieve for prime
> numbers, and the larger the sieve, the faster the program runs (for large
> numbers, it uses Miller-Rabin primality test which is significantly slower than
> seeking a bit in an array, with paging, caching, and other modern RAM techniques
> included). I work with a teammate, we share ideas but write different
> implementations. He uses Delphi, and I decided to try D, but currently my
> implementation sucks 'cause I have no 'legal' way to use an array that big.
> 
> Current problem aside, there are many ways to use my current amount of RAM,
> which is 768M, indexing it with only a couple of arrays. It's not the first time
> I face the restriction even working on the current problem!
> 
> Ivan Kazmenko.

Instead of

T[25_000_000] myarray;

just do

T[] myarray = new T[25_000_000];

and that will take care of the immediate problem you describe and let 
you get on with solving the problem.

Example:

import std.stdio, std.string;

int main(char[][] args)
{
     bool[] flags = new bool[25_000_000];
     int  count;

     count = 0;
     flags[] = 1;
     for(int i = 2; i < flags.length; i++)
     {
         if(flags[i])
         {
             // remove all multiples of prime: i
             for(int j = i + i; j < flags.length; j += i) flags[j] = 0;
             count++;
         }
     }

     writefln("Count: ",count);

     return 0;
}

- Dave