Large memory allocations

[Kagamin]

lol, quot damage!

Janderson Wrote:

> bearophile wrote:
> > While allocating lot of memory for a little memory-hungry program, I have found results that I don't understand. So I have written the following test programs. Maybe someone can give me some information on the matter.
> > I am using a default install of a 32 bit Win XP with 2 GB RAM (so for example I can't allocate 3 GB of RAM). (I presume answers to my questions are Windows-related).
> > 
> > From C (MinGW 4.2.1) this is about the largest memory block I can allocate (even it swaps and requires 7+ seconds to run), 1_920_000_000 bytes:
> > 
> > #include "stdio.h"
> > #include "stdlib.h"
> > #define N 480000000
> > int main() {
> >     unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned int));
> >     unsigned int i;
> >     if (a != NULL)
> >         for (i = 0; i < N; ++i)
> >            a[i] = i;
> >     else
> >         printf("null!");
> >     return 0;
> > }
> > 
> > 
> > But from D this is about the largest memory block I can allocate with std.c.stdlib.malloc, 1_644_000_000 bytes, do you know why the difference?
> > 
> > //import std.gc: malloc;
> > import std.c.stdlib: malloc;
> > void main() {
> >     const uint N = 411_000_000;
> >     uint* a = cast(uint*)malloc(N * uint.sizeof);
> >     if (a !is null)
> >         for (uint i; i < N; ++i)
> >            a[i] = i;
> >     else
> >         printf("null!");
> > }
> > 
> > (If I use std.gc.malloc the situation is different yet, and generally worse).
> > 
> > -----------------------
> > 
> > So I have tried to use a sequence of smaller memory blocks, this is the C code (every block is about 1 MB):
> > 
> > #include "stdio.h"
> > #include "stdlib.h"
> > 
> > #define N 250000
> > 
> > int main(int argc, char** argv) {
> >     unsigned int i, j;
> >     unsigned int m = argc == 2 ? atoi(argv[1]) : 100;
> > 
> >     for (j = 0; j < m; ++j) {
> >         unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned int));
> > 
> >         if (a != NULL) {
> >             for (i = 0; i < N; ++i)
> >                a[i] = i;
> >         } else {
> >             printf("null! %d\n", j);
> >             break;
> >         }
> >     }
> > 
> >     return 0;
> > }
> > 
> > 
> > And the D code:
> > 
> > //import std.gc: malloc;
> > import std.c.stdlib: malloc;
> > import std.conv: toUint;
> > 
> > void main(string[] args) {
> >     const uint N = 250_000;
> >     uint m = toUint(args[1]);
> > 
> >     for (uint j; j < m; ++j) {
> >         uint* a = cast(uint*)malloc(N * uint.sizeof);
> > 
> >         if (a !is null) {
> >             for (uint i; i < N; ++i)
> >                a[i] = i;
> >         } else {
> >             printf("null! %d\n", j);
> >             break;
> >         }
> >     }
> > }
> > 
> > With such code I can allocate 1_708_000_000 bytes from D and up to 2_038_000_000 bytes from C (but near the last 100-200 MB of RAM the C code swaps a lot).
> > So can't I can't use all my RAM from my D code? And do you know why?
> > 
> > Bye,
> > bearophile
> 
> 
> bearophile wrote:
>  > While allocating lot of memory for a little memory-hungry program, I 
> have found results that I don't understand. So I have written the 
> following test programs. Maybe someone can give me some information on 
> the matter.
>  > I am using a default install of a 32 bit Win XP with 2 GB RAM (so for 
> example I can't allocate 3 GB of RAM). (I presume answers to my 
> questions are Windows-related).
>  >
>  > From C (MinGW 4.2.1) this is about the largest memory block I can 
> allocate (even it swaps and requires 7+ seconds to run), 1_920_000_000 
> bytes:
>  >
>  > #include "stdio.h"
>  > #include "stdlib.h"
>  > #define N 480000000
>  > int main() {
>  >     unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned int));
>  >     unsigned int i;
>  >     if (a != NULL)
>  >         for (i = 0; i < N; ++i)
>  >            a[i] = i;
>  >     else
>  >         printf("null!");
>  >     return 0;
>  > }
>  >
>  >
>  > But from D this is about the largest memory block I can allocate with 
> std.c.stdlib.malloc, 1_644_000_000 bytes, do you know why the difference?
>  >
>  > //import std.gc: malloc;
>  > import std.c.stdlib: malloc;
>  > void main() {
>  >     const uint N = 411_000_000;
>  >     uint* a = cast(uint*)malloc(N * uint.sizeof);
>  >     if (a !is null)
>  >         for (uint i; i < N; ++i)
>  >            a[i] = i;
>  >     else
>  >         printf("null!");
>  > }
>  >
>  > (If I use std.gc.malloc the situation is different yet, and generally 
> worse).
>  >
>  > -----------------------
>  >
>  > So I have tried to use a sequence of smaller memory blocks, this is 
> the C code (every block is about 1 MB):
>  >
>  > #include "stdio.h"
>  > #include "stdlib.h"
>  >
>  > #define N 250000
>  >
>  > int main(int argc, char** argv) {
>  >     unsigned int i, j;
>  >     unsigned int m = argc == 2 ? atoi(argv[1]) : 100;
>  >
>  >     for (j = 0; j < m; ++j) {
>  >         unsigned int* a = (unsigned int*)malloc(N * sizeof(unsigned 
> int));
>  >
>  >         if (a != NULL) {
>  >             for (i = 0; i < N; ++i)
>  >                a[i] = i;
>  >         } else {
>  >             printf("null! %d\n", j);
>  >             break;
>  >         }
>  >     }
>  >
>  >     return 0;
>  > }
>  >
>  >
>  > And the D code:
>  >
>  > //import std.gc: malloc;
>  > import std.c.stdlib: malloc;
>  > import std.conv: toUint;
>  >
>  > void main(string[] args) {
>  >     const uint N = 250_000;
>  >     uint m = toUint(args[1]);
>  >
>  >     for (uint j; j < m; ++j) {
>  >         uint* a = cast(uint*)malloc(N * uint.sizeof);
>  >
>  >         if (a !is null) {
>  >             for (uint i; i < N; ++i)
>  >                a[i] = i;
>  >         } else {
>  >             printf("null! %d\n", j);
>  >             break;
>  >         }
>  >     }
>  > }
>  >
>  > With such code I can allocate 1_708_000_000 bytes from D and up to 
> 2_038_000_000 bytes from C (but near the last 100-200 MB of RAM the C 
> code swaps a lot).
>  > So can't I can't use all my RAM from my D code? And do you know why?
>  >
>  > Bye,
>  > bearophile
> 
> Different allocation schemes have different strengths and weaknesses. 
> Some are fast, some fragment less, some have less overhead, some allow 
> larger sized blocks.  Often these things arn't mutual so there are 
> always tradoffs.  For example, to improve speed an allocator may 
> allocate into particular buckets which might restrict the maximum size 
> of one allocation.
> 
> I wonder how Ned-Malloc or Hord perform with your tests?
> 
> -Joel