What is the memory usage of my app?

[Adil via Digitalmars-d-learn]

On Thursday, 16 April 2015 at 20:33:17 UTC, Marc Schütz wrote:
> On Thursday, 16 April 2015 at 17:13:25 UTC, Laeeth Isharc wrote:
>> For CSV files, what I found is that parsing is quite slow (and 
>> memory intensive).
>
> If your sure that CSV reading is the culprit, writing a custom 
> parser could help. It's possible to load a CSV file with almost 
> no memory overhead. What I would do:
>
> - Use std.mmfile with Mode.readCopyOnWrite to map the file into 
> memory.
> - Iterate over the lines, and then over the fields using 
> std.algorithm.splitter.
> - Don't copy, but return slices into the mapped memory.
> - If a field needs to be unescaped, this can be done in-place. 
> Unescaping never makes a string longer, and the original file 
> won't be modified thanks to COW (private mapping).

These are REALLY USEFUL optimizations! Thanks Marc.

Although, i'm still no better with the memory usage. I've reduced 
the application to just loading a CSV file into structs.

Here is void main :

````
void main(string[] args)
{
     auto text = readText(args[1]);

     foreach(record; csvReader!(string[string])(text, null))
     {
         if (!record["RIC"] || !record["TRBCIndCode"]) {
             continue;
         }

         // Add a Security to Securities
         securities.add(record["RIC"], record["TRBCIndCode"], 
record, []);
     }

     delete text;

     GC.collect();

     writefln("%d securities processed", securities.length);
     writefln("Securities : %d MB", securities.bytes/1024/1024);

     import core.thread;
     Thread.sleep(dur!"seconds"(60));
}
````

The output is :

````
make screener-d-simple; ./screener-d data/instruments-clean.csv

dmd -vgc -ofscreener-d source/simplemain.d source/lib/security.d
source/simplemain.d(30): vgc: indexing an associative array may 
cause GC allocation
source/simplemain.d(30): vgc: indexing an associative array may 
cause GC allocation
source/simplemain.d(35): vgc: indexing an associative array may 
cause GC allocation
source/simplemain.d(35): vgc: indexing an associative array may 
cause GC allocation
source/simplemain.d(38): vgc: 'delete' requires GC
source/lib/security.d(105): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(111): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(113): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(115): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(118): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(122): vgc: operator ~= may cause GC 
allocation
source/lib/security.d(123): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(164): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(164): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(173): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(173): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(182): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(182): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(191): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(191): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d(203): vgc: indexing an associative array 
may cause GC allocation
source/lib/security.d-mixin-213(213): vgc: indexing an 
associative array may cause GC allocation
source/lib/security.d-mixin-213(213): vgc: indexing an 
associative array may cause GC allocation
source/lib/security.d-mixin-219(219): vgc: indexing an 
associative array may cause GC allocation
source/lib/security.d-mixin-219(219): vgc: indexing an 
associative array may cause GC allocation
source/lib/security.d-mixin-225(225): vgc: indexing an 
associative array may cause GC allocation
source/lib/security.d-mixin-225(225): vgc: indexing an 
associative array may cause GC allocation
source/lib/security.d-mixin-231(231): vgc: indexing an 
associative array may cause GC allocation
source/lib/security.d-mixin-231(231): vgc: indexing an 
associative array may cause GC allocation

20066 securities processed
Securities : 188 MB
````

And yet memory usage is 617 MB.