Significant GC performance penalty

[Rob T]

On Sunday, 16 December 2012 at 05:37:57 UTC, SomeDude wrote:
>
> Isn't the memory management completely negligible when compared 
> to the database access here ?

Here are the details ...

My test run selects and returns 206,085 records with 14 fields 
per record.

With all dynamic memory allocations disabled that are used to 
create the data structure containing the returned rows, a run 
takes 5 seconds. This does not return any data, but it runs 
exactly through all the records in the same way but returns to a 
temporary stack allocated value of appropriate type.

If I disable the GC before the run and re-enable it immediately 
after, it takes 7 seconds. I presume a full 2 seconds are used to 
disable and re-enable the GC which seems like a lot of time.

With all dynamic memory allocations enabled that are used to 
create the data structure containing the returned rows, a run 
takes 28 seconds. In this case, all 206K records are returned in 
a dynamically generate list.

If I disable the GC before the run and re-enable it immediately 
after, it takes 11 seconds. Since a full 2 seconds are used to 
disable and re-enable the GC, then 9 seconds are used, and since 
5 seconds are used without memory allocations, the allocations 
are using 4 seconds, but I'm doing a lot of allocations.

In my case, the structure is dynamically generated by allocating 
each individual field for each record returned, so there's 
206,085 records x 14 fields = 2,885,190 allocations being 
performed. I can cut the individual allocations down to 206,000 
by allocating the full record in one shot, however this is a 
stress test designed to work D as hard as possible and compare it 
with an identically stressed C++ version.

Both the D and C++ versions perform identically with the GC 
disabled and subtracting the 2 seconds from the D version to 
remove the time used up by enabling and disabling the GC during 
and after the run.

I wonder why 2 seconds are used to disable and enable the GC? 
That seems like a very large amount of time. If I select only 
5,000 records, the time to disable and enable the GC drops 
significantly to negligible levels and it takes the same amount 
of time per run with GC disabled & enabled, or with GC left 
enabled all the time.

During all tests, I do not run out of free RAM, and at no point 
does the memory go to swap.

--rt