Significant GC performance penalty

[SomeDude]

On Sunday, 16 December 2012 at 07:47:48 UTC, Rob T wrote:
> On Sunday, 16 December 2012 at 05:37:57 UTC, SomeDude wrote:
>>
>> Isn't the memory management completely negligible when 
>> compared to the database access here ?
>
> Here are the details ...
>
> My test run selects and returns 206,085 records with 14 fields 
> per record.
>
> With all dynamic memory allocations disabled that are used to 
> create the data structure containing the returned rows, a run 
> takes 5 seconds. This does not return any data, but it runs 
> exactly through all the records in the same way but returns to 
> a temporary stack allocated value of appropriate type.
>
> If I disable the GC before the run and re-enable it immediately 
> after, it takes 7 seconds. I presume a full 2 seconds are used 
> to disable and re-enable the GC which seems like a lot of time.
>
> With all dynamic memory allocations enabled that are used to 
> create the data structure containing the returned rows, a run 
> takes 28 seconds. In this case, all 206K records are returned 
> in a dynamically generate list.
>
> If I disable the GC before the run and re-enable it immediately 
> after, it takes 11 seconds. Since a full 2 seconds are used to 
> disable and re-enable the GC, then 9 seconds are used, and 
> since 5 seconds are used without memory allocations, the 
> allocations are using 4 seconds, but I'm doing a lot of 
> allocations.
>
> In my case, the structure is dynamically generated by 
> allocating each individual field for each record returned, so 
> there's 206,085 records x 14 fields = 2,885,190 allocations 
> being performed. I can cut the individual allocations down to 
> 206,000 by allocating the full record in one shot, however this 
> is a stress test designed to work D as hard as possible and 
> compare it with an identically stressed C++ version.
>

You cannot expect the GC to perform like manual memory 
management. It's a completely unrealistic microbenchmark to 
allocate each individual field, even for manual MM. The least you 
can do to be a little bit realistic is indeed to allocate one row 
at a time. I hope that's what you intend to do. But usually, 
database drivers allow the user to tweak the queries and decide 
how many rows can be fetched at a time, and it's pretty common to 
fetch 50 or 100 rows at a time, meaning one allocation only each 
time. It would be interesting to compare the performance of the 
two languages in these situations, i.e one row at a time, and 50 
rows at a time.