Memory leak - only with large data set

[bearophile]

Roger:

> The cached list consists of about 500000 items 
> (ExtendedOrderItemDTO) and when we use a search criteria that 
> results in a large amount of items we get a memory leak of 
> about 90MB on each run.

This is interesting. Currently the D GC is conservative, this 
means that some pointers coming (inbound) randomly inside the 
large arrays can keep them alive. Until there is more precision 
(and the Summer Of Code has produced something), having 64 but 
pointers reduces this problem. So are you using a 32 bit 
compilation? Are you able to compile it at 64 bit? I think you 
are using Windows, and while the 64 bit Windows DMD2 is coming 
(it already compiles Phobos2) it's not yet usable.



>   private static shared ExtendedOrderItemDTO[] _orderItems;
>   public static @property ExtendedOrderItemDTO[] OrderItems()
>   {
>     return cast(ExtendedOrderItemDTO[]) _orderItems;
>   }

Generally in D method/function names start with a lowercase, 
unlike C#.

What's the purpose of the cast? Generally it's better to minimize 
casts. And maybe this is enough in your case (not tested), and a 
bit safer:

return cast()_orderItems;



> 	auto result=filter!( (ExtendedOrderItemDTO x) =>
> 	  x._SaleDate >= state.StatePeriod.From &&
> 	  x._SaleDate <= state.StatePeriod.To &&
> 	  (
> 	  state.Brand.Level == 0 ||
> 	  (state.Brand.Level == 1 && state.Brand.ID == x.BrandID) ||
> 	  (state.Brand.Level == 2 && state.Brand.ID == x.VariantID) ||
> 	  (state.Brand.Level == 3 && state.Brand.ID == x.ProductID)
> 	) &&
> 	(
> 	  state.Channel.Level == 0 ||
> 	  (state.Channel.Level == 1 && state.Channel.ID == x.Channel) 
> ||
> 	  (state.Channel.Level == 2 && state.Channel.ID == 
> x.SubChannel)
> 	) &&
> 	(
> 	  state.ProductType == 2 ||
> 	  (state.ProductType == 0 && x.IsFMC == true) ||
> 	  (state.ProductType == 1 && x.IsFMC == false)	
> 	)
> 	)(Cache.OrderItems);
>
> 	return array(result);


That's the biggest filtering lambda I have seen so far :o) It's 
an interesting example of real-world code that I've never seen in 
Haskell (or D).

In D there is the (nestable) with() construct that sometimes 
helps reduce the size of similar jungles.

Maybe here using UFCS helps readability a bit:

auto result = Cache.OrderItems.filter!((ExtendedOrderItemDTO x) 
=> ...
     )();

Bye,
bearophile