Performance Issue

[Azi Hassan via Digitalmars-d-learn]

On Tuesday, 5 September 2017 at 09:44:09 UTC, Vino.B wrote:
> Hi,
>
>  The below code is consume more memory and slower can you 
> provide your suggestion on how to over come these issues.

You can start by dropping the .array conversions after 
dirEntries. That way your algorithm will become lazy (as opposed 
to eager), meaning that it won't allocate an entire array of 
DirEntry[]. It will, instead, treat the DirEntries one at a time, 
resulting in less memory consumption.

I didn't understand the join(["\\\\?\\", d[0]]) part, maybe you 
meant to write join("\\\\?\\", d[0]) ?

If appender is too slow, you can experiment with a dynamic array 
whose capacity was preallocated : string[][] Subdata; 
Subdata.reserve(10000); In this case Subdata will hold enough 
space for 10000 string[]s, which will result in better 
performance.

Here's the updated code (sans .array) in case any one wants to 
reproduce the issue :

import std.stdio;
import std.conv;
import std.typecons;
import std.array;
import std.path;
import std.container;
import std.file;
import std.parallelism;
import std.algorithm;

void main()
{
	".".csizeDirList(1024).each!writeln;
}

string[][] csizeDirList (string FFs, int SizeDir) {
	ulong subdirTotal = 0;
	ulong subdirTotalGB;
	auto Subdata = appender!(string[][]);
	auto dFiles = dirEntries(FFs, SpanMode.shallow)
		.filter!(a => a.isDir && !globMatch(a.baseName, "*DND*"))
		.map!(a => tuple(a.name, a.size));

	foreach (d; dFiles) {
		auto SdFiles = dirEntries(join(["\\\\?\\", d[0]]), 
SpanMode.depth)
			.map!(a => tuple(a.size));
		foreach (f; parallel(SdFiles,1)) {
			subdirTotal += f[0];
		}
		subdirTotalGB = (subdirTotal/1024/1024);
		if (subdirTotalGB > SizeDir) {
			Subdata ~= [d[0], to!string(subdirTotalGB)];
		}
		subdirTotal = 0;
	}
	return Subdata.data;
}