task parallelize dirEntries

Arun Chandrasekaran via Digitalmars-d-learn digitalmars-d-learn at puremagic.com
Fri Aug 11 14:33:51 PDT 2017


I've modified the sample from tour.dlang.org to calculate the md5 
digest of the files in a directory using std.parallelism.

When I run this on a dir with huge number of files, I get:

core.exception.OutOfMemoryError at src/core/exception.d(696): Memory 
allocation failed

Since dirEntries returns a range, I thought 
std.parallelism.parallel can make use of that without loading the 
entire file list into the memory.

What am I doing wrong here? Is there a way to achieve what I'm 
expecting?

```
import std.digest.md;
import std.stdio: writeln;
import std.file;
import std.algorithm;
import std.parallelism;

void printUsage()
{
     writeln("Loops through a given directory and calculates the 
md5 digest of each file encountered.");
     writeln("Usage: md <dirname>");
}

void safePrint(T...)(T args)
{
     synchronized
     {
         import std.stdio : writeln;
         writeln(args);
     }
}

void main(string[] args)
{
     if (args.length != 2)
         return printUsage;

     foreach (d; parallel(dirEntries(args[1], 
SpanMode.depth).filter!(f => f.isFile), 1))
     {
         auto md5 = new MD5Digest();
         md5.reset();
         auto data = cast(const(ubyte)[]) read(d.name);
         md5.put(data);
         auto hash = md5.finish();
         import std.array;
         string[] t = split(d.name, '/');
         safePrint(toHexString!(LetterCase.lower)(hash), "  ", 
t[$-1]);
     }
}
```


More information about the Digitalmars-d-learn mailing list