4x speedup of recursive rmdir in std.file

[Jay Norwood]

Andrei Alexandrescu Wrote:
> That's why I'm saying - let's leave the decision to the user. Take a 
> uint parameter for the number of threads to be used, where 0 means leave 
> it to phobos, and default to 0.
> 
> Andrei
> 


ok, here is another version.  I was reading about the std.parallelism library, and I see I can do the parallel removes more cleanly.  Plus the library figures out the number of cores and limits the taskpool size accordingly. It is only a slight bit slower than the other code.  It looks like they choose 7 threads in the taskPool when you have 8 cores.

So, I do the regular files in parallel, then pass it back to the original library code which cleans up the  directory-only  tree non-parallel.  I also added in code to get the directory names from argv.


module main;

import std.stdio;
import std.file;
import std.datetime;
import std.parallelism;

int main(string[] argv)
{
	if (argv.length < 2){
		writeln ("need to specify one or more directories to remove");
		return 0;
	}
	foreach(string dir; argv[1..$]){
		writeln("removing directory: "~ dir );
		auto st1 = Clock.currTime(); //Current time in local time.
		rmdirRecurse2(dir); 
 		auto st2 = Clock.currTime(); //Current time in local time.
		auto dif = st2  - st1 ;
		auto ts= dif.toString();
		writeln("time:"~ts);
	}
	writeln("finished !");
	return 0;
}
void rmdirRecurse2(in char[] pathname){
    DirEntry de = dirEntry(pathname);
    rmdirRecurse2(de);
}
void rmdirRecurse2(ref DirEntry de){ 
	string[] files;

	if(!de.isDir)        
		throw new FileException( de.name, " is not a directory");    
	if(de.isSymlink())        
		remove(de.name);    
	else    { 
		// make an array of the regular files only
 		foreach(DirEntry e; dirEntries(de.name, SpanMode.depth, false)){
             if (!attrIsDir(e.linkAttributes)){
 				 files ~= e.name ;
 			 }
 		} 

		// parallel foreach for regular files
		foreach(fn ; taskPool.parallel(files,1000)) {
			remove(fn);
		}

		// let the original code remove the directories only
		rmdirRecurse(de);
	}
}