How do I limit the number of active threads (queuing spawn calls)

[Andrej Mitovic]

I'm testing out some various compilation schemes with DMD. Right now I'm spawning multiple threads which simply do a `system` call with a string like "DMD -c somefile.d". I'd like to limit the number of active threads to something my CPU can handle (4 in this case since I've got 4 cores..).

How do I go about doing this?

Here's the function which I spawn:
void compileObjfile(string name)
{
    shell(r"dmd -od" ~ r".\cache\" ~ r" -c -version=Unicode -version=WindowsNTonly -version=Windows2000 -version=WindowsXP -I..\ " ~ name ~ " ");
}

So I just need to pass the module name to it. The trouble is, if I spawn this function inside a foreach loop, I'll inadvertently create a few dozen threads. This hogs the system for a while. :) (although this does seem to create some rather impressive compilation speeds, LOL!)

This is what the main function might look like:
void main()
{
    foreach (string name; dirEntries(curdir, SpanMode.shallow))
    {
        if (name.isfile && name.getExt == "d")
        {
            spawn(&compileObjfile, name);
        }
    }
}

Sidenotes: So I've tried compiling the win32 libraries via `DMD -lib`. DMD eats up over 300 Megs of memory, and its quite scary how fast that number grows. It took over 25 seconds to compile a lib file.

On the other hand, compiling .obj files one by one by blocking a single thread on system calls (in other words, single-threaded version), it takes about 15 seconds to create a library file. In each instantiation DMD wastes only about a dozen or so Mbytes, maybe less.

When I spawn an unlimited number of threads via a foreach loop, again compiling object-by-object, the lib file is generated in only 5(!) seconds. I'm running a quad-core on XP32, btw.

So I'm a little perplexed, because according to Tomasz (maker of xfBuild) and his various posts, compiling .obj by .obj file should apparently be really really slow and -lib makes the fastest builds. But I'm getting the exact opposite results.