How do I limit the number of active threads (queuing spawn calls)

[Jonathan M Davis]

On 2011-03-26 18:15, Andrej Mitovic wrote:
> I'm testing out some various compilation schemes with DMD. Right now I'm
> spawning multiple threads which simply do a `system` call with a string
> like "DMD -c somefile.d". I'd like to limit the number of active threads
> to something my CPU can handle (4 in this case since I've got 4 cores..).
> 
> How do I go about doing this?
> 
> Here's the function which I spawn:
> void compileObjfile(string name)
> {
>     shell(r"dmd -od" ~ r".\cache\" ~ r" -c -version=Unicode
> -version=WindowsNTonly -version=Windows2000 -version=WindowsXP -I..\ " ~
> name ~ " "); }
> 
> So I just need to pass the module name to it. The trouble is, if I spawn
> this function inside a foreach loop, I'll inadvertently create a few dozen
> threads. This hogs the system for a while. :) (although this does seem to
> create some rather impressive compilation speeds, LOL!)
> 
> This is what the main function might look like:
> void main()
> {
>     foreach (string name; dirEntries(curdir, SpanMode.shallow))
>     {
>         if (name.isfile && name.getExt == "d")
>         {
>             spawn(&compileObjfile, name);
>         }
>     }
> }
> 
> Sidenotes: So I've tried compiling the win32 libraries via `DMD -lib`. DMD
> eats up over 300 Megs of memory, and its quite scary how fast that number
> grows. It took over 25 seconds to compile a lib file.
> 
> On the other hand, compiling .obj files one by one by blocking a single
> thread on system calls (in other words, single-threaded version), it takes
> about 15 seconds to create a library file. In each instantiation DMD
> wastes only about a dozen or so Mbytes, maybe less.
> 
> When I spawn an unlimited number of threads via a foreach loop, again
> compiling object-by-object, the lib file is generated in only 5(!)
> seconds. I'm running a quad-core on XP32, btw.
> 
> So I'm a little perplexed, because according to Tomasz (maker of xfBuild)
> and his various posts, compiling .obj by .obj file should apparently be
> really really slow and -lib makes the fastest builds. But I'm getting the
> exact opposite results.

I don't believe that std.concurrency has any way to manage the number of 
threads that are running. It gives you the means to communicate between 
threads and gives you a nice to spawn a thread, but it doesn't really do much 
with thread management. You could use core.thread.Thread.getAll to get an 
array of all of the Threads, and spin until the number is below whatever the 
threshold is that you want, but that's not terribly efficient, since then 
you're going to have a thread spinning, eating up CPU as it waits for the 
others to finish.

What I have done when I've wanted to do something like this is to have each 
spawned thread send a message back when it's done. Then, I increment a thread 
count when I spawn a thread and decrement it when I receive a message 
indicating that a thread has terminated. In the loop that I have running which 
is processing whatever list of things I want processed, it will only spawn a 
thread if the thread count is below the chosen threshhold. Otherwise it sits 
there waiting to receive a message. So, it would do something like this

foreach(string name; dirEntries(curdir, SpanMode.shallow))
{
    if(name.isfile && name.getExt == "d")
    {
        if(currThreads < maxThreads)
            receiveTimeout(1, recProc);
        else
            receive(recProc;

        spawn(&compileObjfile, name);
        ++currThreads;
    }
}

recProc is then a function which handles receiving messages, and it decrements 
currThreads when it receives the message that a thread has been terminated.

std.concurrency does not manage threads. It only gives you tools for creating 
them and communicating between them. So, you need to manage the threads 
yourself if you want to manage them.

However, it should be noted that the task that you're looking to solve here 
may be better solved by std.parallelism, which David has been working on, and 
has been being reviewed on the main list.

- Jonathan M Davis