4x speedup of recursive rmdir in std.file

deadalnix deadalnix at gmail.com
Tue Feb 7 03:19:49 PST 2012


Le 05/02/2012 18:38, Andrei Alexandrescu a écrit :
> On 2/5/12 10:16 AM, Nick Sabalausky wrote:
>> "Jay Norwood"<jayn at prismnet.com> wrote in message
>> news:jgm5vh$hbe$1 at digitalmars.com...
>>> == Quote from Nick Sabalausky (a at a.a)'s article
>>>> Interesting. How does it perform when just running on one core?
>>>
>>> The library without the threads is 1 min 5 secs for the 1.5GB
>>> directory structure with about 32k files. This is on an 510
>>> series intel ssd. The win7 os removes it in almost exactly the
>>> same time, and you can see from their task manager it is also
>>> being done single core and only a small percentage of cpu. In
>>> contrast, all 8 threads in the task manager max out for a period
>>> when running this multi-thread remove. The regular file deletes
>>> are occurring in parallel. A single thread removes the directory
>>> structure after waiting for all the regular files to be deleted by
>>> the parallel threads. I attached a screen capture.
>>>
>>
>> What I'm wondering is this:
>>
>> Suppose all the cores but one are already preoccupied with other
>> stuff, or
>> maybe you're even running on a single-core. Does the threading add enough
>> overhead that it would actually go slower than the original
>> single-threaded
>> version?
>>
>> If not, then this would indeed be a fantastic improvement to phobos.
>> Otherwise, I wonder how such a situation could be mitigated?
>
> There's a variety of ways, but the simplest approach is to pass a
> parameter to the function telling how many threads it's allowed to
> spawn. Jay?
>
> Andrei
>
>

That cold be a solution, but this is a bad separation of concerns IMO, 
and should be like that in phobos.

The parameter should be a thread pool or something similar. This allow 
to not only choose the number of thread, but also to choose how the task 
is distributed over threads, eventually mix thoses task with other tasks 
(by using the same thread pool in other places).

It allow to basically separate the problem of deleting and the problem 
of spreading the task over multiple threads and with which policy.


More information about the Digitalmars-d-announce mailing list