parallel threads stalls until all thread batches are finished.
Christian Köstlin
christian.koestlin at gmail.com
Mon Aug 28 10:33:15 UTC 2023
On 26.08.23 05:39, Joe at bloow.edu wrote:
> On Friday, 25 August 2023 at 21:31:37 UTC, Ali Çehreli wrote:
>> On 8/25/23 14:27, Joe at bloow.edu wrote:
>>
>> > "A work unit is a set of consecutive elements of range to be
>> processed
>> > by a worker thread between communication with any other
>> thread. The
>> > number of elements processed per work unit is controlled by
>> the
>> > workUnitSize parameter. "
>> >
>> > So the question is how to rebalance these work units?
>>
>> Ok, your question brings me back from summer hibernation. :)
>>
>> This is what I do:
>>
>> - Sort the tasks in decreasing time order; the ones that will take the
>> most time should go first.
>>
>> - Use a work unit size of 1.
>>
>> The longest running task will start first. You can't get better than
>> that. When I print some progress reporting, I see that most of the
>> time N-1 tasks have finished and we are waiting for that one longest
>> running task.
>>
>> Ali
>> "back to sleep"
>
>
> I do not know the amount of time they will run. They are files that are
> being downloaded and I neither know the file size nor the download
> rate(in fact, the actual download happens externally).
>
> While I could use work unit of size 1 then problem then is I would be
> downloading N files at once and that will cause other problems if N is
> large(and sometimes it is).
>
> There should be a "work unit size" and a "max simultaneous workers".
> Then I could set the work unit size to 1 and say the max simultaneous
> workers to 8 to get 8 simultaneous downloads without stalling.
I think thats what is implemented atm ...
`taskPool` creates a `TaskPool` of size `defaultPoolThreads` (defaulting
to totalCPUs - 1). The work unit size is only there to optimize for
small workloads where task / thread switching would be a big performance
problem (I guess). So in your case a work unit size of 1 should be good.
Did you try this already?
Kind regards,
Christian
More information about the Digitalmars-d-learn
mailing list