parallel threads stalls until all thread batches are finished.

[Joe at bloow.edu]

On Friday, 25 August 2023 at 21:31:37 UTC, Ali Çehreli wrote:
> On 8/25/23 14:27, Joe at bloow.edu wrote:
>
> > "A work unit is a set of consecutive elements of range to be
> processed
> > by a worker thread between communication with any other
> thread. The
> > number of elements processed per work unit is controlled by
> the
> > workUnitSize parameter. "
> >
> > So the question is how to rebalance these work units?
>
> Ok, your question brings me back from summer hibernation. :)
>
> This is what I do:
>
> - Sort the tasks in decreasing time order; the ones that will 
> take the most time should go first.
>
> - Use a work unit size of 1.
>
> The longest running task will start first. You can't get better 
> than that. When I print some progress reporting, I see that 
> most of the time N-1 tasks have finished and we are waiting for 
> that one longest running task.
>
> Ali
> "back to sleep"


I do not know the amount of time they will run. They are files 
that are being downloaded and I neither know the file size nor 
the download rate(in fact, the actual download happens 
externally).

While I could use work unit of size 1 then problem then is I 
would be downloading N files at once and that will cause other 
problems if N is large(and sometimes it is).

There should be a "work unit size" and a "max simultaneous 
workers". Then I could set the work unit size to 1 and say the 
max simultaneous workers to 8 to get 8 simultaneous downloads 
without stalling.