parallel threads stalls until all thread batches are finished.

[Christian Köstlin]

On 29.08.23 00:37, Joe at bloow.edu wrote:
> Well, I have 32 cores so that would spawn 64-1 threads with hyper 
> threading so not really a solution as it is too many simultaneous downs 
> IMO.
> 
> 
> "These properties get and set the number of worker threads in the 
> TaskPool instance returned by taskPool. The default value is totalCPUs - 
> 1. Calling the setter after the first call to taskPool does not changes 
> number of worker threads in the instance returned by taskPool. "
> 
> I guess I could try to see if I can change this but I don't know what 
> the "first call" is(and I'm using parallel to create it).
The call that is interesting in your case is the call to `taskPool` in 
your foreach loop. So if you call `defaultPoolThreads(8)` before your 
loop you should be good.

> Seems that the code should simply be made more robust. Probably a just a 
> few lines of code to change/add at most. Maybe the constructor and 
> parallel should take an argument to set the "totalCPUs" which defaults 
> to getting the total number rather than it being hard coded.
> 
> I currently don't need or have 32+ downlaods to test ATM so...
> 
> 
>     this() @trusted
>      {
>          this(totalCPUs - 1);
>      }
> 
>      /**
>      Allows for custom number of worker threads.
>      */
>      this(size_t nWorkers) @trusted
>      {
> 
> 
> Basically everything is hard coded to use totalCPU's and that is the 
> ultimate problem. Not all tasks should use all CPU' >
> What happens when we get 128 cores? or even 32k at some poin >
> It shouldn't be a hard coded value, it's really that simple and where 
> the problem originates because someone didn't think ahead.
You have the option to not use the default value.

Kind regards,
Christian