std.parallelism: Request for Review

[Don]

Russel Winder wrote:
> On Mon, 2011-02-28 at 10:41 +0100, Don wrote:
> [ . . . ]
>> As the name says, it is * cores per CPU *. That is _not_ the same as the 
>> total number of cores in the machine.
> 
> I guess then the missing extension is to have a function that returns an
> array opf processor references so that the core count can be measured?

Yes.

> 
>> The documentation probably doesn't state that strongly enough, but 
>> currently there is no known bug in the implementation of 
>> core.cpuid.coresPerCPU.
> 
> OK so my complaints were somewhat misdirected, apologies.
> 
> The fact remains though that it seems there is no way in D or Phobos to
> pick up the number of "processors" available to a Linux or Mac OS X
> machine -- I guess the same applies to Windows but I personally don't
> use that OS so don't really care about that one ;-)
> 
>> It's an OS issue, anyway: just because a machine has 64 cores, is no 
>> guarantee that it will give you access to all of them. std.parallelism 
>> shouldn't be using core.cpuid for that purpose.
> 
> I had thought you were arguing that D/Phobos couldn't rely on waiting
> for OS functionality, that it had to use direct access to the processor
> techniques.  Maybe I misunderstood what you were saying.  If so, my
> apologies.

The primary purpose of core.cpuid is to determine which instruction set 
is supported. Obviously this needs to happen at a very early stage in 
the runtime initialization. It turns out that on x86 (and Itanium), it 
is possible to determine the cache sizes as well, and this can be used 
to optimize array operations.
It would very nice if you could get the number of processors at the same 
time, but it's just not possible.

> Linux claims to have 8 processors on my twin-Xeon box.  std.parallelism
> therefore needs some form of support from D and/or Phobos to say "how
> many parallel threads can this platform sustain".  All other parallelism
> frameworks I use handle this no problem.
> 
> I accept your argument about core.cpuid and so will not investigate it
> further, other than to say it needs to work on 64-bit processors as well
> as 32-bit ones.  The campaign now must be to have an OS query capability
> to find the number of processors.

Yes, it definitely needs to be a priority. It's a fundamental function 
for a modern system programming language.

I'm not sure where this stuff should go, probably not into core.cpuid.
I could imagine something with a name like std.sysinfo, which dealt with 
system configuration, and also provided OS-independent abstractions for 
things like setting process affinity.
I think it should go into std.parallelism for now.