std.concurrency wrapper over MPI?

[Russel Winder]

On Fri, 2011-08-05 at 23:51 -0400, dsimcha wrote:
[ . . . ]
> 1.  Is anyone besides me interested in this?

MPI may be ancient, it may be a bit daft in terms of its treatment of
marshalling, unmarshalling and serializing, it may be only a Fortran and
C thing bolted into C++ (quite well) but it is the de facto standard for
HPC.  OK so HPC is about 10% of world-wide computing, probably less than
that of spend despite the enormous per installation price, but it is
about 90% of political marketing.  Any short term parallelism strategy
must include MPI -- and work with OpenMPI and MPICH2.

So I don't think it is a matter of just interest for D, I would say that
if D is to stand with C++, C and Fortran then there has to be an MPI
API.  Even though MPI should be banned going forward.

> 2.  Is anyone already working on something similar.
> 
> 3.  Would this be Phobos material even though it would depend on MPI, or 
> would it better be kept as a 3rd party library?

Given that it requires a transitive dependency then either Phobos goes
forward with optional dependencies or the MPI API is a separate thing.
Given my personal opinion that actor model, dataflow model, agents, etc.
should be the application level concurrency and parallelism model, I
would be quite happy with an MPI API not being in Phobos.  Keep Phobos
for that which every D installation will need.  MPI is a niche market in
that respect.

Optional dependencies sort of work but are sort of a real pain in the
Java/Maven milieu.

> 4.  std.concurrency in its current incarnation doesn't allow objects 
> with mutable indirection to be passed as messages.   This makes sense 
> when passing messages between threads in the same address space. 
> However, for passing between MPI processes, the object is going to be 
> copied anyhow.  Should the restriction be kept (for consistency) or 
> removed (because it doesn't serve much of a purpose in the MPI context)?

At the root of this issue is local thread-based parallelism in a shared
memory context, vs cluster parallelism.  MPI is a cluster solution --
even though it can be used in multicore shared memory situation.  The
point about enforced copying vs. potential sharing is core to this
obviously.  This has to be handled with absolute top notch performance
in mind.  It is arguably a situation where programming language
semantics and purity have to be sacrificed at the altar of performance.
There are already far too many MPI applications that are written with
far too much comms code in the application simply to ensure performance
-- because the MPI infrastructure cannot be trusted to do things fast
enough if you use anything other than the bottom most layer.

> 5.  For passing complex object graphs, serialization would obviously be 
> necessary.  What's the current state of the art in serialization in D? 
> I want something that's efficient and general first and foremost.  I 
> really don't care about human readability or standards compliance (in 
> other words, no XML or JSON or anything like that).

Again performance is everything, so nothing must get in the way of
having something that cannot be made faster.

The main problem here is going to be that when anything gets released
performance will be the only yardstick by which things are measured.
Simplicity of code, ease of evolution of code, all the things
professional developers value, will go out of the window.  It's HPC
after all :-)

I still think D needs a dataflow, CSP and data parallelism strategy, cf.
Go, GPars, Akka, even Haskell.  Having actors is good, but having only
actors is not good, cf. Scala and Akka.

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at russel.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20110806/5fcfcb91/attachment.pgp>