OT On NumPy [was beginner's pyd question - exporting structs to python]

[Russel Winder via Digitalmars-d-learn]

On Mon, 2014-08-18 at 23:12 +0000, Laeeth Isharc via Digitalmars-d-learn
wrote:
[…]
> > For me, NumPy has some serious problems despite being the 
> > accepted norm for computational work.
> 
> If not too offtopic, do you have a link describing, or would you 
> briefly summarize these problems?  I am intrigued.  And what 
> would you suggest in its place?  Fortran?
[…]

I have no benchmark experiment data as proof yet, just anecdotal
evidence forming an hypothesis, but it seems that the underlying data
parallelism model of NumPy has some serious overhead problems: speed-ups
are not as high as they should be, and scaling is not as good as it
should be.

The finance people using Python in London, and indeed the general data
analysis using Python folk (cf. PyData meetings around the world) all
take NumPy as a given, and that it works well enough for them. I guess
those for whom NumPy is not good enough are using Cython, C++ or Fortran
(or even C) for the computationally intensive stuff. Or more likely they
already had the native code in place and so are not using NumPy other
than for data visualization and replacement of Matlab.

I think Numba is a disruptive technology here.

However the danger is that the opaque type approach of NumPy (which is
good) is forgotten as a good abstraction in the face of Numba speeds,
with people reverting to explicit rather than implicit iteration just
because things go faster.

Or maybe this won't happen because all the needed computationally
intensive libraries already exists. I guess the PyData meetings are the
place where al this will be played out.  

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder