Scientific computing with D

[Fawzi Mohamed]

On 2009-01-30 20:55:06 +0100, Bill Baxter <wbaxter at gmail.com> said:

> On Fri, Jan 30, 2009 at 10:23 PM, Lars Kyllingstad
> <public at kyllingen.nospamnet> wrote:
>> I think D is, or at least could be, the scientific programming language of
>> the future.
> 
> Agreed!  Numerics is one place where D fits nicely.
> 
>> [...]
>> The one thing I miss the most, however, and which I think is necessary for D
>> to "take off" as a scientific language, is a native D scientific library.
> 
>> I think it would be really nice if many or all of the above mentioned things
>> could be collected in a single library, together with all kinds of other
>> stuff. Something like the GSL, only written in D.
> 
> ...and not GPL, preferably.  :-)
> 
> The goal sounds great.  I'm certainly willing to help out.
> 
> To me it seems one of the first things you need to get nailed down is
> how to represent multidimensional data.  Both dense, strided, and --
> for 2D -- common sparse and BLAS formats.   Probably this can be done
> at the concept (compile-time interface) level without having to go
> into implementation details.  But it might also be nice to have some
> concrete implementation nailed down too, so not everything is built as
> a giant house of template cards.  Once you have that, then any package
> that can work with that concept or array format will be able to share
> data pretty easily.
> 
> But it's not so easy.

indeed probably one might need
A: access
1) access to any element by index
1.b) setting any element
2) subslicing
3) looping on elements + indexes
3.b) looping of row, then cols
3.c) looping on col then rows

B: vector space structure
zero & 1 vector & diag matrixes
scalar multiplication
dot product

in some cases maybe even wedge product

A and B are rather separated

depending on the exact structure different ways to do this might be 
more or less efficient or not possible at all (setting an element 
outside the sparsity pattern, row looping for some sparse formats,...).

for A I think that one could have an interface for 1-3 that works, and 
a dot product could be implemented in a generic way on the top of it, 
but this interface will probably not be the most efficient (but 
probably ok)

for B maybe templates, with a class wrapper that exposes all the stuff 
(one could also use only templates, but it might be more cumbersome

In any case not easy...

>   Even now I use 3 different libraries at
> different times for different things.
> * I have MatrixT/VectorT,  fixed-size dense matrix and vector, whose
> 2D and 3D versions have some special operations like "cross product"
> * I have Dflat, which provides dynamically-sized Matrix types that
> match what BLAS accepts.  And also a few dynamically-sized sparse
> formats that work with various sparse libs.
> * I have MultiArray which is a generic container for N-dimensional
> strided data, with dynamic number of dimensions and size.
> 
> On the last one ... I consider that one a bit of a failed experiment.
> Dynamic number of dimensions causes too many headaches and is not
> really useful very often.  I think Fawzi is brewing a fix-dim N-d
> array of his own, or was at one point.

yes it is in blip, it works quite well, it uses compile time rank (as 
you had suggested) and has a nice interface to lapack

I haven't announced it officially yet because I wanted to make the SMP 
parallelization better, and add binary serialization but anyway here is 
what blip has:
- N dimensional arrays with arbitrary strides, and subslicing without 
copying, and a nice interface to lapack, and fair performance.
- random testing framework (that does tests in parallel)
- serialization (at the moment to/from json format, but very extensible 
to other formats, can be activated with minimal effort if one uses 
Xpose), and is usabe also as input file
- SMP parallelization (that is used by the random testing framework, 
and should improve drastically soon, and be used also by NArray)

it is usable already now, so if you want to give a look...

Fawzi
> 
> But anyway uniting all these things under a single library or even a
> single concept would be significant work.
> 
> --bb