My Framework wishlist For D

[jmh530]

On Wednesday, 28 April 2021 at 15:20:40 UTC, bachmeier wrote:
> On Wednesday, 28 April 2021 at 12:47:49 UTC, bioinfornatics 
> wrote:
>> Firstly my needs it is around data processing and knowledge 
>> extraction so It is no a generalization of the needs. Moreover 
>> some tools/frameworks have an alternative in D (often not 
>> enough mature)
>>
>> Data computing:
>>  - job scheduling (yarn from hadoop, celery from python or 
>> slurm from HPC world)
>>  - data storage at least read and write to parquet file 
>> (through apache arrow lib)
>>  - Multinode processing such it is done by Ray: 
>> https://docs.ray.io/en/master/
>>  - Data processing «à la» Pandas/Dask
>>  - scipy and numpy library
>>  - a web project generator such it is done with jhipster: 
>> https://www.jhipster.tech/
>>  - IA library (maybe), if we can store to parquet that imply 
>> we are able to load them from python and run tensorfow, 
>> pytorch or other …
>>
>> and may others things
>
> Which of these can be done by calling other languages (easy to 
> handle) and which would need to be written in D (probably won't 
> happen)? Is Windows support necessary or is WSL sufficient?

For some of these, the OP references libraries, like 
numpy/scipy/pandas, that are largely user-friendly wrappers over 
some C or Fortran libraries, at least for the most 
computationally intensive parts of the libraries. While something 
like lapack is doing the same thing, you get the benefit of 
keeping it all in D (the scripting languages also may have used 
different default settings which trade performance for accuracy).

Apache Arrow is a C++ library. I don't have any idea how 
difficult it would be to get working in D, but there is a GLIB 
version with a C API that should be easy enough to get working in 
D.