My Framework wishlist For D
Andre Pany
andre at s-e-a-p.de
Thu Apr 29 13:20:10 UTC 2021
On Wednesday, 28 April 2021 at 19:23:44 UTC, bioinfornatics wrote:
> On Wednesday, 28 April 2021 at 18:44:52 UTC, Andre Pany wrote:
>> On Wednesday, 28 April 2021 at 12:47:49 UTC, bioinfornatics
>> wrote:
>>> Firstly my needs it is around data processing and knowledge
>>> extraction so It is no a generalization of the needs.
>>> Moreover some tools/frameworks have an alternative in D
>>> (often not enough mature)
>>>
>>> Data computing:
>>> - job scheduling (yarn from hadoop, celery from python or
>>> slurm from HPC world)
>>> - data storage at least read and write to parquet file
>>> (through apache arrow lib)
>>> - Multinode processing such it is done by Ray:
>>> https://docs.ray.io/en/master/
>>> - Data processing «à la» Pandas/Dask
>>> - scipy and numpy library
>>> - a web project generator such it is done with jhipster:
>>> https://www.jhipster.tech/
>>> - IA library (maybe), if we can store to parquet that imply
>>> we are able to load them from python and run tensorfow,
>>> pytorch or other …
>>>
>>> and may others things
>>
>> Regarding reading and writing Parquet files using Apache
>> arrow, this is more or less easily possible. You can use DPP,
>> but you have some small effort afterwards, see here
>>
>> https://github.com/atilaneves/dpp/issues/242
>>
>> Kind regards
>> Andre
>
> Yes of course with some effort it is possible that means the
> ecosystem is not ready for
> and you will loose Every possible candidate to choose D
>
> in c++ you have apache arrow and Dataframe
> (https://github.com/hosseinmoein/DataFrame)
> in place of python (pyarrow/pandas) . And it is ready to use
Yes, I working in the area of big data / cloud with Python
(numpy/ pandas) and D. And yes, you are right, while the dub
packages list is growing, the scientific area is really small.
You have to leave here the happy path and have to invest 3 to 4
hours to get Parquet working with D.
This is my personal opinion: Every minute I had to invest
additionally to get everything running in D was a success. I was
bitten so many times by python issues and invested many hours to
solve them...
Now I have smoothly running D code working hand in hand with
python code. And yes, every few week something else, breaks in
the python, while D continues to run smoothly.
You are right, at the moment you need to be enthusiastic about
D,to setup a scientific application, but it totally worths it.
For me, the biggest blocker to get a numpy like library in D is
missing named arguments feature (dip is accepted).
Kind regards
Andre
More information about the Digitalmars-d
mailing list