Machine learning framework in D

[mw]

On Tuesday, 25 May 2021 at 01:04:19 UTC, jmh530 wrote:
> On Tuesday, 25 May 2021 at 00:20:26 UTC, Tim wrote:
>> Hi all,
>>
>> I'm needing to do a bit of ML soon for a project I'm involved 
>> in and I can't find any ML libs for D. Most of the libs 
>> available are for python (sci-kit learn, TensorFlow, etc.). I 
>> originally looked at using TensorFlow's C API but it doesn't 
>> include the actual ML stuff, just the tensor things. Could 
>> anybody help me out here?
>>
>> [snip]
>
> Grain is a deep learning framework for D.
>
> https://github.com/ShigekiKarita/grain

Also, there is grain2[1], and tfd: tensorflow for D[2].

But seriously, all these ML libraries in Dlang do not have many 
needed features you can easily find in Python ML libraries. If we 
step back: when those big companies like Google (TensorFlow 
etc.), Facebook (PyTorch etc.), Amazon (MxNet etc.), M$ (CNTK 
etc.) have already invested billions of dollars in either 
building these libraries directly or sponsoring them, which have 
been used by millions of users worldwide, why we want to build & 
use yet another one just in D?

Few D users also means you have high probability to hit bugs in 
some very basic stuff, e.g. bug in grain's Adam optimizer was 
only fixed 24 days ago [3].

D is already late for the ML game, maybe the best thing to do is 
concentrate the limited resource of the D communality on wrapping 
other mature libraries, e.g. tfd.


Given the current situation, on the practical side I'm using D to 
pre-processing data, and feed them into Python's ML libraries.

To communicate between D & Python:

-- grpc-dlang cannot talk to Python [4], as you have noted, and 
this issue has been around for half year now. (As a workaround, I 
added a Python pyd client talk to D server [8])

-- apache-thrift has build failure [5], which I just send a PR 
[6] to fix it today.

(again, I'm not sure how many real users these 2 libraries have, 
they are broken at some very basic stuff, not sure how other 
people can use them)

-- pyd: this is what I'm using right now. There are some rough 
edges [7], but at least I can get the basic stuff working 
finally, i.e. D & Python can talk.


Anyway, it's not easy, good luck!


[1]: https://github.com/ShigekiKarita/grain2
[2]: https://github.com/ShigekiKarita/tfd
[3]: https://github.com/ShigekiKarita/grain/pull/15
[4]: https://github.com/huntlabs/grpc-dlang/issues/15
[5]: https://github.com/D-Programming-Deimos/openssl/issues/63
[6]: https://github.com/apache/thrift/pull/2397
[7]: https://github.com/ariovistus/pyd/issues
[8]: https://github.com/huntlabs/grpc-dlang/pull/19