Subject: scrubbed – a native web-data cleaning pipeline written in D

[Dejan Lekic]

On Monday, 5 October 2026 at 19:56:41 UTC, Shammah Chancellor 
wrote:
> Some parts of it definitely are; I just haven't published them 
> to dub yet.
>
> There's a warc-reader, parquet library, working on an s3 
> client, and a few other things. I'm eventually going to publish 
> them once they're more well tested.
>
> What part are you interested in using as a library? My thought 
> was more that this would integrate as a binary with things like 
> DataTrove and spark.

gid project has bindings for arrow libraries, parquet included. 
You may find that useful. I am interested in web scraping part as 
that is sometimes we all need to do... Currently I use Python's 
BeautifulSoup when I need something like that.