dub: JSON, SDL, YAML, TOML, what color should we paint it ?

[Atila Neves]

On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:
> Currently, Dub uses two formats: JSON and SDL.
>
> JSON was there from the beginning, I assume due to Dub's npm 
> inspiration.
> In fact, we still very much depend on JSON internally: until 
> recently, JSON object were passed around in quite a few places. 
> When we download a package, regardless of the format the 
> package recipe is in, we store it as `dub.json`.
>
> But JSON is a terrible format to write configurations in, given 
> how verbose it is, and it lacking support for comments.
> So around 2014, we added support for SDL (no, not [this 
> one](https://www.libsdl.org/), or [this 
> one](https://web.fe.up.pt/~mricardo/05_06/amsr/artigos/sdlTutorial.pdf), but [this one](https://sdlang.org/), and it took me 10 minutes to find the link because Wikipedia doesn't even know about it).
>
> At some point last year, I did some heavy refactoring on dub 
> along with quite a few improvements. In the process of doing 
> so, I took advantage of the fact that YAML is a subset of JSON 
> to add [a YAML 
> parser](https://github.com/dlang-community/D-YAML/), on which I 
> have built a 
> [library](https://github.com/dlang-community/configy) to do 
> automatic configuration parsing, because let's be honest, it's 
> 2023 and we're using the language with the best compile-time 
> reflection capabilities, so we really don't want to manually 
> parse a standard file format to fill a struct.
>
> This parser is used to parse all JSON files:
> - `dub.json` (but not `dub.sdl`);
> - `dub.selections.json`;
> - `dub.settings.json`;
>
> Now, I think that having `dub.settings.json` being written in 
> JSON doesn't make sense, and I would like to [change 
> this](https://github.com/dlang/dub/pull/2546). In fact, I have 
> been wanting this for more than 3 years, and the people that 
> [took part in the 
> discussion](https://github.com/dlang/dub/issues/1832) broadly 
> agreed.
>
> Here are the options that have been floated around:
> 1) Use SDL everywhere;
> 2) Add support for YAML;
> 3) Add support for another format (E.g. TOML);
>
> I would like to go with YAML, not because it is the greatest 
> format on earth, but because SDLang is not well known (it means 
> newcomer have to learn yet another format, and while not a big 
> deal, it adds friction), while YAML is everywhere. In the 
> [thread were SDL was 
> discussed](https://forum.dlang.org/thread/gnfctbuhiemidetngrzi@forum.dlang.org), people even thought we invented it!
>
> In practice, this would mean adding support for the file format 
> with a YAML extension, giving it priority over the old format, 
> then after a number of releases (most likely 10, probably more 
> for `dub.json`), deprecating the JSON / SDL equivalent.
>
> Obviously such a change would not happen overnight, and would 
> need broad support from the community. Opinions ?

I have several thoughts on this post and the replies to it.

At the time SDL was introduced, I didn't think it was a good 
idea. I'd still rather it hadn't happened, but the truth is that 
once one gets used to it, it's a lot nicer to write and read than 
JSON. But: it's obscure, and now we have both, and two different 
pages to look at for dub documentation. I don't think it's a 
great place to be.

Having said that, having *3* formats is even worse. I don't think 
we can talk about changing the format without having a plan for 
migrating every single dub.json/dub.sdl out there. I don't know 
how that'd work with issuing PRs to all of them, especially if 
the maintainers ignore them.

To all who doubt that using a turing-complete language is a bad 
idea: it's not. Any complicated project ends up needing it anyway 
and getting around the limitation by generating the declarative 
description or something like that. I've written enough CMake to 
know. I've worked around dub's limitations more times than I can 
count.

Using D sounds great, until one realises that reflecting can take 
a while, and unless there's a generation step followed by an 
actual simple build system, is too annoying to use. I *might* 
know what I'm talking about here: 
https://github.com/atilaneves/reggae/.
It's gotten to the point that I'm thinking of transitioning the 
build descriptions to Python at work to get around 2-3s of 
compile-time. The main reason I haven't done it yet is our builds 
are so complicated and dub takes so long to actually give me the 
information I need that it frequently dwarfs those seconds I'd 
save anyway.

I wish I had a conclusion but I don't.