marmos v0.1.0 (alpha state) - Documentation Generation using external tooling

[Bradley Chatha]

On Wednesday, 3 July 2024 at 03:19:12 UTC, Steven Schveighoffer 
wrote:
> Do I understand that this is something that generates a JSON 
> file for consumption by other doc generators?

Yeah basically, the process is basically (in the case of the 
reference docfx stuff):

```
mod/one.d -> marmos -> mod.one.json -> marmos-docfx -> docfx YML 
files... -> docfx
mod/two.d -> marmos -> mod.two.json ^
mod/ten.d -> marmos -> mod.ten.json ^
```

Each module gets it's own JSON file, and then these JSON files 
can be fed into some sort of converter tool (e.g. marmos-docfx) 
to generate more files for some external tooling.

So in this case, marmos-docfx will create [API 
Pages](https://dotnet.github.io/docfx/docs/api-page.html) and 
[Table of 
Contents](https://dotnet.github.io/docfx/docs/table-of-contents.html) files that docfx can natively process.

> It's very cool that you are using the parser as a library!

Yeah I was surprised that it was actually kind of painless to 
use! The main annoyance was having to go through what felt like 
50 layers of nesting to find what I want, but that's likely due 
to a lack of familiarity with dmd's source.

The annoying part was getting Meson setup to compile dmd-fe... I 
did it before a few years ago but forgot that I needed to set a 
bunch of versions to disable the backend stuff.

> Can you elaborate on why one would want to use this instead of 
> ddoc or something like adrdox or ddox? Do you think this might 
> be in a state some day to replace the D docs (which I've heard 
> complaints about recently)?
>
> -Steve

For flexibility I guess, and to use existing site generators.

With raw ddoc for example you don't really get an awful lot out 
of it:

* No search bar
* No tables of contents/navigation
* Other quirks such as it only supporting specific section names 
(unless that's changed since I last used it). E.g. if I were to 
do `Workflow:` I remember that ddoc doesn't generate a header for 
it, but just renders it as normal paragraph text.

ddox I've admittedly written off mentally due to past bad 
experiences, without even really considering it for the modern 
day:

* Back in my Windows days I struggled to even get ddox to compile 
or work with dub.
* When I _did_ managed to get it to work, it was failing to deal 
with my project's codebase (can't remember the exact issues), and 
so wouldn't generate the full site.
* I'd looked over some existing ddox sites and honestly I just 
didn't like how it looks by default. I remember trying to use 
some premade theme that looked better but ran into other issues 
with that as well...
* Do I remember correctly that it requires you to correctly pass 
in things like import paths/it does some light semantic pass? I 
feel like I remember getting some issues around that...

It might be better nowadays, but when I used it, it was just pure 
frustration. I was also still stuck with using ddoc's syntax 
(prior to modern ddoc adopting markdown-like features).

It did have better handling of some things though, it had 
cross-referencing stuff that mostly worked out of the box.

adrdox is awesome; handles comments well; has good features, and 
looks good, however:

* (kinda minor but) Similar to ddox it doesn't use dmd-fe but 
instead a (forked?) community parser, so if your code uses newer 
syntax then you kinda just can't get any docs out of it until the 
community parser(s) are updated.
   * I had an issue like this with the shorthand `int a() => 1;` 
syntax for a while as an example.
* I'm wary about its future, especially it's ability to support 
future changes to D if D and OpenD begin to diverge a ton.
* I think it's unable to support the use case where you can use 
the same generator for both an API reference, as well as other 
forms of documentation?
   * I guess ddoc and ddox can support this via standalone .ddoc 
files.
   * For example, C#'s doc site can support [hand written 
pages](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/apis?view=aspnetcore-8.0) as well as normal [API reference pages](https://learn.microsoft.com/en-us/dotnet/api/microsoft.csharp.csharpcodeprovider?view=net-6.0)
   * I will definitely note that this is not needed for a lot of 
projects, but is a deficiency either way.

---

Ultimately I decided that documentation sites are very 
subjective, so I set the goal of leaving the core marmos tool to 
extract all the important bits and pieces, and then leaving it up 
to the converter tools to format everything into how they want it 
to look, and letting the end user decide what site generator they 
wanted to use.

So I started to look into making use of external tooling that 
already has a ton of features ready to use so long as I could 
convert D stuff into the tooling's stuff, and docfx seemed like a 
good fit for a reference/dogfood target.

I initially wanted to explore built in options for generating a 
dump of the AST + documentation, but my [learn 
post](https://forum.dlang.org/post/ktmdbnfjopjqqgjydlpu@forum.dlang.org) had no answers, and the AST dump had other issues as I mention in marmos' README.

I don't really know how useful I'll be able to get the generation 
to be in the end, and it's definitely not as good as adrdox yet, 
but my goal is to basically be adrdox-level of quality but with 
the bonus of being able to use external generators & themes 
(rather than being tied down to default ddoc, ddox, adrdox, or 
having to spend ages on custom CSS styling + JS integration for 
basic stuff like searchbars).

In other words I feel there's value in having the ability to 
extract the structure and documentation of D code for external 
processing purposes, which I don't believe is something that 
fully exists yet (at least, isn't purpose built for the task)?

I guess in theory it could one day generate the D doc site, by 
virtue of having an appropriate converter for the models, but I 
don't think it's all that close to such a thing yet.

And on one other level is that the `Juptune` project as a whole 
that marmos belongs to has an underlying theme of me not really 
wanting to place trust in other people's work in D now, so having 
my own tooling gives me the level of personal flexibility I'm 
looking for.

:sweat: Sorry for the mini essay, but I wanted to write my full 
thoughts down so I can get corrections where needed by people who 
know better than myself.