Better docs for D (WIP)

Wed Dec 30 17:32:56 PST 2015

On Thursday, 31 December 2015 at 00:04:03 UTC, bachmeier wrote:
> It's the process that requires so much overhead that nobody
> wants to contribute. I really tried to do so myself, but I'm 
> busy, and it is senseless that 95% (or more) of the time I 
> spend on it is wasted due to a system that is flawed from top 
> to bottom. The only thing that surprises me is that there are 
> any contributions.

Exactly. I've put my time into ddoc. Let me give you a brief 
history of my efforts:

~2010: I had just written this awesome dom.d library and wanted 
to document it and release it to the world. I write stuff like:

/// Returns the text in the element. For example, innerText of 
<span>foo<span> is "foo" (without quotes)
string innerText();

And it came out mangled because of ddocs embedded HTML "feature" 
which doesn't encode the output. I proposed a fix, using ddoc's 
existing ESCAPES macro, and wrote the patch for dmd. Embedded 
html would be automatically encoded, but you can still define it 
in macro definitions, which can be inline with the comment, so it 
isn't hard to use when you want it.

It was rejected. Walter didn't see what the problem was and I was 
told to just write $(LT)span$(GT)foo$(LT)/span$(GT). Seriously.

Well, that was unacceptable so I decided to take matters into my 
own hands. On Feb 20, 2010, I registered dpldocs.info and started 
writing my own web service to delver dom.d's docs, encoding the 
output correctly, getting the docs out of dmd's JSON output, 
using -D -X.

Well, it worked, but it sucked. The JSON output didn't give me 
enough information to do cross referencing, so I wrote a little 
search engine to paper over it... and that didn't help much.

I moved on to other things, every so often still advocating to 
fix ddoc's escaping, but mostly just writing my docs such that 
they were readable in the source code and not bothering with 
generating them at all. (I'm told my source tends to be fairly 
readable anyway and I like to narrate my thoughts a bit in the 
comments including of implementation details so they understand 
the pros and cons of my decisions.)

At some point around here, I wrote a crappy Javascript table of 
contents for the D website (then hosted at digitalmars.com still) 
- the infamous "Jump list". This was meant to be a temporary 
measure until ddoc got proper table of contents support.

That sucky Javascript hack is still live today.

2011: We had a proposal for a website redesign. Remember this 
one? http://arsdnet.net/d-web-site/index.html

The jump list hack garnered a lot of hate. It was a blob of text 
that we couldn't read, but ddoc's design made a user-defined 
table of contents extremely difficult to implement. A few 
attempts were started, but none finished.

I wrote a program, called improveddoc.d, still on the 'net: 
http://arsdnet.net/d-web-site/improveddoc.d that post-processed 
the generated HTML to create a cheat-sheet table.

This is what it made of std.algorithm at the time:

http://arsdnet.net/d-web-site/std_algorithm.html

I offered to integrate it into the website tools... and got my 
first look at the horror that is that build process. 
Nevertheless, running a little post process script to run that on 
the html was doable. It also would read tag macros and use them 
to organize the content.

The idea (and working program) was rejected because the team felt 
a post-processor was the wrong way to do it.

Instead, Andrei manually wrote a std.algorithm cheat sheet and we 
added a bullet between the links in the jump list hack. Both of 
these have remained basically unchanged to this day and dmd still 
cannot generate a table of contents in ddoc.

We did get a website redesign after that, 
d-programming-language.org was launched, and Vladimir's forum got 
integrated (I wrote one too but his was so much better than mine 
that I gladly yielded). ddoc, however, barely changed.

For the next couple years, despite ddoc being unusable for half 
my libraries because they are web libs and showing HTML examples 
was practically impossible, I still was basically on its side, 
while still working on improving the json support. I had tried 
the json thing and found the compiler approach to be better, but 
the json output is good for a few things and Ddoc's syntax isn't 
all that bad if you aim for source readability and clever use of 
recursive macros can do some pretty cool things.

I put my support behind dmd. With the vibe.d team also pushing 
for improved json support, a few things happened there... though 
little of consequence. My opinion of the json output hasn't 
significantly changed since 2010: cool for a few things, I like 
that we have it, but it is not a substitute for 
compiler-integrated ddoc.

We got another website redesign in these years, and moved to 
dlang.org. The build process finally got documented after I and 
others complained for a while, but it was never actually cleaned 
up.

At the end of 2014, yea, even a year before this writing, people 
were complaining about ddoc's syntax again and asking for some 
Markdown variant to be introduced. Seeing an opportunity to 
revive my old encoding problem and finally write some ddoc for 
myself, I jumped on the `code` syntax and implemented it, 
including proper encoding of the output. Finally, I can write 
`<span>foo</span>` and have it legible to the user!

And with it also alleviating the pressure for syntax sugar from 
the community... this patch was actually accepted this time! For 
the first time in five years, I was excited about ddoc again.

I started using it to write This Week in D (and noticing the 
limits the macro syntax place on legible prose source...) and 
wrote at some length about my simpledisplay.d, cgi.d, and others 
(though not really much on dom.d, ironically. I still haven't 
gotten around to updating that.)

Just a few months ago, I got the simpledisplay docs looking 
pretty good with ddoc: http://arsdnet.net/arsd/simpledisplay.html

2015 was a good year for ddoc and I, we basically got along, 
despite its warts.

Then, October rolls around. A friend of a friend is looking for 
work and I say maybe you can help me with some D stuff. Don't 
know D? Not a problem, it is easy!

It wasn't easy. From installation, problems arose. I'd then show 
how to fix it and explain why.

She'd say "I understand now, but why don't the docs just say what 
you just said?"

Good question. I decide to start editing it in... and quickly hit 
a brick wall that kills my momentum. I couldn't get through the 
nonsense before having to get back to work.

I handle hundreds of D support requests in a year. Sometimes, I 
spend so much time on the chat, the learn forum, or Stack 
Overflow that I feel like I'm a full-time D support engineer. 
Time and time again, people ask things that ought to be pretty 
easy to find, yet they ask us anyway.

Maybe they just don't like searching and ask first. I actually 
don't mind that - I tell my coworkers that if their attempts at 
Google don't yield results in one minute, send me an IM or email 
(and continue looking). There's a good chance I'll have an answer 
in my brain somewhere, or at least a helpful pointer, and I don't 
mind the interruptions at all (if I didn't want them, I'd just 
close the chat client!). Better to ping me than waste hours on a 
fruitless search.

But still, some people say "I tried searching and couldn't find 
it"... and I totally believe that with D. I used it think it was 
just because we were young, but it has now been almost SIX YEARS 
since my first dive into the website issues and it has barely 
changed. There must be something systemic.

Fast-forward to December. We have that base64 question. I go to 
edit the site and get stopped by the process again. Nevertheless, 
I blamed the makefile and saw new documentation Andrei wrote on 
this. Maybe there's still hope.

Then another friend of mine tells me he finally started looking 
at the D Cookbook I gave him last year and is writing some code. 
I watch him work through a couple problems... and notice that the 
website was only useful to him insofar as he could copy/paste 
examples. As soon as he modified them and got an error message or 
tried to put functions together without pre-written examples, 
he'd slow way down. He'd get through it but not easily.

The D leaders know how important examples are. We are often told 
adding more is low hanging fruit. I completely agree. But that's 
not ALL we need. He wants examples to get started, yes, but he 
also wants understanding to go beyond examples. That's where text 
helps. That's where understanding the function signatures help. 
Eventually, when people go to write their own libraries, they 
might want to do Phobos style genericity. You can't do that 
without understanding the function signatures... and those blobs 
of text are not understandable.

Shift to the forum. We got talking about website redesign on the 
forum again a couple weeks ago and I remark how I think just 
whitespace formatting of the constraints would help a lot and 
almost went in dmd to implement it.

I just got sidetracked doing a quick paragraph fix for Andrei... 
and then wanted to see how far I could go. I got it working, 
despite the obstacles that ddoc's freeform macros bring, and 
surprise, it was rejected. Even the simpler patch that just 
collapses blank lines sits unanswered. A tool to look for broken 
links sits unanswered. There's always a call to contributions, 
but when you do push through the painful process to get the code 
up (and the tester, the f$^%$^ing tester), nobody seems to care.

The combined experience over these last six years tells me that 
the website sucks and it keeps sucking because changing it sucks 
even more than using it. A new strategy was required.

And here we are.

I still need to figure out how I want to add new docs and link 
them all in, but I guarantee you it won't consist of editing SRCS 
and MANIFEST and posix.mak and win32.mak and std.ddoc and 
latex.ddoc and <voice trails off>...