Better docs for D (WIP)

[Adam D. Ruppe via Digitalmars-d-announce]

On Tuesday, 5 January 2016 at 14:18:32 UTC, Andrei Alexandrescu 
wrote:
> There's been a recent discussion with Walter and Martin about 
> using wildcards in makefiles (which would obviate the necessity 
> of being explicit about files).

I actually do NOT use wildcards in most of my own makefiles for 
exactly these reasons, I tend to keep dead files around.

But I also list the file once and only once. There's no separate 
SRC, MANIFEST, DOCS, whatever. I don't even know what the 
difference is between each of those.

However, that's for building code which is separate from building 
documentation. Building code needs a bit more care because it has 
to actually compile. Docs don't, it can make a few extra html 
files without lots of worry.

This is one of ddoc's fundamental technical flaws, by the way, 
being integrated with the compiler. The compiler *needs* 
everything set up right to function properly. You can't compile 
code without the proper conditional compilation settings, for 
example. You can't compile a program with two main()'s.

But docs are different. You don't want a function not to appear 
just because it is under a version(cool_feature) block.


Notice I said "fundamental" rather than "fatal" because we have a 
workaround: the version(D_DDOC) thing. But that sucks. You now 
have to write the whole function prototype again to show up in 
the docs and if it happens to be different on the other 
platforms, well, you might as well just write the docs by hand. 
You've lost all benefits of auto-generation now.


Oh yeah, and the website also has a navbar that is manually 
maintained, and an index page which is often neglected, probably 
because people don't expect to have SIX different places to add a 
new file! (The makefile parts, these macros, and, of course, the 
git add to the filesystem.)

My nav bar is automatic. Click around std.socket to see what I 
did with it today: 
http://dpldocs.info/experimental-docs-2/std.socket.Socket.setOption.4.html


> to produce incomplete builds if some files are missing by 
> mistake.

That would surely be caught by the compiler itself (module file 
not found) or the test runner. In the case of documentation, a 
bunch of broken links would be detected. For druntime, the Phobos 
build would fail, etc.



> Would you recommend switching to wildcards in our makefiles and 
> assume people use git to keep their directories in good shape?

It'd probably work fine in git; worth an exploration. Especially 
for doc building which wants everything in /import, without a 
doubt. Phobos isn't as clear cut because it builds right from 
/src and has some internal files in there, but still, a wildcard 
that excludes the internals beats one that is wrong in the common 
case.

> * Each library version must be built with the same compiler 
> version. The appropriate compiler version might be missing, so 
> it needs to be downloaded, installed, and run on the side 
> without affecting any existing installation.

And this is again because everything is so strongly coupled to 
dmd. I'm not even convinced Phobos ought to be this closely 
coupled (my libs outside Phobos are compatible with at least a 
year of previous versions, sometimes more, and my users find this 
very valuable for stability against the fear of regressions), but 
the documentation certainly shouldn't be.

With an external doc generator, you can get much broader version 
compatibility since it isn't responsible for actually compiling 
the entire language like dmd. It just needs to parse enough of it 
to get useful information to attach to comments, and understand 
just enough to attach some useful text.

For example, my doc generator (built on Brian Schott's libdparse 
btw) knows that "@safe" is worth pointing out to the user. But it 
doesn't need to know what it actually means. It doesn't run that 
level of semantic analysis and is thus isolated by changes in the 
safed spec. dmd doesn't have that luxury.


> My understanding is that your system does not do any of these 
> things.

I ask you to consider the importance of compromising website 
quality in favor of those things from a strategic viewpoint.

If the pdf is really popular enough that its continuance is 
warranted (tbh, I can't understand why it would be, it is 
equivalent to or inferior to even the current website in almost 
every way), you should find a way to do it that doesn't hurt the 
main site.

I certainly won't be concerning myself with it. I do care about 
offline viewing, and have a simple solution: I can offer a 
zip/tarball of the website html files. They will just work 
without a network connection as well.

BTW: generating those other formats from html is also pretty easy 
and can be done after the fact.

> The script 
> https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak stands at 498 lines. What ways there are to make it considerably simpler and smaller?

I've looked at this before and didn't even understand what it is 
trying to do, much less why it is doing that. I don't feel 
comfortable modifying code that I don't understand.

And frankly, I have zero interest in dealing with complicated 
build systems. My view is if they aren't nearly invisible, they 
have failed me.

And I deal with failing subordinates in a similar manner to Darth 
Vader:

https://www.youtube.com/watch?v=aV2DLkDPwM8&feature=youtu.be&t=33s

> Indeed the web forms sound like a great idea. Would you want to 
> work on such for the mainline website as well? It's not a 
> contest; the better everything is, the better off everybody is.

It's not the web form that's difficult though, it is getting the 
new file integrated with the build and actually running it. The 
existing site makes that exceedingly difficult.

> Nice. What could we do in ddoc and ddox to have a similar 
> improvement?

I implemented it as a magic macro, they could do the same.

This kind of expansion cannot be implemented as a ddoc macro 
itself because it needs more info than text expansion (and mine 
has an if statement if name lookup fails), but it is fairly easy 
to express in D.

> How do you envision scaling it up as the system becomes more 
> successful and you get more requests than you can handle 
> reasonably quick?

I'll cross that bridge when I get to it. Let's not let tomorrow's 
potential hurdle be today's showstopper.


> Would you be willing to set up such a system for dlang.org as 
> well? It would be absolutely fantastic.

People can (and often do) email me all kinds of things. But I'm 
not the god-emperor of dlang.org so what I can easily do there is 
limited.

> Anyone could do the same today for all pull requests - just 
> fetch them, fix them, and resubmit with credit. Additionally, 
> team members could just pull the PR, fix it, and make an 
> additional commit.

Indeed, that's not a technical barrier. But it is a cultural 
barrier - the D PR process (which btw is strongly encouraged by 
the GitHub UI, it isn't just us) has more emphasis on leaving 
comments than on fixing it yourself.

> I'll note that you are already a member of that team so you 
> already had the rights to exact quite a few of the improvements 
> you intend to make going forward. Would you please join me in 
> taking a look at older PRs (oldest is from 2014) and pull them 
> in?

The technical access and the feeling of ownership are two 
different things.

If you gave me a key to your house, would it be fine for me to do 
in some day when you're not home and start remodeling it?

That's the problem with the "lieutenant" thing that's been 
discussed before. Lieutenants are junior officers who just keep 
things running according to regulation while the captain is busy 
elsewhere. If anything extraordinary happens, they report it up 
the chain of command.

What we need is a greater amount of delegation. 
Lieutenant-generals perhaps: individuals who have full 
authorization and broad autonomy to run an entire sector of 
operations as they see fit.

> Overall my understanding of your message is "My system would be 
> better not for a fundamental technical reason, but because I am 
> willing to pour in it time and talent." This is the kind of 
> argument I have a lot of respect for.

Well, there are technical reasons too. Tying ddoc to dmd was a 
short-term win, it got to reuse the compiler to do hard work at a 
time when there were no realistic alternatives.

But, today, it is limiting. Sharing code with the compiler means 
ddoc is subject to the compiler's trade-offs.... which, given a 
choice that strengthens code or strengths docs, code is going to 
win every time. As it should, that's the compiler's job. Moving 
ddoc to a separate path will result in code duplication at best, 
so at that point, you don't gain much from staying in dmd anymore.

Ddoc's text macro system, while an elegant implementation and 
good enough for many things, is also a fundamentally limiting 
factor (and now tied up a bit due to backward-compatibility with 
arbitrary user-defined extensions), and prone to complications. 
It puts a limit on how much analysis of the source text we can 
realistically do because anything could expand to anything else. 
It is a  mistake. (The missing word there btw is $1, disappeared 
because ddoc mistakes it for syntax and just erases it. 
Seriously, try it.)

Ddoc's total failure to understand what it is generating means it 
cannot automatically link. And since it lacks conditionals, you 
can't even program it to link to user-defined things. (And no, I 
am not suggesting we add $(IF). We don't need yet another 
programming language... nobody wants to write their documentation 
in uncommon Lisp. We want it to just work.)


Fixing these will mean breaking backward compatibility. If you're 
willing to do that, great! I've been writing a ddoc 2.0 generator 
:P