[Dlang-internal] Potential changes to DDoc

[David Gileadi]

(I don't know if this is the right forum for this post, so if not please 
let me know.)

I'm working on adding Markdown support to DDoc. I'm doing it by 
modifying the highlightText method in doc.d [1].

I've been trying to make my changes non-breaking, in that existing DDoc 
files should be rendered the same as they currently are. However I've 
realized that there may need to be some breakage, and I'd like to 
discuss it.

First of all, the CSS styles in the default DDoc theme remove all 
styling from headers, blockquotes, list items, pre and code sections, 
etc. [2] The effect of this is that Markdown features like headers, 
quotes, etc. don't display correctly under the default theme. I'd like 
to modify the default DDoc theme to avoid de-styling these elements.

Secondly, HTML is treated differently under Markdown and DDoc. In DDoc 
anything directly within an HTML tag (i.e. between < and >) is passed 
through unchanged. However if it's not part of the tag it is treated as 
regular DDoc text and subject to regular processing. In Markdown all 
text between a given set of HTML tags is passed through unchanged [3]. 
I'd prefer to switch to the CommonMark handling, allowing whole blocks 
of HTML to be passed through directly.

Thirdly and most controversially is how DDoc handles blank lines between 
paragraphs. Markdown wraps paragraphs in <p></p> tags and collapses 
multiple blank lines between paragraphs (i.e. ten blank lines is treated 
the same as one). DDoc inserts $(DDOC_BLANKLINE) macros on each blank 
line it encounters, which in the default theme inserts <br><br>, in the 
Phobos theme inserts <div class="blankline"/>, and for LaTeX inserts 
/par. Phobos docs also have the common practice of hand-wrapping 
paragraphs in $(P) macros which inserts <p></p> tags.

The result of DDoc's handling is that under the default theme if you 
have, say, three blank lines between paragraphs then it results in five 
blank lines being rendered between them.

I'm not yet sure it's feasible, but I'd really like to change DDoc to 
treat paragraphs the way Markdown does--wrap them in, say, a $(P) macro 
and collapse multiple blank lines. Any desired extra blank lines could 
be inserted by hand, via the $(BR) macro or similar. However this could 
change the rendering of existing documentation which depends on the 
$(DDOC_BLANKLINE) macro.

What do you think about the proposed changes? Should I try to implement 
any or all of them?

-Dave


Appendix A: Planned Markdown support
------------------------------------

I plan to support the following Markdown features, as specified by 
CommonMark [4]:

- Headings, both the ATX and setext kind
- Styling text with * (but not with _)
- Quoting text blocks with >
- Code blocks fenced with ```, ~~~ and DDoc's existing ---
- Backtick-quoted code as it currently works in DDoc
- Nesting lists
- Thematic breaks (<hr>)
- Inline links, although I'm not sure if I will support reference links [5]
- Markdown escapes with \
- HTML, although that's a discussion item (see above)

I don't plan to support the following Markdown features:

- Indented code blocks because existing DDoc comments are often already 
indented
- Using --- for HRs and second-level headings because DDoc uses it as a 
code block fence
- Tables
- Backtick-quoting code across multiple lines
- Github-specific features like task lists, mentioning people and teams, 
referencing issues and pull requests, and emoji


[1] https://github.com/dgileadi/dmd/tree/ddoc-markdown
[2] https://github.com/dlang/dmd/blob/master/res/default_ddoc_theme.ddoc#L70
[3] http://spec.commonmark.org/0.28/#html-blocks
[4] http://spec.commonmark.org/0.28/
[5] http://spec.commonmark.org/0.28/#reference-link