Docbook to CommonMark Best Practices

[ryantm]

This is a meta-issue for details about how to convert the manual from DocBook to Markdown. We'll try to update this top post to be the most recent best practices.

No hard-wrapping

Use editor soft wrapping to edit:

• visual-line-mode in Emacs

Automatically convert DocBook to Markdown

$ nix-shell -I nixpkgs=channel:nixos-unstable -p pandoc --run 'pandoc -f docbook -t markdown --tab-stop=2 --wrap=none doc/languages-frameworks/ruby.section.xml > doc/languages-frameworks/ruby.section.md'

Target format is markdown instead of commonmark since it allows using extra syntactical extensions.

This can save a lot of work, manual changes are still needed, though.

Format Markdown with prettier

$ nix-shell -I nixpkgs=channel:nixos-unstable -p nodePackages.prettier --run 'prettier --prose-wrap never --write doc/languages-frameworks/ruby.section.md'

Note that prettier does not support some syntax like admonitions (blocks delimited by :::) so you will need to manually correct those if they appear in the document.

Preserve section anchor links

Match existing DocBook anchors on headers so that links are not broken:

# header {#docbook-anchor-name}

Use fenced code blocks instead of indented ones

Fenced code blocks allow specifying the used language for syntax highlighting.

Use ShellSession for shell sessions

```ShellSession
$ nix-env -iA hello
```

For now, keep DocBook tags when functionality can't be reproduced

```{=docbook}
Put any docbook code here
``

Using a `shell.nix` as described (see <xref linkend="creating-a-shell"/>) should just work.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/happy-docsgiving/10214/14

[fzakaria]

What about including how to do a quick pass with pandoc

nix run nixpkgs.pandoc --command pandoc -s --from docbook doc/builders/packages/weechat.xml --to commonmark -o doc/builders/packages/weechat.section.md

[fzakaria]

@ryantm should an issue be opened to make Prettier part of the CI for markdown if we are asking to do it.
(similar to the editorconfig)

[fzakaria]

I also don't have nix shell enabled; you can use nix run instead however:

[bqv]

I also don't have nix shell enabled; you can use nix run instead however:

Maybe you can nix --experimental-features "nix-command" with a recent nix

[ryantm]

#! /usr/bin/env nix-shell
#! nix-shell -i bash -p git meld gitAndTools.gh

trap "kill 0" SIGINT SIGTERM EXIT

PR="$1"

gh pr checkout "$PR"
MB="$(git merge-base HEAD master)"

nix-build pkgs/top-level/release.nix -A manual
CONVERTED_MAN="$(readlink result)"

git checkout "$MB"

nix-build pkgs/top-level/release.nix -A manual
MB_MAN="$(readlink result)"

nix-shell -p python3 --run "python3 -m http.server 8000 --directory $MB_MAN/share/doc/nixpkgs" &
nix-shell -p python3 --run "python3 -m http.server 8001 --directory $CONVERTED_MAN/share/doc/nixpkgs" &

"$BROWSER" localhost:8000/manual.html
"$BROWSER" localhost:8001/manual.html

meld "$MB_MAN/share/doc/nixpkgs/manual.html" "$CONVERTED_MAN/share/doc/nixpkgs/manual.html" &

sleep 5
read -r -p "Press enter to exit."

Requires that you have GitHub's gh CLI program set up.

Then you call it with a PR number.

./review-docs.sh 105021

I made a script to review doc PRs. cc @jtojnar

[ryantm]

I updated the nix shell command to use nix-shell. Thanks for pointing that out @fzakaria!

[stale]

I marked this as stale due to inactivity. → More info

[Artturin]

lets convert the manpages to markdown too https://github.com/NixOS/nixpkgs/tree/master/nixos/doc/manual

nixpkgs/nixos/doc/manual/default.nix

Lines 235 to 253 in 443b46d

	# Generate the NixOS manpages.
	manpages = runCommand "nixos-manpages"
	{ inherit sources;
	nativeBuildInputs = [ buildPackages.libxml2.bin buildPackages.libxslt.bin ];
	allowedReferences = ["out"];
	}
	''
	# Generate manpages.
	mkdir -p $out/share/man
	xsltproc --nonet \
	--maxdepth 6000 \
	--param man.output.in.separate.dir 1 \
	--param man.output.base.dir "'$out/share/man/'" \
	--param man.endnotes.are.numbered 0 \
	--param man.break.after.slash 1 \
	--stringparam target.database.document "${olinkDB}/olinkdb.xml" \
	${docbook_xsl_ns}/xml/xsl/docbook/manpages/docbook.xsl \
	${manual-combined}/man-pages-combined.xml
	'';

[jtojnar]

NixOS manpages are generated from the options subset of the manual and if we produced a markdown as the intermediate, we would need markdown to *roff converter. But we would have to convert option descriptions to markdown first, which is one of the most complicated parts given our closure requirements.

When we have done that, it might even make sense to use NixOS options with Markdown → DocBook → *roff, since nothing really beats the closure size of docbook-xsl for manpage generation.

[Artturin]

im talking about these https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/man-nixos-install.xml
which dont have anything to do with the nixos manual afaik

[ryantm]

The purpose of this issue is to talk about how we should be using CommonMark syntax and formatting our files. Let's make a new issue for man pages. It would be fine to link it to the same project.

[Artturin]

#161122

[milahu]

first stab at nixpkgs/doc/functions/library/attrsets.xml

https://github.com/milahu/docbook2md

obviously not complete, but javascript is much more "hackable" than haskell (pandoc)

[jtojnar]

@milahu Nice, though do not forget to preserve section anchor links, as mentioned in the OP.

[milahu]

preserve section anchor links

of course. current version:

attrsets.md (github) (gitlab) (gitea) (srht)

[jtojnar]

No need to create links to headings, we have a script for that in the manual. Only the anchor definitions (using the syntax described in the Preserve section anchor links section in the top of this issue or in the manual) is necessary.

[milahu]

yeah i know. my goal is

the generated markdown should be pretty and functional on the github blob API

obviously its redundant to have headings like

# [Attribute-Set Functions](#sec-functions-library-attrset)<a id="sec-functions-library-attrset"></a>

but this can be solved with a markdown linter

to add a new heading, i write

# Attribute-Set Functions {#sec-functions-library-attrset}

and the linter converts that to github-compatible markdown

a linter makes it much easier to refactor code

[jtojnar]

The RFC 0072 explicitly mentions the use of extensions, which implies that rendering on GitHub is a non-goal. It also explicitly mentions that HTML should be avoided.

We also already have a tooling (pandoc filters and various tool-specific parsers) that allow transformations of the current format, as documented in the Nixpkgs manual. So I do not think there should be need to introduce yet another syntax, especially one going against the settled RFC.

Edit: Just randomly clicked on the # link next to a heading on your srht example and it creates anchors like this: a-hreffunction-library-libattrsetattrbypath-example-value-existsextracting-a-value-from-a-nested-attribute-setaa-idfunction-library-libattrsetattrbypath-example-value-existsa 🙀

[noamraph]

@ryantm perhaps edit the post to use '''console instead of '''ShellSession? mdbook highlights it. See rust-lang/book#1449