docgen: flexible page structure

[FedericoCeratto]

Current state

nim doc enforces a fixed order (Imports, Types, Consts, Procs, Templates) and sorts elements alphanumerically within each category.

Yet, often procs/types/templates are grouped together in a source file based on what they do and the order is meaningful. Different groups are separated by comments.

Feature request

nim doc could parse a flag in the source file and structure the generated HTML page in different ways:

1. Current behavior (default if the flag is missing)
2. Split the generated documentation in chapters based on comments with a special syntax. Items within each chapter are orders as usual (Imports, Types, Consts, ... plus alphanum order).
3. Like 2 but the items retain the same order as in the source file

[ire4ever1190]

Maybe be able to group by type (of the first parameter) for procs/macros/templates also?
Default docs has 'group by type' via js but it doesn't seem to work and would be nice if the page could be generated like that without the need for user intervention

[a-mr]

That sounds like a module divided into weakly-coupled sub-modules.

It would be logical to split the module into a few files explicitly. And then use include for each.

What about proposing another feature instead of p.2:
to add an option to form different sections for each included file
?

In that case there is no need for adding special comments to every symbol.

[FedericoCeratto]

@a-mr having to split a source file to satisfy the needs of a documentation tool sounds quite heavy-handed and awkward. Also it would not satisfy the use-case 3.

[a-mr]

@FedericoCeratto ,

I can agree that we do need this feature without regard to that particular use case.

Though adding a dedicated comment to each symbol still seems non-optimal.
How about reverting this linking by adding the appropriate directive to module description (first doc comment at the top of file)? I suggest to introduce a custom RST directive symbols which would accept a list of symbols to include into its place.

An example at the top of file:

## Chapter 1 name
## --------------
## .. symbols:: f1 f2 f3
##
## text text text...
##
## Chapter 2 name
## --------------
## .. symbols:: f4 f5 f6

...
proc f1*() = ...
const f2* = ...

This directive can accept

• :sort: option with values this (default: provided in the list), on (alphanumeric), off (source file order)
• :headings: option that turns on/off Imports, Types, ... headings. E.g. if only one symbol is included into a particular place then printing a heading seems redundant so :headings: off may be assumed by default

The syntax will be normal RST, e.g. for case 3 this will work:

## .. symbols::
##    :sort: on
##    :headings: on
##    f1 f2 f3 f4 f5 f6

So all symbols mentioned in any symbols directive will fall into its place (in module description = 1st half of generated HTML) while not mentioned ones will still fall back to the 2nd half of generated HTML file as it happens now. If we need to put all not mentioned symbols in some particular chapter we can also add :other: option to signify the fall-back place (which can be only one in a file):

## .. symbols::
##    :other:

This way there is no need in any flags for nim doc also.

W.r.t. to Table of Contents (the left column) I suppose that it will always contain all sorted symbols. In case of :headings: on option given then it makes sense to also include the mentioned symbols into a part of TOC dedicated for module description. In that case the same symbol will appear twice in TOC (module description part of TOC and final alphabetic list) but the contents will still appear only 1 time in the body — in module description.

What are you thoughts?

[a-mr]

The proposed mechanism is similar to .. kernel-doc:: directive that is used in Linux kernel RST documentation: https://www.kernel.org/doc/html/v5.17/doc-guide/kernel-doc.html#including-kernel-doc-comments

It's worth reusing syntax, e.g. using this in a .rst file

.. nim-doc:: lib/pure/os.nim
   :identifiers: walkDir

instead of symbols directive proposed above.

This syntax is supposed to be used externally from .rst.

We can do this inside .nim file also by omitting file name:

.. nim-doc::
   :identifiers: walkDir

[a-mr]

After pondering a little, I changed my opinion. It's worth to add an ordering without too much hustle.

May be preferable solution is to get away with even simpler methods, without additional options:
Just adding additional headings in comments and making symbols (appearing below these headings) follow them.

## Module description

## Auxiliary code section
## -------------------------

proc a*() = ...

## Main code section
## --------------------

proc f*() = ...
proc g*() = ...

Of course, we will probably want to change the order of sections — normally we have most important procs in the end of file because of Nim's requirement of usage after declaration. For that sake it's enough to introduce additional directive for reordering of sections, that could look like this:

## .. sections::
##    :order:
##       - Main code section
##       - Auxiliary code section

placed in Module description.

I need to check feasibility of such approach esp. w.r.t. how forward declarations are represented in Nim AST. But anyway it seems the most simple and logical one.

Generally speaking the problem is decomposed into 2 independent ones:

1. Attaching symbols to some (HTML) sections
2. Representing & ordering symbols inside the sections

Here we will solve p.1. by making symbols follow RST/Markdown comment structure in 1st PR. Regarding p.2. — it can be implemented separately in 2nd PR, by addition of the following directive:

## .. symbols::
##    :order: f g

for precise order,
or e.g.:

## .. symbols::
##    :order-follows-code:
##    :type-sections: off

for the same order as code + turning off symbol kind headings (proc, const)