init support for doc-comments

[hsjobeki]

@infinisil

Adds support for doc-comments. (see: NixOS/rfcs#145)

• support for doc-comments in parallel.
• Shift all markdown headings down one two levels, so the nixpkgs manual rendering still works.
• Backwards compatibility with arguments
• Native compatibilty with structured arguments (referred as 'lambda formals' within rfc145)
• Unit Tests
• Documentation & Changelog update

Doesn't discover new items, it just renders, what was rendered before. (e.g. lib.lists.map keeps name and section heading)
But If the content is a new doc-comment (surrounded by /** */) then it doesn't perform the 'nixdoc' rendering with Example: and Type: sections. Instead the section headings are written in explicit markdown by the user/documentation writer.
All section headings are automatically shifted down two levels. (Means you can and should currently write H1 to H4 in your doc comments)

This is certainly a limitation, to resolve this we need to change the nixpkgs manual rendering tool (nixos_render_docs) which could be one of the next steps.

Moved some rust functions into separate files. Having everything in main.rs makes it hard to maintain two separate feature sets.

• functions that are only needed for legacy support: legacy.rs
• shared functions that are only needed formatting content: format.rs
• unit tests: test.rs

Version 3.0.0

This PR bumps the version to 3.0.0

• Added Changelog Entry
• Added Migration tutorial
• Reformulate Readme

[infinisil]

Nice! We should definitely wait for the RFC to be accepted, which should be pretty soon. After that I'm generally in favor of merging it, but we definitely need tests, documentation and a changelog entry for this.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-01-04-documentation-team-meeting-notes-102/37754/1

[hsjobeki]

@infinisil could i ask you for some review & suggestions?

[infinisil]

Nice, I've been eagerly awaiting a ping for it to be ready :D. I can't get to it immediately, but I'll take a close look when I get to it!

[infinisil]

Overall looking really good!

I'd like to see some extra tests:

• One to make sure that the headings get indented properly

• One to make sure omitting a doc comment from an attribute doesn't duplicate the previous one:

  {
    /** Comment */
    foo = 0;
  
    # This should not have any docs
    bar = 1;
  }

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-02-15-documentation-team-meeting-notes-109/39826/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-54/39990/1

[hsjobeki]

@infinisil. Just worked your feedback in.

[hsjobeki]

@infinisil to keep you updated.
I just bootstrapped a cli tool https://github.com/hsjobeki/flutsch. Which in combination with a static analysis tool could replace nixdoc entirely.

Features such as #98 make more sense on nix value level. Unfortunately the comments are not part of the c++nix ast which requires invoking an addtional (e.g. rnix based) tool.

[infinisil]

Just some minor bits now

I just bootstrapped a cli tool https://github.com/hsjobeki/flutsch. Which in combination with a static analysis tool could replace nixdoc entirely.

Nice! I generally don't mind archiving nixdoc if there's something better. Though I definitely can't say I'm a fan of C++ 😅

[hsjobeki]

Just some minor bits now

I just bootstrapped a cli tool https://github.com/hsjobeki/flutsch. Which in combination with a static analysis tool could replace nixdoc entirely.

Nice! I generally don't mind archiving nixdoc if there's something better. Though I definitely can't say I'm a fan of C++ 😅

Yes. C++ is horrible.
My vision is to use it only for the information that we need to pull out of the native evaluator. (Using rust with the tvix evaluator would be a nice alternative but we wouldn't want nixpkgs docs to depend on tvix^^)

Then using a static rust/rnix based tool on top of that (introspection information) to generate the markdown docs.