General architecture changes / getting involved

[dylans]

Not sure if this is the best place to raise this, but here it goes:

We've been working on documentation for Dojo 2 and are interested in making some hopefully useful changes to improve TypeDoc. We'd rather reach out to collaborate than start yet another project.

Overall, some of our goals include:

Decouple the parsing from rendering approach

The main reason for this is that we would probably prefer a different approach to how we render our API docs, and a fairly substantial part of TypeDoc is focused on how things will be rendered. This would also make it easier to include partial API docs, e.g. say you want to embed a small API subset into a tutorial.

We think that decoupling doc parsing and rendering into separate packages would lead to better architecture decisions by having one package that parses docs into an intermediate format, and then the separate rendering package that reads content in this format. We realize they are in separate modules today, but the dependency chain for the rendering side of things is pretty large. It would also likely remove a few patterns we see today such as copying backbone/events directly into the project, and including all of lodash rather than just the sub-packages that are needed.

Another goal with this would be to keep things smaller and easier for someone to follow, rather than a large, somewhat monolithic package for all things related to generating API documentation from TypeScript (I say somewhat, because things are separated into modules, but the assumption with the current architecture is that you want to use the current approach and opinions around rendering).

Override code with comments

We have an implementation of an object composition system, dojo/compose, and TypeDoc currently makes supporting this nearly impossible because TypeDoc doesn't know what to do with objects that are created using our composition system. So we’d like to propose a way to extend TypeDoc to introduce other object type systems that can be re-used. Our initial thoughts here are to offer an optional ability to look to function and variable comments (like jsdoc) for type information, and override the inferred type if they are present.

Model based design, annotations, and decorators

It does feel like there's quite a bit of complexity and code to model the model. It feels like there's a pretty strong drive to leverage Domain Driven Design (DDD) and annotations/decorators are commonplace. This approach is not widely used in JavaScript, which we feel may limit third party contributions due to the complexity it is adding currently.

Also Decorators are used pretty heavily, and the TS implementation and the version in ES.next are starting to diverge. It's probably ok, but it may be worth a rethink with this approach.

Some minor nitpicks

• Remove remaining uses of var
• Use util/fs consistently
• Directory exports with index.js, we would probably change this approach

POC

We created a couple of POCs ( https://github.com/rishson/api-doc and https://github.com/kitsonk/api-doc ) that leverage TypeScript Language Services (similar to how TypeDoc does), to determine how we would address our dojo/compose challenge, and to try to identify a fairly simple alternative to TypeDoc.

The general approach of this POC is to follow a publishing pathway:

• grunt doc (task creates the documentation by running the Collector.. obviously it doesn't need to be Grunt, we just found this to be an easy option)
• Collector (finds all repos in a top level dir, by finding .tsconfig files)
• Runner (bootstraps the Parser with repo files)
• Parser (takes TS code from repo files and creates Doc object)
• Generator (takes Doc object and creates Markdown files)
• grunt doc -publish (task takes markdown files and creates git tag of changes and pushes to a repo)

The above solution produces markdown (.md) files, which we can store to then be leveraged elsewhere (for example, use something like hexo to go from markdown to HTML).

This POC is relatively simple and small, and of course doesn't come close to solving all of the needs of TypeDoc. But it might provide something useful in thinking through how a refactored TypeDoc might work?

We would much rather work together than build something separate as there aren't likely to be that many people in the world interested in working on something like this, assuming there's some agreement with the things we're proposing. :)

Please let us know your thoughts and feedback. We look forward to getting involved!

[blakeembrey]

@dylans A brief comment, but hopefully it's enough: I'd love to see someone take this project in the direction it needs to go. I'm not a fan of the current architecture, I feel it's a real pain point for anyone trying to get fully involved. Definitely open to refactoring it. I'd like to keep things are together as we can, so if you are working on this anyway I'd love for you to be involved with TypeDoc instead of diverging the ecosystem.

We can figure out the best way to move this forward, but I think we'd need some parity with the existing TypeDoc. Once that is achieved, we can merge a PR into master and tidy up for a new major release. I'd be happy to add you and others as maintainers at that point too. With the default themes, I'd be happy to move them into this repo with the PR as making these two major changes may be tricking with them being separate - but I'll leave that up to you to decide.

If there's anything I can help with or you'd like to chat more, you can also email me (email on my GitHub profile) and we can figure things out quicker that way.

Related: #256.

[dylans]

Thanks @blakeembrey , we'll be in touch shortly and we'll start working on some ideas as well. I agree that the goal with any minor or major changes would be to keep TypeDoc parity, otherwise we'll break the existing user base, etc.

[aciccarello]

I think it'd be great to see someone continue developing this project. It has some really great features already but needs some work to make it more usable.

We think that decoupling doc parsing and rendering into separate packages would lead to better architecture decisions by having one package that parses docs into an intermediate format,

This would help with issues like #174 and #251. Even if the parser and renderer are separate packages, would you want to keep them in the same GitHub repo?

Our initial thoughts here are to offer an optional ability to look to function and variable comments (like jsdoc) for type information, and override the inferred type if they are present.

I thought TypeScript already does this. I know the Salsa feature added a jsdoc parser for JavaScript projects using the TS compiler but IDK how/if it is used on TypeScript projects or if TypeDoc is looking at the wrong properties.

This approach is not widely used in JavaScript, which we feel may limit third party contributions due to the complexity it is adding currently.

👍 I agree that this is a barrier to contributions so I'd be happy to see the structure changed to simplify things.

[dylans]

Thanks. We have a big need for Dojo 2 and Intern 4, so we're happy to help.

I think it'd be great to see someone continue developing this project. It has some really great features already but needs some work to make it more usable.

We think that decoupling doc parsing and rendering into separate packages would lead to better architecture decisions by having one package that parses docs into an intermediate format,

This would help with issues like #174 and #251. Even if the parser and renderer are separate packages, would you want to keep them in the same GitHub repo?

My initial thought was to make them separate repos. The main reason being that the renderer requires quite a few dependencies that aren't needed if you're not using the default renderer. It might also encourage people to create alternative renderers (I know we will be creating one).

Our initial thoughts here are to offer an optional ability to look to function and variable comments (like jsdoc) for type information, and override the inferred type if they are present.

I thought TypeScript already does this. I know the Salsa feature added a jsdoc parser for JavaScript projects using the TS compiler but IDK how/if it is used on TypeScript projects or if TypeDoc is looking at the wrong properties.

Indeed, it is supported via TS Language Services, but from what we can tell does not currently work with TypeDoc.

This approach is not widely used in JavaScript, which we feel may limit third party contributions due to the complexity it is adding currently.

👍 I agree that this is a barrier to contributions so I'd be happy to see the structure changed to simplify things.

Thanks! 👍

[dylans]

Note that #393, #398, and #401 are our early efforts into working on this... would love some feedback on those issues/PRs for these initial minor changes, before we start doing more invasive things. ;) cc @blakeembrey @aciccarello

[nicknisi]

@blakeembrey @aciccarello I'm very interested in getting more involved in this project! I've submitted a few issues and I'm working on fixes for them. In addition to the comments above, I took a stab at breaking out some of the things we could do to improve the codebase and make it easier for new contributors to get up and running more quickly.

Improvement Ideas

Improvement Ideas

Codebase Refactor

1. Remove var use in favor of const/let.
2. Add more documentation comments.
3. Update interface names to follow the current TypeScript best practices, removing the I prefix.
4. Remove index files and directly import the needed exports directly into the modules where they are needed.
5. Consider switching to default exports where applicable as it helps to self-document the most important exports of a module (if any).
6. Use util/fs consistently.
7. Consider adding tslint to the project.
8. Consider refactoring the converter code to just convert nodes. Currently, the node converters are also responsible for registering the nodes to the project and for triggering events such as EVENT_CREATE_DECLARATION. It also can return null from the converter based on the node and the TypeDoc configuration.
9. Consider refactoring the dependency injection through decorators and how classes such as converter plugins are imported and automatically loaded as the current method, as it’s kind of tough to follow along.
10. Add tests for more configuration options.

Converter / Renderer Split

1. Split the renderer and its dependencies into its own repository
2. Come up with a output format from the converter (could just be the —json output) that the renderer (and other renderers) can consume to produce the documentation output.
3. Figure out a way to tie in which renderer to use through the TypeDoc config or cli flags.

I'd love to discuss these and more ideas to improve TypeDoc, along with next priorities!

[cancerberoSgx]

somewhat related, this template works fine for generating markdown output: This works fine and is active: https://github.com/tgreyjs/typedoc-plugin-markdown and the project is active. My two cents