Generate core API docs from TSDoc comments #32148

rudolf · 2019-02-27T19:58:37Z

Summary

Uses api-extractor and api-documenter to generate documentation for the Kibana core API from TSDoc comments in the source code. This does not integrate the generated docs into the kibana documentation workflow and won't get published to the website. Instead, these markdown files are meant to be viewed in your editor or on Github. Example of viewing docs in Github

Documentation can be generated using node scripts/check_core_api_changes --docs

Todo:

Take a first stab at fleshing out docs like parameter and method descriptions so they're actually useful. Also add a few missing documentation tags that currently causes api-extractor to fail with a non-zero exit code.
Add test that checks for core api changes by comparing the API signature with common/core_api_review/kibana.api.ts. Running node scripts/check_core_api_changes --accept will update the review file and documentation and has to be commited to pass the test.

Details

To generate the documentation the following steps are executed:

npm run build:types (pre-existing functionality in our build system)
Uses src/type_exports.ts to create target/types/type_exports.d.ts. These types are exported when someone does import {...} from 'kibana' (package.json has a "types" property pointing to ./kibana.d.ts which re-exports target/types/type_exports.d.ts)
api-extractor uses target/types/type_exports.d.ts to determine the surface area of the public API and generates ./build/kibana.api.json.
This is like an AST of the API and it's TSDoc comments used in (3) by api-documenter to render markdown documentation.
It also generates an API review file build/kibana.api.ts and compares this file to src/core/public/kibana.api.md. A difference in the type signatures means the public API signature has been changed which will print out a warning. A developer has to "accept" the change by running node scripts/check_core_api_changes --accept and then committing the updated API review file and documentation.
api-documenter creates documentation by "rendering" the contents of ./build/{public/server}/kibana.api.json as markdown files which we commit to the repository.

Since api-exporter doesn't support import * as Module from __, a technique we use to prevent type collisions in src/core/index.ts we treat the public and server api's as two different packages and run api-extractor twice. The biggest drawback is that api-extractor uses the name field in package.json as for api review files and documentation.

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

~~This was checked for cross-browser compatibility, including a check against IE11~~
~~Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~Documentation was added for features that require explanation or tutorials~~
~~Unit or functional tests were updated or added to match the most common scenarios~~
~~This was checked for keyboard-only and screenreader accessibility~~

For maintainers

~~This was checked for breaking API changes and was labeled appropriately~~
~~This includes a feature addition or change that requires a release note and was labeled appropriately~~

elasticmachine · 2019-02-27T19:59:15Z

Pinging @elastic/kibana-platform

elasticmachine · 2019-03-01T09:52:41Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

Uses api-extractor and api-documenter to generate documentation for the Kibana core API from TSDoc comments in the source code. Documentation can be generated using `npm run docs:api`. I used --no-verify to ignore the following pre-commit hook errors: 1. Filenames MUST use snake_case - api-extractor.json It's possible to specify a different config file, but I prefer to keep the "standard" config file name. 2. UNHANDLED ERROR: Unable to find tsconfig.json file selecting "common/core_api_review/kibana.api.ts". Ensure one exists and it is listed in "src/dev/typescript/projects.ts" This is not a source file, so safe to ignore.

elasticmachine · 2019-03-01T13:28:13Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

I initially wanted to include this as a precommit hook, but it takes quite long to execute (~12s) so might be better suited as a test or as part of the release process. The script currently fails because api-extractor uses an older version of typescript.

rudolf · 2019-03-01T14:12:05Z

Checking for core API changes takes quite long, which can get annoying in a pre-commit hook, so I think it's better suited as a test.

~~api-exporter doesn't yet use the latest typescript compiler introduced in (#32063) so it returns a non-zero exit code which means we can't enforce it as a test yet.~~

I suggest that we don't block on api-exporter and merge this so long, manually keeping docs updated. Once it supports the latest typescript we can update api-exporter and automate the detection of api changes as a test.

elasticmachine · 2019-03-01T14:46:17Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

src/dev/typescript/get_ts_project_for_absolute_path.ts

rudolf · 2019-04-02T07:57:11Z

@monfera Thanks for the review. This shouldn't impact any other teams, but due to a large merge commit, other teams were incorrectly tagged as code owners.

elasticmachine · 2019-04-02T10:30:31Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-04-02T12:24:32Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

mshustov · 2019-04-02T20:08:34Z

docs/development/core/server/kibana-plugin-server.pluginsetupcontext.md

@@ -14,5 +14,5 @@ export interface PluginSetupContext

 |  Property | Type | Description |
 |  --- | --- | --- |
-|  [elasticsearch](./kibana-plugin-server.pluginsetupcontext.elasticsearch.md) | `{`<p/>`        adminClient$: Observable<ClusterClient>;`<p/>`        dataClient$: Observable<ClusterClient>;`<p/>`    }` |  |
+|  [elasticsearch](./kibana-plugin-server.pluginsetupcontext.elasticsearch.md) | <code>{`<p/>`        adminClient$: Observable&lt;ClusterClient&gt;;`<p/>`        dataClient$: Observable&lt;ClusterClient&gt;;`<p/>`    }</code> |  |


ooooh. they fix one thing and break another. let's don't block this PR

mshustov · 2019-04-02T20:12:48Z

docs/development/core/server/kibana-plugin-server.clusterclient.asscoped.md

+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  req | `{`<p/>`        headers?: Headers;`<p/>`    }` | Request the `ScopedClusterClient` instance will be scoped to. |


this is bummer 😞 ... probably we should file some issues in their repo and track progress.
not a blocker for the current PR

mshustov

I think there are several issues in documenter, so it's not a blocker for the current PR, but nice to fix later

cross links in types. when a typeA is mentioned in TypeB.md, there should be a link to typeA.md page
weird alignments in typescript code blocks
escaped line break for object types { adminClient$: Observable<ClusterClient>; dataClient$: Observable<ClusterClient>; }
warning comment for autogenerated documentation page

only problem in the code
https://github.com/elastic/kibana/pull/32148/files/51b8bc1f09219075eeb7209e5a93c3db6c91474e#r270865425

elasticmachine · 2019-04-03T08:54:20Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rudolf · 2019-04-03T09:39:29Z

After the latest merge there are still undocumented classes and missing exports. Some of them are related to #34416 . I want to merge this to plug the leaky bucket of new changes being committed to master which causes api-extractor to fail. After #34416 we can do a cleanup of the exported types and documentation.

elasticmachine · 2019-04-03T10:14:59Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* Generate core API docs from TSDoc comments Uses api-extractor and api-documenter to generate documentation for the Kibana core API from TSDoc comments in the source code. Documentation can be generated using `npm run docs:api`. I used --no-verify to ignore the following pre-commit hook errors: 1. Filenames MUST use snake_case - api-extractor.json It's possible to specify a different config file, but I prefer to keep the "standard" config file name. 2. UNHANDLED ERROR: Unable to find tsconfig.json file selecting "common/core_api_review/kibana.api.ts". Ensure one exists and it is listed in "src/dev/typescript/projects.ts" This is not a source file, so safe to ignore. * Flesh out API docs a little bit * Ignore snake_case check for api-extractor.json * Ignore api-extractor's review file from pre-commit check * Try to fix build failing by using masters yarn.lock * I'm being stupid * Found a better home for ignoring common/core_api_review/kibana.api.ts * Node script for detecting core API changes I initially wanted to include this as a precommit hook, but it takes quite long to execute (~12s) so might be better suited as a test or as part of the release process. The script currently fails because api-extractor uses an older version of typescript. * Fix tslint precommit hook ignore condition * Write tsdoc-metadata.json into ./build * Add LogMeta and ElasticSearch to exported types & docs * Suppress logging when running api-extractor from script * Improve check_core_api_changes script and run as test * Inline api-extractor.json config * Fix check_core_api_changes --help flag * LogMeta TSDoc comments * check_core_api_changes: fail if api-extractor produces warnings or errors And print more useful messages to the console * Move ignored ts files list into dev/file * Add back build:types since api-exporter cannot operate on source files * Upgrade api-exporter/documenter * api-extractor: independantly analyze core/public and core/server Becasue of microsoft/rushstack#1029 api-extractor can't use core/index.ts as a single entry point for analyzing the public and server API's as isolated namespaces. Instead we analyze these projects separately. This introduces other problems like the api review files and documentation always being called "kibana." from the package.json filename. * Build types as part of build task * Include types in typescript browser compilation * Force inclusion of core/public for building types * Fix api review filename in api-exporter errors * Update docs and API review files * Fix api-extractor warnings * Remove ts file ignored list since it's no longer necessary * Rename exported api package name * Review comments * Export other missing types * Upgrade api-documenter to latest beta * Export more missing types * Fix warnings and add api-exporter to Jenkins tests * Correctly handle runBuildTypes() exceptions * Fix another swallowed exception * Fix api-extractor warnings after master merge

* Generate core API docs from TSDoc comments (#32148) * Generate core API docs from TSDoc comments Uses api-extractor and api-documenter to generate documentation for the Kibana core API from TSDoc comments in the source code. Documentation can be generated using `npm run docs:api`. I used --no-verify to ignore the following pre-commit hook errors: 1. Filenames MUST use snake_case - api-extractor.json It's possible to specify a different config file, but I prefer to keep the "standard" config file name. 2. UNHANDLED ERROR: Unable to find tsconfig.json file selecting "common/core_api_review/kibana.api.ts". Ensure one exists and it is listed in "src/dev/typescript/projects.ts" This is not a source file, so safe to ignore. * Flesh out API docs a little bit * Ignore snake_case check for api-extractor.json * Ignore api-extractor's review file from pre-commit check * Try to fix build failing by using masters yarn.lock * I'm being stupid * Found a better home for ignoring common/core_api_review/kibana.api.ts * Node script for detecting core API changes I initially wanted to include this as a precommit hook, but it takes quite long to execute (~12s) so might be better suited as a test or as part of the release process. The script currently fails because api-extractor uses an older version of typescript. * Fix tslint precommit hook ignore condition * Write tsdoc-metadata.json into ./build * Add LogMeta and ElasticSearch to exported types & docs * Suppress logging when running api-extractor from script * Improve check_core_api_changes script and run as test * Inline api-extractor.json config * Fix check_core_api_changes --help flag * LogMeta TSDoc comments * check_core_api_changes: fail if api-extractor produces warnings or errors And print more useful messages to the console * Move ignored ts files list into dev/file * Add back build:types since api-exporter cannot operate on source files * Upgrade api-exporter/documenter * api-extractor: independantly analyze core/public and core/server Becasue of microsoft/rushstack#1029 api-extractor can't use core/index.ts as a single entry point for analyzing the public and server API's as isolated namespaces. Instead we analyze these projects separately. This introduces other problems like the api review files and documentation always being called "kibana." from the package.json filename. * Build types as part of build task * Include types in typescript browser compilation * Force inclusion of core/public for building types * Fix api review filename in api-exporter errors * Update docs and API review files * Fix api-extractor warnings * Remove ts file ignored list since it's no longer necessary * Rename exported api package name * Review comments * Export other missing types * Upgrade api-documenter to latest beta * Export more missing types * Fix warnings and add api-exporter to Jenkins tests * Correctly handle runBuildTypes() exceptions * Fix another swallowed exception * Fix api-extractor warnings after master merge * Update yarn.lock * Fix erraneous type * Revert "Update yarn.lock" This reverts commit 85a8093. * Revert "Fix erraneous type" This reverts commit 7f0550c. * Backport #32440 * Update core api signature and docs

rudolf · 2019-04-16T09:31:25Z

cross links in types. when a typeA is mentioned in TypeB.md, there should be a link to typeA.md page

I've created microsoft/rushstack#1237

weird alignments in typescript code blocks

I created microsoft/rushstack#1239

escaped line break for object types { adminClient$: Observable<ClusterClient>; dataClient$: Observable<ClusterClient>; }

Upstream PR microsoft/rushstack#1234

warning comment for autogenerated documentation page

Fixed in #34817

rudolf self-assigned this Feb 27, 2019

rudolf added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:New Platform labels Feb 27, 2019

rudolf added non-issue Indicates to automation that a pull request should not appear in the release notes v8.0.0 labels Feb 27, 2019

rudolf requested a review from spalger February 28, 2019 14:06

rudolf force-pushed the rm-core-api-docs branch 2 times, most recently from 9134c67 to 12c3bdf Compare March 1, 2019 09:09

rudolf added 4 commits March 1, 2019 11:29

Flesh out API docs a little bit

ee431ce

Ignore snake_case check for api-extractor.json

70e25d7

Ignore api-extractor's review file from pre-commit check

8bcc4a2

rudolf force-pushed the rm-core-api-docs branch from 12c3bdf to 8bcc4a2 Compare March 1, 2019 11:25

rudolf added 3 commits March 1, 2019 13:12

Try to fix build failing by using masters yarn.lock

4cc5f34

I'm being stupid

9926ab4

Found a better home for ignoring common/core_api_review/kibana.api.ts

8a0db42

elastic deleted a comment from elasticmachine Mar 1, 2019

rudolf added the review label Mar 1, 2019

spalger reviewed Mar 1, 2019

View reviewed changes

src/dev/typescript/get_ts_project_for_absolute_path.ts Show resolved Hide resolved

rudolf removed request for a team April 2, 2019 07:55

rudolf added 4 commits April 2, 2019 11:00

Review comments

e6660ab

Export other missing types

c325a56

Upgrade api-documenter to latest beta

21f5abf

Export more missing types

c69c601

Fix warnings and add api-exporter to Jenkins tests

74bc082

rudolf requested a review from mshustov April 2, 2019 12:03

mshustov reviewed Apr 2, 2019

View reviewed changes

rudolf added 3 commits April 3, 2019 10:22

Merge remote-tracking branch 'upstream/master' into rm-core-api-docs

979c57a

Correctly handle runBuildTypes() exceptions

0615c25

Fix another swallowed exception

f64751c

Fix api-extractor warnings after master merge

cfcfb4e

rudolf merged commit 5c45797 into elastic:master Apr 3, 2019

rudolf deleted the rm-core-api-docs branch April 3, 2019 10:26

rudolf mentioned this pull request Apr 3, 2019

[7.x] Generate core API docs from TSDoc comments (#32148) #34421

Merged

mshustov mentioned this pull request Apr 11, 2019

[new-platform] Explicitly define core setup types #34817

Merged

1 task

lukeelmers mentioned this pull request May 12, 2020

[expressions] AST Builder #64395

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate core API docs from TSDoc comments #32148

Generate core API docs from TSDoc comments #32148

rudolf commented Feb 27, 2019 •

edited

Loading

elasticmachine commented Feb 27, 2019

elasticmachine commented Mar 1, 2019

elasticmachine commented Mar 1, 2019

rudolf commented Mar 1, 2019 •

edited

Loading

elasticmachine commented Mar 1, 2019

rudolf commented Apr 2, 2019

elasticmachine commented Apr 2, 2019

elasticmachine commented Apr 2, 2019

mshustov Apr 2, 2019

mshustov Apr 2, 2019

mshustov left a comment

elasticmachine commented Apr 3, 2019

rudolf commented Apr 3, 2019

elasticmachine commented Apr 3, 2019

rudolf commented Apr 16, 2019 •

edited

Loading

Generate core API docs from TSDoc comments #32148

Generate core API docs from TSDoc comments #32148

Conversation

rudolf commented Feb 27, 2019 • edited Loading

Summary

Details

Checklist

For maintainers

elasticmachine commented Feb 27, 2019

elasticmachine commented Mar 1, 2019

💔 Build Failed

elasticmachine commented Mar 1, 2019

💚 Build Succeeded

rudolf commented Mar 1, 2019 • edited Loading

elasticmachine commented Mar 1, 2019

💚 Build Succeeded

rudolf commented Apr 2, 2019

elasticmachine commented Apr 2, 2019

💚 Build Succeeded

elasticmachine commented Apr 2, 2019

💚 Build Succeeded

mshustov Apr 2, 2019

Choose a reason for hiding this comment

mshustov Apr 2, 2019

Choose a reason for hiding this comment

mshustov left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 3, 2019

💔 Build Failed

rudolf commented Apr 3, 2019

elasticmachine commented Apr 3, 2019

💚 Build Succeeded

rudolf commented Apr 16, 2019 • edited Loading

rudolf commented Feb 27, 2019 •

edited

Loading

rudolf commented Mar 1, 2019 •

edited

Loading

rudolf commented Apr 16, 2019 •

edited

Loading