-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution] Provide an utility to merge domain-specific OpenAPI bundles with the other Kibana bundles #186356
Comments
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Pinging @elastic/security-solution (Team: SecuritySolution) |
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
api_docs
folderoas_docs
folder
oas_docs
folderoas_docs
folder
oas_docs
folder**Addresses:** #186356 ## Summary This PR adds OpenAPI spec files merger utility (programmatic API). It provides a similar functionality as `npx @redocly/cli join` does and takes into account [discussion results](#183019 (comment)) - provides a simple way to produce a single Kibana OpenAPI bundle - extends `requestBody` and `responses` MIME types with a version parameters `Elastic-Api-Version=<version>` to avoid different API endpoint versions conflicts - has flexibility to adjust Kibana needs The utility is exposed from `kbn-openapi-bundler` package. ## Details **OpenAPI merger** is a tool for merging multiple OpenAPI specification files. It's useful to merge already processed specification files to produce a result bundle. **OpenAPI bundler** uses the merger under the hood to merge bundled OpenAPI specification files. Exposed externally merger is a wrapper of the bundler's merger but extended with an ability to parse JSON files and forced to produce a single result file. It is able to read OpenAPI spec files defined in JSON and YAML formats. The result file is always written in YAML format where every `requestBody` and response in `responses` extended with document's `info.version` value added as a MIME type parameter `Elastic-Api-Version=<version>`. Currently package supports only programmatic API. As the next step you need to create a JavaScript script file like below ```ts require('../../src/setup_node_env'); const { resolve } = require('path'); const { merge } = require('@kbn/openapi-bundler'); const { REPO_ROOT } = require('@kbn/repo-info'); (async () => { await merge({ sourceGlobs: [ `${REPO_ROOT}/my/path/to/spec1.json`, `${REPO_ROOT}/my/path/to/spec2.yml`, `${REPO_ROOT}/my/path/to/spec3.yaml`, ], outputFilePath: `${REPO_ROOT}/oas_docs/bundle.yaml`, mergedSpecInfo: { title: 'My merged OpenAPI specs', version: '1.0.0', }, }); })(); ``` Finally you should be able to run OpenAPI merger via ```bash node ./path/to/the/script.js ``` or it could be added to `package.json` and run via `yarn`. After running the script it will log different information and write a merged OpenAPI specification to a the provided path. ## Caveats Items below don't look critical at the moment and can be addressed later on. - It doesn't support merging of specs having different OpenAPI versions (Kibana's OpenAPI specs use version `3.0.x` but we should keep an eye on that) - It doesn't support top level `$ref` for - Path item - Request body - Responses
An utility to merge OpenAPI spec from different sources was implemented in #188110. I'll keep this ticked open util I get a feedback from the Docs Engineering team. There is a chance some little adjustment will be needed n and it can be tracked in this ticket. |
**Addresses:** #186356 **Relates to:** #184428 ## Summary This PR adds a merging JS script based on the utility implemented in #186356. Resulted OpenAPI bundle as committed in `oas_docs/output/kibana.serverless.bundled.yaml`. ## Details #188110 implements and exposes `merge` utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below ```js const { merge } = require('@kbn/openapi-bundler'); (async () => { await merge({ sourceGlobs: [/* a list of source globs goes here */], outputFilePath: 'path/to/the/output/file.yaml', }); })(); ``` The JS script added in this PR includes source OpenAPI specs presented in `oas_docs/makefile` plus Security Solution OpenAPI specs based on #184428. **To run** the script use the following command from Kibana root folder ```bash node ./oas_docs/scripts/merge_serverless_oas.js ``` ## Known linting issues with Security Solution OpenAPI specs Running Spectral OpenAPI linter on the result bundle shows a number of errors caused by `no-$ref-siblings` rule. This caused by the current code generator implementation which requires `default` property to be set next to `$ref` though it's not correct for OpenAPI `3.0.3` while it's allowed in `3.1`. It seems that Bump.sh handles such cases properly though by properly showing a default value. We need to analyze the problem and decide if/when we should fix it. The rest of warnings look fixable and will be addressed in the next stage after setting up linter rules. ## Next steps Since `@kbn/openapi-bundler` package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means `oas_docs/makefile` should be superseded by JS script(s) using `merge` utility form `@kbn/openapi-bundler` package. `@kbn/openapi-bundler` SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting `spectral-cli` for linting purposes.
OpenAPI Merge utility was enabled in #189262. |
…c#189262) **Addresses:** elastic#186356 **Relates to:** elastic#184428 ## Summary This PR adds a merging JS script based on the utility implemented in elastic#186356. Resulted OpenAPI bundle as committed in `oas_docs/output/kibana.serverless.bundled.yaml`. ## Details elastic#188110 implements and exposes `merge` utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below ```js const { merge } = require('@kbn/openapi-bundler'); (async () => { await merge({ sourceGlobs: [/* a list of source globs goes here */], outputFilePath: 'path/to/the/output/file.yaml', }); })(); ``` The JS script added in this PR includes source OpenAPI specs presented in `oas_docs/makefile` plus Security Solution OpenAPI specs based on elastic#184428. **To run** the script use the following command from Kibana root folder ```bash node ./oas_docs/scripts/merge_serverless_oas.js ``` ## Known linting issues with Security Solution OpenAPI specs Running Spectral OpenAPI linter on the result bundle shows a number of errors caused by `no-$ref-siblings` rule. This caused by the current code generator implementation which requires `default` property to be set next to `$ref` though it's not correct for OpenAPI `3.0.3` while it's allowed in `3.1`. It seems that Bump.sh handles such cases properly though by properly showing a default value. We need to analyze the problem and decide if/when we should fix it. The rest of warnings look fixable and will be addressed in the next stage after setting up linter rules. ## Next steps Since `@kbn/openapi-bundler` package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means `oas_docs/makefile` should be superseded by JS script(s) using `merge` utility form `@kbn/openapi-bundler` package. `@kbn/openapi-bundler` SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting `spectral-cli` for linting purposes.
…88110) **Addresses:** elastic#186356 ## Summary This PR adds OpenAPI spec files merger utility (programmatic API). It provides a similar functionality as `npx @redocly/cli join` does and takes into account [discussion results](elastic#183019 (comment)) - provides a simple way to produce a single Kibana OpenAPI bundle - extends `requestBody` and `responses` MIME types with a version parameters `Elastic-Api-Version=<version>` to avoid different API endpoint versions conflicts - has flexibility to adjust Kibana needs The utility is exposed from `kbn-openapi-bundler` package. ## Details **OpenAPI merger** is a tool for merging multiple OpenAPI specification files. It's useful to merge already processed specification files to produce a result bundle. **OpenAPI bundler** uses the merger under the hood to merge bundled OpenAPI specification files. Exposed externally merger is a wrapper of the bundler's merger but extended with an ability to parse JSON files and forced to produce a single result file. It is able to read OpenAPI spec files defined in JSON and YAML formats. The result file is always written in YAML format where every `requestBody` and response in `responses` extended with document's `info.version` value added as a MIME type parameter `Elastic-Api-Version=<version>`. Currently package supports only programmatic API. As the next step you need to create a JavaScript script file like below ```ts require('../../src/setup_node_env'); const { resolve } = require('path'); const { merge } = require('@kbn/openapi-bundler'); const { REPO_ROOT } = require('@kbn/repo-info'); (async () => { await merge({ sourceGlobs: [ `${REPO_ROOT}/my/path/to/spec1.json`, `${REPO_ROOT}/my/path/to/spec2.yml`, `${REPO_ROOT}/my/path/to/spec3.yaml`, ], outputFilePath: `${REPO_ROOT}/oas_docs/bundle.yaml`, mergedSpecInfo: { title: 'My merged OpenAPI specs', version: '1.0.0', }, }); })(); ``` Finally you should be able to run OpenAPI merger via ```bash node ./path/to/the/script.js ``` or it could be added to `package.json` and run via `yarn`. After running the script it will log different information and write a merged OpenAPI specification to a the provided path. ## Caveats Items below don't look critical at the moment and can be addressed later on. - It doesn't support merging of specs having different OpenAPI versions (Kibana's OpenAPI specs use version `3.0.x` but we should keep an eye on that) - It doesn't support top level `$ref` for - Path item - Request body - Responses
…c#189262) **Addresses:** elastic#186356 **Relates to:** elastic#184428 This PR adds a merging JS script based on the utility implemented in elastic#186356. Resulted OpenAPI bundle as committed in `oas_docs/output/kibana.serverless.bundled.yaml`. elastic#188110 implements and exposes `merge` utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below ```js const { merge } = require('@kbn/openapi-bundler'); (async () => { await merge({ sourceGlobs: [/* a list of source globs goes here */], outputFilePath: 'path/to/the/output/file.yaml', }); })(); ``` The JS script added in this PR includes source OpenAPI specs presented in `oas_docs/makefile` plus Security Solution OpenAPI specs based on elastic#184428. **To run** the script use the following command from Kibana root folder ```bash node ./oas_docs/scripts/merge_serverless_oas.js ``` Running Spectral OpenAPI linter on the result bundle shows a number of errors caused by `no-$ref-siblings` rule. This caused by the current code generator implementation which requires `default` property to be set next to `$ref` though it's not correct for OpenAPI `3.0.3` while it's allowed in `3.1`. It seems that Bump.sh handles such cases properly though by properly showing a default value. We need to analyze the problem and decide if/when we should fix it. The rest of warnings look fixable and will be addressed in the next stage after setting up linter rules. Since `@kbn/openapi-bundler` package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means `oas_docs/makefile` should be superseded by JS script(s) using `merge` utility form `@kbn/openapi-bundler` package. `@kbn/openapi-bundler` SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting `spectral-cli` for linting purposes.
…189262) (#190541) # Backport This will backport the bundler aspects of the following commits from `main` to `8.15`: - [[HTTP/OAS] Merge OpenAPI specs by using `kbn-openapi-bundler` (#189262)](#189262) - #189621 - #189482 - #189348 - #189472 - #188110 - #188812 <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-08-13T10:45:35Z","message":"[HTTP/OAS] Merge OpenAPI specs by using `kbn-openapi-bundler` (#189262)\n\n**Addresses:** https://github.com/elastic/kibana/issues/186356\r\n**Relates to:** https://github.com/elastic/kibana/issues/184428\r\n\r\n## Summary\r\n\r\nThis PR adds a merging JS script based on the utility implemented in #186356. Resulted OpenAPI bundle as committed in `oas_docs/output/kibana.serverless.bundled.yaml`.\r\n\r\n## Details\r\n\r\nhttps://github.com//pull/188110 implements and exposes `merge` utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below\r\n\r\n```js\r\nconst { merge } = require('@kbn/openapi-bundler');\r\n\r\n(async () => {\r\n await merge({\r\n sourceGlobs: [/* a list of source globs goes here */],\r\n outputFilePath: 'path/to/the/output/file.yaml',\r\n });\r\n})();\r\n```\r\n\r\nThe JS script added in this PR includes source OpenAPI specs presented in `oas_docs/makefile` plus Security Solution OpenAPI specs based on https://github.com/elastic/kibana/issues/184428.\r\n\r\n**To run** the script use the following command from Kibana root folder\r\n\r\n```bash\r\nnode ./oas_docs/scripts/merge_serverless_oas.js \r\n```\r\n\r\n## Known linting issues with Security Solution OpenAPI specs\r\n\r\nRunning Spectral OpenAPI linter on the result bundle shows a number of errors caused by `no-$ref-siblings` rule. This caused by the current code generator implementation which requires `default` property to be set next to `$ref` though it's not correct for OpenAPI `3.0.3` while it's allowed in `3.1`. It seems that Bump.sh handles such cases properly though by properly showing a default value.\r\n\r\nWe need to analyze the problem and decide if/when we should fix it.\r\n\r\nThe rest of warnings look fixable and will be addressed in the next stage after setting up linter rules.\r\n\r\n## Next steps\r\n\r\nSince `@kbn/openapi-bundler` package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means `oas_docs/makefile` should be superseded by JS script(s) using `merge` utility form `@kbn/openapi-bundler` package.\r\n\r\n`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting `spectral-cli` for linting purposes.","sha":"7a2e7bef9644eaf04fd820d16b8ee137b3c00f2b","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","backport:skip","docs","Team: SecuritySolution","Feature:OAS","v8.16.0"],"number":189262,"url":"https://github.com/elastic/kibana/pull/189262","mergeCommit":{"message":"[HTTP/OAS] Merge OpenAPI specs by using `kbn-openapi-bundler` (#189262)\n\n**Addresses:** https://github.com/elastic/kibana/issues/186356\r\n**Relates to:** https://github.com/elastic/kibana/issues/184428\r\n\r\n## Summary\r\n\r\nThis PR adds a merging JS script based on the utility implemented in #186356. Resulted OpenAPI bundle as committed in `oas_docs/output/kibana.serverless.bundled.yaml`.\r\n\r\n## Details\r\n\r\nhttps://github.com//pull/188110 implements and exposes `merge` utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below\r\n\r\n```js\r\nconst { merge } = require('@kbn/openapi-bundler');\r\n\r\n(async () => {\r\n await merge({\r\n sourceGlobs: [/* a list of source globs goes here */],\r\n outputFilePath: 'path/to/the/output/file.yaml',\r\n });\r\n})();\r\n```\r\n\r\nThe JS script added in this PR includes source OpenAPI specs presented in `oas_docs/makefile` plus Security Solution OpenAPI specs based on https://github.com/elastic/kibana/issues/184428.\r\n\r\n**To run** the script use the following command from Kibana root folder\r\n\r\n```bash\r\nnode ./oas_docs/scripts/merge_serverless_oas.js \r\n```\r\n\r\n## Known linting issues with Security Solution OpenAPI specs\r\n\r\nRunning Spectral OpenAPI linter on the result bundle shows a number of errors caused by `no-$ref-siblings` rule. This caused by the current code generator implementation which requires `default` property to be set next to `$ref` though it's not correct for OpenAPI `3.0.3` while it's allowed in `3.1`. It seems that Bump.sh handles such cases properly though by properly showing a default value.\r\n\r\nWe need to analyze the problem and decide if/when we should fix it.\r\n\r\nThe rest of warnings look fixable and will be addressed in the next stage after setting up linter rules.\r\n\r\n## Next steps\r\n\r\nSince `@kbn/openapi-bundler` package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means `oas_docs/makefile` should be superseded by JS script(s) using `merge` utility form `@kbn/openapi-bundler` package.\r\n\r\n`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting `spectral-cli` for linting purposes.","sha":"7a2e7bef9644eaf04fd820d16b8ee137b3c00f2b"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/189262","number":189262,"mergeCommit":{"message":"[HTTP/OAS] Merge OpenAPI specs by using `kbn-openapi-bundler` (#189262)\n\n**Addresses:** https://github.com/elastic/kibana/issues/186356\r\n**Relates to:** https://github.com/elastic/kibana/issues/184428\r\n\r\n## Summary\r\n\r\nThis PR adds a merging JS script based on the utility implemented in #186356. Resulted OpenAPI bundle as committed in `oas_docs/output/kibana.serverless.bundled.yaml`.\r\n\r\n## Details\r\n\r\nhttps://github.com//pull/188110 implements and exposes `merge` utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below\r\n\r\n```js\r\nconst { merge } = require('@kbn/openapi-bundler');\r\n\r\n(async () => {\r\n await merge({\r\n sourceGlobs: [/* a list of source globs goes here */],\r\n outputFilePath: 'path/to/the/output/file.yaml',\r\n });\r\n})();\r\n```\r\n\r\nThe JS script added in this PR includes source OpenAPI specs presented in `oas_docs/makefile` plus Security Solution OpenAPI specs based on https://github.com/elastic/kibana/issues/184428.\r\n\r\n**To run** the script use the following command from Kibana root folder\r\n\r\n```bash\r\nnode ./oas_docs/scripts/merge_serverless_oas.js \r\n```\r\n\r\n## Known linting issues with Security Solution OpenAPI specs\r\n\r\nRunning Spectral OpenAPI linter on the result bundle shows a number of errors caused by `no-$ref-siblings` rule. This caused by the current code generator implementation which requires `default` property to be set next to `$ref` though it's not correct for OpenAPI `3.0.3` while it's allowed in `3.1`. It seems that Bump.sh handles such cases properly though by properly showing a default value.\r\n\r\nWe need to analyze the problem and decide if/when we should fix it.\r\n\r\nThe rest of warnings look fixable and will be addressed in the next stage after setting up linter rules.\r\n\r\n## Next steps\r\n\r\nSince `@kbn/openapi-bundler` package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means `oas_docs/makefile` should be superseded by JS script(s) using `merge` utility form `@kbn/openapi-bundler` package.\r\n\r\n`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting `spectral-cli` for linting purposes.","sha":"7a2e7bef9644eaf04fd820d16b8ee137b3c00f2b"}}]}] BACKPORT--> --------- Co-authored-by: Maxim Palenov <[email protected]>
**Relates to:** #186356 ## Summary This PR implements proper tags sorting when a custom tag's `x-displayName` property is set. It allows to display tags properly at the API reference documentation page where `x-displayName` instead of tag's name when it's presented.
…ic#191732) **Relates to:** elastic#186356 ## Summary This PR implements proper tags sorting when a custom tag's `x-displayName` property is set. It allows to display tags properly at the API reference documentation page where `x-displayName` instead of tag's name when it's presented. (cherry picked from commit 3c4e502)
… it is set (#191732) (#192203) # Backport This will backport the following commits from `main` to `8.15`: - [[HTTP/OAS] Sort OpenAPI tags by `x-displayName` when it is set (#191732)](#191732) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-05T15:48:01Z","message":"[HTTP/OAS] Sort OpenAPI tags by `x-displayName` when it is set (#191732)\n\n**Relates to:** https://github.com/elastic/kibana/issues/186356\r\n\r\n## Summary\r\n\r\nThis PR implements proper tags sorting when a custom tag's `x-displayName` property is set. It allows to display tags properly at the API reference documentation page where `x-displayName` instead of tag's name when it's presented.","sha":"3c4e50249d87ee4721d628c5255b9e51a8857a2d","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","docs","Team: SecuritySolution","Feature:OAS","v8.16.0","v8.15.1"],"title":"[HTTP/OAS] Sort OpenAPI tags by `x-displayName` when it is set","number":191732,"url":"https://github.com/elastic/kibana/pull/191732","mergeCommit":{"message":"[HTTP/OAS] Sort OpenAPI tags by `x-displayName` when it is set (#191732)\n\n**Relates to:** https://github.com/elastic/kibana/issues/186356\r\n\r\n## Summary\r\n\r\nThis PR implements proper tags sorting when a custom tag's `x-displayName` property is set. It allows to display tags properly at the API reference documentation page where `x-displayName` instead of tag's name when it's presented.","sha":"3c4e50249d87ee4721d628c5255b9e51a8857a2d"}},"sourceBranch":"main","suggestedTargetBranches":["8.15"],"targetPullRequestStates":[{"branch":"main","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191732","number":191732,"mergeCommit":{"message":"[HTTP/OAS] Sort OpenAPI tags by `x-displayName` when it is set (#191732)\n\n**Relates to:** https://github.com/elastic/kibana/issues/186356\r\n\r\n## Summary\r\n\r\nThis PR implements proper tags sorting when a custom tag's `x-displayName` property is set. It allows to display tags properly at the API reference documentation page where `x-displayName` instead of tag's name when it's presented.","sha":"3c4e50249d87ee4721d628c5255b9e51a8857a2d"}},{"branch":"8.15","label":"v8.15.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Maxim Palenov <[email protected]>
Epic: https://github.com/elastic/security-team/issues/9401 (internal)
Summary
Provide an utility to merge Security Solution domain-specific bundles stored in
/x-pack/plugins/security_solution/docs/openapi
and OpenAPI packages with Kibana-wide bundles stored in/oas_docs
. As a result, we should have two final bundles:As the next step, we (Kibana Core, most likely) will need to set up a CI/CD pipeline for deploying these two bundles to bump.sh.
Details
kbn-openapi-bundler
is capable of merging OpenAPI specs as a part of the bundling process. It should be modified to expose the merger functionality.To do
merge
utility function formkbn-openapi-bundler
package (implemented in PR)The text was updated successfully, but these errors were encountered: