[Fleet] Automate the generation of Fleet's OpenAPI spec through @kbn/config-schema

[kpollich]

Today, Fleet's OpenAPI spec file is generated and maintained through a manual process documented in the README file in x-pack/fleet/plugins/fleet/common/openapi/.

Kibana as a whole is moving towards support for fully-automated OpenAPI specs for plugin APIs, as tracked here: #180056. Fleet's manual process should be replaced by a fully automated one in which the OAS spec is inferred from metadata on our server-side routes and request/response contracts defined via @kbn/config-schema.

We had previously done some investigation into whether @kbn/openapi-generator was ready to meet Fleet's needs and the answer at the time was "no". However, with a few months of progress since then, we should start the process of moving towards automating our schema generation. Manually generating and maintaining our OpenAPI spec is labor intensive and error prone, so shedding this friction will speed up our development process on the team.

Here is an example of the /api/status Kibana API's schema written with @kbn/config-schema that results in an OpenAPI spec here:

kibana/oas_docs/bundle.json

Lines 451 to 525 in 3efa595

	"/api/status": {
	"get": {
	"operationId": "/api/status#0",
	"parameters": [
	{
	"description": "The version of the API to use",
	"in": "header",
	"name": "elastic-api-version",
	"schema": {
	"default": "2023-10-31",
	"enum": [
	"2023-10-31"
	],
	"type": "string"
	}
	},
	{
	"description": "Set to \"true\" to get the response in v7 format.",
	"in": "query",
	"name": "v7format",
	"required": false,
	"schema": {
	"type": "boolean"
	}
	},
	{
	"description": "Set to \"true\" to get the response in v8 format.",
	"in": "query",
	"name": "v8format",
	"required": false,
	"schema": {
	"type": "boolean"
	}
	}
	],
	"responses": {
	"200": {
	"content": {
	"application/json; Elastic-Api-Version=2023-10-31": {
	"schema": {
	"anyOf": [
	{
	"$ref": "#/components/schemas/core_status_response"
	},
	{
	"$ref": "#/components/schemas/core_status_redactedResponse"
	}
	],
	"description": "Kibana's operational status. A minimal response is sent for unauthorized users."
	}
	}
	},
	"description": "Get Kibana's current status."
	},
	"503": {
	"content": {
	"application/json; Elastic-Api-Version=2023-10-31": {
	"schema": {
	"anyOf": [
	{
	"$ref": "#/components/schemas/core_status_response"
	},
	{
	"$ref": "#/components/schemas/core_status_redactedResponse"
	}
	],
	"description": "Kibana's operational status. A minimal response is sent for unauthorized users."
	}
	}
	},
	"description": "Get Kibana's current status."
	}
	},
	"summary": "Get Kibana's current status."
	}

What we'll actually need to do to accomplish this is to replace each entry we have today in Fleet's OpenAPI spec files with a more advanced set of types created with @kbn/config-schema alongside each route we register.

An example of this for the /api/fleet/setup API might look like this:

diff --git a/x-pack/plugins/fleet/server/routes/setup/index.ts b/x-pack/plugins/fleet/server/routes/setup/index.ts
index f09ff70e145..054fb7a2754 100644
--- a/x-pack/plugins/fleet/server/routes/setup/index.ts
+++ b/x-pack/plugins/fleet/server/routes/setup/index.ts
@@ -13,11 +13,13 @@ import { API_VERSIONS } from '../../../common/constants';
 import type { FleetConfigType } from '../../../common/types';
 
 import { getFleetStatusHandler, fleetSetupHandler } from './handlers';
+import { schema } from '@kbn/config-schema';
 
 export const registerFleetSetupRoute = (router: FleetAuthzRouter) => {
   router.versioned
     .post({
       path: SETUP_API_ROUTE,
+      description: 'Initiate Fleet setup',
       fleetAuthz: {
         fleet: { setup: true },
       },
@@ -25,7 +27,51 @@ export const registerFleetSetupRoute = (router: FleetAuthzRouter) => {
     .addVersion(
       {
         version: API_VERSIONS.public.v1,
-        validate: false,
+        validate: {
+          request: {},
+          response: {
+            200: {
+              body: () =>
+                schema.object(
+                  {
+                    isInitialized: schema.boolean(),
+                    nonFatalErrors: schema.object({
+                      name: schema.string(),
+                      message: schema.string(),
+                    }),
+                  },
+                  {
+                    meta: {
+                      description:
+                        "A summary of the result of Fleet's `setup` lifecycle. If `isInitialized` is true, Fleet is ready to accept agent enrollment. `nonFatalErrors` may include useful insight into non-blocking issues with Fleet's health.",
+                    },
+                  }
+                ),
+            },
+            400: {
+              body: () =>
+                schema.object(
+                  {
+                    statusCode: schema.number(),
+                    error: schema.string(),
+                    message: schema.string(),
+                  },
+                  {
+                    meta: { description: 'Generic error' },
+                  }
+                ),
+            },
+            500: {
+              body: () =>
+                schema.object(
+                  {
+                    message: schema.string(),
+                  },
+                  { meta: { description: 'Internal server error' } }
+                ),
+            },
+          },
+        },
       },
       fleetSetupHandler
     );

Each documented status code, response body, query parameter, etc can be documented directly alongside the route definition. We'd need to take what we have today in the various OpenAPI spec files and translate them to this approach for every Fleet API endpoint. This will be no small undertaking, but it will greatly improve our ability to create or iterate on API endpoints It will also greatly empower our docs team to generate better API documentation.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

New fleet API docs: https://www.elastic.co/docs/api/doc/kibana/operation/operation-get-enrollment-api-keys

Currently this reads from the existing bundled.yaml fleet openapi docs: #190571

When we move all APIs to use @kbn/config-schema, we can delete the fleet/openapi folder in kibana, and rely on the generated openapi spec.
We'd have to remove this line from each script:

kibana/oas_docs/scripts/merge_ess_oas.js

Line 23 in f15e825

`${REPO_ROOT}/x-pack/plugins/fleet/common/openapi/bundled.yaml`,

[juliaElastic]

It seems not supported yet to add examples yet, there is an open issue for it: #188926

[juliaElastic]

As discussed we are targeting 9.x with this change.

[juliaElastic]

Opened #193698 as the last chunk of APIs to define schema, still working on the last epm routes.
After this is ready and merged, the remaining task is to remove the manually created yaml schemas: #184685 (comment)