[Security Solution] Implement OpenAPI specs merger utility (#188110)

**Addresses:** #186356

## Summary

This PR adds OpenAPI spec files merger utility (programmatic API). It provides a similar functionality as `npx @redocly/cli join` does and takes into account [discussion results](#183019 (comment))

- provides a simple way to produce a single Kibana OpenAPI bundle
- extends `requestBody` and `responses` MIME types with a version parameters `Elastic-Api-Version=<version>` to avoid different API endpoint versions conflicts
- has flexibility to adjust Kibana needs

The utility is exposed from `kbn-openapi-bundler` package.

## Details

**OpenAPI merger** is a tool for merging multiple OpenAPI specification files. It's useful to merge already processed specification files to produce a result bundle. **OpenAPI bundler** uses the merger under the hood to merge bundled OpenAPI specification files. Exposed externally merger is a wrapper of the bundler's merger but extended with an ability to parse JSON files and forced to produce a single result file.

It is able to read OpenAPI spec files defined in JSON and YAML formats. The result file is always written in YAML format where every `requestBody` and response in `responses` extended with document's `info.version` value added as a MIME type parameter `Elastic-Api-Version=<version>`.

Currently package supports only programmatic API. As the next step you need to create a JavaScript script file like below

```ts
require('../../src/setup_node_env');
const { resolve } = require('path');
const { merge } = require('@kbn/openapi-bundler');
const { REPO_ROOT } = require('@kbn/repo-info');

(async () => {
  await merge({
    sourceGlobs: [
      `${REPO_ROOT}/my/path/to/spec1.json`,
      `${REPO_ROOT}/my/path/to/spec2.yml`,
      `${REPO_ROOT}/my/path/to/spec3.yaml`,
    ],
    outputFilePath: `${REPO_ROOT}/oas_docs/bundle.yaml`,
    mergedSpecInfo: {
      title: 'My merged OpenAPI specs',
      version: '1.0.0',
    },
  });
})();
```

Finally you should be able to run OpenAPI merger via

```bash
node ./path/to/the/script.js
```

or it could be added to `package.json` and run via `yarn`.

After running the script it will log different information and write a merged OpenAPI specification to a the provided path.

## Caveats

Items below don't look critical at the moment and can be addressed later on.

- It doesn't support merging of specs having different OpenAPI versions (Kibana's OpenAPI specs use version `3.0.x` but we should keep an eye on that)
- It doesn't support top level `$ref` for
  - Path item
  - Request body
  - Responses

packages/kbn-openapi-bundler/README.md

@@ -1,20 +1,24 @@
 # OpenAPI Specs Bundler for Kibana
 
-`@kbn/openapi-bundler` is a tool for transforming multiple OpenAPI specification files (source specs) into a bundled specification file(s) (target spec). The number of resulting bundles depends on a number of versions
-used in the OpenAPI specification files. The package can be used for API documentation generation purposes. This approach allows you to:
-
-- Abstract away the knowledge of where you keep your OpenAPI specs, how many specs are there, and how to find them. Consumer should only know where result files (bundles) are located.
-- Omit internal API endpoints from the bundle.
-- Omit API endpoints that are hidden behind a feature flag and haven't been released yet.
-- Omit parts of schemas that are hidden behind a feature flag (e.g. a new property added to an existing response schema).
-- Omit custom OpenAPI attributes from the bundle, such as `x-codegen-enabled`, `x-internal`, `x-modify` and `x-labels`.
-- Include only dedicated OpenAPI operation objects (a.k.a HTTP verbs) into the result bundle by labeling them via `x-labels`
-  and using `includeLabels` bundler option, e.g. produce separate ESS and Serverless bundles
-- Transform the target schema according to the custom OpenAPI attributes, such as `x-modify`.
-- Resolve references, inline some of them and merge `allOf` object schemas for better readability. The bundled file contains only local references and paths.
-- Group OpenAPI specs by version (OpenAPI's `info.version`) and produce a separate bundle for each group
-
-## Getting started
+This packages provides tooling for manipulating OpenAPI endpoint specifications. It has two tools exposes
+
+- **OpenAPI bundler** is a tool for transforming multiple OpenAPI specification files (source specs) into a bundled specification file(s) (target spec). The number of resulting bundles depends on a number of versions
+  used in the OpenAPI specification files. The package can be used for API documentation generation purposes. This approach allows you to:
+
+  - Abstract away the knowledge of where you keep your OpenAPI specs, how many specs are there, and how to find them. Consumer should only know where result files (bundles) are located.
+  - Omit internal API endpoints from the bundle.
+  - Omit API endpoints that are hidden behind a feature flag and haven't been released yet.
+  - Omit parts of schemas that are hidden behind a feature flag (e.g. a new property added to an existing response schema).
+  - Omit custom OpenAPI attributes from the bundle, such as `x-codegen-enabled`, `x-internal`, `x-modify` and `x-labels`.
+  - Include only dedicated OpenAPI operation objects (a.k.a HTTP verbs) into the result bundle by labeling them via `x-labels`
+    and using `includeLabels` bundler option, e.g. produce separate ESS and Serverless bundles
+  - Transform the target schema according to the custom OpenAPI attributes, such as `x-modify`.
+  - Resolve references, inline some of them and merge `allOf` object schemas for better readability. The bundled file contains only local references and paths.
+  - Group OpenAPI specs by version (OpenAPI's `info.version`) and produce a separate bundle for each group
+
+- **OpenAPI merger** is a tool for merging multiple OpenAPI specification files. It's useful to merge already processed specification files to produce a result bundle. **OpenAPI bundler** uses the merger under the hood to merge bundled OpenAPI specification files. Exposed externally merger is a wrapper of the bundler's merger but extended with an ability to parse JSON files and forced to produce a single result file.
+
+## Getting started with OpenAPI bundling
 
 To let this package help you with bundling your OpenAPI specifications you should have OpenAPI specification describing your API endpoint request and response schemas along with common types used in your API. Refer [@kbn/openapi-generator](../kbn-openapi-generator/README.md) and [OpenAPI 3.0.3](https://swagger.io/specification/v3/) (support for [OpenAPI 3.1.0](https://swagger.io/specification/) is planned to be added later) for more details.
 
@@ -163,6 +167,48 @@ components:
     securitySchemes: ...
 ```
 
+## Getting started with OpenAPI merger
+
+To let this package help you with merging OpenAPI specifications you should have valid OpenAPI specifications version `3.0.x`. OpenAPI `3.1` is not supported currently.
+
+Currently package supports only programmatic API. As the next step you need to create a JavaScript script file like below
+
+```ts
+require('../../src/setup_node_env');
+const { resolve } = require('path');
+const { merge } = require('@kbn/openapi-bundler');
+const { REPO_ROOT } = require('@kbn/repo-info');
+
+(async () => {
+  await merge({
+    sourceGlobs: [
+      `${REPO_ROOT}/my/path/to/spec1.json`,
+      `${REPO_ROOT}/my/path/to/spec2.yml`,
+      `${REPO_ROOT}/my/path/to/spec3.yaml`,
+    ],
+    outputFilePath: `${REPO_ROOT}/oas_docs/bundle.serverless.yaml`,
+    mergedSpecInfo: {
+      title: 'Kibana Serverless',
+      version: '1.0.0',
+    },
+  });
+})();
+```
+
+Finally you should be able to run OpenAPI merger via
+
+```bash
+node ./path/to/the/script.js
+```
+
+or it could be added to a `package.json` and run via `yarn`.
+
+After running the script it will log different information and write a merged OpenAPI specification to a the provided path.
+
+### Caveats
+
+Merger shows an error when it's unable to merge some OpenAPI specifications. There is a possibility that references with the same name are defined in two or more files or there are endpoints of different versions and different parameters. Additionally top level `$ref` in path items, path item's `requestBody` and each response in `responses` aren't supported.
+
 ## Multiple API versions declared via OpenAPI's `info.version`
 
 Serverless brought necessity for versioned HTTP API endpoints. We started with a single `2023-10-31` version. In some point

packages/kbn-openapi-bundler/index.ts

@@ -7,3 +7,4 @@
  */
 
 export * from './src/openapi_bundler';
+export * from './src/openapi_merger';

packages/kbn-openapi-bundler/src/bundler/bundle_document.ts

@@ -26,17 +26,14 @@ import {
 import { createIncludeLabelsProcessor } from './process_document/document_processors/include_labels';
 import { BundleRefProcessor } from './process_document/document_processors/bundle_refs';
 import { RemoveUnusedComponentsProcessor } from './process_document/document_processors/remove_unused_components';
+import { insertRefByPointer } from '../utils/insert_by_json_pointer';
 
 export class SkipException extends Error {
   constructor(public documentPath: string, message: string) {
     super(message);
   }
 }
 
-export interface BundledDocument extends ResolvedDocument {
-  bundledRefs: ResolvedRef[];
-}
-
 interface BundleDocumentOptions {
   includeLabels?: string[];
 }
@@ -58,7 +55,7 @@ interface BundleDocumentOptions {
 export async function bundleDocument(
   absoluteDocumentPath: string,
   options?: BundleDocumentOptions
-): Promise<BundledDocument> {
+): Promise<ResolvedDocument> {
   if (!isAbsolute(absoluteDocumentPath)) {
     throw new Error(
       `bundleDocument expects an absolute document path but got "${absoluteDocumentPath}"`
@@ -114,7 +111,9 @@ export async function bundleDocument(
     );
   }
 
-  return { ...resolvedDocument, bundledRefs: Array.from(bundleRefsProcessor.getBundledRefs()) };
+  injectBundledRefs(resolvedDocument, bundleRefsProcessor.getBundledRefs());
+
+  return resolvedDocument;
 }
 
 interface MaybeObjectWithPaths {
@@ -128,3 +127,12 @@ function hasPaths(document: MaybeObjectWithPaths): boolean {
     Object.keys(document.paths).length > 0
   );
 }
+
+function injectBundledRefs(
+  resolvedDocument: ResolvedDocument,
+  refs: IterableIterator<ResolvedRef>
+): void {
+  for (const ref of refs) {
+    insertRefByPointer(ref.pointer, ref.refNode, resolvedDocument.document);
+  }
+}

packages/kbn-openapi-bundler/src/bundler/merge_documents/enrich_with_version_mime_param.ts

@@ -0,0 +1,82 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { OpenAPIV3 } from 'openapi-types';
+import { ResolvedDocument } from '../ref_resolver/resolved_document';
+import { isRefNode } from '../process_document';
+import { getOasDocumentVersion } from '../../utils/get_oas_document_version';
+import { KNOWN_HTTP_METHODS } from './http_methods';
+
+export function enrichWithVersionMimeParam(resolvedDocuments: ResolvedDocument[]): void {
+  for (const resolvedDocument of resolvedDocuments) {
+    const version = getOasDocumentVersion(resolvedDocument);
+    const paths = resolvedDocument.document.paths as OpenAPIV3.PathsObject;
+
+    for (const path of Object.keys(paths ?? {})) {
+      const pathItemObj = paths[path];
+
+      for (const httpVerb of KNOWN_HTTP_METHODS) {
+        const operationObj = pathItemObj?.[httpVerb];
+
+        if (operationObj?.requestBody && !isRefNode(operationObj.requestBody)) {
+          const requestBodyContent = operationObj.requestBody.content;
+
+          enrichContentWithVersion(requestBodyContent, version);
+        }
+
+        enrichCollection(operationObj?.responses ?? {}, version);
+      }
+    }
+
+    if (resolvedDocument.document.components) {
+      const components = resolvedDocument.document.components as OpenAPIV3.ComponentsObject;
+
+      if (components.requestBodies) {
+        enrichCollection(components.requestBodies, version);
+      }
+
+      if (components.responses) {
+        enrichCollection(components.responses, version);
+      }
+    }
+  }
+}
+
+function enrichCollection(
+  collection: Record<
+    string,
+    { content?: Record<string, OpenAPIV3.MediaTypeObject> } | OpenAPIV3.ReferenceObject
+  >,
+  version: string
+) {
+  for (const name of Object.keys(collection)) {
+    const obj = collection[name];
+
+    if (!obj || isRefNode(obj) || !obj.content) {
+      continue;
+    }
+
+    enrichContentWithVersion(obj.content, version);
+  }
+}
+
+function enrichContentWithVersion(
+  content: Record<string, OpenAPIV3.MediaTypeObject>,
+  version: string
+): void {
+  for (const mimeType of Object.keys(content)) {
+    if (mimeType.includes('; Elastic-Api-Version=')) {
+      continue;
+    }
+
+    const mimeTypeWithVersionParam = `${mimeType}; Elastic-Api-Version=${version}`;
+
+    content[mimeTypeWithVersionParam] = content[mimeType];
+    delete content[mimeType];
+  }
+}

packages/kbn-openapi-bundler/src/bundler/merge_documents/http_methods.ts

@@ -0,0 +1,20 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { OpenAPIV3 } from 'openapi-types';
+
+export const KNOWN_HTTP_METHODS = [
+  OpenAPIV3.HttpMethods.HEAD,
+  OpenAPIV3.HttpMethods.GET,
+  OpenAPIV3.HttpMethods.POST,
+  OpenAPIV3.HttpMethods.PATCH,
+  OpenAPIV3.HttpMethods.PUT,
+  OpenAPIV3.HttpMethods.OPTIONS,
+  OpenAPIV3.HttpMethods.DELETE,
+  OpenAPIV3.HttpMethods.TRACE,
+] as const;

packages/kbn-openapi-bundler/src/bundler/merge_documents/merge_arrays.ts

@@ -0,0 +1,39 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+/**
+ * Merges source arrays by merging array items and omitting duplicates.
+ * Duplicates checked by exacts match.
+ */
+export function mergeArrays<T>(sources: Array<readonly T[]>): T[] {
+  const merged: T[] = [];
+  const seen = new Set<string>();
+
+  for (const itemsSource of sources) {
+    for (const item of itemsSource) {
+      const searchableItem = toString(item);
+
+      if (seen.has(searchableItem)) {
+        continue;
+      }
+
+      merged.push(item);
+      seen.add(searchableItem);
+    }
+  }
+
+  return merged;
+}
+
+function toString(value: unknown): string {
+  try {
+    return JSON.stringify(value);
+  } catch {
+    throw new Error('Unable to merge arrays - encountered value is not serializable');
+  }
+}