[SavedObjects] Add aggregations support (#96292)

* step 1 to add aggs in the find function of saved object * setp 2 - add specific unit test to aggs + fix bug found during integrations * step 3 - add security api_integration arounds aggs * fix types * unit test added for aggs_utils * add documentation * fix docs * review I * doc * try to fix test * add the new property to the saved object globaltype * fix types * delete old files * fix types + test api integration * type fix + test * Update src/core/server/saved_objects/types.ts Co-authored-by: Rudolf Meijering <[email protected]> * review I * change our validation to match discussion with Pierre and Rudolph * Validate multiple items nested filter query through KueryNode * remove unused import * review + put back test * migrate added tests to new TS file * fix documentation * fix license header * move stuff * duplicating test mappings * rename some stuff * move ALL the things * cast to aggregation container * update generated doc * add deep nested validation * rewrite the whole validation mechanism * some cleanup * minor cleanup * update generated doc * adapt telemetry client * fix API integ tests * fix doc * TOTO-less * remove xpack tests * list supported / unsupported aggregations * typo fix * extract some validation function * fix indent * add some unit tests * adapt FTR assertions * update doc * fix doc * doc again * cleanup test names * improve tsdoc on validation functions * perf nit Co-authored-by: Xavier Mouligneau <[email protected]> Co-authored-by: Rudolf Meijering <[email protected]> Co-authored-by: Kibana Machine <[email protected]>
elastic · Apr 16, 2021 · 106afd4 · 106afd4
1 parent 15e8ca1
commit 106afd4
Show file tree

Hide file tree

Showing 34 changed files with 1,369 additions and 69 deletions.
diff --git a/docs/api/saved-objects/find.asciidoc b/docs/api/saved-objects/find.asciidoc
@@ -53,9 +53,14 @@ experimental[] Retrieve a paginated set of {kib} saved objects by various condit
   (Optional, object) Filters to objects that have a relationship with the type and ID combination.
 
 `filter`::
-  (Optional, string) The filter is a KQL string with the caveat that if you filter with an attribute from your type saved object.
-  It should look like that savedObjectType.attributes.title: "myTitle". However, If you used a direct attribute of a saved object like `updatedAt`,
-  you will have to define your filter like that savedObjectType.updatedAt > 2018-12-22.
+  (Optional, string) The filter is a KQL string with the caveat that if you filter with an attribute from your saved object type,
+  it should look like that: `savedObjectType.attributes.title: "myTitle"`. However, If you use a root attribute of a saved 
+  object such as `updated_at`, you will have to define your filter like that: `savedObjectType.updated_at > 2018-12-22`.
+
+`aggs`::
+  (Optional, string) **experimental** An aggregation structure, serialized as a string. The field format is similar to `filter`, meaning
+  that to use a saved object type attribute in the aggregation, the `savedObjectType.attributes.title`: "myTitle"` format
+  must be used. For root fields, the syntax is `savedObjectType.rootField`
 
 NOTE: As objects change in {kib}, the results on each page of the response also
 change. Use the find API for traditional paginated results, but avoid using it to export large amounts of data.

diff --git a/docs/development/core/public/kibana-plugin-core-public.savedobjectsclient.find.md b/docs/development/core/public/kibana-plugin-core-public.savedobjectsclient.find.md
@@ -9,5 +9,5 @@ Search for objects
 <b>Signature:</b>
 
 ```typescript
-find: <T = unknown>(options: SavedObjectsFindOptions) => Promise<SavedObjectsFindResponsePublic<T>>;
+find: <T = unknown, A = unknown>(options: SavedObjectsFindOptions) => Promise<SavedObjectsFindResponsePublic<T, unknown>>;
 ```
diff --git a/docs/development/core/public/kibana-plugin-core-public.savedobjectsclient.md b/docs/development/core/public/kibana-plugin-core-public.savedobjectsclient.md
@@ -24,7 +24,7 @@ The constructor for this class is marked as internal. Third-party code should no
 |  [bulkGet](./kibana-plugin-core-public.savedobjectsclient.bulkget.md) |  | <code>(objects?: Array&lt;{</code><br/><code>        id: string;</code><br/><code>        type: string;</code><br/><code>    }&gt;) =&gt; Promise&lt;SavedObjectsBatchResponse&lt;unknown&gt;&gt;</code> | Returns an array of objects by id |
 |  [create](./kibana-plugin-core-public.savedobjectsclient.create.md) |  | <code>&lt;T = unknown&gt;(type: string, attributes: T, options?: SavedObjectsCreateOptions) =&gt; Promise&lt;SimpleSavedObject&lt;T&gt;&gt;</code> | Persists an object |
 |  [delete](./kibana-plugin-core-public.savedobjectsclient.delete.md) |  | <code>(type: string, id: string, options?: SavedObjectsDeleteOptions &#124; undefined) =&gt; ReturnType&lt;SavedObjectsApi['delete']&gt;</code> | Deletes an object |
-|  [find](./kibana-plugin-core-public.savedobjectsclient.find.md) |  | <code>&lt;T = unknown&gt;(options: SavedObjectsFindOptions) =&gt; Promise&lt;SavedObjectsFindResponsePublic&lt;T&gt;&gt;</code> | Search for objects |
+|  [find](./kibana-plugin-core-public.savedobjectsclient.find.md) |  | <code>&lt;T = unknown, A = unknown&gt;(options: SavedObjectsFindOptions) =&gt; Promise&lt;SavedObjectsFindResponsePublic&lt;T, unknown&gt;&gt;</code> | Search for objects |
 |  [get](./kibana-plugin-core-public.savedobjectsclient.get.md) |  | <code>&lt;T = unknown&gt;(type: string, id: string) =&gt; Promise&lt;SimpleSavedObject&lt;T&gt;&gt;</code> | Fetches a single object |
 
 ## Methods

diff --git a/...public/kibana-plugin-core-public.savedobjectsfindresponsepublic.aggregations.md b/...public/kibana-plugin-core-public.savedobjectsfindresponsepublic.aggregations.md
@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-public](./kibana-plugin-core-public.md) &gt; [SavedObjectsFindResponsePublic](./kibana-plugin-core-public.savedobjectsfindresponsepublic.md) &gt; [aggregations](./kibana-plugin-core-public.savedobjectsfindresponsepublic.aggregations.md)
+
+## SavedObjectsFindResponsePublic.aggregations property
+
+<b>Signature:</b>
+
+```typescript
+aggregations?: A;
+```
diff --git a/...lopment/core/public/kibana-plugin-core-public.savedobjectsfindresponsepublic.md b/...lopment/core/public/kibana-plugin-core-public.savedobjectsfindresponsepublic.md
@@ -11,13 +11,14 @@ Return type of the Saved Objects `find()` method.
 <b>Signature:</b>
 
 ```typescript
-export interface SavedObjectsFindResponsePublic<T = unknown> extends SavedObjectsBatchResponse<T> 
+export interface SavedObjectsFindResponsePublic<T = unknown, A = unknown> extends SavedObjectsBatchResponse<T> 
 ```
 
 ## Properties
 
 |  Property | Type | Description |
 |  --- | --- | --- |
+|  [aggregations](./kibana-plugin-core-public.savedobjectsfindresponsepublic.aggregations.md) | <code>A</code> |  |
 |  [page](./kibana-plugin-core-public.savedobjectsfindresponsepublic.page.md) | <code>number</code> |  |
 |  [perPage](./kibana-plugin-core-public.savedobjectsfindresponsepublic.perpage.md) | <code>number</code> |  |
 |  [total](./kibana-plugin-core-public.savedobjectsfindresponsepublic.total.md) | <code>number</code> |  |

diff --git a/docs/development/core/server/kibana-plugin-core-server.savedobjectsclient.find.md b/docs/development/core/server/kibana-plugin-core-server.savedobjectsclient.find.md
@@ -9,7 +9,7 @@ Find all SavedObjects matching the search query
 <b>Signature:</b>
 
 ```typescript
-find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T>>;
+find<T = unknown, A = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T, A>>;
 ```
 
 ## Parameters
@@ -20,5 +20,5 @@ find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindRes
 
 <b>Returns:</b>
 
-`Promise<SavedObjectsFindResponse<T>>`
+`Promise<SavedObjectsFindResponse<T, A>>`
 
diff --git a/.../core/server/kibana-plugin-core-server.savedobjectsfindresponse.aggregations.md b/.../core/server/kibana-plugin-core-server.savedobjectsfindresponse.aggregations.md
@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [SavedObjectsFindResponse](./kibana-plugin-core-server.savedobjectsfindresponse.md) &gt; [aggregations](./kibana-plugin-core-server.savedobjectsfindresponse.aggregations.md)
+
+## SavedObjectsFindResponse.aggregations property
+
+<b>Signature:</b>
+
+```typescript
+aggregations?: A;
+```
diff --git a/docs/development/core/server/kibana-plugin-core-server.savedobjectsfindresponse.md b/docs/development/core/server/kibana-plugin-core-server.savedobjectsfindresponse.md
@@ -11,13 +11,14 @@ Return type of the Saved Objects `find()` method.
 <b>Signature:</b>
 
 ```typescript
-export interface SavedObjectsFindResponse<T = unknown> 
+export interface SavedObjectsFindResponse<T = unknown, A = unknown> 
 ```
 
 ## Properties
 
 |  Property | Type | Description |
 |  --- | --- | --- |
+|  [aggregations](./kibana-plugin-core-server.savedobjectsfindresponse.aggregations.md) | <code>A</code> |  |
 |  [page](./kibana-plugin-core-server.savedobjectsfindresponse.page.md) | <code>number</code> |  |
 |  [per\_page](./kibana-plugin-core-server.savedobjectsfindresponse.per_page.md) | <code>number</code> |  |
 |  [pit\_id](./kibana-plugin-core-server.savedobjectsfindresponse.pit_id.md) | <code>string</code> |  |

diff --git a/...evelopment/core/server/kibana-plugin-core-server.savedobjectsrepository.find.md b/...evelopment/core/server/kibana-plugin-core-server.savedobjectsrepository.find.md
@@ -7,7 +7,7 @@
 <b>Signature:</b>
 
 ```typescript
-find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T>>;
+find<T = unknown, A = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T, A>>;
 ```
 
 ## Parameters
@@ -18,7 +18,7 @@ find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindRes
 
 <b>Returns:</b>
 
-`Promise<SavedObjectsFindResponse<T>>`
+`Promise<SavedObjectsFindResponse<T, A>>`
 
 {<!-- -->promise<!-- -->} - { saved\_objects: \[{ id, type, version, attributes }<!-- -->\], total, per\_page, page }
 
diff --git a/...e/server/kibana-plugin-core-server.savedobjectsutils.createemptyfindresponse.md b/...e/server/kibana-plugin-core-server.savedobjectsutils.createemptyfindresponse.md
@@ -9,5 +9,5 @@ Creates an empty response for a find operation. This is only intended to be used
 <b>Signature:</b>
 
 ```typescript
-static createEmptyFindResponse: <T>({ page, perPage, }: SavedObjectsFindOptions) => SavedObjectsFindResponse<T>;
+static createEmptyFindResponse: <T, A>({ page, perPage, }: SavedObjectsFindOptions) => SavedObjectsFindResponse<T, A>;
 ```
diff --git a/docs/development/core/server/kibana-plugin-core-server.savedobjectsutils.md b/docs/development/core/server/kibana-plugin-core-server.savedobjectsutils.md
@@ -15,7 +15,7 @@ export declare class SavedObjectsUtils
 
 |  Property | Modifiers | Type | Description |
 |  --- | --- | --- | --- |
-|  [createEmptyFindResponse](./kibana-plugin-core-server.savedobjectsutils.createemptyfindresponse.md) | <code>static</code> | <code>&lt;T&gt;({ page, perPage, }: SavedObjectsFindOptions) =&gt; SavedObjectsFindResponse&lt;T&gt;</code> | Creates an empty response for a find operation. This is only intended to be used by saved objects client wrappers. |
+|  [createEmptyFindResponse](./kibana-plugin-core-server.savedobjectsutils.createemptyfindresponse.md) | <code>static</code> | <code>&lt;T, A&gt;({ page, perPage, }: SavedObjectsFindOptions) =&gt; SavedObjectsFindResponse&lt;T, A&gt;</code> | Creates an empty response for a find operation. This is only intended to be used by saved objects client wrappers. |
 |  [namespaceIdToString](./kibana-plugin-core-server.savedobjectsutils.namespaceidtostring.md) | <code>static</code> | <code>(namespace?: string &#124; undefined) =&gt; string</code> | Converts a given saved object namespace ID to its string representation. All namespace IDs have an identical string representation, with the exception of the <code>undefined</code> namespace ID (which has a namespace string of <code>'default'</code>). |
 |  [namespaceStringToId](./kibana-plugin-core-server.savedobjectsutils.namespacestringtoid.md) | <code>static</code> | <code>(namespace: string) =&gt; string &#124; undefined</code> | Converts a given saved object namespace string to its ID representation. All namespace strings have an identical ID representation, with the exception of the <code>'default'</code> namespace string (which has a namespace ID of <code>undefined</code>). |
 

diff --git a/src/core/public/public.api.md b/src/core/public/public.api.md
@@ -1224,7 +1224,7 @@ export class SavedObjectsClient {
     // Warning: (ae-forgotten-export) The symbol "SavedObjectsClientContract" needs to be exported by the entry point index.d.ts
     delete: (type: string, id: string, options?: SavedObjectsDeleteOptions | undefined) => ReturnType<SavedObjectsClientContract_2['delete']>;
     // Warning: (ae-forgotten-export) The symbol "SavedObjectsFindOptions" needs to be exported by the entry point index.d.ts
-    find: <T = unknown>(options: SavedObjectsFindOptions_2) => Promise<SavedObjectsFindResponsePublic<T>>;
+    find: <T = unknown, A = unknown>(options: SavedObjectsFindOptions_2) => Promise<SavedObjectsFindResponsePublic<T, unknown>>;
     get: <T = unknown>(type: string, id: string) => Promise<SimpleSavedObject<T>>;
     update<T = unknown>(type: string, id: string, attributes: T, { version, migrationVersion, references }?: SavedObjectsUpdateOptions): Promise<SimpleSavedObject<T>>;
 }
@@ -1244,6 +1244,8 @@ export interface SavedObjectsCreateOptions {
 
 // @public (undocumented)
 export interface SavedObjectsFindOptions {
+    // @alpha
+    aggs?: Record<string, estypes.AggregationContainer>;
     defaultSearchOperator?: 'AND' | 'OR';
     fields?: string[];
     // Warning: (ae-forgotten-export) The symbol "KueryNode" needs to be exported by the entry point index.d.ts
@@ -1284,7 +1286,9 @@ export interface SavedObjectsFindOptionsReference {
 }
 
 // @public
-export interface SavedObjectsFindResponsePublic<T = unknown> extends SavedObjectsBatchResponse<T> {
+export interface SavedObjectsFindResponsePublic<T = unknown, A = unknown> extends SavedObjectsBatchResponse<T> {
+    // (undocumented)
+    aggregations?: A;
     // (undocumented)
     page: number;
     // (undocumented)

diff --git a/src/core/public/saved_objects/saved_objects_client.ts b/src/core/public/saved_objects/saved_objects_client.ts
@@ -103,7 +103,9 @@ export interface SavedObjectsDeleteOptions {
  *
  * @public
  */
-export interface SavedObjectsFindResponsePublic<T = unknown> extends SavedObjectsBatchResponse<T> {
+export interface SavedObjectsFindResponsePublic<T = unknown, A = unknown>
+  extends SavedObjectsBatchResponse<T> {
+  aggregations?: A;
   total: number;
   perPage: number;
   page: number;
@@ -310,7 +312,7 @@ export class SavedObjectsClient {
    * @property {object} [options.hasReference] - { type, id }
    * @returns A find result with objects matching the specified search.
    */
-  public find = <T = unknown>(
+  public find = <T = unknown, A = unknown>(
     options: SavedObjectsFindOptions
   ): Promise<SavedObjectsFindResponsePublic<T>> => {
     const path = this.getPath(['_find']);
@@ -326,6 +328,7 @@ export class SavedObjectsClient {
       sortField: 'sort_field',
       type: 'type',
       filter: 'filter',
+      aggs: 'aggs',
       namespaces: 'namespaces',
       preference: 'preference',
     };
@@ -342,13 +345,20 @@ export class SavedObjectsClient {
       query.has_reference = JSON.stringify(query.has_reference);
     }
 
+    // `aggs` is a structured object. we need to stringify it before sending it, as `fetch`
+    // is not doing it implicitly.
+    if (query.aggs) {
+      query.aggs = JSON.stringify(query.aggs);
+    }
+
     const request: ReturnType<SavedObjectsApi['find']> = this.savedObjectsFetch(path, {
       method: 'GET',
       query,
     });
     return request.then((resp) => {
       return renameKeys<SavedObjectsFindResponse, SavedObjectsFindResponsePublic>(
         {
+          aggregations: 'aggregations',
           saved_objects: 'savedObjects',
           total: 'total',
           per_page: 'perPage',

diff --git a/src/core/server/saved_objects/routes/find.ts b/src/core/server/saved_objects/routes/find.ts
@@ -44,6 +44,7 @@ export const registerFindRoute = (router: IRouter, { coreUsageData }: RouteDepen
           has_reference_operator: searchOperatorSchema,
           fields: schema.maybe(schema.oneOf([schema.string(), schema.arrayOf(schema.string())])),
           filter: schema.maybe(schema.string()),
+          aggs: schema.maybe(schema.string()),
           namespaces: schema.maybe(
             schema.oneOf([schema.string(), schema.arrayOf(schema.string())])
           ),
@@ -59,6 +60,20 @@ export const registerFindRoute = (router: IRouter, { coreUsageData }: RouteDepen
       const usageStatsClient = coreUsageData.getClient();
       usageStatsClient.incrementSavedObjectsFind({ request: req }).catch(() => {});
 
+      // manually validation to avoid using JSON.parse twice
+      let aggs;
+      if (query.aggs) {
+        try {
+          aggs = JSON.parse(query.aggs);
+        } catch (e) {
+          return res.badRequest({
+            body: {
+              message: 'invalid aggs value',
+            },
+          });
+        }
+      }
+
       const result = await context.core.savedObjects.client.find({
         perPage: query.per_page,
         page: query.page,
@@ -72,6 +87,7 @@ export const registerFindRoute = (router: IRouter, { coreUsageData }: RouteDepen
         hasReferenceOperator: query.has_reference_operator,
         fields: typeof query.fields === 'string' ? [query.fields] : query.fields,
         filter: query.filter,
+        aggs,
         namespaces,
       });
 

diff --git a/src/core/server/saved_objects/service/lib/aggregations/aggs_types/bucket_aggs.ts b/src/core/server/saved_objects/service/lib/aggregations/aggs_types/bucket_aggs.ts
@@ -0,0 +1,86 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { schema as s, ObjectType } from '@kbn/config-schema';
+
+/**
+ * Schemas for the Bucket aggregations.
+ *
+ * Currently supported:
+ * - filter
+ * - histogram
+ * - terms
+ *
+ * Not implemented:
+ * - adjacency_matrix
+ * - auto_date_histogram
+ * - children
+ * - composite
+ * - date_histogram
+ * - date_range
+ * - diversified_sampler
+ * - filters
+ * - geo_distance
+ * - geohash_grid
+ * - geotile_grid
+ * - global
+ * - ip_range
+ * - missing
+ * - multi_terms
+ * - nested
+ * - parent
+ * - range
+ * - rare_terms
+ * - reverse_nested
+ * - sampler
+ * - significant_terms
+ * - significant_text
+ * - variable_width_histogram
+ */
+export const bucketAggsSchemas: Record<string, ObjectType> = {
+  filter: s.object({
+    term: s.recordOf(s.string(), s.oneOf([s.string(), s.boolean(), s.number()])),
+  }),
+  histogram: s.object({
+    field: s.maybe(s.string()),
+    interval: s.maybe(s.number()),
+    min_doc_count: s.maybe(s.number()),
+    extended_bounds: s.maybe(
+      s.object({
+        min: s.number(),
+        max: s.number(),
+      })
+    ),
+    hard_bounds: s.maybe(
+      s.object({
+        min: s.number(),
+        max: s.number(),
+      })
+    ),
+    missing: s.maybe(s.number()),
+    keyed: s.maybe(s.boolean()),
+    order: s.maybe(
+      s.object({
+        _count: s.string(),
+        _key: s.string(),
+      })
+    ),
+  }),
+  terms: s.object({
+    field: s.maybe(s.string()),
+    collect_mode: s.maybe(s.string()),
+    exclude: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
+    include: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
+    execution_hint: s.maybe(s.string()),
+    missing: s.maybe(s.number()),
+    min_doc_count: s.maybe(s.number()),
+    size: s.maybe(s.number()),
+    show_term_doc_count_error: s.maybe(s.boolean()),
+    order: s.maybe(s.oneOf([s.literal('asc'), s.literal('desc')])),
+  }),
+};
diff --git a/src/core/server/saved_objects/service/lib/aggregations/aggs_types/index.ts b/src/core/server/saved_objects/service/lib/aggregations/aggs_types/index.ts
@@ -0,0 +1,15 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { bucketAggsSchemas } from './bucket_aggs';
+import { metricsAggsSchemas } from './metrics_aggs';
+
+export const aggregationSchemas = {
+  ...metricsAggsSchemas,
+  ...bucketAggsSchemas,
+};