[Ingest Manager] Support limiting integrations on an agent config #70542

jen-huang · 2020-07-02T01:08:35Z

Summary

Resolves #64371. This PR supports the package definition config_templates.[0].multiple: false field that indicates whether or not a package is allowed to be added to an agent config more than once. Currently the only package that uses this field is endpoint.

Server-side changes:

Adds API endpoint GET /api/ingest_manager/epm/packages/limited that retrieves a list of package names which are limited
Adds a query parameter full to agent config list endpoint GET /api/ingest_manager/agent_configs?full=true to retrieve expanded package_configs information on each agent config
- The default behavior (i.e. without the parameter), remains unchanged: list endpoint will return an array of SO IDs for package_configs field
Adjusts API endpoint POST /api/ingest_manager/package_configs to return an error if a package config is using a 1) limited package and 2) is being added to an agent config which already has another package config using the same limited package
- In addition, some functionality which were being executed by the handler for this endpoint, have been moved to packageConfigService.create() instead

Client-side changes:

On Agent config > Add integration page, remove limited integrations which have already been added to that agent config from the list of available integrations
On Integration > Add [integration name page, for limited integrations, disable agent configs which already have that integration added

API integration changes:

Dockerized EPM tests were moved to their own directory: /ingest_manager_api_integration/apis/epm
New directory created for package config API integration tests: /ingest_manager_api_integration/apis/package_config
- create.ts file created containing basic tests covering creating package configs
- Note: a test was written to cover the erroring case for limited packages described above, but is skipped ATM due to the docker image not containing recent package registry changes needed for the test to succeed
  - Unskipped as the docker image was updated!

Testing

Testing can be done in the UI by opening Agent config > Add integration page in two tabs, selecting Endpoint in both and submitting the first tab. Attempting to submit the second tab should result in a toast error message.
Similar steps for the Integration flow.

Screenshots

…ns available for an agent config

…s, re

…eady from list of agent configs available for an integration

…ings

… tests to their own directory

…grations

elasticmachine · 2020-07-02T21:13:27Z

Pinging @elastic/ingest-management (Team:Ingest Management)

…gService.create()

hbharding · 2020-07-02T22:08:38Z

Thanks Jen. LGTM visually

…grations

jfsiii · 2020-07-06T14:46:30Z

x-pack/plugins/ingest_manager/common/types/rest_spec/common.ts

@@ -3,6 +3,7 @@
 * or more contributor license agreements. Licensed under the Elastic License;
 * you may not use this file except in compliance with the Elastic License.
 */
+import { HttpFetchQuery } from 'src/core/public';


Do we need this? It appears unused

bad merge :)

jfsiii · 2020-07-06T14:57:40Z

x-pack/plugins/ingest_manager/server/services/epm/packages/get.ts

+    (pkg) => (pkg.status = InstallationStatus.installed)
+  );
+  const installedPackagesInfo = await Promise.all(
+    installedPackages.map(async (pkgInstall) => {


nit: can drop async since we're not using await and getPackageInfo is already async

jfsiii · 2020-07-06T14:59:09Z

x-pack/plugins/ingest_manager/server/services/epm/packages/get.ts

+    })
+  );
+  return installedPackagesInfo
+    .filter((pkgInfo) => isPackageLimited(pkgInfo))


nit: no need for the lambda

Suggested change

.filter((pkgInfo) => isPackageLimited(pkgInfo))

.filter(isPackageLimited)

jfsiii · 2020-07-06T15:03:38Z

x-pack/plugins/ingest_manager/server/services/package_config.ts

    packageConfig: NewPackageConfig,
    options?: { id?: string; user?: AuthenticatedUser }
  ): Promise<PackageConfig> {
+    // Make sure the associated package is installed


Can we do these requests in parallel?

Do we have to wait for ensureInstalledPackage to complete before calling getPackageInfo?

jfsiii · 2020-07-06T15:05:32Z

x-pack/test/ingest_manager_api_integration/apis/index.js

+
+    // EPM
+    loadTestFile(require.resolve('./epm/list'));
+    loadTestFile(require.resolve('./epm/list'));


Seems like this should be changed to import a different file or removed

…grations

jen-huang · 2020-07-06T17:19:13Z

Updated PR with review feedback. I also re-enabled the skipped API integration test as the docker image was updated in #70716 🎉

jfsiii · 2020-07-06T18:59:16Z

x-pack/plugins/ingest_manager/server/services/epm/packages/get.ts

+}): Promise<string[]> {
+  const { savedObjectsClient } = options;
+  const allPackages = await getPackages({ savedObjectsClient });
+  const installedPackages = allPackages.filter(


I think we can skip the get & filter by calling getPackageSavedObjects. I don't know if it'll be a drop-in replacement, but it will start with the installed packages instead of getting all from the registry & filtering.

Not a blocker, just a comment

It will end up being the same number of calls because getPackageSavedObjects just returns the saved objects, which do not contain the necessary PackageInfo fields (from the registry) to detect if the package is limited or not

…grations

kibanamachine · 2020-07-06T21:10:58Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: fa768a1

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
ingestManager	358	+1	357

History

💚 Build #59075 succeeded c18ec76
💔 Build #58968 failed 0d86922
💚 Build #58532 succeeded 6d3006e
💚 Build #58491 succeeded d32eb27
💚 Build #58117 succeeded 441f331

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jen-huang · 2020-07-06T21:14:59Z

Holding off on backporting until backport #70862 is merged, to avoid conflicts.

…0542) (#70895) * Add API endpoint and hook for retrieving restricted packages * Filter out restricted packages already in use from list of integrations available for an agent config * Allow list agent configs to optionally return expanded package configs, re * Filter out agent configs which already use the restricted package already from list of agent configs available for an integration * Allow more than 20 agent configs to be shown * Rename restricted to limited; add some common methods to DRY * Add limited package check on server side * Adjust copy wording * Fix typings * Add some package config api integration tests, update es archive mappings * Move test to dockerized integation tests directory; move existing epm tests to their own directory * Remove extra assignPackageConfigs() - already handled in packageConfigService.create() * Review fixes * Fix type, reenabled skipped test * Move new EPM integration test file

* master: (53 commits) [Composable template] Details panel + delete functionality (elastic#70814) [Uptime] Ping list body scroll (elastic#70781) moving indexPattern.delete() to indexPatterns.delete(indexPattern) (elastic#70430) Adapt expected response of advanced settings feature control for cloud tests (elastic#70793) skip flaky suite (elastic#70885) skip flaky suite (elastic#67814) skip flaky suite (elastic#70906) Revert "reenable regression and classification functional tests (elastic#70661)" (elastic#70908) Added UI validation when creating a Webhook connector with invalid URL (elastic#70025) [Security Solution] Change default index pattern (elastic#70797) ServiceNow push to Incident generic implementation (supporting both Case specific and generic Alerts) (elastic#68464) add button link to ingest (elastic#70142) reenable regression and classification functional tests (elastic#70661) [Component templates] Form wizard (elastic#69732) [Ingest Manager] Copy changes (elastic#70828) Adding test user to maps functional tests - PR 1 (elastic#70649) [Ingest Manager] Support limiting integrations on an agent config (elastic#70542) skip flaky suite (elastic#70880) [Metrics UI] Fix a bug in Metric Threshold query filter construction (elastic#70672) upgrade caniuse-lite database (elastic#70833) ...

* master: (46 commits) [Composable template] Details panel + delete functionality (elastic#70814) [Uptime] Ping list body scroll (elastic#70781) moving indexPattern.delete() to indexPatterns.delete(indexPattern) (elastic#70430) Adapt expected response of advanced settings feature control for cloud tests (elastic#70793) skip flaky suite (elastic#70885) skip flaky suite (elastic#67814) skip flaky suite (elastic#70906) Revert "reenable regression and classification functional tests (elastic#70661)" (elastic#70908) Added UI validation when creating a Webhook connector with invalid URL (elastic#70025) [Security Solution] Change default index pattern (elastic#70797) ServiceNow push to Incident generic implementation (supporting both Case specific and generic Alerts) (elastic#68464) add button link to ingest (elastic#70142) reenable regression and classification functional tests (elastic#70661) [Component templates] Form wizard (elastic#69732) [Ingest Manager] Copy changes (elastic#70828) Adding test user to maps functional tests - PR 1 (elastic#70649) [Ingest Manager] Support limiting integrations on an agent config (elastic#70542) skip flaky suite (elastic#70880) [Metrics UI] Fix a bug in Metric Threshold query filter construction (elastic#70672) upgrade caniuse-lite database (elastic#70833) ...

* actions/feature: (46 commits) [Composable template] Details panel + delete functionality (elastic#70814) [Uptime] Ping list body scroll (elastic#70781) moving indexPattern.delete() to indexPatterns.delete(indexPattern) (elastic#70430) Adapt expected response of advanced settings feature control for cloud tests (elastic#70793) skip flaky suite (elastic#70885) skip flaky suite (elastic#67814) skip flaky suite (elastic#70906) Revert "reenable regression and classification functional tests (elastic#70661)" (elastic#70908) Added UI validation when creating a Webhook connector with invalid URL (elastic#70025) [Security Solution] Change default index pattern (elastic#70797) ServiceNow push to Incident generic implementation (supporting both Case specific and generic Alerts) (elastic#68464) add button link to ingest (elastic#70142) reenable regression and classification functional tests (elastic#70661) [Component templates] Form wizard (elastic#69732) [Ingest Manager] Copy changes (elastic#70828) Adding test user to maps functional tests - PR 1 (elastic#70649) [Ingest Manager] Support limiting integrations on an agent config (elastic#70542) skip flaky suite (elastic#70880) [Metrics UI] Fix a bug in Metric Threshold query filter construction (elastic#70672) upgrade caniuse-lite database (elastic#70833) ...

jen-huang added 11 commits July 1, 2020 17:46

Add API endpoint and hook for retrieving restricted packages

8439e2a

Filter out restricted packages already in use from list of integratio…

0a2009b

…ns available for an agent config

Allow list agent configs to optionally return expanded package config…

7deee92

…s, re

Filter out agent configs which already use the restricted package alr…

845b7ed

…eady from list of agent configs available for an integration

Allow more than 20 agent configs to be shown

d9a59ea

Rename restricted to limited; add some common methods to DRY

03aa609

Add limited package check on server side

f307922

Adjust copy wording

a44ec20

Fix typings

441f331

Add some package config api integration tests, update es archive mapp…

e88f80b

…ings

Move test to dockerized integation tests directory; move existing epm…

7da46ac

… tests to their own directory

jen-huang self-assigned this Jul 2, 2020

Merge remote-tracking branch 'upstream/master' into ingest/limit-inte…

d32eb27

…grations

jen-huang marked this pull request as ready for review July 2, 2020 21:13

jen-huang requested a review from a team July 2, 2020 21:13

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Jul 2, 2020

jen-huang added release_note:skip Skip the PR/issue when compiling release notes v7.9.0 v8.0.0 labels Jul 2, 2020

jen-huang changed the title ~~[WIP][Ingest Manager] Support limiting integrations on an agent config~~ [Ingest Manager] Support limiting integrations on an agent config Jul 2, 2020

Remove extra assignPackageConfigs() - already handled in packageConfi…

6d3006e

…gService.create()

Merge remote-tracking branch 'upstream/master' into ingest/limit-inte…

0d86922

…grations

jfsiii approved these changes Jul 6, 2020

View reviewed changes

jen-huang added 4 commits July 6, 2020 09:49

Merge remote-tracking branch 'upstream/master' into ingest/limit-inte…

18a72f3

…grations

Review fixes

b4e522f

Merge remote-tracking branch 'upstream/master' into ingest/limit-inte…

cdef2b6

…grations

Fix type, reenabled skipped test

c18ec76

jfsiii reviewed Jul 6, 2020

View reviewed changes

jen-huang added 2 commits July 6, 2020 12:17

Merge remote-tracking branch 'upstream/master' into ingest/limit-inte…

2c13944

…grations

Move new EPM integration test file

fa768a1

jen-huang merged commit 7debf4d into elastic:master Jul 6, 2020

jen-huang deleted the ingest/limit-integrations branch July 6, 2020 21:12

jen-huang mentioned this pull request Jul 6, 2020

[7.x] [Ingest Manager] Support limiting integrations on an agent config (#70542) #70895

Merged

jen-huang mentioned this pull request Jul 8, 2020

[Ingest Manager] Data source names should be unique #68019

Closed

jen-huang mentioned this pull request Jul 16, 2020

[Ingest Manager] perceived performance drop in /api/ingest_manager/epm/packages api response which is what loads Integrations to be added to a configuration #71289

Closed

jen-huang mentioned this pull request Mar 30, 2021

[discuss] Limited packages & multiple policy templates elastic/package-spec#155

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Support limiting integrations on an agent config #70542

[Ingest Manager] Support limiting integrations on an agent config #70542

jen-huang commented Jul 2, 2020 •

edited

Loading

elasticmachine commented Jul 2, 2020

hbharding commented Jul 2, 2020

jfsiii Jul 6, 2020

jen-huang Jul 6, 2020

jfsiii Jul 6, 2020

jfsiii Jul 6, 2020

jfsiii Jul 6, 2020

jfsiii Jul 6, 2020

jen-huang commented Jul 6, 2020 •

edited

Loading

jfsiii Jul 6, 2020

jen-huang Jul 6, 2020 •

edited

Loading

kibanamachine commented Jul 6, 2020

jen-huang commented Jul 6, 2020

	.filter((pkgInfo) => isPackageLimited(pkgInfo))
	.filter(isPackageLimited)

[Ingest Manager] Support limiting integrations on an agent config #70542

[Ingest Manager] Support limiting integrations on an agent config #70542

Conversation

jen-huang commented Jul 2, 2020 • edited Loading

Summary

Server-side changes:

Client-side changes:

API integration changes:

Testing

Screenshots

elasticmachine commented Jul 2, 2020

hbharding commented Jul 2, 2020

jfsiii Jul 6, 2020

Choose a reason for hiding this comment

jen-huang Jul 6, 2020

Choose a reason for hiding this comment

jfsiii Jul 6, 2020

Choose a reason for hiding this comment

jfsiii Jul 6, 2020

Choose a reason for hiding this comment

jfsiii Jul 6, 2020

Choose a reason for hiding this comment

jfsiii Jul 6, 2020

Choose a reason for hiding this comment

jen-huang commented Jul 6, 2020 • edited Loading

jfsiii Jul 6, 2020

Choose a reason for hiding this comment

jen-huang Jul 6, 2020 • edited Loading

Choose a reason for hiding this comment

kibanamachine commented Jul 6, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

History

jen-huang commented Jul 6, 2020

jen-huang commented Jul 2, 2020 •

edited

Loading

jen-huang commented Jul 6, 2020 •

edited

Loading

jen-huang Jul 6, 2020 •

edited

Loading