[Fleet] Make upload and registry package info consistent #126915

kpollich · 2022-03-04T14:50:33Z

Alters our validation/processing flow for package archives to bring them in line with the processing we do in the package registry.

elasticmachine · 2022-03-04T14:50:38Z

Pinging @elastic/fleet (Team:Fleet)

kpollich · 2022-03-04T15:11:28Z

x-pack/test/fleet_api_integration/apis/epm/index.js

@@ -14,7 +14,7 @@ export default function loadTests({ loadTestFile }) {
    loadTestFile(require.resolve('./file'));
    loadTestFile(require.resolve('./template'));
    loadTestFile(require.resolve('./ilm'));
-    // loadTestFile(require.resolve('./install_bundled'));
+    loadTestFile(require.resolve('./install_bundled'));


This was somehow commented out and missed in a previous PR.

I had to make some additional changes to get these tests passing. Since they were commented out incidentally, there were some gotchas to sort out with bundled package tests in CI. Namely, we needed a way to place fixture test files in the bundled package directory, which proved difficult in an integration test environment where we'd need to mutate the build/ directory in order to properly copy over fixture files.

The solution I landed on with some input from the ops folks was to introduce a developer.bundledPackageLocation option that allows us to override Fleet's default bundled package location with one /tmp to prevent mutation of the build artifact during CI. It was quite the runaround (the commit log for this PR is a sight to behold), but I think this is probably the cleanest solution we can have to support this very specific use case in tests.

kpollich · 2022-03-04T15:19:38Z

x-pack/plugins/fleet/server/services/epm/archive/validation.ts

+    }
+
+    if (elasticsearch?.index_template?.mappings) {
+      parsedElasticsearchEntry['index_template.mappings'] = expandDottedEntries(


These mappings/settings fields come through the manifest with dotted.key.names, but the package info object expects them to be expanded to nested object structures.

kpollich · 2022-03-04T15:20:54Z

x-pack/plugins/fleet/server/services/epm/archive/validation.ts

+      );
+    }
+
+    // Build up the stream object here so we can conditionally insert nullable fields. The package registry omits undefined


A lot of the changes here are related to this concept. The registry service omits undefined values. We could probably get away with not doing this and more loosely comparing values, treating an explicit undefined the same as an omitted value, but I opted to be as one-to-one as possible here.

Makes sense to be conservative for now. When we're done with this PR, could we add notes to #115032 about potential any cleanup like this?

kpollich · 2022-03-04T15:21:18Z

@elasticmachine merge upstream

…-consistent

joshdover

Tested locally with the apm bundled package and saw the ingest pipelines and settings to be complete and correct.

x-pack/plugins/fleet/server/services/epm/archive/validation.ts

joshdover · 2022-03-04T16:43:13Z

x-pack/plugins/fleet/server/services/epm/archive/validation.ts

+      );
+    }
+
+    // Build up the stream object here so we can conditionally insert nullable fields. The package registry omits undefined


Makes sense to be conservative for now. When we're done with this PR, could we add notes to #115032 about potential any cleanup like this?

joshdover · 2022-03-04T16:44:58Z

As we discussed offline, I'm also +1 to renaming this code to something other than validation. What do you think about generatePackageInfo?

kpollich · 2022-03-07T13:50:06Z

@elasticmachine merge upstream

…-consistent

kpollich · 2022-03-09T06:00:43Z

x-pack/test/fleet_api_integration/apis/epm/install_by_upload.ts

@@ -75,7 +75,7 @@ export default function (providerContext: FtrProviderContext) {
        .type('application/gzip')
        .send(buf)
        .expect(200);
-      expect(res.body.items.length).to.be(27);
+      expect(res.body.items.length).to.be(29);


These responses have ingest pipelines, etc now, so they have additional elements.

kpollich · 2022-03-09T14:02:28Z

@elasticmachine merge upstream

…-consistent

nchaulet · 2022-03-09T14:10:13Z

x-pack/plugins/fleet/server/integration_tests/validate_bundled_packages.test.ts

+        'application/zip'
+      );
+
+      expect(archivePackageInfo.packageInfo.data_streams).toEqual(


can we test more things than the data_streams property here?

I am taking a look at other properties to place under test here and there don't seem to be any good candidates. policy_templates and vars were my first thoughts, but the current registry logic "fills out" variable values that we infer default for in Fleet, so these fields are unlikely to match for now. I am going to merge this PR for now, and these tests will likely change in #115032

nchaulet

🚀

kpollich · 2022-03-09T14:43:35Z

@elasticmachine merge upstream

…-consistent

joshdover

LGTM, manually tested with this config (and bundled packages correctly added to directory):

xpack.fleet.packages:
  - package: apm
    version: 8.1.0

Got these logs:

[2022-03-09T15:58:24.821+01:00][DEBUG][plugins.fleet] Setting up Fleet Elasticsearch assets
[2022-03-09T15:58:24.821+01:00][DEBUG][plugins.fleet] Creating Fleet component template and ingest pipeline
[2022-03-09T15:58:25.279+01:00][DEBUG][plugins.fleet] Setting up initial Fleet packages
[2022-03-09T15:58:25.280+01:00][DEBUG][plugins.fleet] kicking off bulk install of apm
[2022-03-09T15:58:25.287+01:00][DEBUG][plugins.fleet] found bundled package for requested install of apm-8.1.0 - installing from bundled package archive
[2022-03-09T15:58:25.361+01:00][DEBUG][plugins.fleet] setting file list to the cache for apm-8.1.0
[2022-03-09T15:58:25.361+01:00][DEBUG][plugins.fleet] setting package info to the cache for apm-8.1.0
[2022-03-09T15:58:34.785+01:00][DEBUG][plugins.fleet] retrieved installed package apm-8.1.0 from cache

And APM assets were correctly installed, including ingest pipelines 👍

joshdover · 2022-03-09T14:26:02Z

x-pack/plugins/fleet/server/services/epm/packages/bundled_packages.ts

+  const BUNDLED_PACKAGE_DIRECTORY = config?.developer?.bundledPackageLocation
+    ? config.developer.bundledPackageLocation
+    : path.join(__dirname, '../../../../target/bundled_packages');


I think it'd be ideal if we put this default in the config schema in x-pack/plugins/fleet/server/index.ts:

bundledPackageLocation: schema.string({ defaultValue: '...' }),

joshdover · 2022-03-09T14:34:32Z

x-pack/plugins/fleet/server/services/epm/archive/parse.ts

+// Ingest pipelines are specified in a `data_stream/<name>/elasticsearch/ingest_pipeline/` directory where a `default`
+// ingest pipeline should be specified by one of these filenames.


Thank you for these comments 💯

kpollich · 2022-03-09T17:27:08Z

@elasticmachine merge upstream

…-consistent

kpollich · 2022-03-09T18:06:03Z

@elasticmachine merge upstream

…-consistent

…' of github.com:kpollich/kibana into 126695-make-upload-and-registry-package-info-consistent

kibana-ci · 2022-03-09T23:22:31Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1237	1240	+3

Unknown metric groups

API count

id	before	after	diff
`fleet`	1354	1357	+3

History

💔 Build #28990 failed f0e9da1
💔 Build #28974 failed bfffa79
💔 Build #28955 failed 9122a63
💔 Build #28929 failed 5d97961
💔 Build #28877 failed dadfe20

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

kibanamachine · 2022-03-10T14:00:29Z

⚪ Backport skipped

The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.

Manual backport

To create the backport manually run:

node scripts/backport --pr 126915

Questions ?

Please refer to the Backport tool documentation

…move-pdf-generation-to-screenshotting * 'main' of github.com:elastic/kibana: (62 commits) [Lens] Drop partial buckets option (elastic#127153) chore(NA): remove unused translation xpack.ml.management.jobsSpacesList.objectNoun from fr-FR (elastic#127457) Add data to user details page (elastic#127019) [Fleet] Make upload and registry package info consistent (elastic#126915) [Reporting] Capture browser errors (elastic#127135) Initial readme commit with some stub articles (elastic#127420) skip flaky suite (elastic#121482) skip flaky suite (elastic#127416) Tests to ensure Kibana is handling multi-space import of saved objects correctly (elastic#127229) [Aggs] remove toAngularJson (elastic#127267) [i18n] Integrate 8.2.0 Translations (elastic#127309) [Security Solution] [Endpoint] Creates generic policy tab artifact component to be used for all of our artifacts (elastic#126685) [Kibana React] Fix Page Template `solutionNav` propagation (elastic#127140) [Cases] Export getRelatedCases API from cases client (elastic#127065) [Cloud Posture]add support for sorting benchmark page (elastic#126983) [User experience] Fix filters for the app (elastic#127295) [Fleet] Fix timeserie dimension mapping (elastic#127328) [data view mgmt] fix data view name wrap (elastic#127319) [kbn/optimizer] extract string diffing logic (elastic#127394) [ResponseOps] Add pagination and sorting to the alerts search strategy (elastic#126813) ... # Conflicts: # x-pack/plugins/screenshotting/common/errors.ts # x-pack/plugins/screenshotting/common/index.ts # x-pack/plugins/screenshotting/server/screenshots/observable.ts

kibanamachine · 2022-03-14T14:06:23Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 126915 or prevent reminders by adding the backport:skip label.

kibanamachine · 2022-03-15T14:16:18Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 126915 or prevent reminders by adding the backport:skip label.

kpollich added 2 commits March 3, 2022 12:52

Update docker image + set up initial validation test

cf96755

Get validation test passing

625f582

kpollich requested a review from a team as a code owner March 4, 2022 14:50

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Mar 4, 2022

kpollich changed the title ~~[Fleet] ake upload and registry package info consistent~~ [Fleet] Make upload and registry package info consistent Mar 4, 2022

kpollich self-assigned this Mar 4, 2022

kpollich marked this pull request as draft March 4, 2022 14:51

kpollich added release_note:skip Skip the PR/issue when compiling release notes auto-backport Deprecated - use backport:version if exact versions are needed v8.2.0 Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project labels Mar 4, 2022

kpollich commented Mar 4, 2022

View reviewed changes

kibanamachine and others added 2 commits March 4, 2022 10:21

Merge branch 'main' into 126695-make-upload-and-registry-package-info…

5ff6ee6

…-consistent

Remove erroneous test load call

c564592

joshdover reviewed Mar 4, 2022

View reviewed changes

joshdover mentioned this pull request Mar 4, 2022

[Fleet] Rely on the content of a package when installing it #115032

Closed

2 tasks

Address PR review + improve comments + rename validation.ts -> parse.ts

586c08c

Merge branch 'main' into 126695-make-upload-and-registry-package-info…

6a9c1e9

…-consistent

kpollich marked this pull request as ready for review March 7, 2022 14:25

kpollich added 7 commits March 7, 2022 10:17

Replace packages in fleet_packages.json

62a45cc

Add temp debug log to debug CI failures

cae823e

Use a non-colliding package in bundled package tests

37f60f3

(debug) Add logging output

4212c27

(debug) More logging

0f779d9

(debug) Log bundled package dir in module

000f3a1

Use absolute path for bundled packages

6fdfd1f

kpollich added 2 commits March 8, 2022 21:49

Fix test hopefully 🤞

14e7b87

Fix other failing tests

eadab35

kpollich commented Mar 9, 2022

View reviewed changes

Merge branch 'main' into 126695-make-upload-and-registry-package-info…

1da1eba

…-consistent

nchaulet reviewed Mar 9, 2022

View reviewed changes

nchaulet approved these changes Mar 9, 2022

View reviewed changes

Merge branch 'main' into 126695-make-upload-and-registry-package-info…

a1254e7

…-consistent

joshdover approved these changes Mar 9, 2022

View reviewed changes

Move default for bundled package dir to schema definition

dadfe20

Merge branch 'main' into 126695-make-upload-and-registry-package-info…

5d97961

…-consistent

kibanamachine and others added 5 commits March 9, 2022 13:06

Merge branch 'main' into 126695-make-upload-and-registry-package-info…

9122a63

…-consistent

Fix schema default value for bundledPackageLocation

bd2e6c2

Merge branch '126695-make-upload-and-registry-package-info-consistent…

bfffa79

…' of github.com:kpollich/kibana into 126695-make-upload-and-registry-package-info-consistent

Fix snapshot

f0e9da1

Fix regression in bundled packages fetch

29a30e4

kpollich merged commit 88f12fd into elastic:main Mar 10, 2022

kpollich deleted the 126695-make-upload-and-registry-package-info-consistent branch March 10, 2022 14:02

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 14, 2022

kpollich added the backport:skip This commit does not require backporting label Mar 16, 2022

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 16, 2022

kpollich mentioned this pull request Jan 11, 2024

Failing test: X-Pack EPM API Integration Tests.x-pack/test/fleet_api_integration/apis/epm/list·ts - EPM Endpoints EPM - list list api tests lists all limited packages from the registry #167188

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Make upload and registry package info consistent #126915

[Fleet] Make upload and registry package info consistent #126915

kpollich commented Mar 4, 2022 •

edited

Loading

elasticmachine commented Mar 4, 2022

kpollich Mar 4, 2022

kpollich Mar 9, 2022

kpollich Mar 4, 2022

kpollich Mar 4, 2022

joshdover Mar 4, 2022

kpollich commented Mar 4, 2022

joshdover left a comment

joshdover Mar 4, 2022

joshdover commented Mar 4, 2022

kpollich commented Mar 7, 2022

kpollich Mar 9, 2022

kpollich commented Mar 9, 2022

nchaulet Mar 9, 2022

kpollich Mar 10, 2022

nchaulet left a comment

kpollich commented Mar 9, 2022

joshdover left a comment

joshdover Mar 9, 2022

joshdover Mar 9, 2022

kpollich commented Mar 9, 2022

kpollich commented Mar 9, 2022

kibana-ci commented Mar 9, 2022

API count

kibanamachine commented Mar 10, 2022

kibanamachine commented Mar 14, 2022

kibanamachine commented Mar 15, 2022

		// Ingest pipelines are specified in a `data_stream/<name>/elasticsearch/ingest_pipeline/` directory where a `default`
		// ingest pipeline should be specified by one of these filenames.

[Fleet] Make upload and registry package info consistent #126915

[Fleet] Make upload and registry package info consistent #126915

Conversation

kpollich commented Mar 4, 2022 • edited Loading

elasticmachine commented Mar 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kpollich commented Mar 4, 2022

joshdover left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshdover commented Mar 4, 2022

kpollich commented Mar 7, 2022

Choose a reason for hiding this comment

kpollich commented Mar 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nchaulet left a comment

Choose a reason for hiding this comment

kpollich commented Mar 9, 2022

joshdover left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kpollich commented Mar 9, 2022

kpollich commented Mar 9, 2022

kibana-ci commented Mar 9, 2022

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

History

kibanamachine commented Mar 10, 2022

⚪ Backport skipped

Manual backport

Questions ?

kibanamachine commented Mar 14, 2022

kibanamachine commented Mar 15, 2022

kpollich commented Mar 4, 2022 •

edited

Loading