[Profiling] Manage indices via Elasticsearch

[jbcrail]

Summary

This PR sets up the profiling indices via Elasticsearch instead of using the single-click installation in Kibana.

Notable changes

• Set Elasticsearch flag (xpack.profiling.templates.enabled) to true so that indices are bootstrapped in Elasticsearch
• Remove component templates and mappings (handled in Elasticsearch plugin now)
• Remove six steps from Kibana's single-click installation (handled in Elasticsearch plugin now)
• Use new status API in Elasticsearch to check if resources (templates, indices, etc) are ready
• Invoke creation of collector and symbolizer package policies in parallel to updating APM policy
• Refactor initialization and verification steps so that we can later give the user more granular insight into the status of steps

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[jbcrail]

This check can likely be replaced by querying the new status API after it is merged.

[jbcrail]

The latest Cypress tests fail with the following error:

Failed to fetch latest version of profiler_symbolizer from registry: Cannot read properties of null (reading 'version')

@inge4pres @florianl Is this error related to the configuration in x-pack/plugins/profiling/server/lib/setup/steps/get_fleet_policy_step.ts? I updated to the latest version for the symbolizer package. However, the tests failed both before and after this change.

[florianl]

The latest Cypress tests fail with the following error:

Failed to fetch latest version of profiler_symbolizer from registry: Cannot read properties of null (reading 'version')

maybe @joshdover or @kpollich can tell more about this error?

The package profiler_symbolizer exists now for some time and wasn't touched since its creation. And the bundling with kibana did also not change:

kibana/fleet_packages.json

Lines 43 to 48 in 9d02c63

	{
	"name": "profiler_symbolizer",
	"version": "8.8.0-preview",
	"forceAlignStackVersion": true,
	"allowSyncToPrerelease": true
	},

[rockdaboot]

To understand the error, the code around here is key:

kibana/x-pack/plugins/fleet/server/services/epm/registry/index.ts

Line 129 in b7bd376

`Failed to fetch latest version of ${packageName} from registry: ${error.message}`

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[florianl]

With the following command I verified that the package is available:

$ curl -s "https://epr.elastic.co/search?package=profiler_symbolizer&prerelease=true&all=true"
[
  {
    "name": "profiler_symbolizer",
    "title": "Universal Profiling Symbolizer",
    "version": "8.8.0-preview",
    ...
    }
 ]

Not sure where and why it fails here.

[rockdaboot]

With the following command I verified that the package is available:

$ curl -s "https://epr.elastic.co/search?package=profiler_symbolizer&prerelease=true&all=true"
[
  {
    "name": "profiler_symbolizer",
    "title": "Universal Profiling Symbolizer",
    "version": "8.8.0-preview",
    ...
    }
 ]

Not sure where and why it fails here.

Yes, the point is that the prerelease is set to false for some reasons.
(I logged the URL in a commit that I meanwhile removed.)

This corrects the URL, but then the tests the fail later on, not sure if that is related to UP.

diff --git a/x-pack/plugins/fleet/server/services/epm/registry/index.ts b/x-pack/plugins/fleet/server/services/epm/registry/index.ts
index 3b9b37f31d9..ff3768e3118 100644
--- a/x-pack/plugins/fleet/server/services/epm/registry/index.ts
+++ b/x-pack/plugins/fleet/server/services/epm/registry/index.ts
@@ -100,7 +100,7 @@ async function _fetchFindLatestPackage(
 
     // temporary workaround to allow synthetics package beta version until there is a GA available
     // needed because synthetics is installed by default on kibana startup
-    const prereleaseAllowedExceptions = ['synthetics'];
+    const prereleaseAllowedExceptions = ['synthetics', 'profiler_symbolizer'];
 
     const prereleaseEnabled = prerelease || prereleaseAllowedExceptions.includes(packageName);

Maybe someone knows how to set prerelease=true without changing the code !?

[jbcrail]

I found out today that the prerelease error was not the reason why the tests were failing. It turns out that the steps needed to set up the policies, permissions, and settings were running at the same time as Elasticsearch was setting up the templates, indices, etc, so it was likely a timing issue instead.

Unfortunately one of the UI tests is failing after I fixed several things, but at least we have progress. I worked on these today:

• use new status API in Elasticsearch to check if resources (templates, indices, etc) are ready
• invoke creation of collector and symbolizer package policies in parallel to updating APM policy
• refactor initialization and verification steps so that we can later give the user more granular insight into the status of steps

Here's what left to do next week:

• address any feedback
• fix failed UI test
• remove duplication
• add checks to see if the collector and symbolizer package policies were created

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 43f4404

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
securitySolution	413	417	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
securitySolution	497	501	+4
total			+6

History

• 💚 Build #132321 succeeded a7cff1f4e7c366afc54675f796bcd191b8ed594a
• 💔 Build #132297 failed 69e9a654e1b7fb22662bc70dbbf50868b80c5379
• 💚 Build #132264 succeeded 4b575aef65d726b1f07f7a8eb91005f90689f60e
• 💚 Build #132231 succeeded fd210ce23834760321b9f93b00435072096c1e47
• 💚 Build #131947 succeeded d5e21de9c349bc4f8d8b0eb197ded3e01e48f851
• 💛 Build #131925 was flaky fa477a41305ed3b8133bdfc011e3fa13bd437b84

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @inge4pres