[Publishing] Load ES image for ftest from DRA artifact

[jedrazb]

Closes https://github.com/elastic/search-team/issues/8273

Changes

This PR updates the functional test pipeline to download and load es docker images from DRA artifacts instead of relying on the snapshot image in the registry. This ensures compatibility during the release process when the ES snapshot image is not yet available in the elastic docker registry.

Changes:

• Resolves the appropriate ES tarball URL based on the architecture (see available arch) using the Elasticsearch DRA manifest.
• Removed Kibana and Enterprise Search from docker-compose.yml fixtures as they are not required for our ftests.

Validation:

• Tested manually different versions
• How you can validate? Go to buildkite build and check e.g. smoke test steps to confirm that the image is coming from DRA artifact

Error handling

• Added a lot of logs to make sure that we know why we failed, e.g. when we need to produce an artifact and ES still not produced one we will see message like:

+++ curl -sS -f https://artifacts-snapshot.elastic.co/elasticsearch/latest/9.0.1-SNAPSHOT.json
++ RESPONSE='curl: (22) The requested URL returned error: 404'
++ CURL_EXIT_CODE=22
++ '[' 22 -ne 0 ']'
++ echo 'Error: Failed to fetch DRA manifest for artifact elasticsearch and version 9.0.1-SNAPSHOT.'

[artem-shelkovnikov]

How well would that work if you run ftests locally on different branches (9.0 and 8.x) for example?

Since there's no version it can have weird conflicts, right? Or is it resolved somewhere else?

[jedrazb]

It works well, I tried it with e.g. 8.16

We are fetching the manifest for the right version that depends on the connector version (in the VERSION file) see this line https://github.com/elastic/connectors/pull/2979/files#diff-2c88fda54d210cfb99f46dd547063eb3b88f8f41296ce60881535313cd7b281cR105 and you can backtrack where the version variable comes from.

Unfortunately the elasticsearch images loaded from tarball do not have the "stack version" tag, instead they are just in format elasticsearch:{aarch64|x86_64}, so this is why we reference image in such a way.

But given that we are fetching it from the right manifest URL it is the correct version.

[jedrazb]

E.g. this is for SPO ftests in 9.0.0, note the ES img created by DRA is only 1 hr old

[jedrazb]

If you run first on main branch, then checkout to 8.16 and run ftest again, the docker load will override the the image for your arch and the logs will show:

[FMWK][12:10:39][INFO] Elasticsearch 8.16.0-SNAPSHOT and Connectors 8.16.0 are compatible

( I tried this sequence of steps locally by tweaking the version in the file) so no weird conflicts

[artem-shelkovnikov]

🚀 🌕

[jedrazb]

@seanstory 👋 you might want to take a look as well as you have been involved with onboarding connectors to DRA

[github-actions]

💔 Failed to create backport PR(s)

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

To backport manually run:
backport --pr 2979 --autoMerge --autoMergeMethod squash

[jedrazb]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation