Replacing es_archives/reporting/ecommerce_kibana with kbn_archiver/reporting/ecommerce.json as part of migrating to kbn_archiver

[bhavyarm]

Part of #102552

These are the steps after you find out where is the .kibana data getting loaded in the test ( sometimes you can find it by name other times look at the file)

1. Load ecommerce_kibana data by running

NODE_TLS_REJECT_UNAUTHORIZED=0 node --no-warnings scripts/es_archiver load x-pack/test/functional/es_archives/reporting/ecommerce_kibana --config test/functional/config.js  --es-url=http://elastic:changeme@localhost:9220 --kibana-url=http://elastic:changeme@localhost:5620

2. Find all the different types of saved objects in .kibana loaded in the first step by running

cat  x-pack/test/functional/es_archives/reporting/ecommerce_kibana/data.json | awk '{$1=$1};1' | fgrep '"type":' | fgrep -v '"type": "doc"' | sort | uniq

3. Now save .kibana json into KBN archiver by running (please feed the types from step 2 here - dashboard,index-pattern,search,visualization )

node scripts/kbn_archiver.js --config x-pack/test/functional/config.js save x-pack/test/functional/fixtures/kbn_archiver/reporting/ecommerce.json --type dashboard,index-pattern,search,visualization

4. Make the changes in the test file reporting.ts. Replace esArchiver loading .kibana and unloading:

await kibanaServer.importExport.load('x-pack/test/functional/fixtures/kbn_archiver/reporting/ecommerce.json');

await esArchiver.unload('x-pack/test/functional/es_archives/reporting/ecommerce');

5. Run the test - make sure there are no errors and test runs without any issues
6. Find which other tests are using the esArchiver from step 1 and replace it with kbnArchiver data path
7. Delete the corresponding .kibana esArchiver file - we are replacing that with the new file from step 3
8. Commit your changes

[bhavyarm]

@elasticmachine merge upstream

[bhavyarm]

@elasticmachine merge upstream

[bhavyarm]

@elasticmachine merge upstream

[dmlemeshko]

We got an image comparison failure:

     │      Error: expected 0.09129735815213762 to be below 0.09
     │       at Assertion.assert (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/expect/expect.js:100:11)
     │       at Assertion.lessThan.Assertion.below (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/expect/expect.js:336:8)
     │       at Function.lessThan (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/expect/expect.js:531:15)
     │       at Context.<anonymous> (test/functional/apps/dashboard/reporting/screenshots.ts:162:35)
     │       at Object.apply (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

0.09129735815213762 is something that we are getting constantly, so at least it is not a flakiness.

I checked failure screenshot and obviously there is no significant difference with the baseline, I can suggest running the test locally and see output.

@marius-dr might have other ideas

[marius-dr]

Looking at the screenshots, there seem to be a few problems:

1. the screenshot needs updating due to the font change. I think the baseline was added before last week when we got the proper font in.
2. there's a toast with "unable to update UI setting" which shouldn't be there.
3. The area chart seems completely different as well, maybe it was changed from Vislib to Lens.
4. same with the donut chart.

Easy links for reference:
https://storage.googleapis.com/kibana-ci-artifacts/jobs/elastic%2Bkibana%2Bpipeline-pull-request/132986/ci-worker/parallel/20/kibana/x-pack/test/functional/screenshots/session/small_dashboard_preserve_layout_actual-session-resized.png

https://storage.googleapis.com/kibana-ci-artifacts/jobs/elastic%2Bkibana%2Bpipeline-pull-request/132986/ci-worker/parallel/20/kibana/x-pack/test/functional/screenshots/session/small_dashboard_preserve_layout_baseline-baseline-resized.png

Also 9% is a lot of difference. I'd ask the reporting team if they want to update the baseline and lower that percentage.

[bhavyarm]

@elasticmachine merge upstream

[bhavyarm]

Thanks @dmlemeshko @marius-dr I have skipped that test - created an issue #102911 and starting work on it.

[elasticmachine]

Pinging @elastic/kibana-qa (Team:QA)

[bhavyarm]

cc @rashmivkulkarni

[bhavyarm]

This skipped test is running without any issues in my local and I don't have ui settings error anymore - #102911 . Once I merge this - I will unskip the screenshots.ts test under dashboard run it multiple times in CI and merge it.

[bhavyarm]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: ed2abb5
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #133134 succeeded c8ebb16e134b8d4106d0377798cf138612ee0847
• 💔 Build #132986 failed dd86478
• 💔 Build #132964 failed 4f1130c
• 💔 Build #132975 failed d94be95
• 💔 Build #132944 failed 3cc35f4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @bhavyarm

[bhavyarm]

@wayneseymour are these steps still valid? #102825 (comment)

Thanks!

[wayneseymour]

kinda yes, kinda no as things have changed. Normally that should work, but there can be many "gotchas" that make things difficult.