[Upgrade Assistant] Reindexing progress

[yuliacech]

Summary

This PR adds a progress display to the reindexing process. The progress is shown in percentages and is calculated as follows: There are 4 steps in the reindexing process, those are making the index read-only, creating a new index, reindexing documents in the old index into the new index and swapping an alias for the new index with the old index.

1. The reindex process started - 0%
2. The index has been made read only - 5%
3. A new index has been created - 10%
4. Reindexing documents - 10%-95% depending on the progress of this step in ES
5. Alias has been swapped - 100%

The reindexing progress is displayed in the reindex flyout and in the deprecations table.

How to test

1. Download and unzip this file
   data.zip

2. start your 7.x cluster with the command yarn es snapshot -E path.data=path/to/data/folder

3. start Kibana with the command yarn start

Screenshot

Reindex flyout

Deprecations table

Screencast

Screen.Recording.2021-10-07.at.14.01.08.mov

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

Release Note

Upgrade Assistant now displays the progress of your reindexing tasks in the Elasticsearch deprecations table and also in the reindex flyout.

[yuliacech]

getReindexState is used with data from both getReindexStatus and startReindexTask endpoints. The status endpoint returns data in the format

{ reindexOp: { ...reindexOpProperties }, warnings, hasPrivileges }

and start reindex return data in the format

{ ...reindexOpProperties } 

[elasticmachine]

Pinging @elastic/kibana-stack-management (Team:Stack Management)

[yuliacech]

@elasticmachine merge upstream

[sabarasaba]

Thanks for working on this @yuliacech! Tested locally and everything works as expected, comments and steps to test were very helpful! Left 2 small comments only that I wanted to get your opinion on

[sabarasaba]

I know this was here from before, but I don't think we need to specify empty value objects for all these FormattedMessage components 🤔

[yuliacech]

I regenerated those snapshots with node scripts/jest.js -i plugins/upgrade_assistant/ -u command, so I think this is how the component is 'shallow rendered'.

[sabarasaba]

I noticed we call this reindexingDocumentsProgress here but in the reindex state we call it reindexTaskPercComplete. I think we should try to use the same naming convention in both places

[yuliacech]

yeah, good call! let me update this real quick

[yuliacech]

@elasticmachine merge upstream

[sabarasaba]

Latest changes LGTM!

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request

• Commit: a4bbc28

• Storybooks Preview

• Flaky suites:

  • xpack-kibana-ciGroup8

---

Test Failures

Kibana Pipeline / general / Performance Tests.x-pack/test/performance/tests/reporting_dashboard·ts.performance reporting dashbaord downloaded PDF has OK status

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 3 times on tracked branches: https://github.com/elastic/kibana/issues/110470

[00:00:00]     │
[00:00:00]       └-: performance
[00:00:00]         └-> "before all" hook in "performance"
[00:00:00]         └-: reporting dashbaord
[00:00:00]           └-> "before all" hook for "downloaded PDF has OK status"
[00:00:00]           └-> "before all" hook for "downloaded PDF has OK status"
[00:00:00]             │ debg resolved import for x-pack/test/performance/kbn_archives/reporting_dashboard to /dev/shm/workspace/parallel/12/kibana/x-pack/test/performance/kbn_archives/reporting_dashboard.json
[00:00:00]             │ info importing 4 saved objects { space: undefined }
[00:00:00]             │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_7.16.0_001/a8azJuosSzmfSXaSPzVqGQ] update_mapping [_doc]
[00:00:00]             │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.ds-ilm-history-5-2021.10.12-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [ilm-history-ilm-policy]
[00:00:00]             │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.ds-ilm-history-5-2021.10.12-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [ilm-history-ilm-policy]
[00:00:00]             │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_7.16.0_001/a8azJuosSzmfSXaSPzVqGQ] update_mapping [_doc]
[00:00:00]             │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_7.16.0_001/a8azJuosSzmfSXaSPzVqGQ] update_mapping [_doc]
[00:00:00]             │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_7.16.0_001/a8azJuosSzmfSXaSPzVqGQ] update_mapping [_doc]
[00:00:00]             │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_7.16.0_001/a8azJuosSzmfSXaSPzVqGQ] update_mapping [_doc]
[00:00:01]             │ succ import success
[00:00:01]             │ info [x-pack/test/performance/es_archives/reporting_dashboard] Loading "mappings.json"
[00:00:01]             │ info [x-pack/test/performance/es_archives/reporting_dashboard] Loading "data.json.gz"
[00:00:01]             │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [foo] creating index, cause [api], templates [], shards [1]/[1]
[00:00:01]             │ info [x-pack/test/performance/es_archives/reporting_dashboard] Created index "foo"
[00:00:01]             │ debg [x-pack/test/performance/es_archives/reporting_dashboard] "foo" settings {"index":{"number_of_replicas":"1","number_of_shards":"1"}}
[00:00:02]             │ info [x-pack/test/performance/es_archives/reporting_dashboard] Indexed 10000 docs into "foo"
[00:00:02]           └-> downloaded PDF has OK status
[00:00:02]             └-> "before each" hook: global before each for "downloaded PDF has OK status"
[00:00:02]             │ debg navigating to dashboards url: http://localhost:61121/app/dashboards
[00:00:02]             │ debg navigate to: http://localhost:61121/app/dashboards
[00:00:03]             │ debg browser[INFO] http://localhost:61121/login?next=%2Fapp%2Fdashboards%3F_t%3D1634045441139 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:03]             │
[00:00:03]             │ debg browser[INFO] http://localhost:61121/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:03]             │ debg ... sleep(700) start
[00:00:03]             │ debg ... sleep(700) end
[00:00:03]             │ debg returned from get, calling refresh
[00:00:05]             │ debg browser[INFO] http://localhost:61121/login?next=%2Fapp%2Fdashboards%3F_t%3D1634045441139 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:05]             │
[00:00:05]             │ debg browser[INFO] http://localhost:61121/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:05]             │ debg currentUrl = http://localhost:61121/login?next=%2Fapp%2Fdashboards%3F_t%3D1634045441139
[00:00:05]             │          appUrl = http://localhost:61121/app/dashboards
[00:00:05]             │ debg TestSubjects.find(kibanaChrome)
[00:00:05]             │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:00:06]             │ debg Found login page
[00:00:06]             │ debg TestSubjects.setValue(loginUsername, test_user)
[00:00:06]             │ debg TestSubjects.click(loginUsername)
[00:00:06]             │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:00:06]             │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:00:06]             │ERROR browser[SEVERE] http://localhost:61121/api/licensing/info - Failed to load resource: the server responded with a status of 401 (Unauthorized)
[00:00:06]             │ debg TestSubjects.setValue(loginPassword, changeme)
[00:00:06]             │ debg TestSubjects.click(loginPassword)
[00:00:06]             │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:00:06]             │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:00:06]             │ debg TestSubjects.click(loginSubmit)
[00:00:06]             │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:00:06]             │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:00:06]             │ debg Find.waitForDeletedByCssSelector('.kibanaWelcomeLogo') with timeout=10000
[00:00:06]             │ proc [kibana]   log   [13:30:44.834] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:00:07]             │ debg browser[INFO] http://localhost:61121/app/dashboards?_t=1634045441139 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:07]             │
[00:00:07]             │ debg browser[INFO] http://localhost:61121/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:07]             │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:00:09]             │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"] nav:not(.ng-hide)') with timeout=60000
[00:00:10]             │ debg browser[INFO] http://localhost:61121/app/dashboards?_t=1634045447631 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:10]             │
[00:00:10]             │ debg browser[INFO] http://localhost:61121/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:10]             │ debg Finished login process currentUrl = http://localhost:61121/app/dashboards
[00:00:10]             │ debg ... sleep(501) start
[00:00:11]             │ debg ... sleep(501) end
[00:00:11]             │ debg in navigateTo url = http://localhost:61121/app/dashboards
[00:00:11]             │ debg Waiting up to 20000ms for dashboard landing page...
[00:00:11]             │ debg onDashboardLandingPage
[00:00:11]             │ debg TestSubjects.exists(dashboardLandingPage)
[00:00:11]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="dashboardLandingPage"]') with timeout=5000
[00:00:11]             │ debg Load Saved Dashboard dashboard
[00:00:11]             │ debg gotoDashboardLandingPage
[00:00:11]             │ debg onDashboardLandingPage
[00:00:11]             │ debg TestSubjects.exists(dashboardLandingPage)
[00:00:11]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="dashboardLandingPage"]') with timeout=5000
[00:00:11]             │ debg searchForItemWithName: dashboard
[00:00:11]             │ debg TestSubjects.find(tableListSearchBox)
[00:00:11]             │ debg Find.findByCssSelector('[data-test-subj="tableListSearchBox"]') with timeout=10000
[00:00:11]             │ debg isGlobalLoadingIndicatorVisible
[00:00:11]             │ debg TestSubjects.exists(globalLoadingIndicator)
[00:00:11]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:00:11]             │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:00:11]             │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:00:14]             │ debg TestSubjects.click(dashboardListingTitleLink-dashboard)
[00:00:14]             │ debg Find.clickByCssSelector('[data-test-subj="dashboardListingTitleLink-dashboard"]') with timeout=10000
[00:00:14]             │ debg Find.findByCssSelector('[data-test-subj="dashboardListingTitleLink-dashboard"]') with timeout=10000
[00:00:14]             │ debg isGlobalLoadingIndicatorVisible
[00:00:14]             │ debg TestSubjects.exists(globalLoadingIndicator)
[00:00:14]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:00:14]             │ debg browser[INFO] http://localhost:61121/app/dashboards#/view/37b49c50-2dc6-11eb-8af3-cb3aa84dbabd?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now)) 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:14]             │
[00:00:14]             │ debg browser[INFO] http://localhost:61121/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:16]             │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:00:16]             │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:00:16]             │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:00:16]             │ debg TestSubjects.missingOrFail(dashboardLandingPage)
[00:00:16]             │ debg Find.waitForDeletedByCssSelector('[data-test-subj="dashboardLandingPage"]') with timeout=10000
[00:00:17]             │ debg openPdfReportingPanel
[00:00:17]             │ debg openShareMenuItem title:PDF Reports
[00:00:17]             │ debg TestSubjects.exists(shareContextMenu)
[00:00:17]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="shareContextMenu"]') with timeout=2500
[00:00:18]             │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [.async-search] creating index, cause [api], templates [], shards [1]/[0]
[00:00:20]             │ debg --- retry.tryForTime error: [data-test-subj="shareContextMenu"] is not displayed
[00:00:20]             │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_7.16.0_001/a8azJuosSzmfSXaSPzVqGQ] update_mapping [_doc]
[00:00:20]             │ debg TestSubjects.click(shareTopNavButton)
[00:00:20]             │ debg Find.clickByCssSelector('[data-test-subj="shareTopNavButton"]') with timeout=10000
[00:00:20]             │ debg Find.findByCssSelector('[data-test-subj="shareTopNavButton"]') with timeout=10000
[00:00:21]             │ debg Find.findByCssSelector('div.euiContextMenuPanel') with timeout=10000
[00:00:22]             │ debg TestSubjects.click(sharePanel-PDFReports)
[00:00:22]             │ debg Find.clickByCssSelector('[data-test-subj="sharePanel-PDFReports"]') with timeout=10000
[00:00:22]             │ debg Find.findByCssSelector('[data-test-subj="sharePanel-PDFReports"]') with timeout=10000
[00:00:23]             │ debg Find.waitForElementStale with timeout=10000
[00:00:23]             │ debg TestSubjects.click(generateReportButton)
[00:00:23]             │ debg Find.clickByCssSelector('[data-test-subj="generateReportButton"]') with timeout=10000
[00:00:23]             │ debg Find.findByCssSelector('[data-test-subj="generateReportButton"]') with timeout=10000
[00:00:23]             │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_7.16.0_001/a8azJuosSzmfSXaSPzVqGQ] update_mapping [_doc]
[00:00:23]             │ debg getReportURL
[00:00:23]             │ debg TestSubjects.getAttribute(downloadCompletedReportButton, href, tryTimeout=120000, findTimeout=60000)
[00:00:23]             │ debg TestSubjects.find(downloadCompletedReportButton)
[00:00:23]             │ debg Find.findByCssSelector('[data-test-subj="downloadCompletedReportButton"]') with timeout=60000
[00:01:25]             │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="downloadCompletedReportButton"])
[00:01:25]             │      Wait timed out after 61179ms
[00:01:25]             │ debg TestSubjects.find(downloadCompletedReportButton)
[00:01:25]             │ debg Find.findByCssSelector('[data-test-subj="downloadCompletedReportButton"]') with timeout=60000
[00:02:26]             │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="downloadCompletedReportButton"])
[00:02:26]             │      Wait timed out after 61163ms
[00:02:27]             │ info Taking screenshot "/dev/shm/workspace/parallel/12/kibana/x-pack/test/functional/screenshots/failure/performance reporting dashbaord downloaded PDF has OK status.png"
[00:02:27]             │ info Current URL is: http://localhost:61121/app/dashboards#/view/37b49c50-2dc6-11eb-8af3-cb3aa84dbabd?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))
[00:02:27]             │ info Saving page source to: /dev/shm/workspace/parallel/12/kibana/x-pack/test/performance/failure_debug/html/performance reporting dashbaord downloaded PDF has OK status.html
[00:02:27]             └- ✖ fail: performance reporting dashbaord downloaded PDF has OK status
[00:02:27]             │      Error: retry.tryForTime timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="downloadCompletedReportButton"])
[00:02:27]             │ Wait timed out after 61163ms
[00:02:27]             │     at /dev/shm/workspace/parallel/12/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
[00:02:27]             │     at runMicrotasks (<anonymous>)
[00:02:27]             │     at processTicksAndRejections (internal/process/task_queues.js:95:5)
[00:02:27]             │       at onFailure (/dev/shm/workspace/parallel/12/kibana/test/common/services/retry/retry_for_success.ts:17:9)
[00:02:27]             │       at retryForSuccess (/dev/shm/workspace/parallel/12/kibana/test/common/services/retry/retry_for_success.ts:59:13)
[00:02:27]             │       at RetryService.tryForTime (/dev/shm/workspace/parallel/12/kibana/test/common/services/retry/retry.ts:22:12)
[00:02:27]             │       at TestSubjects.getAttribute (/dev/shm/workspace/parallel/12/kibana/test/functional/services/common/test_subjects.ts:190:12)
[00:02:27]             │       at ReportingPageObject.getReportURL (test/functional/page_objects/reporting_page.ts:36:17)
[00:02:27]             │       at Context.<anonymous> (test/performance/tests/reporting_dashboard.ts:50:7)
[00:02:27]             │       at Object.apply (/dev/shm/workspace/parallel/12/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:02:27]             │ 
[00:02:27]             │ 

Stack Trace

Error: retry.tryForTime timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="downloadCompletedReportButton"])
Wait timed out after 61163ms
    at /dev/shm/workspace/parallel/12/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at onFailure (/dev/shm/workspace/parallel/12/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/12/kibana/test/common/services/retry/retry_for_success.ts:59:13)
    at RetryService.tryForTime (/dev/shm/workspace/parallel/12/kibana/test/common/services/retry/retry.ts:22:12)
    at TestSubjects.getAttribute (/dev/shm/workspace/parallel/12/kibana/test/functional/services/common/test_subjects.ts:190:12)
    at ReportingPageObject.getReportURL (test/functional/page_objects/reporting_page.ts:36:17)
    at Context.<anonymous> (test/performance/tests/reporting_dashboard.ts:50:7)
    at Object.apply (/dev/shm/workspace/parallel/12/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

---

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
upgradeAssistant	125.6KB	126.2KB	+589.0B

History

• 💚 Build #159858 succeeded a4bbc28
• 💚 Build #159852 succeeded 3eb7aa3
• 💔 Build #159380 failed a62cbed
• 💛 Build #158889 was flaky c77c8ab

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream