Failing test: X-Pack Usage Collection Functional Tests.x-pack/test/usage_collection/test_suites/application_usage - Application Usage keys in the schema match the registered application IDs

[kibanamachine]

A test failed on a tracked branch

Error: retry.tryForTime timeout: Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="kibanaChrome"])
Wait timed out after 61159ms
    at /dev/shm/workspace/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at onFailure (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry_for_success.ts:57:13)
    at Retry.try (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry.ts:32:14)
    at /dev/shm/workspace/parallel/2/kibana/test/functional/page_objects/common_page.ts:227:23
    at runAttempt (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry_for_success.ts:27:15)
    at retryForSuccess (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry_for_success.ts:66:21)
    at Retry.tryForTime (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry.ts:23:14)
    at CommonPage.navigateToApp (/dev/shm/workspace/parallel/2/kibana/test/functional/page_objects/common_page.ts:226:7)
    at Context.<anonymous> (test/usage_collection/test_suites/application_usage/index.ts:19:7)
    at Object.apply (/dev/shm/workspace/parallel/2/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
    at onFailure (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry_for_success.ts:57:13)
    at Retry.tryForTime (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry.ts:23:14)
    at CommonPage.navigateToApp (/dev/shm/workspace/parallel/2/kibana/test/functional/page_objects/common_page.ts:226:7)
    at Context.<anonymous> (test/usage_collection/test_suites/application_usage/index.ts:19:7)
    at Object.apply (/dev/shm/workspace/parallel/2/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

First failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[afharo]

Yesterday's failures, both of them are related to Kibana failing to validate the licensing info when navigating to the home page.

08:09:05         └-: Application Usage
08:09:05           └-> "before all" hook for "keys in the schema match the registered application IDs"
08:09:05           └-> keys in the schema match the registered application IDs
08:09:05             └-> "before each" hook: global before each for "keys in the schema match the registered application IDs"
08:09:08             │ERROR browser[SEVERE] http://localhost:61211/api/licensing/info - Failed to load resource: the server responded with a status of 401 (Unauthorized)
08:09:09             │ proc [kibana]   log   [06:09:09.136] [info][plugins][routes][security] Logging in with provider "basic" (basic)
08:09:14             └- ✖ fail: Application Usage keys in the schema match the registered application IDs
08:09:14             │      TypeError: Application Usage's schema is not up-to-date with the actual registered apps. Please update it at src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts.
08:09:14             │ Cannot read property 'sort' of null
08:09:14             │       at Context.<anonymous> (test/usage_collection/test_suites/application_usage/index.ts:22:74)
08:09:14             │       at processTicksAndRejections (internal/process/task_queues.js:95:5)
08:09:14             │       at Object.apply (/dev/shm/workspace/parallel/21/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
08:09:14             │ 
08:09:14             │ 
08:09:14           └-> "after all" hook for "keys in the schema match the registered application IDs"

Not related to Application Usage as such.

I'll update the test to reflect errors others than the actual schema check.

[afharo]

Nevermind, I noticed that we show the same error even in the successful runs. Digging further into this.

[afharo]

Found the bug! The tests are navigating to home. However, they are not waiting for the loading screen to be away, meaning, apps may have not been fully registered and loaded yet. Pushing a PR for that.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[kibanamachine]

New failure: CI Build - master

[spalger]

Looks like this is still flaky, but after looking at the source it seems like this line isn't doing what it's supposed to.

Skipped

master/8.1: 8a34170

[afharo]

it seems like this line isn't doing what it's supposed to.

do you mean that the components changed and that we need to change that check again?

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[afharo]

It looks like a corrupted build that couldn't load the Kibana UI:

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[dokmic]

Temporary failure that was caused by unrelated PR and resolved later. The issue can be safely closed.

[kibanamachine]

New failure: kibana-on-merge - main

[afharo]

It looks like "enterpriseSearchRelevance" should be removed from src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts

[kibanamachine]

New failure: kibana-on-merge - main

[afharo]

Maybe the license fetching system is flaky in the test?

I can see that this app is only registered if a certain level of the license is met:

kibana/x-pack/plugins/enterprise_search/public/plugin.ts

Line 476 in e523deb

if (license.isActive && license.hasAtLeast('enterprise')) {

I'll close this issue for now because it hasn't happened in 2 days. But we might want to investigate that license issue if it happens again.

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - main

[afharo]

Again "enterpriseSearchRelevance" was missing from the list of registered apps.

@elastic/search-kibana, are you aware of any flakiness in the app not being registered sometimes?

[TattdCodeMonkey]

@afharo yes we've had issues with the way that app is registered since the license check is async, we have an outstanding bug that we think could be caused by how it's registered as well. We are investigating refactoring the app registration to use a KibanaFeature for license gating instead of doing a manual license check. But I'm not sure what the timeline on that is.

cc: @saikatsarkar056

[TattdCodeMonkey]

@afharo tangentially I think we have a few other apps we might need to add to src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts namely search_playground and searchNotebooks

[afharo]

@afharo tangentially I think we have a few other apps we might need to add to src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts namely search_playground and searchNotebooks

oh! that's a good point and I think it highlights a flaw in this test: it runs with the default Kibana configuration instead of enabling all plugins. It's not a blocker, since the schema is mostly for documentation purposes and validation. But we may need to look at refactoring this test to find all potentially registered apps (maybe AST instead of a runtime test?).

I created #191059 to tackle that rewrite.