Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/security_and_spaces/tests/alerting/alerts·ts - alerting api integration security and spaces enabled Alerting alerts superuser at space1 should unmute all instances when unmuting an alert

[kibanamachine]

A test failed on a tracked branch

Error: retry.try timeout: Error: Expected 0 tasks but received 1
    at retry.try (test/alerting_api_integration/common/lib/task_manager_utils.ts:42:15)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at onFailure (/dev/shm/workspace/kibana/test/common/services/retry/retry_for_success.ts:28:9)
    at retryForSuccess (/dev/shm/workspace/kibana/test/common/services/retry/retry_for_success.ts:68:13)

First failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-test-triage (failed-test)

[kibanamachine]

New failure: Jenkins Build

[spalger]

Test has failed 11 times across master and PRs in the last week

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[brianseeders]

Skipped on master 51d96e5

[pmuellr]

The skipped test is here:

kibana/x-pack/test/alerting_api_integration/security_and_spaces/tests/alerting/alerts.ts

Lines 765 to 774 in 51d96e5

	it.skip(`should unmute all instances when unmuting an alert`, async () => {
	const testStart = new Date();
	const reference = alertUtils.generateReference();
	const response = await alertUtils.createAlwaysFiringAction({
	reference,
	overwrites: {
	enabled: false,
	schedule: { interval: '1s' },
	},
	});

Here's a bit of the ci log from the test. Note the test before the one that reported a failure included a message from ES about using an invalidated API key, which given the log for the actual test failure, makes me think that was the cause.

ci log from test

[00:10:13]                 └-> shouldn't schedule actions when alert instance is muted
[00:10:13]                   └-> "before each" hook: global before each
[00:10:14]                   │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1578403513476867716] Authentication using apikey failed - api key has been invalidated
[00:10:14]                   │ proc [kibana]   log   [14:38:30.411] [error][task_manager] Task actions:test.rate-limit "3c9b0a00-315b-11ea-bae7-037313b790e0" failed: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } } :: {"path":"/_security/user/_has_privileges","query":{},"body":"{\"applications\":[{\"application\":\"kibana-.kibana\",\"resources\":[\"space:space1\"],\"privileges\":[\"version:8.0.0-SNAPSHOT\",\"login:\",\"saved_object:8.0.0-SNAPSHOT:action/get\"]}]}","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/_security/user/_has_privileges]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/_security/user/_has_privileges]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}},\"status\":401}","wwwAuthenticateDirective":"Bearer realm=\"security\", ApiKey, Basic realm=\"security\" charset=\"UTF-8\""}
[00:10:16]                   │ debg --- retry.try error: Expected 2 but received 0.
[00:10:17]                   │ debg --- retry.try failed again with the same message...
[00:10:17]                   │ debg --- retry.try failed again with the same message...
[00:10:18]                   │ debg --- retry.try error: Expected 2 but received 1.
[00:10:18]                   │ debg --- retry.try failed again with the same message...
...
[00:10:21]                   └- ✓ pass  (8.2s) "alerting api integration security and spaces enabled Alerting alerts superuser at space1 shouldn't schedule actions when alert instance is muted"
[00:10:21]                 └-> "after each" hook

[00:10:22]                 └-> should unmute all instances when unmuting an alert

[00:10:22]                   └-> "before each" hook: global before each

[00:10:27]                   │ debg --- retry.try error: Expected 1 but received 0.

[00:10:28]                   │ debg --- retry.try failed again with the same message...

...

[00:10:29]                   │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1578403513476867716] Authentication using apikey failed - api key has been invalidated

[00:10:29]                   │ proc [kibana]   log   [14:38:45.418] [error][task_manager] Task actions:test.index-record "6744d290-315b-11ea-bae7-037313b790e0" failed: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm="security"" & 1="ApiKey" & 2="Basic realm="security" charset="UTF-8"" } } } :: {"path":"/_security/user/_has_privileges","query":{},"body":"{"applications":[{"application":"kibana-.kibana","resources":["space:space1"],"privileges":["version:8.0.0-SNAPSHOT","login:","saved_object:8.0.0-SNAPSHOT:action/get"]}]}","statusCode":401,"response":"{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_security/user/_has_privileges]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_security/user/_has_privileges]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}","wwwAuthenticateDirective":"Bearer realm="security", ApiKey, Basic realm="security" charset="UTF-8""}

[00:10:29]                   │ debg --- retry.try failed again with the same message...

[00:10:30]                   │ debg --- retry.try failed again with the same message...

[00:11:31]                   │ debg --- retry.try error: Expected 0 tasks but received 1

[00:11:31]                   │ debg --- retry.try failed again with the same message...

...

[00:13:31]                   └- ✖ fail: "alerting api integration security and spaces enabled Alerting alerts superuser at space1 should unmute all instances when unmuting an alert"

[pmuellr]

Note that @FrankHassanabad has seen some of the missing authentication credentials for REST request log messages as well, in the SIEM tests - guessing it's the same symptom, whatever the cause happens to be.

[pmuellr]

I'm doing some unrelated stress testing ATM, and seeing the ES Authentication using apikey failed - api key has been invalidated message, but I think just as I'm deleting 100's of alerts I've been testing with.

I had been wondering if somehow the alert is getting deleted after an action is scheduled, and the alert deletion "cleans up" the API key, which then means the action which runs later sees it as invalidated. Should be able to do some focused debug logging in that area anyway.

[kibanamachine]

New failure: Jenkins Build

[brianseeders]

Skipped on master/7.x, alerting team said they will probably address this week

[kibanamachine]

New failure: Jenkins Build

[mikecote]

@brianseeders we may need to skip the test in 7.6 as well. I think that's where this comment (#54125 (comment)) came from.