[Uptime] Unskip alerting functional tests #72963

justinkambic · 2020-07-22T20:33:16Z

Summary

Fixes #65948.

~~WIP,~~ fixing skipped tests in the Uptime alerting test suite.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
This was checked for keyboard-only and screenreader accessibility
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

justinkambic · 2020-07-23T18:34:27Z

One successful run of functional tests on the flaky test runner: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/651/

justinkambic · 2020-07-23T21:49:36Z

Second successful flaky test run: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/657/

elasticmachine · 2020-07-23T21:50:18Z

Pinging @elastic/uptime (Team:uptime)

justinkambic · 2020-07-26T13:28:36Z

@elasticmachine merge upstream

justinkambic · 2020-07-27T13:27:34Z

A third 100x flaky test run: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/664/

There were failures, but not these tests.

shahzad31 · 2020-07-27T17:16:59Z

x-pack/test/functional/services/uptime/navigation.ts

@@ -17,7 +17,7 @@ export function UptimeNavigationProvider({ getService, getPageObjects }: FtrProv
      if (await testSubjects.exists('uptimeSettingsToOverviewLink', { timeout: 0 })) {
        await testSubjects.click('uptimeSettingsToOverviewLink');
        await testSubjects.existOrFail('uptimeOverviewPage', { timeout: 2000 });
-      } else if (!(await testSubjects.exists('uptimeOverviewPage', { timeout: 0 }))) {


I think this isn't related to the alert test flakiness, it's related to the EuiLink not being able to click.

I am pretty sure when I was testing this, skipping the contents of this block were causing the test to fail. I can push a commit that reverts this change to see if it keeps succeeding.

@shahzad31 I am going to need to wait a bit to test this because Jenkins maintenance is happening right now.

spalger

I don't see any issues, but I'm deferring to @shahzad31 for a review of the actual fix

elasticmachine · 2020-07-27T18:39:10Z

Pinging @elastic/apm-ui (Team:apm)

justinkambic · 2020-07-28T15:40:51Z

@shahzad31 looking at the results the most recent flaky test run, vs the one I ran yesterday, I'd say that re-introducing the code in that else if condition contributed to the flakiness. If it's an optimization to save time I'm fine with refactoring it but my present goal is to get this test running again as it's a blocker.

We could refactor that code in a follow-up and re-test it but right now I'd like to move on from this issue.

shahzad31 · 2020-07-28T15:44:32Z

@shahzad31 looking at the results the most recent flaky test run, vs the one I ran yesterday, I'd say that re-introducing the code in that else if condition contributed to the flakiness. If it's an optimization to save time I'm fine with refactoring it but my present goal is to get this test running again as it's a blocker.

We could refactor that code in a follow-up and re-test it but right now I'd like to move on from this issue.

@justinkambic sure makes sense. The only part i am concerned about is that, does this also solves the flakiness where we can't click links in alert popover ? i might be wrong, but i thought alert flakiness was bcoz of that.

justinkambic · 2020-07-28T16:15:47Z

The only part i am concerned about is that, does this also solves the flakiness where we can't click links in alert popover ? i might be wrong, but i thought alert flakiness was bcoz of that.

@shahzad31 that's something I was concerned over as well but I am not seeing that problem recur at all. I'm wondering if there was some EUI issue that has been subsequently solved. I will run locally and see if I can make that happen again by running the test many times and reply back here.

shahzad31

LGTM, i am also not able to reproduce that unclickable eui link state.

justinkambic · 2020-07-28T16:21:35Z

I am also not able to reproduce that unclickable eui link state.

I'm going to try to smoke test it now as a best effort to make sure we don't knowingly commit a buggy solution, but I am expecting the tests to pass.

justinkambic · 2020-07-29T19:06:57Z

This PR is blocked by #73527. Once a fix is merged, we'll be ready to merge this. And once this unskip is merged, bugs like the one fixed in #73528 will be less likely.

justinkambic · 2020-07-29T19:07:06Z

@elasticmachine merge upstream

justinkambic · 2020-07-30T15:10:18Z

@elasticmachine merge upstream

justinkambic · 2020-07-30T17:32:30Z

@elasticmachine merge upstream

kibanamachine · 2020-07-30T19:22:02Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 19c8781

Build metrics

✅ unchanged

History

💔 Build #65548 failed 470cc23
💔 Build #65339 failed 8f5b96b
💔 Build #64776 failed ff2f6a3
💚 Build #64752 succeeded ff2f6a3
💚 Build #64678 succeeded ff2f6a3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

justinkambic · 2020-07-31T14:38:53Z

I ran a final flaky test check and this passed. I think we are good to merge now. https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/718/

* Unskip monitor status alert test. * Trying to resolve flakiness. * Remove commented code. * Simplify test expect. * Revert conditional block change. * Remove line in question. Co-authored-by: Elastic Machine <[email protected]>

* Unskip monitor status alert test. * Trying to resolve flakiness. * Remove commented code. * Simplify test expect. * Revert conditional block change. * Remove line in question. Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

justinkambic · 2020-07-31T18:46:45Z

Backported to:

7.x/7.10.0 fefbbd5
[Uptime] Unskip alerting functional tests #72963
7.9/7.9.0 9b4f26e
[7.9] [Uptime] Unskip alerting functional tests (#72963) #73955

kibanamachine · 2020-08-04T14:48:45Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

kibanamachine · 2020-08-05T14:58:44Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

kibanamachine · 2020-08-06T15:08:44Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

* Unskip monitor status alert test. * Trying to resolve flakiness. * Remove commented code. * Simplify test expect. * Revert conditional block change. * Remove line in question. Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

justinkambic added bug Fixes for quality problems that affect the customer experience test_xpack_functional v8.0.0 Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability v7.10.0 v7.9.0 labels Jul 22, 2020

justinkambic self-assigned this Jul 22, 2020

justinkambic force-pushed the uptime_unskip-alerting-tests branch from c515c09 to ba78cdf Compare July 23, 2020 18:13

justinkambic marked this pull request as ready for review July 23, 2020 21:50

justinkambic added release_note:skip Skip the PR/issue when compiling release notes blocker labels Jul 23, 2020

justinkambic requested review from spalger and shahzad31 July 26, 2020 15:32

shahzad31 reviewed Jul 27, 2020

View reviewed changes

spalger approved these changes Jul 27, 2020

View reviewed changes

justinkambic requested review from a team as code owners July 27, 2020 18:39

justinkambic requested a review from a team July 27, 2020 18:39

botelastic bot added the Team:APM All issues that need APM UI Team support label Jul 27, 2020

Unskip monitor status alert test.

dac4ab6

justinkambic added 2 commits July 28, 2020 08:42

Merge branch 'master' into uptime_unskip-alerting-tests

5b5fc08

Remove line in question.

ff2f6a3

shahzad31 approved these changes Jul 28, 2020

View reviewed changes

justinkambic mentioned this pull request Jul 28, 2020

[Uptime] Alert flyout is broken #73527

Closed

Merge branch 'master' into uptime_unskip-alerting-tests

8f5b96b

Merge branch 'master' into uptime_unskip-alerting-tests

470cc23

Merge branch 'master' into uptime_unskip-alerting-tests

19c8781

justinkambic merged commit df3e209 into elastic:master Jul 31, 2020

This was referenced Jul 31, 2020

[7.x] [Uptime] Unskip alerting functional tests (#72963) #73954

Merged

[7.9] [Uptime] Unskip alerting functional tests (#72963) #73955

Merged

justinkambic deleted the uptime_unskip-alerting-tests branch July 31, 2020 15:03

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Aug 4, 2020

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Aug 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Uptime] Unskip alerting functional tests #72963

[Uptime] Unskip alerting functional tests #72963

justinkambic commented Jul 22, 2020 •

edited

Loading

justinkambic commented Jul 23, 2020

justinkambic commented Jul 23, 2020

elasticmachine commented Jul 23, 2020

justinkambic commented Jul 26, 2020

justinkambic commented Jul 27, 2020

shahzad31 Jul 27, 2020

justinkambic Jul 27, 2020

justinkambic Jul 27, 2020

spalger left a comment

elasticmachine commented Jul 27, 2020

justinkambic commented Jul 28, 2020 •

edited

Loading

shahzad31 commented Jul 28, 2020

justinkambic commented Jul 28, 2020

shahzad31 left a comment

justinkambic commented Jul 28, 2020

justinkambic commented Jul 29, 2020

justinkambic commented Jul 29, 2020

justinkambic commented Jul 30, 2020

justinkambic commented Jul 30, 2020

kibanamachine commented Jul 30, 2020

justinkambic commented Jul 31, 2020

justinkambic commented Jul 31, 2020 •

edited

Loading

kibanamachine commented Aug 4, 2020

kibanamachine commented Aug 5, 2020

kibanamachine commented Aug 6, 2020

[Uptime] Unskip alerting functional tests #72963

[Uptime] Unskip alerting functional tests #72963

Conversation

justinkambic commented Jul 22, 2020 • edited Loading

Summary

Checklist

For maintainers

justinkambic commented Jul 23, 2020

justinkambic commented Jul 23, 2020

elasticmachine commented Jul 23, 2020

justinkambic commented Jul 26, 2020

justinkambic commented Jul 27, 2020

shahzad31 Jul 27, 2020

Choose a reason for hiding this comment

justinkambic Jul 27, 2020

Choose a reason for hiding this comment

justinkambic Jul 27, 2020

Choose a reason for hiding this comment

spalger left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 27, 2020

justinkambic commented Jul 28, 2020 • edited Loading

shahzad31 commented Jul 28, 2020

justinkambic commented Jul 28, 2020

shahzad31 left a comment

Choose a reason for hiding this comment

justinkambic commented Jul 28, 2020

justinkambic commented Jul 29, 2020

justinkambic commented Jul 29, 2020

justinkambic commented Jul 30, 2020

justinkambic commented Jul 30, 2020

kibanamachine commented Jul 30, 2020

💚 Build Succeeded

Build metrics

History

justinkambic commented Jul 31, 2020

justinkambic commented Jul 31, 2020 • edited Loading

kibanamachine commented Aug 4, 2020

kibanamachine commented Aug 5, 2020

kibanamachine commented Aug 6, 2020

justinkambic commented Jul 22, 2020 •

edited

Loading

justinkambic commented Jul 28, 2020 •

edited

Loading

justinkambic commented Jul 31, 2020 •

edited

Loading