Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: X-Pack Cloud Security Posture Functional Tests.x-pack/test/cloud_security_posture_functional/pages/cis_integrations/cnvm/cis_integration_cnvm·ts - Cloud Security Posture Test adding Cloud Security Posture Integrations CNVM CNVM AWS Clicking on Launch CloudFormation on post intall modal should lead user to Cloud Formation page #186302

Open
kibanamachine opened this issue Jun 17, 2024 · 18 comments · Fixed by #187784
Open

Failing test: X-Pack Cloud Security Posture Functional Tests.x-pack/test/cloud_security_posture_functional/pages/cis_integrations/cnvm/cis_integration_cnvm·ts - Cloud Security Posture Test adding Cloud Security Posture Integrations CNVM CNVM AWS Clicking on Launch CloudFormation on post intall modal should lead user to Cloud Formation page #186302

kibanamachine opened this issue Jun 17, 2024 · 18 comments · Fixed by #187784
Assignees
@opauloh
Labels
automation failed-test A test failure on a tracked branch, potentially flaky-test Team:Cloud Security Cloud Security team related
Milestone
8.18

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Jun 17, 2024 •
edited
Loading

A test failed on a tracked branch

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="confirmCloudFormationModalConfirmButton"])
Wait timed out after 104037ms
    at /opt/buildkite-agent/builds/bk-agent-prod-gcp-1718634920926064467/elastic/kibana-on-merge/kibana/node_modules/selenium-webdriver/lib/webdriver.js:911:22
    at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  remoteStacktrace: ''
}

First failure: kibana-on-merge - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Jun 17, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Jun 17, 2024
@kibanamachine kibanamachine added the Team:Cloud Security Cloud Security team related label Jun 17, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Jun 17, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@opauloh
Copy link
Contributor

opauloh commented Jun 18, 2024

I was able to reproduce the issue by adding a second integration and setting the internet to slow (to simulate the machine clicking on the button as soon it’s available)

I think we need to review how we can disable the “Save and Continue” button while the loading to detect the integration name happens (as slow internet conditions can also happen for the users). Also, it would be good to review the useEffects logic that is currently occurring in parallel.

Screen.Recording.2024-06-18.at.12.27.28.PM.mov

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

mistic added a commit that referenced this issue Jun 19, 2024
@mistic
skip flaky suite (#186302)
fe07ada
@mistic
Copy link
Member

mistic commented Jun 19, 2024

Skipped.

main: fe07ada

@animehart animehart self-assigned this Jul 8, 2024
@animehart animehart mentioned this issue Jul 9, 2024
Merged
@animehart animehart linked a pull request Jul 9, 2024 that will close this issue
[Cloud Security]Fix for flaky FTRs caused by duplicate names #187784
Merged
animehart added a commit that referenced this issue Jul 9, 2024
@animehart
[Cloud Security]Fix for flaky FTRs caused by duplicate names (#187784)
45b0c25 
## Summary

This PR is to address the following flaky FTRs
#186302
#186315
#186387
#186389

The flakiness are being caused by having duplicate names for the
integration when adding them. We tried adding wait before clicking on
the Save button however that doesn't seem to work as it starts failing
in the flaky test runner

This PR gives the Integration a specific name (not relying on auto
naming)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Jul 9, 2024
@animehart
[Cloud Security]Fix for flaky FTRs caused by duplicate names (elastic…
64e2038 
…#187784)

## Summary

This PR is to address the following flaky FTRs
elastic#186302
elastic#186315
elastic#186387
elastic#186389

The flakiness are being caused by having duplicate names for the
integration when adding them. We tried adding wait before clicking on
the Save button however that doesn't seem to work as it starts failing
in the flaky test runner

This PR gives the Integration a specific name (not relying on auto
naming)

(cherry picked from commit 45b0c25)
kibanamachine added a commit that referenced this issue Jul 9, 2024
@kibanamachine @animehart
[8.15] [Cloud Security]Fix for flaky FTRs caused by duplicate names (#…
85619da 
…187784) (#187916)

# Backport

This will backport the following commits from `main` to `8.15`:
- [[Cloud Security]Fix for flaky FTRs caused by duplicate names
(#187784)](#187784)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Rickyanto
Ang","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-07-09T20:01:00Z","message":"[Cloud
Security]Fix for flaky FTRs caused by duplicate names (#187784)\n\n##
Summary\r\n\r\nThis PR is to address the following flaky
FTRs\r\nhttps://github.com//issues/186302\r\nhttps://github.com//issues/186315\r\nhttps://github.com//issues/186387\r\nhttps://github.com//issues/186389\r\n\r\nThe
flakiness are being caused by having duplicate names for
the\r\nintegration when adding them. We tried adding wait before
clicking on\r\nthe Save button however that doesn't seem to work as it
starts failing\r\nin the flaky test runner\r\n\r\nThis PR gives the
Integration a specific name (not relying on
auto\r\nnaming)","sha":"45b0c251913b6978195291a5d317972fe8c83ddb","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Cloud
Security","v8.15.0","v8.16.0"],"title":"[Cloud Security]Fix for flaky
FTRs caused by duplicate
names","number":187784,"url":"https://github.com/elastic/kibana/pull/187784","mergeCommit":{"message":"[Cloud
Security]Fix for flaky FTRs caused by duplicate names (#187784)\n\n##
Summary\r\n\r\nThis PR is to address the following flaky
FTRs\r\nhttps://github.com//issues/186302\r\nhttps://github.com//issues/186315\r\nhttps://github.com//issues/186387\r\nhttps://github.com//issues/186389\r\n\r\nThe
flakiness are being caused by having duplicate names for
the\r\nintegration when adding them. We tried adding wait before
clicking on\r\nthe Save button however that doesn't seem to work as it
starts failing\r\nin the flaky test runner\r\n\r\nThis PR gives the
Integration a specific name (not relying on
auto\r\nnaming)","sha":"45b0c251913b6978195291a5d317972fe8c83ddb"}},"sourceBranch":"main","suggestedTargetBranches":["8.15"],"targetPullRequestStates":[{"branch":"8.15","label":"v8.15.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/187784","number":187784,"mergeCommit":{"message":"[Cloud
Security]Fix for flaky FTRs caused by duplicate names (#187784)\n\n##
Summary\r\n\r\nThis PR is to address the following flaky
FTRs\r\nhttps://github.com//issues/186302\r\nhttps://github.com//issues/186315\r\nhttps://github.com//issues/186387\r\nhttps://github.com//issues/186389\r\n\r\nThe
flakiness are being caused by having duplicate names for
the\r\nintegration when adding them. We tried adding wait before
clicking on\r\nthe Save button however that doesn't seem to work as it
starts failing\r\nin the flaky test runner\r\n\r\nThis PR gives the
Integration a specific name (not relying on
auto\r\nnaming)","sha":"45b0c251913b6978195291a5d317972fe8c83ddb"}}]}]
BACKPORT-->

Co-authored-by: Rickyanto Ang <[email protected]>
@kibanamachine kibanamachine reopened this Jul 22, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kfirpeled
Copy link
Contributor

@animehart we agreed this can be more suitable to run on our sanity tests instead of on every PR
There we can also assert the cloud formation page is rendered as expected without the fear it will be flaky and block kibana's pipeline

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kfirpeled
Copy link
Contributor

Moved back to triage board, @elastic/kibana-cloud-security-posture feel free to pick it up on your next triage

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@maxcold
Copy link
Contributor

maxcold commented Aug 19, 2024

/skip

@maxcold maxcold mentioned this issue Aug 19, 2024
Merged
maxcold added a commit that referenced this issue Aug 20, 2024
@maxcold
[Cloud Security] skip flaky CNVM test (#190718)
6f392ee 
## Summary

Skipping flaky test

Part of:
- #186302
@maxcold
Copy link
Contributor

maxcold commented Aug 20, 2024

skipped in #190718

@acorretti acorretti added this to the 8.18 milestone Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@opauloh opauloh

Labels
automation failed-test A test failure on a tracked branch, potentially flaky-test Team:Cloud Security Cloud Security team related
Projects
None yet
Milestone
8.18
Development

Successfully merging a pull request may close this issue.

[Cloud Security]Fix for flaky FTRs caused by duplicate names animehart/kibana
8 participants
@acorretti @maxcold @mistic @animehart @elasticmachine @opauloh @kibanamachine @kfirpeled