[EDR Workflows][E2E] Increase the timeout of agent check in #168438
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…into fix-e2e-agent-check-in-timeout
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]
History
To update your PR or re-run it, just comment with: |
szwarckonrad
added
release_note:skip
|
Pinging @elastic/security-defend-workflows (Team:Defend Workflows) |
ashokaditya
approved these changes
Oct 11, 2023
dasansol92
reviewed
Oct 11, 2023
| @@ -17,6 +17,6 @@ export const createEndpointHost = ( | |||
| { | |||
| agentPolicyId, | |||
| }, | |||
| { timeout: timeout ?? 600000 } | |||
| { timeout: timeout ?? 900000 } // 15 minutes, since setup can take 10 minutes and more. Task will time out if is not resolved within this time. | |||
There was a problem hiding this comment.
Is there any other way to speed up this instead of increasing the timeout?
There was a problem hiding this comment.
I don't think so. If cypress task that set ups and enrols agent doesn't resolve within this timeout the task and test fails. We have noticed, that at some occurrences this 10 minutes is not enough to download, install, enroll and wait for agent to show up in fleet and test fails making it flaky not because of the test itself but this setup task. We could investigate a way to make this task faster or break it into several tasks but that should be a spike, this PR aims at stabilizing our test runs now.
dasansol92
approved these changes
Oct 11, 2023
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 11, 2023
…168438) This pull request extends the agent fleet check timeout from 2 minutes to 4 minutes. We've identified a number of unreliable tests that fail during the `beforeAll` stage while executing the `createEndpointHost` task. The following logs appear before the timeout: ``` info Enrolling Elastic Agent with Fleet | Installing service....... DONE | Starting service... DONE | Enrolling Elastic Agent with Fleet..........Successfully enrolled the Elastic Agent. | Elastic Agent has been successfully installed. | info Waiting for Agent to check-in with Fleet ``` The error message we encounter is `> Timed out waiting for host [test-host-4981] to appear in Fleet.` It appears that all the preceding steps are successful, and only the final one fails due to either the agent not checking in with the fleet for 2 minutes or the agent being unhealthy for two minutes. Since I haven't been able to replicate this behavior locally, and there isn't a way to inspect what's happening on the agent, I believe the best course of action at this point is to extend the timeout and monitor the results. Reports of this error: closes elastic#168427 closes elastic#168394 closes elastic#168393 closes elastic#168390 closes elastic#168363 closes elastic#168362 closes elastic#168361 closes elastic#168360 closes elastic#168359 Affected CI runs: https://buildkite.com/elastic/kibana-on-merge/builds/36483 https://buildkite.com/elastic/kibana-on-merge/builds/36497 https://buildkite.com/elastic/kibana-on-merge/builds/36501 https://buildkite.com/elastic/kibana-on-merge/builds/36526 Another time out happens from time to time when previously set 10 minutes timeout on `createEndpointHost` task is not enough to set up the environment. Its portrayed below, timeout happens during agent setup ``` | default: Running: inline script | default: Reading package lists... | default: Building dependency tree... | default: Reading state information... | default: Suggested packages: | default: zip | default: The following NEW packages will be installed: | default: unzip | default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. | default: Need to get 174 kB of archives. | default: After this operation, 385 kB of additional disk space will be used. | default: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] | default: dpkg-preconfigure: unable to re-open stdin: No such file or directory | default: Fetched 174 kB in 1s (210 kB/s) | default: Selecting previously unselected package unzip. | (Reading database ... 63961 files and directories currently installed.) | default: Preparing to unpack .../unzip_6.0-26ubuntu3.1_amd64.deb ... | default: Unpacking unzip (6.0-26ubuntu3.1) ... | default: Setting up unzip (6.0-26ubuntu3.1) ... | default: Processing triggers for man-db (2.10.2-1) ... | | CypressError: `cy.task('createEndpointHost')` timed out after waiting `600000ms`. ``` (cherry picked from commit 91cdbe2)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Oct 11, 2023
…168438) (#168614) # Backport This will backport the following commits from `main` to `8.11`: - [[EDR Workflows][E2E] Increase the timeout of agent check in (#168438)](#168438) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Konrad Szwarc","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-10-11T14:26:45Z","message":"[EDR Workflows][E2E] Increase the timeout of agent check in (#168438)\n\nThis pull request extends the agent fleet check timeout from 2 minutes\r\nto 4 minutes. We've identified a number of unreliable tests that fail\r\nduring the `beforeAll` stage while executing the `createEndpointHost`\r\ntask. The following logs appear before the timeout:\r\n\r\n```\r\ninfo Enrolling Elastic Agent with Fleet\r\n | Installing service....... DONE\r\n | Starting service... DONE\r\n | Enrolling Elastic Agent with Fleet..........Successfully enrolled the Elastic Agent.\r\n | Elastic Agent has been successfully installed.\r\n | info Waiting for Agent to check-in with Fleet\r\n```\r\n\r\nThe error message we encounter is `> Timed out waiting for host\r\n[test-host-4981] to appear in Fleet.`\r\n\r\nIt appears that all the preceding steps are successful, and only the\r\nfinal one fails due to either the agent not checking in with the fleet\r\nfor 2 minutes or the agent being unhealthy for two minutes. Since I\r\nhaven't been able to replicate this behavior locally, and there isn't a\r\nway to inspect what's happening on the agent, I believe the best course\r\nof action at this point is to extend the timeout and monitor the\r\nresults.\r\n\r\nReports of this error:\r\ncloses https://github.com/elastic/kibana/issues/168427\r\ncloses https://github.com/elastic/kibana/issues/168394\r\ncloses https://github.com/elastic/kibana/issues/168393\r\ncloses https://github.com/elastic/kibana/issues/168390\r\ncloses https://github.com/elastic/kibana/issues/168363\r\ncloses https://github.com/elastic/kibana/issues/168362\r\ncloses https://github.com/elastic/kibana/issues/168361\r\ncloses https://github.com/elastic/kibana/issues/168360\r\ncloses https://github.com/elastic/kibana/issues/168359\r\n\r\nAffected CI runs:\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36483\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36497\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36501\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36526\r\n\r\nAnother time out happens from time to time when previously set 10\r\nminutes timeout on `createEndpointHost` task is not enough to set up the\r\nenvironment. Its portrayed below, timeout happens during agent setup\r\n```\r\n | default: Running: inline script\r\n | default: Reading package lists...\r\n | default: Building dependency tree...\r\n | default: Reading state information...\r\n | default: Suggested packages:\r\n | default: zip\r\n | default: The following NEW packages will be installed:\r\n | default: unzip\r\n | default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\r\n | default: Need to get 174 kB of archives.\r\n | default: After this operation, 385 kB of additional disk space will be used.\r\n | default: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB]\r\n | default: dpkg-preconfigure: unable to re-open stdin: No such file or directory\r\n | default: Fetched 174 kB in 1s (210 kB/s)\r\n | default: Selecting previously unselected package unzip.\r\n | (Reading database ... 63961 files and directories currently installed.)\r\n | default: Preparing to unpack .../unzip_6.0-26ubuntu3.1_amd64.deb ...\r\n | default: Unpacking unzip (6.0-26ubuntu3.1) ...\r\n | default: Setting up unzip (6.0-26ubuntu3.1) ...\r\n | default: Processing triggers for man-db (2.10.2-1) ...\r\n | \r\n | CypressError: `cy.task('createEndpointHost')` timed out after waiting `600000ms`.\r\n```","sha":"91cdbe2d354100683b5d8670de88e0b2cf665ba9","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Defend Workflows","v8.11.0","v8.12.0"],"number":168438,"url":"https://github.com/elastic/kibana/pull/168438","mergeCommit":{"message":"[EDR Workflows][E2E] Increase the timeout of agent check in (#168438)\n\nThis pull request extends the agent fleet check timeout from 2 minutes\r\nto 4 minutes. We've identified a number of unreliable tests that fail\r\nduring the `beforeAll` stage while executing the `createEndpointHost`\r\ntask. The following logs appear before the timeout:\r\n\r\n```\r\ninfo Enrolling Elastic Agent with Fleet\r\n | Installing service....... DONE\r\n | Starting service... DONE\r\n | Enrolling Elastic Agent with Fleet..........Successfully enrolled the Elastic Agent.\r\n | Elastic Agent has been successfully installed.\r\n | info Waiting for Agent to check-in with Fleet\r\n```\r\n\r\nThe error message we encounter is `> Timed out waiting for host\r\n[test-host-4981] to appear in Fleet.`\r\n\r\nIt appears that all the preceding steps are successful, and only the\r\nfinal one fails due to either the agent not checking in with the fleet\r\nfor 2 minutes or the agent being unhealthy for two minutes. Since I\r\nhaven't been able to replicate this behavior locally, and there isn't a\r\nway to inspect what's happening on the agent, I believe the best course\r\nof action at this point is to extend the timeout and monitor the\r\nresults.\r\n\r\nReports of this error:\r\ncloses https://github.com/elastic/kibana/issues/168427\r\ncloses https://github.com/elastic/kibana/issues/168394\r\ncloses https://github.com/elastic/kibana/issues/168393\r\ncloses https://github.com/elastic/kibana/issues/168390\r\ncloses https://github.com/elastic/kibana/issues/168363\r\ncloses https://github.com/elastic/kibana/issues/168362\r\ncloses https://github.com/elastic/kibana/issues/168361\r\ncloses https://github.com/elastic/kibana/issues/168360\r\ncloses https://github.com/elastic/kibana/issues/168359\r\n\r\nAffected CI runs:\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36483\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36497\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36501\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36526\r\n\r\nAnother time out happens from time to time when previously set 10\r\nminutes timeout on `createEndpointHost` task is not enough to set up the\r\nenvironment. Its portrayed below, timeout happens during agent setup\r\n```\r\n | default: Running: inline script\r\n | default: Reading package lists...\r\n | default: Building dependency tree...\r\n | default: Reading state information...\r\n | default: Suggested packages:\r\n | default: zip\r\n | default: The following NEW packages will be installed:\r\n | default: unzip\r\n | default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\r\n | default: Need to get 174 kB of archives.\r\n | default: After this operation, 385 kB of additional disk space will be used.\r\n | default: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB]\r\n | default: dpkg-preconfigure: unable to re-open stdin: No such file or directory\r\n | default: Fetched 174 kB in 1s (210 kB/s)\r\n | default: Selecting previously unselected package unzip.\r\n | (Reading database ... 63961 files and directories currently installed.)\r\n | default: Preparing to unpack .../unzip_6.0-26ubuntu3.1_amd64.deb ...\r\n | default: Unpacking unzip (6.0-26ubuntu3.1) ...\r\n | default: Setting up unzip (6.0-26ubuntu3.1) ...\r\n | default: Processing triggers for man-db (2.10.2-1) ...\r\n | \r\n | CypressError: `cy.task('createEndpointHost')` timed out after waiting `600000ms`.\r\n```","sha":"91cdbe2d354100683b5d8670de88e0b2cf665ba9"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/168438","number":168438,"mergeCommit":{"message":"[EDR Workflows][E2E] Increase the timeout of agent check in (#168438)\n\nThis pull request extends the agent fleet check timeout from 2 minutes\r\nto 4 minutes. We've identified a number of unreliable tests that fail\r\nduring the `beforeAll` stage while executing the `createEndpointHost`\r\ntask. The following logs appear before the timeout:\r\n\r\n```\r\ninfo Enrolling Elastic Agent with Fleet\r\n | Installing service....... DONE\r\n | Starting service... DONE\r\n | Enrolling Elastic Agent with Fleet..........Successfully enrolled the Elastic Agent.\r\n | Elastic Agent has been successfully installed.\r\n | info Waiting for Agent to check-in with Fleet\r\n```\r\n\r\nThe error message we encounter is `> Timed out waiting for host\r\n[test-host-4981] to appear in Fleet.`\r\n\r\nIt appears that all the preceding steps are successful, and only the\r\nfinal one fails due to either the agent not checking in with the fleet\r\nfor 2 minutes or the agent being unhealthy for two minutes. Since I\r\nhaven't been able to replicate this behavior locally, and there isn't a\r\nway to inspect what's happening on the agent, I believe the best course\r\nof action at this point is to extend the timeout and monitor the\r\nresults.\r\n\r\nReports of this error:\r\ncloses https://github.com/elastic/kibana/issues/168427\r\ncloses https://github.com/elastic/kibana/issues/168394\r\ncloses https://github.com/elastic/kibana/issues/168393\r\ncloses https://github.com/elastic/kibana/issues/168390\r\ncloses https://github.com/elastic/kibana/issues/168363\r\ncloses https://github.com/elastic/kibana/issues/168362\r\ncloses https://github.com/elastic/kibana/issues/168361\r\ncloses https://github.com/elastic/kibana/issues/168360\r\ncloses https://github.com/elastic/kibana/issues/168359\r\n\r\nAffected CI runs:\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36483\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36497\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36501\r\nhttps://buildkite.com/elastic/kibana-on-merge/builds/36526\r\n\r\nAnother time out happens from time to time when previously set 10\r\nminutes timeout on `createEndpointHost` task is not enough to set up the\r\nenvironment. Its portrayed below, timeout happens during agent setup\r\n```\r\n | default: Running: inline script\r\n | default: Reading package lists...\r\n | default: Building dependency tree...\r\n | default: Reading state information...\r\n | default: Suggested packages:\r\n | default: zip\r\n | default: The following NEW packages will be installed:\r\n | default: unzip\r\n | default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\r\n | default: Need to get 174 kB of archives.\r\n | default: After this operation, 385 kB of additional disk space will be used.\r\n | default: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB]\r\n | default: dpkg-preconfigure: unable to re-open stdin: No such file or directory\r\n | default: Fetched 174 kB in 1s (210 kB/s)\r\n | default: Selecting previously unselected package unzip.\r\n | (Reading database ... 63961 files and directories currently installed.)\r\n | default: Preparing to unpack .../unzip_6.0-26ubuntu3.1_amd64.deb ...\r\n | default: Unpacking unzip (6.0-26ubuntu3.1) ...\r\n | default: Setting up unzip (6.0-26ubuntu3.1) ...\r\n | default: Processing triggers for man-db (2.10.2-1) ...\r\n | \r\n | CypressError: `cy.task('createEndpointHost')` timed out after waiting `600000ms`.\r\n```","sha":"91cdbe2d354100683b5d8670de88e0b2cf665ba9"}}]}] BACKPORT--> Co-authored-by: Konrad Szwarc <[email protected]>
dej611
pushed a commit
to dej611/kibana
that referenced
this pull request
Oct 17, 2023
…168438) This pull request extends the agent fleet check timeout from 2 minutes to 4 minutes. We've identified a number of unreliable tests that fail during the `beforeAll` stage while executing the `createEndpointHost` task. The following logs appear before the timeout: ``` info Enrolling Elastic Agent with Fleet | Installing service....... DONE | Starting service... DONE | Enrolling Elastic Agent with Fleet..........Successfully enrolled the Elastic Agent. | Elastic Agent has been successfully installed. | info Waiting for Agent to check-in with Fleet ``` The error message we encounter is `> Timed out waiting for host [test-host-4981] to appear in Fleet.` It appears that all the preceding steps are successful, and only the final one fails due to either the agent not checking in with the fleet for 2 minutes or the agent being unhealthy for two minutes. Since I haven't been able to replicate this behavior locally, and there isn't a way to inspect what's happening on the agent, I believe the best course of action at this point is to extend the timeout and monitor the results. Reports of this error: closes elastic#168427 closes elastic#168394 closes elastic#168393 closes elastic#168390 closes elastic#168363 closes elastic#168362 closes elastic#168361 closes elastic#168360 closes elastic#168359 Affected CI runs: https://buildkite.com/elastic/kibana-on-merge/builds/36483 https://buildkite.com/elastic/kibana-on-merge/builds/36497 https://buildkite.com/elastic/kibana-on-merge/builds/36501 https://buildkite.com/elastic/kibana-on-merge/builds/36526 Another time out happens from time to time when previously set 10 minutes timeout on `createEndpointHost` task is not enough to set up the environment. Its portrayed below, timeout happens during agent setup ``` | default: Running: inline script | default: Reading package lists... | default: Building dependency tree... | default: Reading state information... | default: Suggested packages: | default: zip | default: The following NEW packages will be installed: | default: unzip | default: 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. | default: Need to get 174 kB of archives. | default: After this operation, 385 kB of additional disk space will be used. | default: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] | default: dpkg-preconfigure: unable to re-open stdin: No such file or directory | default: Fetched 174 kB in 1s (210 kB/s) | default: Selecting previously unselected package unzip. | (Reading database ... 63961 files and directories currently installed.) | default: Preparing to unpack .../unzip_6.0-26ubuntu3.1_amd64.deb ... | default: Unpacking unzip (6.0-26ubuntu3.1) ... | default: Setting up unzip (6.0-26ubuntu3.1) ... | default: Processing triggers for man-db (2.10.2-1) ... | | CypressError: `cy.task('createEndpointHost')` timed out after waiting `600000ms`. ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request extends the agent fleet check timeout from 2 minutes to 4 minutes. We've identified a number of unreliable tests that fail during the
beforeAllstage while executing thecreateEndpointHosttask. The following logs appear before the timeout:The error message we encounter is
> Timed out waiting for host [test-host-4981] to appear in Fleet.It appears that all the preceding steps are successful, and only the final one fails due to either the agent not checking in with the fleet for 2 minutes or the agent being unhealthy for two minutes. Since I haven't been able to replicate this behavior locally, and there isn't a way to inspect what's happening on the agent, I believe the best course of action at this point is to extend the timeout and monitor the results.
Reports of this error:
closes #168427
closes #168394
closes #168393
closes #168390
closes #168363
closes #168362
closes #168361
closes #168360
closes #168359
Affected CI runs:
https://buildkite.com/elastic/kibana-on-merge/builds/36483
https://buildkite.com/elastic/kibana-on-merge/builds/36497
https://buildkite.com/elastic/kibana-on-merge/builds/36501
https://buildkite.com/elastic/kibana-on-merge/builds/36526
Another time out happens from time to time when previously set 10 minutes timeout on
createEndpointHosttask is not enough to set up the environment. Its portrayed below, timeout happens during agent setup