Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Apply backpressure to the task poller whenever Elasticsearch requests respond with 503 errors #195134

Closed
3 tasks
mikecote opened this issue Oct 4, 2024 · 1 comment · Fixed by #196900
Closed
3 tasks

Apply backpressure to the task poller whenever Elasticsearch requests respond with 503 errors #195134

mikecote opened this issue Oct 4, 2024 · 1 comment · Fixed by #196900
Assignees
@ersin-erdal
Labels
Feature:Task Manager Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@mikecote
Copy link
Contributor

mikecote commented Oct 4, 2024 •
edited by ersin-erdal
Loading

When Elasticsearch is under stress we should receive HTTP 429 status codes. However, the proxy can also return 503 errors whenever Elasticsearch becomes unresponsive. Whenever that is the case, we should apply backpressure to the task poller by polling less frequently and claiming fewer tasks like we do for 429 status codes.

Definition of Done
@mikecote mikecote added Feature:Task Manager Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels Oct 4, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@mikecote mikecote changed the title Apply backpressure to the task poller whenever Elasticsearch requests respond with 503 errors Apply backpressure to the task poller whenever Elasticsearch requests respond with 501+ errors Oct 7, 2024
@ersin-erdal ersin-erdal self-assigned this Oct 18, 2024
@ersin-erdal ersin-erdal changed the title Apply backpressure to the task poller whenever Elasticsearch requests respond with 501+ errors Apply backpressure to the task poller whenever Elasticsearch requests respond with 503 errors Oct 18, 2024
@ersin-erdal ersin-erdal mentioned this issue Oct 18, 2024
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Oct 23, 2024
@ersin-erdal
Apply backpressure to the task poller whenever Elasticsearch requests…
7fe619a 
… respond with 503 errors (elastic#196900)

Resolves: elastic#195134

This PR adds 503 error check to the error filter of
`createManagedConfiguration` function, besides the 501 error .
So it applies backpressure to the task poller for 503 errors as well.

(cherry picked from commit 292a7d3)
kibanamachine added a commit that referenced this issue Oct 24, 2024
@kibanamachine @ersin-erdal
[8.x] Apply backpressure to the task poller whenever Elasticsearch re…
e2001ca 
…quests respond with 503 errors (#196900) (#197544)

# Backport

This will backport the following commits from `main` to `8.x`:
- [Apply backpressure to the task poller whenever Elasticsearch requests
respond with 503 errors
(#196900)](#196900)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ersin
Erdal","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-23T23:16:45Z","message":"Apply
backpressure to the task poller whenever Elasticsearch requests respond
with 503 errors (#196900)\n\nResolves: #195134\r\n\r\nThis PR adds 503
error check to the error filter of\r\n`createManagedConfiguration`
function, besides the 501 error .\r\nSo it applies backpressure to the
task poller for 503 errors as
well.","sha":"292a7d384e51ca9e76d09f817f583bd0b201d9e0","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","v9.0.0","backport:prev-minor"],"title":"Apply
backpressure to the task poller whenever Elasticsearch requests respond
with 503
errors","number":196900,"url":"https://github.com/elastic/kibana/pull/196900","mergeCommit":{"message":"Apply
backpressure to the task poller whenever Elasticsearch requests respond
with 503 errors (#196900)\n\nResolves: #195134\r\n\r\nThis PR adds 503
error check to the error filter of\r\n`createManagedConfiguration`
function, besides the 501 error .\r\nSo it applies backpressure to the
task poller for 503 errors as
well.","sha":"292a7d384e51ca9e76d09f817f583bd0b201d9e0"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196900","number":196900,"mergeCommit":{"message":"Apply
backpressure to the task poller whenever Elasticsearch requests respond
with 503 errors (#196900)\n\nResolves: #195134\r\n\r\nThis PR adds 503
error check to the error filter of\r\n`createManagedConfiguration`
function, besides the 501 error .\r\nSo it applies backpressure to the
task poller for 503 errors as
well.","sha":"292a7d384e51ca9e76d09f817f583bd0b201d9e0"}}]}] BACKPORT-->

Co-authored-by: Ersin Erdal <[email protected]>
ersin-erdal added a commit to ersin-erdal/kibana that referenced this issue Oct 24, 2024
@ersin-erdal
Apply backpressure to the task poller whenever Elasticsearch requests…
c8a8b4b 
… respond with 503 errors (elastic#196900)

Resolves: elastic#195134

This PR adds 503 error check to the error filter of
`createManagedConfiguration` function, besides the 501 error .
So it applies backpressure to the task poller for 503 errors as well.

(cherry picked from commit 292a7d3)
ersin-erdal added a commit that referenced this issue Oct 24, 2024
@ersin-erdal
[8.16] Apply backpressure to the task poller whenever Elasticsearch r…
6bde962 
…equests respond with 503 errors (#196900) (#197705)

# Backport

This will backport the following commits from `main` to `8.16`:
- [Apply backpressure to the task poller whenever Elasticsearch requests
respond with 503 errors
(#196900)](#196900)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ersin
Erdal","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-23T23:16:45Z","message":"Apply
backpressure to the task poller whenever Elasticsearch requests respond
with 503 errors (#196900)\n\nResolves: #195134\r\n\r\nThis PR adds 503
error check to the error filter of\r\n`createManagedConfiguration`
function, besides the 501 error .\r\nSo it applies backpressure to the
task poller for 503 errors as
well.","sha":"292a7d384e51ca9e76d09f817f583bd0b201d9e0","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","v9.0.0","backport:prev-minor","v8.16.0","v8.17.0"],"number":196900,"url":"https://github.com/elastic/kibana/pull/196900","mergeCommit":{"message":"Apply
backpressure to the task poller whenever Elasticsearch requests respond
with 503 errors (#196900)\n\nResolves: #195134\r\n\r\nThis PR adds 503
error check to the error filter of\r\n`createManagedConfiguration`
function, besides the 501 error .\r\nSo it applies backpressure to the
task poller for 503 errors as
well.","sha":"292a7d384e51ca9e76d09f817f583bd0b201d9e0"}},"sourceBranch":"main","suggestedTargetBranches":["8.16"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196900","number":196900,"mergeCommit":{"message":"Apply
backpressure to the task poller whenever Elasticsearch requests respond
with 503 errors (#196900)\n\nResolves: #195134\r\n\r\nThis PR adds 503
error check to the error filter of\r\n`createManagedConfiguration`
function, besides the 501 error .\r\nSo it applies backpressure to the
task poller for 503 errors as
well.","sha":"292a7d384e51ca9e76d09f817f583bd0b201d9e0"}},{"branch":"8.16","label":"v8.16.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","labelRegex":"^v8.17.0$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/197544","number":197544,"state":"MERGED","mergeCommit":{"sha":"e2001cadd96d112108ee60780bf3708d5fa48a14","message":"[8.x]
Apply backpressure to the task poller whenever Elasticsearch requests
respond with 503 errors (#196900) (#197544)\n\n# Backport\n\nThis will
backport the following commits from `main` to `8.x`:\n- [Apply
backpressure to the task poller whenever Elasticsearch requests\nrespond
with 503
errors\n(#196900)](https://github.com/elastic/kibana/pull/196900)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Ersin\nErdal\",\"email\":\"[email protected]\"},\"sourceCommit\":{\"committedDate\":\"2024-10-23T23:16:45Z\",\"message\":\"Apply\nbackpressure
to the task poller whenever Elasticsearch requests respond\nwith 503
errors (#196900)\\n\\nResolves: #195134\\r\\n\\r\\nThis PR adds
503\nerror check to the error filter
of\\r\\n`createManagedConfiguration`\nfunction, besides the 501 error
.\\r\\nSo it applies backpressure to the\ntask poller for 503 errors
as\nwell.\",\"sha\":\"292a7d384e51ca9e76d09f817f583bd0b201d9e0\",\"branchLabelMapping\":{\"^v9.0.0$\":\"main\",\"^v8.17.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"Team:ResponseOps\",\"v9.0.0\",\"backport:prev-minor\"],\"title\":\"Apply\nbackpressure
to the task poller whenever Elasticsearch requests respond\nwith
503\nerrors\",\"number\":196900,\"url\":\"https://github.com/elastic/kibana/pull/196900\",\"mergeCommit\":{\"message\":\"Apply\nbackpressure
to the task poller whenever Elasticsearch requests respond\nwith 503
errors (#196900)\\n\\nResolves: #195134\\r\\n\\r\\nThis PR adds
503\nerror check to the error filter
of\\r\\n`createManagedConfiguration`\nfunction, besides the 501 error
.\\r\\nSo it applies backpressure to the\ntask poller for 503 errors
as\nwell.\",\"sha\":\"292a7d384e51ca9e76d09f817f583bd0b201d9e0\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v9.0.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/196900\",\"number\":196900,\"mergeCommit\":{\"message\":\"Apply\nbackpressure
to the task poller whenever Elasticsearch requests respond\nwith 503
errors (#196900)\\n\\nResolves: #195134\\r\\n\\r\\nThis PR adds
503\nerror check to the error filter
of\\r\\n`createManagedConfiguration`\nfunction, besides the 501 error
.\\r\\nSo it applies backpressure to the\ntask poller for 503 errors
as\nwell.\",\"sha\":\"292a7d384e51ca9e76d09f817f583bd0b201d9e0\"}}]}]
BACKPORT-->\n\nCo-authored-by: Ersin Erdal
<[email protected]>"}}]}] BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ersin-erdal ersin-erdal

Labels
Feature:Task Manager Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
None yet
Milestone
No milestone
3 participants
@mikecote @elasticmachine @ersin-erdal