Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs:Add support rule evaluation via query frontend docs #6200

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs:Add support rule evaluation via query frontend docs #6200

wants to merge 1 commit into from

Conversation

SungJin1212
Copy link
Contributor

What this PR does:
Add docs on configuring Ruler to evaluate rules via Query Frontend and the pros and cons of this feature.
Which issue(s) this PR fixes:
Fixes #6198

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@SungJin1212
Copy link
Contributor Author

@yeya24
Could you take a look?

@SungJin1212 SungJin1212 force-pushed the Add-support-rule-evalution-via-query-frontend-docs branch 5 times, most recently from d79a2cf to cfb6ed4 Compare September 9, 2024 11:53
friedrichg
friedrichg approved these changes Sep 9, 2024
View reviewed changes
Copy link
Member

@friedrichg friedrichg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

yeya24
yeya24 reviewed Sep 10, 2024
View reviewed changes
docs/guides/rule-evaluations-via-query-frontend.md Outdated
frontend_address: query-frontend.svc.cluster.local:9095
```

In addition, you can configure gRPC client (Ruler -> Query Frontend) config via args:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is unnecessary to list all configuration flags. This adds extra complexity for users who read this doc as I expect that most of the case default values should be good enough.

Is it possible to use a link to redirect to the frontend client configurations?

docs/guides/rule-evaluations-via-query-frontend.md Outdated
backoff_config:
min_period: 100ms
max_period: 10s
max_retries: 10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

docs/guides/rule-evaluations-via-query-frontend.md Outdated
## Pros and Cons

### Pros
The rule evaluation performance is improved as we can use Query Frontend features like the vertical query sharding.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In terms of performance, I think it is complex and it is not always true to say evaluation performance is improved.

By evaluating rules through Query Frontend, there are more hops in this path. Ruler -> Query Frontend -> Query Scheduler -> Querier -> Ingester/Store Gateway.

Query Frontend/Query Scheduler has a query queue and Querier pulls queries from the queue to execute. If there are not enough queriers, add rule queries to this path could cause query starvation problem and rules cannot get evaluated. I think this is something to call out in this doc.

docs/guides/rule-evaluations-via-query-frontend.md Outdated

### Pros
The rule evaluation performance is improved as we can use Query Frontend features like the vertical query sharding.
The ruler can use fewer resources as it doesn't need to initialize the querier anymore.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds vague to me. You mean in the code Ruler doesn't need to intialize the querier struct? I suggest rewording it to something like:

Ruler can use fewer resources as it doesn't need to run query engine to evaluate rules.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, what I mean is that. Let me fix it.

docs/guides/rule-evaluations-via-query-frontend.md Outdated

## How to enable

By default, the Ruler queries to the Ingesters for evaluating rules (alerting rules or recording rules). If you have set `-ruler.frontend-address` then the Ruler query to the Query Frontend for evaluation rules.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the Ruler queries the Ingesters.
the Ruler queries the Query Frontend.

Ruler queries both Ingesters and Store Gateway depending on the Rule time range.

@SungJin1212 SungJin1212 force-pushed the Add-support-rule-evalution-via-query-frontend-docs branch from cfb6ed4 to 438d722 Compare September 10, 2024 06:52
@SungJin1212 SungJin1212 force-pushed the Add-support-rule-evalution-via-query-frontend-docs branch 2 times, most recently from 428b05c to 6849497 Compare September 10, 2024 07:04
@SungJin1212
Copy link
Contributor Author

@yeya24
could you review it?

@SungJin1212 SungJin1212 force-pushed the Add-support-rule-evalution-via-query-frontend-docs branch 2 times, most recently from f4d0a80 to 298ef9e Compare September 16, 2024 10:37
@SungJin1212 SungJin1212 force-pushed the Add-support-rule-evalution-via-query-frontend-docs branch from 298ef9e to dfd861f Compare September 24, 2024 01:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@friedrichg friedrichg friedrichg approved these changes

@yeya24 yeya24 Awaiting requested review from yeya24

Assignees
No one assigned
Labels
size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add documentation for Ruler remote evaluation via Query Frontend
3 participants
@SungJin1212 @friedrichg @yeya24