-
Notifications
You must be signed in to change notification settings - Fork 25k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CCR] Refactore auto follow coordinator #35895
[CCR] Refactore auto follow coordinator #35895
Conversation
* AutofollowCoordinator now fetches cluster states per remote cluster. * AutofollowCoordinator makes use of the `wait_for_metadata_version` feature in cluster state API. * Removed the poll_interval setting, because it is no longer needed. Waiting is done via cluster state api for changes in remote clusters and auto follow pattern changes in the local cluster are immediately processed. * Improved auto follow stats to keep track of last time the remote clusters were checked for changes per remote cluster. Relates to elastic#33007
Pinging @elastic/es-distributed |
@@ -33,6 +33,9 @@ | |||
private static final ParseField RECENT_AUTO_FOLLOW_ERRORS = new ParseField("recent_auto_follow_errors"); | |||
private static final ParseField LEADER_INDEX = new ParseField("leader_index"); | |||
private static final ParseField AUTO_FOLLOW_EXCEPTION = new ParseField("auto_follow_exception"); | |||
private static final ParseField TRACKING_REMOTE_CLUSTERS = new ParseField("tracking_remote_clusters"); | |||
private static final ParseField CLUSTER_NAME = new ParseField("cluster_name"); | |||
private static final ParseField TIME_SINCE_LAST_AUTO_FOLLOW_STARTED_MILLIS = new ParseField("time_since_last_auto_follow_started_millis"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe time_since_last_remote_cluster_state_api_call
is a better name here.
run the gradle build tests 2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Heya @martijnvg I think this PR can be broken down into smaller PRs. I would suggest the following changes:
- remove the poll_interval setting, hard coded the poll_interval to the current default (this will show up a separate change in the change log, that's a good thing)
- add the use of the wait_for_metdata_version parameter on the cluster state API, replacing the hard coded poll_interval
- add the last requested time to the stats API
Closing this PR. The changes in this PR will be added in several other PRs. |
…te cluster and replaced poll interval setting with a hardcoded poll interval. The hard coded interval will be removed in a follow up change to make use of cluster state API's wait_for_metatdata_version. Originates from elastic#35895 Relates to elastic#33007
…ster (#36031) and replaced poll interval setting with a hardcoded poll interval. The hard coded interval will be removed in a follow up change to make use of cluster state API's wait_for_metatdata_version. Before the auto following was bootstrapped from thread pool scheduler, but now auto followers for new remote clusters are bootstrapped when a new cluster state is published. Originates from #35895 Relates to #33007
…ster (#36031) and replaced poll interval setting with a hardcoded poll interval. The hard coded interval will be removed in a follow up change to make use of cluster state API's wait_for_metatdata_version. Before the auto following was bootstrapped from thread pool scheduler, but now auto followers for new remote clusters are bootstrapped when a new cluster state is published. Originates from #35895 Relates to #33007
Changed AutofollowCoordinator makes use of the wait_for_metadata_version feature in cluster state API and removed hard coded poll interval. Originates from elastic#35895 Relates to elastic#33007
For each remote cluster the auto follow coordinator, starts an auto follower that checks the remote cluster state and determines whether an index needs to be auto followed. The time since last auto follow is reported per remote cluster and gives insight whether the auto follow process is alive. Relates to elastic#33007 Originates from elastic#35895
) For each remote cluster the auto follow coordinator, starts an auto follower that checks the remote cluster state and determines whether an index needs to be auto followed. The time since last auto follow is reported per remote cluster and gives insight whether the auto follow process is alive. Relates to #33007 Originates from #35895
) For each remote cluster the auto follow coordinator, starts an auto follower that checks the remote cluster state and determines whether an index needs to be auto followed. The time since last auto follow is reported per remote cluster and gives insight whether the auto follow process is alive. Relates to #33007 Originates from #35895
wait_for_metadata_version
feature in cluster state API.
Waiting is done via cluster state api for changes in remote clusters and
auto follow pattern changes in the local cluster are immediately processed.
were checked for changes per remote cluster.
Relates to #33007