-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Trim unreferenced translog when the safe commit advanced #32967
Conversation
Since elastic#28140 when the global checkpoint is advanced, we try to move the safe commit forward, and clean old index commits if possible. However, we forget to trim unreferenced translog. This change makes sure that we prune both translog and index commits when the safe commit advanced. Relates elastic#28140 Closes elastic#32089
Pinging @elastic/es-distributed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I checked for other callers of indexWriter.deleteUnusedFiles()
, and there is one more, namely releaseIndexCommit
. Should we also do the trimming there?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I checked for other callers of indexWriter.deleteUnusedFiles(), and there is one more, namely releaseIndexCommit. Should we also do the trimming there?
+1
LGTM 2
Since #28140 when the global checkpoint is advanced, we try to move the safe commit forward, and clean up old index commits if possible. However, we forget to trim unreferenced translog. This change makes sure that we prune both old translog and index commits when the safe commit advanced. Relates #28140 Closes #32089
Since #28140 when the global checkpoint is advanced, we try to move the safe commit forward, and clean up old index commits if possible. However, we forget to trim unreferenced translog. This change makes sure that we prune both old translog and index commits when the safe commit advanced. Relates #28140 Closes #32089
* master: Generalize remote license checker (#32971) Trim translog when safe commit advanced (#32967) Fix an inaccuracy in the dynamic templates documentation. (#32890) Logging: Use settings when building daemon threads (#32751) All Translog inner closes should happen after tragedy exception is set (#32674) HLREST: AwaitsFix ML Test Pass DiscoveryNode to initiateChannel (#32958) Add mzn and dz to unsupported locales (#32957) Use settings from the context in BootstrapChecks (#32908) Update docs for node specifications (#30468) HLRC: Forbid all Elasticsearch logging infra (#32784) Only configure publishing if it's applied externally (#32351) Fixes libs:dissect when in eclipse Protect ScriptedMetricIT test cases against failures on 0-doc shards (#32959) (#32968) [Kerberos] Add documentation for Kerberos realm (#32662) Watcher: Properly find next valid date in cron expressions (#32734) Fix some small issues in the getting started docs (#30346) Set forbidden APIs target compatibility to compiler java version (#32935) Move connection listener to ConnectionManager (#32956)
Since #28140 when the global checkpoint is advanced, we try to move the safe commit forward, and clean up old index commits if possible. However, we forget to trim unreferenced translog. This change makes sure that we prune both old translog and index commits when the safe commit advanced. Relates #28140 Closes #32089
…e-types * elastic/master: (89 commits) Fix assertion in AbstractSimpleTransportTestCase (elastic#32991) [DOC] Splits role mapping APIs into separate pages (elastic#32797) HLRC: ML Close Job (elastic#32943) Generalize remote license checker (elastic#32971) Trim translog when safe commit advanced (elastic#32967) Fix an inaccuracy in the dynamic templates documentation. (elastic#32890) Logging: Use settings when building daemon threads (elastic#32751) All Translog inner closes should happen after tragedy exception is set (elastic#32674) HLREST: AwaitsFix ML Test Pass DiscoveryNode to initiateChannel (elastic#32958) Add mzn and dz to unsupported locales (elastic#32957) Use settings from the context in BootstrapChecks (elastic#32908) Update docs for node specifications (elastic#30468) HLRC: Forbid all Elasticsearch logging infra (elastic#32784) Only configure publishing if it's applied externally (elastic#32351) Fixes libs:dissect when in eclipse Protect ScriptedMetricIT test cases against failures on 0-doc shards (elastic#32959) (elastic#32968) [Kerberos] Add documentation for Kerberos realm (elastic#32662) Watcher: Properly find next valid date in cron expressions (elastic#32734) Fix some small issues in the getting started docs (elastic#30346) ...
Since #28140 when the global checkpoint is advanced, we try to move the safe commit forward, and clean up old index commits if possible. However, we forget to trim unreferenced translog. This change makes sure that we prune both old translog and index commits when the safe commit advanced. Relates #28140 Closes #32089
* commit '9088d811ce9cff922e6ef1befbeb0f1e0c27016a': (23 commits) Generalize remote license checker (elastic#32971) Trim translog when safe commit advanced (elastic#32967) Fix an inaccuracy in the dynamic templates documentation. (elastic#32890) HLREST: AwaitsFix ML Test Make Geo Context Mapping Parsing More Strict (elastic#32862) Add mzn and dz to unsupported locales (elastic#32957) Use settings from the context in BootstrapChecks (elastic#32908) Update docs for node specifications (elastic#30468) HLRC: Forbid all Elasticsearch logging infra (elastic#32784) Fix use of deprecated apis Only configure publishing if it's applied externally (elastic#32351) Protect ScriptedMetricIT test cases against failures on 0-doc shards (elastic#32959) (elastic#32968) Scripted metric aggregations: add deprecation warning and system (elastic#32944) Watcher: Properly find next valid date in cron expressions (elastic#32734) Fix some small issues in the getting started docs (elastic#30346) Set forbidden APIs target compatibility to compiler java version (elastic#32935) [TEST] Add "ne" as an unsupported SimpleKdc locale (elastic#32700) MINOR: Remove `IndexTemplateFilter` (elastic#32841) (elastic#32974) INGEST: Create Index Before Pipeline Execute (elastic#32786) (elastic#32975) NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS (elastic#32764) (elastic#32976) ...
Since #28140 when the global checkpoint is advanced, we try to move the
safe commit forward, and clean up old index commits if possible. However,
we forget to trim unreferenced translog.
This change makes sure that we prune both old translog and index commits
when the safe commit advanced.
Relates #28140
Closes #32089
/cc @bleskes