Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Snapshot Repository Scalability #74350

Closed
16 tasks done
original-brownbear opened this issue Jun 21, 2021 · 2 comments
Closed
16 tasks done

Improve Snapshot Repository Scalability #74350

original-brownbear opened this issue Jun 21, 2021 · 2 comments
Assignees
@original-brownbear
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement Meta Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

Comments

@original-brownbear
Copy link
Member

original-brownbear commented Jun 21, 2021 •
edited by jrodewig
Loading

Snapshot repositories do not scale well with large numbers of snapshots and indices in them. With the introduction of the cold and frozen tier compelling use cases for repositories containing tens of thousands of snapshots have emerged. This issue tracks the work required to enable larger snapshot repositories.
@original-brownbear original-brownbear added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Meta labels Jun 21, 2021
@original-brownbear original-brownbear self-assigned this Jun 21, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Jun 21, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@original-brownbear original-brownbear mentioned this issue Jun 21, 2021
Closed
@yuliacech yuliacech mentioned this issue Jul 5, 2021
Closed
@cjcenizal cjcenizal mentioned this issue Jul 15, 2021
Open
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Aug 9, 2021
@original-brownbear
Implement Numeric Offset Parameter in Get Snapshots API
4d37610 
Add numeric offset parameter to this API.

Relates elastic#74350
@original-brownbear original-brownbear mentioned this issue Aug 9, 2021
Merged
original-brownbear added a commit that referenced this issue Aug 9, 2021
@original-brownbear
Implement Numeric Offset Parameter in Get Snapshots API (#76233)
ffaa0f2 
Add numeric offset parameter to this API.

Relates #74350
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Aug 14, 2021
@original-brownbear
Implement Numeric Offset Parameter in Get Snapshots API (elastic#76233)
db29bec 
Add numeric offset parameter to this API.

Relates elastic#74350
original-brownbear added a commit that referenced this issue Aug 14, 2021
@original-brownbear
Implement Numeric Pagination in Get Snapshots API (#76532)
d356a4b 
* Return Total Result Count and Remaining Count in Get Snapshots Response (#76150)

Add total result count and remaining count to get snapshots response.

* Implement Numeric Offset Parameter in Get Snapshots API (#76233)

Add numeric offset parameter to this API.

Relates #74350
@yuliacech yuliacech mentioned this issue Aug 26, 2021
Merged
4 tasks
original-brownbear added a commit that referenced this issue Aug 30, 2021
@original-brownbear
Add Sort By Shard Count and Failed Shard Count to Get Snapshots API (#…
48f3784 
…77011)

It's in the title. As requested by the Kibana team, adding these two additional sort columns.

relates #74350
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Aug 30, 2021
@original-brownbear
Add Sort By Shard Count and Failed Shard Count to Get Snapshots API (e…
209fa7c 
…lastic#77011)

It's in the title. As requested by the Kibana team, adding these two additional sort columns.

relates elastic#74350
@original-brownbear original-brownbear mentioned this issue Aug 30, 2021
Merged
original-brownbear added a commit that referenced this issue Aug 30, 2021
@original-brownbear
Add Sort By Shard Count and Failed Shard Count to Get Snapshots API (#…
84d7831 
…77011) (#77018)

It's in the title. As requested by the Kibana team, adding these two additional sort columns.

relates #74350
This was referenced Sep 7, 2021
Merged
Merged
@dakrone dakrone mentioned this issue Sep 9, 2021
Closed
@original-brownbear original-brownbear mentioned this issue Sep 15, 2021
Closed
@Leaf-Lin Leaf-Lin mentioned this issue Sep 16, 2021
Closed
@original-brownbear original-brownbear mentioned this issue Sep 27, 2021
Merged
This was referenced Nov 15, 2021
Merged
Merged
jrodewig added a commit that referenced this issue Nov 16, 2021
@jrodewig
[DOCS] Remove soft limit for snapshot repositories (#80745)
a480603 
As part of #74350, snapshot repositories no longer have a 200 snapshot soft limit. This removes docs related to the limit.
elasticsearchmachine pushed a commit that referenced this issue Nov 16, 2021
@jrodewig
[DOCS] Remove soft limit for snapshot repositories (#80745) (#80769)
ff04cad 
As part of #74350, snapshot repositories no longer have a 200 snapshot soft limit. This removes docs related to the limit.
elasticsearchmachine pushed a commit that referenced this issue Nov 16, 2021
@jrodewig
[DOCS] Remove soft limit for snapshot repositories (#80745) (#80770)
41232f4 
As part of #74350, snapshot repositories no longer have a 200 snapshot soft limit. This removes docs related to the limit.
elasticsearchmachine pushed a commit that referenced this issue Nov 16, 2021
@jrodewig
[DOCS] Remove soft limit for snapshot repositories (#80745) (#80771)
3ee776b 
As part of #74350, snapshot repositories no longer have a 200 snapshot soft limit. This removes docs related to the limit.
# Conflicts:
#	docs/reference/slm/apis/slm-put.asciidoc
#	docs/reference/snapshot-restore/take-snapshot.asciidoc
elasticsearchmachine pushed a commit that referenced this issue Nov 16, 2021
@jrodewig
[DOCS] Remove soft limit for snapshot repositories (#80745) (#80772)
a23dce2 
As part of #74350, snapshot repositories no longer have a 200 snapshot soft limit. This removes docs related to the limit.
# Conflicts:
#	docs/reference/slm/apis/slm-put.asciidoc
#	docs/reference/snapshot-restore/take-snapshot.asciidoc
jrodewig added a commit to elastic/kibana that referenced this issue Nov 17, 2021
@jrodewig
Remove soft limit for snapshot repositories (#118634)
38e3c09 
With elastic/elasticsearch#74350, we no longer recommend a 200 snapshot soft limit for snapshot repositories or SLM policies. This removes a related sentence from the Snapshot Policy UI.

It also updates the docs link to point to the [create snapshot docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/snapshots-take-snapshot.html), which include guidance on snapshot retention, rather than the [put SLM policy API docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/slm-api-put-policy.html).

Relates to elastic/elasticsearch#80745
jrodewig added a commit to elastic/kibana that referenced this issue Nov 17, 2021
@jrodewig
Remove soft limit for snapshot repositories (#118634) (#118888)
30629fd 
With elastic/elasticsearch#74350, we no longer recommend a 200 snapshot soft limit for snapshot repositories or SLM policies. This removes a related sentence from the Snapshot Policy UI.

It also updates the docs link to point to the [create snapshot docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/snapshots-take-snapshot.html), which include guidance on snapshot retention, rather than the [put SLM policy API docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/slm-api-put-policy.html).

Relates to elastic/elasticsearch#80745
jrodewig added a commit to elastic/kibana that referenced this issue Nov 17, 2021
@jrodewig
Remove soft limit for snapshot repositories (#118634) (#118889)
103c451 
With elastic/elasticsearch#74350, we no longer recommend a 200 snapshot soft limit for snapshot repositories or SLM policies. This removes a related sentence from the Snapshot Policy UI.

It also updates the docs link to point to the [create snapshot docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/snapshots-take-snapshot.html), which include guidance on snapshot retention, rather than the [put SLM policy API docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/slm-api-put-policy.html).

Relates to elastic/elasticsearch#80745
@jrodewig
Copy link
Contributor

I'm going to close as it looks like all tasks are complete.merged. Please re-open if I missed anything.

TinLe pushed a commit to TinLe/kibana that referenced this issue Nov 20, 2021
@jrodewig @TinLe
Remove soft limit for snapshot repositories (elastic#118634)
6ba3d2d 
With elastic/elasticsearch#74350, we no longer recommend a 200 snapshot soft limit for snapshot repositories or SLM policies. This removes a related sentence from the Snapshot Policy UI.

It also updates the docs link to point to the [create snapshot docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/snapshots-take-snapshot.html), which include guidance on snapshot retention, rather than the [put SLM policy API docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/slm-api-put-policy.html).

Relates to elastic/elasticsearch#80745
dmlemeshko pushed a commit to elastic/kibana that referenced this issue Nov 29, 2021
@jrodewig @dmlemeshko
Remove soft limit for snapshot repositories (#118634)
08d952d 
With elastic/elasticsearch#74350, we no longer recommend a 200 snapshot soft limit for snapshot repositories or SLM policies. This removes a related sentence from the Snapshot Policy UI.

It also updates the docs link to point to the [create snapshot docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/snapshots-take-snapshot.html), which include guidance on snapshot retention, rather than the [put SLM policy API docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/slm-api-put-policy.html).

Relates to elastic/elasticsearch#80745
@Leaf-Lin Leaf-Lin mentioned this issue Jun 2, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement Meta Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@original-brownbear @elasticmachine @jrodewig