Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make soft-deletes mandatory #50775

Closed
14 tasks done
dnhatn opened this issue Jan 9, 2020 · 1 comment · Fixed by #51697
Closed
14 tasks done

Make soft-deletes mandatory #50775

dnhatn opened this issue Jan 9, 2020 · 1 comment · Fixed by #51697
Assignees
@dnhatn
Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. Meta

Comments

@dnhatn
Copy link
Member

dnhatn commented Jan 9, 2020 •
edited
Loading

Peer recoveries become faster and use less storage (i.e., no more extra translog) with soft-deletes. Soft-deletes has been enabled by default since 7.0. We should make it mandatory in 8.0, so we can simplify the logic in the engine, translog, recoveries, and other components. This meta issue tracks works to be done on this story.

More tests for corner cases

Cleanup:
@dnhatn dnhatn added Meta :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. labels Jan 9, 2020
@dnhatn dnhatn self-assigned this Jan 9, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Engine)

This was referenced Jan 16, 2020
Merged
Merged
Merged
dnhatn added a commit that referenced this issue Jan 20, 2020
@dnhatn
Account soft-deletes in FrozenEngine (#51192)
c8e9f57 
Currently, we do not exclude soft-deleted documents when opening index 
reader in the FrozenEngine.

Relates #50775
dnhatn added a commit that referenced this issue Jan 20, 2020
@dnhatn
Clean soft-deletes setting in ccr tests (#51113)
698b1e9 
We no longer need to explicitly enable soft-deletes in CCR tests.

Relates #50775
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Jan 20, 2020
@dnhatn
Account soft-deletes in FrozenEngine (elastic#51192)
eae84f8 
Currently, we do not exclude soft-deleted documents when opening index
reader in the FrozenEngine.

Relates elastic#50775
dnhatn added a commit that referenced this issue Jan 20, 2020
@dnhatn
Use Lucene index in peer recovery and resync (#51189)
6e2f7b4 
We can use Lucene index exclusively in peer recoveries and
primary-replica resync in 8.0.

Relates #50775
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this issue Jan 23, 2020
@dnhatn @SivagurunathanV
Account soft-deletes in FrozenEngine (elastic#51192)
d804ff0 
Currently, we do not exclude soft-deleted documents when opening index 
reader in the FrozenEngine.

Relates elastic#50775
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this issue Jan 23, 2020
@dnhatn @SivagurunathanV
Clean soft-deletes setting in ccr tests (elastic#51113)
8ca83e0 
We no longer need to explicitly enable soft-deletes in CCR tests.

Relates elastic#50775
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Jan 23, 2020
@dnhatn
Account soft-deletes in FrozenEngine
6d4b39b 
Currently, we do not exclude soft-deleted documents
 when opening index reader in the FrozenEngine.

Relates elastic#50775
@dnhatn dnhatn mentioned this issue Jan 23, 2020
Merged
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Jan 23, 2020
@dnhatn
Clean soft-deletes setting in ccr tests (elastic#51113)
b4c0439 
We no longer need to explicitly enable soft-deletes in CCR tests.

Relates elastic#50775
@dnhatn dnhatn mentioned this issue Jan 23, 2020
Merged
dnhatn added a commit that referenced this issue Jan 23, 2020
@dnhatn
Clean soft-deletes setting in ccr tests (#51113) (#51372)
072203c 
We no longer need to explicitly enable soft-deletes in CCR tests.

Relates #50775
Backport of #51113
dnhatn added a commit that referenced this issue Jan 23, 2020
@dnhatn
Account soft-deletes in FrozenEngine (#51362)
a388cf7 
Currently, we do not exclude soft-deleted documents
when opening index reader in the FrozenEngine.

Relates #50775
Backport of #51192
@dnhatn dnhatn mentioned this issue Jan 24, 2020
Merged
dnhatn added a commit that referenced this issue Jan 24, 2020
@dnhatn
Remove translog retention policy (#51417)
b034d1e 
We no longer need to retain the extra translog for peer recovery as we 
have switched using Lucene index exclusively in 8.0. This change removes
the translog retention policy.

Relates #50775
@dnhatn dnhatn mentioned this issue Jan 29, 2020
Merged
dnhatn added a commit that referenced this issue Jan 29, 2020
@dnhatn
Deprecate translog retention settings (#51588)
2aa650c 
This change deprecates the translog retention settings as they are 
effectively ignored since 7.4.

Relates #50775
Relates #45473
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Jan 29, 2020
@dnhatn
Deprecate translog retention settings (elastic#51588)
014d79e 
This change deprecates the translog retention settings as they are
effectively ignored since 7.4.

Relates elastic#50775
Relates elastic#45473
@dnhatn dnhatn mentioned this issue Jan 29, 2020
Merged
dnhatn added a commit that referenced this issue Jan 30, 2020
@dnhatn
Deprecate translog retention settings (#51588) (#51638)
f0fad5b 
This change deprecates the translog retention settings as they are
effectively ignored since 7.4.

Relates #50775
Relates #45473
@dnhatn dnhatn mentioned this issue Jan 30, 2020
Merged
dnhatn added a commit that referenced this issue Jan 31, 2020
@dnhatn
Remove translog retention settings (#51697)
6e0fbbd 
The translog retention settings index.translog.retention.size and 
index.translog.retention.age were effectively ignored in 7.4, 
deprecated in 7.7, and now removed in 8.0 in favor of soft-deletes.

Closes #50775
@mfussenegger mfussenegger mentioned this issue Mar 24, 2020
Closed
37 tasks
This was referenced Apr 1, 2020
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. Meta
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove translog retention settings dnhatn/elasticsearch
2 participants
@dnhatn @elasticmachine