Shard history retention leases

[jasontedor]

When a shard of a follower index is consuming shard history from its corresponding shard of its leader index, it could be that the history operations is no longer available on any of the leader shard copies. This can happen if some operations were soft deleted and subsequently merged away before the shard of the following index had a chance to replicate these operations. This has catastrophic consequences for the follower index though as now the only option for the follower index to recover is a full file-based recovery. In the context of cross-cluster replication, this can potentially be over a WAN with limited networking resources. During this file-based recovery, the follower index becomes unavailable, defeating the purpose of being an available copy of the leader index in another cluster.

One idea towards solving this problem is for the shard of a follower index to be able to leave a marker on the corresponding shard of its leader index to notate where in shard history the following shard is. This marker would prevent any operations with sequence number at least at that marker from being eligible to be merged away.

And thus was born the idea of shard history retention leases. Shard history retention leases are aimed at preventing shard history consumers from having to fallback to expensive file copy operations if shard history is not available from a certain point. These consumers include following indices in cross-cluster replication, and local shard recoveries. A future consumer will be the changes API.

Further, index lifecycle management requires coordinating with some of these consumers otherwise it could remove the source before all consumers have finished reading all operations. The notion of shard history retention leases that we are introducing here will also be used to address this problem.

Shard history retention leases are a property of the replication group managed under the authority of the primary. A shard history retention lease is a combination of an identifier, a retaining sequence number, a timestamp indicating when the lease was acquired or renewed, and a string indicating the source of the lease. Being leases they have a limited lifespan that will expire if not renewed. The idea of these leases is that all operations above the minimum of all retaining sequence numbers will be retained during merges (which would otherwise clear away operations that are soft deleted). These leases will be periodically persisted to Lucene a dedicated state file and restored during recovery, and broadcast to replicas under certain circumstances.

This issue is a meta-issue for tracking the progress of implementing shard history retention leases. We will proceed with implementing shard history retention leases along the following rough plan:

• introduce the basics of shard history retention leases @jasontedor Introduce shard history retention leases #37167
• introduce timestamps on and expiration of shard history retention leases @jasontedor Introduce retention lease expiration #37195
• persist retention leases in Lucene switched to persisting in a dedicated state file Introduce retention lease state file #39004
  • introduce persistence to and recovery from Lucene @jasontedor Introduce retention lease persistence #37375
  • soft deletes policy should always fetch the latest shard history retention leases @dnhatn Soft-deletes policy should always fetch latest leases #37940
  • copy retention leases when trimming unsafe commits @dnhatn Copy retention leases when trim unsafe commits #37995
• share shard history retention leases with replicas @jasontedor
  • introduce shard history retention lease syncing when retention leases are created @jasontedor Introduce retention lease syncing #37398
  • sync shard history retention leases when a retention lease expires @jasontedor Sync retention leases on expiration #37902
  • introduce periodic background syncing of shard history retention leases (to capture renewals) @jasontedor Introduce retention lease background sync #38262 Lift retention lease expiration to index shard #38380
• enable removal of retention leases @jasontedor Enable removal of retention leases #38751
• introduce dedicated exceptions for retention lease already exists and retention lease not found @jasontedor Add dedicated retention lease exceptions #38754
• introduce external APIs for retention lease management (add/renew/remove) @jasontedor Introduce retention lease actions #38756
• add retention lease infrastructure to index level replication test case infrastructure @dnhatn Add retention leases replication tests #38857
• introduce version on retention lease collections @jasontedor Introduce retention leases versioning #37951
• handoff shard history retention leases during recovery @jasontedor Recover retention leases during peer recovery #38435
• expose shard history retention leases in shard-level statistics @jasontedor Expose retention leases in shard stats #37991
• integrate shard history retention leases with cross-cluster replication @jasontedor
  • integrate shard history retention leases with recovery from remote @jasontedor Integrate retention leases to recovery from remote #38829
  • integrate shard history retention leases with index following @jasontedor Renew retention leases while following #39335
  • when the unfollow API is invoked the shard history retention leases should be removed @jasontedor Remove retention leases when unfollowing #39088
• switch retention leases to be persisted in a dedicated state file @jasontedor Introduce retention lease state file #39004
• integrate shard history retention leases with recovery @DaveCTurner
• integrate shard history retention leases with index lifecycle management @gwbrown @martijnvg Ensure ILM policies run safely on leader indices  #38140

[elasticmachine]

Pinging @elastic/es-distributed

[elasticmachine]

Pinging @elastic/es-core-features

[dnhatn]

Closing since the work to integrate shard history retention leases with recovery is tracked #41536.