[BUG] [Remote Store] Getting old sequence number after restoring data from remote segment store

[sachinpkale]

Describe the bug

• Restoring data from remtoe store for an index sets sequence number back to last refresh number. Even though primary term changes and (primary term, sequence number) combination is still unique, this is a regression on the current behavior of seq number.
• With remote translog, this bug is mitigated as we replay the operations from remote translog from last refresh to max sequence number in remote translog.

To Reproduce
Steps to reproduce the behavior:

1. Setup 3 node cluster.
2. Create index with remote segment store enabled (do not enable remote translog)
3. Index should have 1 shard and 0 replica.
4. Index 20 documents. Sequence number of last operation should be 19, with primary term 1.
5. Explicit Refresh using API
6. Index 20 more documents. Sequence number of last operation should be 39, with primary term 1.
7. Explicit Refresh using API
8. Bring down the node with primary shard.
9. This should turn the index red.
10. Close the index and restore from remote store.
11. Index would turn GREEN and number of docs would show as 40.
12. Index 1 more document, the index response will have sequence number as 20, with primary term 2.

Expected behavior

1. The sequence number in the response of last index operation should be 40, with primary term 2.

[ashking94]

lets say if the do the following -
index 20 docs, refresh -> primary term 1, seq no 19
index 20 docs, refresh -> primary term 1, seq no 39
index 20 docs, refresh -> primary term 1, seq no 59
stop node where the index shard is located.
Now restore. Index a doc, will this lead to seq no 19? or will it be 39?

[sachinpkale]

lets say if the do the following - index 20 docs, refresh -> primary term 1, seq no 19 index 20 docs, refresh -> primary term 1, seq no 39 index 20 docs, refresh -> primary term 1, seq no 59 stop node where the index shard is located. Now restore. Index a doc, will this lead to seq no 19? or will it be 39?

It would be 40.

[sachinpkale]

Fixed in #5970