s/segment: fixed tracking offsets of empty segments

[mmaslankaprv]

When a segment is rolled due to the segment.ms property its base offset is set to the committed offset of last segment plus one. In a segment constructor all segment offset tracker offsets were set to the base offset passed as the constructor argument. This way an empty segment had exactly the same set of offsets as the segment containing single batch containing one record and base offset equal to the segment base offset.

Incorrect offset accounting in empty segments lead to the situation in which eviction driven log::truncate_prefix was called with an incorrect offset, leading to the situation in which a log wasn't truncated at the segment boundary. This in the end resulted in the disconnection between the offset translator truncation point and first log segment start offset.

For example:

(we represent segment as [base_offset,end_offset])

In the case of log with segments:

[0,10][11,15]

After segment ms a new segment is created like this:

[0,10][11,15][16,16]

When eviction point is established the last offset is checked in this case if all the segments are to be evicted it will be equal to 16. 16 is a last offset that is going to be evicted (last included in raft snapshot). Hence the log::truncate_prefix is called with offset 17.

If some batches are appended to the rolled segment then the log will contain f.e.

 [0,10][11,15][16,25]

Now if the log is prefix truncated at offset 17 it starts offset is updated to 17 but the underlying segment is kept.

[16,25]

Now when the reader start reading the log it will request to start from the log start offset which is 17 but it will have to skip over the batch starting at 16. If a batch at 16 has more than one record it will still be returned to the reader and it will have to translate its base offset to kafka offset space. This is impossible as the offset_translator was already truncated to 16. i.e. there is no information in the translator to correctly translate the offset.

A fix is setting the empty segment dirty, committed and stable offset to its base_offset minus one. This way all the semantics of log operation holds as the offset returned is equal to previous segment last offset which would be the case if there would be no empty segment at the head of the log.

Fixes: #11403

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.1.x
• v22.3.x
• v22.2.x

Release Notes

Bug Fixes

• fixes rare situation in which consumer may stuck due to incorrect truncation point

[andijcr]

lgtm, the only concern I have is if some code expects that committed_offset >= base_offset

[mmaslankaprv]

ci failures:

• CI Failure (Timeout: Leadership did not stablize) in MultiTopicAutomaticLeadershipBalancingTest.test_topic_aware_rebalance #11454
• CI Failure (timeout in kafka-topics.sh create topic) in RackAwarePlacementTest.test_replica_placement in DEBUG builds #11276

[vbotbuildovich]

/backport v23.1.x