rptest: perform full log reads with KgoVerifierSeqConsumer #11759
Conversation
VladLazar
force-pushed
the
kgo-consumer-full-reads
branch
from
cbbc4ce
to
9b092ab
Compare
|
/ci-repeat 5 |
VladLazar
force-pushed
the
kgo-consumer-full-reads
branch
from
9b092ab
to
546d30b
Compare
VladLazar
force-pushed
the
kgo-consumer-full-reads
branch
6 times, most recently
from
090b8b7
to
3d1ea57
Compare
VladLazar
force-pushed
the
kgo-consumer-full-reads
branch
2 times, most recently
from
dd6cb5a
to
e0c6bd0
Compare
|
All failures apart from one in |
|
/ci-repeat 5 |
VladLazar
force-pushed
the
kgo-consumer-full-reads
branch
from
e0c6bd0
to
16c6486
Compare
|
/ci-repeat 5 |
VladLazar
force-pushed
the
kgo-consumer-full-reads
branch
from
16c6486
to
23450ec
Compare
|
/ci-repeat 5 |
VladLazar
force-pushed
the
kgo-consumer-full-reads
branch
from
23450ec
to
b03032f
Compare
|
/ci-repeat 5 |
|
Timing tests are stable now! Once #11846, this can be rebased and merged. |
|
/ci-repeat |
Previously, whenever a KgoVerifierSequentialConsumer and a KgoVerifierProducer ran in parallel (which a number of tests do), there was no guarantee that the consumer would read all partitions entirely. Whenever `wait` is called on the consumer, the `last_pass` command is triggered without taking into account if the producer has completed. This is problematic because it can hide stuck consumer issues. This patch makes use of the enhanced kgo-verifier status which now contains the high watermark for each partition. When creating a sequential consumer, a reference to the producer can be provided. If that was the case, when `wait` is called on the consumer we will wait until the producer has finished and the high watermarks match.
This commit updates the SI configuration usesd for the CloudStorageTimingStressTest(s) to upload the manifest at each oportunity. The change should make the test more robust.
VladLazar
force-pushed
the
kgo-consumer-full-reads
branch
from
b03032f
to
2893f9a
Compare
|
/ci-repeat |
2 similar comments
|
/ci-repeat |
|
/ci-repeat |
|
#11698 (https://buildkite.com/redpanda/redpanda/builds/32526#01892623-a6bf-4ba0-930a-d33a5617d84f) #11884 (https://buildkite.com/redpanda/redpanda/builds/32526#018925de-2328-4968-949c-780edc7e43e9) - new issue but not related to these changes. |
Based on #11846
Previously, whenever a KgoVerifierSequentialConsumer and a
KgoVerifierProducer ran in parallel (which a number of tests do), there
was no guarantee that the consumer would read all partitions entirely.
Whenever
waitis called on the consumer, thelast_passcommand istriggered without taking into account if the producer has completed.
This is problematic because it can hide stuck consumer issues. This
patch makes use of the enhanced kgo-verifier status which now contains
the high watermark for each partition. When creating a sequential
consumer, a reference to the producer can be provided. If that was the
case, when
waitis called on the consumer we will wait until theproducer has finished and the high watermarks match.
Backports Required
Release Notes