[v22.3.x] CI Failure (slightly high HWMs in read replicas) in TestReadReplicaService.test_identical_hwms.partition_count=5

[andrwng]

https://ci-artifacts.dev.vectorized.cloud/redpanda/24358/01877d40-e001-482b-9ec4-804abaa03d94/vbuild/ducktape/results/2023-04-14--001/report.html

Module: rptest.tests.read_replica_e2e_test
Class:  TestReadReplicaService
Method: test_identical_hwms
Arguments:
{
  "partition_count": 5
}

test_id:    rptest.tests.read_replica_e2e_test.TestReadReplicaService.test_identical_hwms.partition_count=5
status:     FAIL
run time:   59.818 seconds


    TimeoutError('')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/read_replica_e2e_test.py", line 242, in test_identical_hwms
    wait_until(clusters_report_identical_hwms,
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

I'm not aware of any differences on dev that would fix this, so this may also exist on dev.

From test logs:

[INFO  - 2023-04-14 01:18:03,568 - read_replica_e2e_test - hwms_are_identical - lineno:71]: {0: 251, 1: 251, 2: 254, 3: 302, 4: 185} vs {0: 252, 1: 253, 2: 255, 3: 303, 4: 186}
[DEBUG - 2023-04-14 01:18:04,569 - rpk - _execute - lineno:637]: Executing command: ['/var/lib/buildkite-agent/builds/buildkite-amd64-xfs-builders-i-0b138675aa805a7f3-1/redpanda/redpanda/vbuild/redpanda_installs/ci/bin/rpk', 'topic', '--brokers', 'docker-rp-2:9092,docker-rp-3:9092,docker-rp-1:9092', 'describe', 'panda-topic', '-p']
[DEBUG - 2023-04-14 01:18:04,593 - rpk - _execute - lineno:650]: 
PARTITION  LEADER  EPOCH  REPLICAS  LOG-START-OFFSET  HIGH-WATERMARK
0          2       1      [1 2 3]   0                 251
1          1       2      [1 2 3]   0                 251
2          1       1      [1 2 3]   0                 254
3          3       1      [1 2 3]   0                 302
4          3       1      [1 2 3]   0                 185

[DEBUG - 2023-04-14 01:18:04,593 - rpk - _execute - lineno:637]: Executing command: ['/var/lib/buildkite-agent/builds/buildkite-amd64-xfs-builders-i-0b138675aa805a7f3-1/redpanda/redpanda/vbuild/redpanda_installs/ci/bin/rpk', 'topic', '--brokers', 'docker-rp-6:9092,docker-rp-7:9092,docker-rp-5:9092', 'describe', 'panda-topic', '-p']
[DEBUG - 2023-04-14 01:18:04,616 - rpk - _execute - lineno:650]: 
PARTITION  LEADER  EPOCH  REPLICAS  LOG-START-OFFSET  HIGH-WATERMARK
0          3       1      [3]       0                 252
1          1       1      [1]       0                 253
2          2       1      [2]       0                 255
3          1       1      [1]       0                 303
4          3       1      [3]       0                 186

[INFO  - 2023-04-14 01:18:04,616 - read_replica_e2e_test - hwms_are_identical - lineno:71]: {0: 251, 1: 251, 2: 254, 3: 302, 4: 185} vs {0: 252, 1: 253, 2: 255, 3: 303, 4: 186}
[ERROR - 2023-04-14 01:18:05,617 - cluster - wrapped - lineno:41]: Test failed, doing failure checks...
Traceback (most recent call last):
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/read_replica_e2e_test.py", line 242, in test_identical_hwms
    wait_until(clusters_report_identical_hwms,
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError