CI Failure (failed to hydrate chunk) in ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy

[VladLazar]

https://buildkite.com/redpanda/vtools/builds/7686#01883fee-00cd-4c52-9646-ad67032b974f

Module: rptest.tests.e2e_shadow_indexing_test
Class:  ShadowIndexingWhileBusyTest
Method: test_create_or_delete_topics_while_busy
Arguments:
{
  "cloud_storage_type": 1,
  "short_retention": true
}

test_id:    rptest.tests.e2e_shadow_indexing_test.ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy.short_retention=True.cloud_storage_type=CloudStorageType.S3
status:     FAIL
run time:   6 minutes 51.836 seconds


    <BadLogLines nodes=ip-172-31-15-0(1) example="ERROR 2023-05-21 21:48:50,234 [shard 0] cloud_storage - [fiber3813 01e0fdce/kafka/topic-aafbfeergf/33_51/336-377-20977637-1-v1.log.1] - segment_chunk_data_source.cc:84 - failed to hydrate chunk starting at 16782165, error: cloud_storage::download_exception (NotFound)">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 87, in wrapped
    redpanda.raise_on_bad_logs(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2294, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=ip-172-31-15-0(1) example="ERROR 2023-05-21 21:48:50,234 [shard 0] cloud_storage - [fiber3813 01e0fdce/kafka/topic-aafbfeergf/33_51/336-377-20977637-1-v1.log.1] - segment_chunk_data_source.cc:84 - failed to hydrate chunk starting at 16782165, error: cloud_storage::download_exception (NotFound)">

[NyaliaLui]

https://buildkite.com/redpanda/vtools/builds/7746#01884cc9-73fb-4853-9761-b684659daee9

[NyaliaLui]

This may be related to #10868

[andijcr]

https://buildkite.com/redpanda/vtools/builds/7916#0188856f-e4e9-4ee3-944e-95a01dd43a36

[abhijat]

should be fixed with #11093, although the last failure mentioned here is a day after the fix was merged, but has not been seen since.