CI Failure (Count of blobs for topic did not decrease sizes) in TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test

[dotnwat]

https://buildkite.com/redpanda/redpanda/builds/32042#0188fd08-a9cd-42c0-8a4d-271ceee94d3f

Module: rptest.tests.topic_delete_test
Class:  TopicDeleteCloudStorageTest
Method: topic_delete_cloud_storage_test
Arguments:
{
  "cloud_storage_type": 2,
  "disable_delete": false
}

====================================================================================================
test_id:    rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=False.cloud_storage_type=CloudStorageType.ABS
status:     FAIL
run time:   3 minutes 11.554 seconds


    AssertionError('Count of blobs for topic did not decrease sizes=[1, 1, 1, 1]')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/utils/mode_checks.py", line 63, in f
    return func(*args, **kwargs)
  File "/root/tests/rptest/services/cluster.py", line 79, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/topic_delete_test.py", line 770, in topic_delete_cloud_storage_test
    self._validate_topic_deletion(self.topic, cloud_storage_type)
  File "/root/tests/rptest/tests/topic_delete_test.py", line 648, in _validate_topic_deletion
    assert len(sizes) > 1 and (
AssertionError: Count of blobs for topic did not decrease sizes=[1, 1, 1, 1]

[dotnwat]

https://buildkite.com/redpanda/vtools/builds/8260#0188fe7a-d00b-4ce1-ae29-a696c1e7a375

[StephanDollberg]

https://buildkite.com/redpanda/redpanda/builds/32106

[jcsp]

Closed #11784 as a dupe of this

https://buildkite.com/redpanda/redpanda/builds/32168#018904b5-dbc4-4cae-b10e-c4873e22a847

[VladLazar]

I looked at this a while back and it was a race between the scrubber and remote partition finalise. Currently, the test uses a 5ms grace period for the purge. That should be increased if this is still failing.

[mmaslankaprv]

Happened again here: #12504

[rystsov]

According to pandatriage the issue is stale, please reopen it only if pandatriage mark it as should reopen otherwise create a new issue.

| failure id | issue id | freq | total | last occ.  | test                                                >
| ---------- | -------- | ---- | ----- | ---------- | ---------------------------------------------------->
| 359        | 11736    | 166  | 5     | 2023-06-27 | TopicDeleteCloudStorageTest.topic_delete_cloud_stora>
| 516        | 11665    | 245  | 2     | 2023-06-26 | TopicDeleteStressTest.stress_test                   >
| 473        | 9052     | 174  | 5     | 2023-06-15 | RandomNodeOperationsTest.test_node_operations       >
| 484        | 9052     | 208  | 6     | 2023-06-14 | RandomNodeOperationsTest.test_node_operations       >
| 478        | 9052     | 69   | 2     | 2023-06-12 | RandomNodeOperationsTest.test_node_operations