CI Failure (_topic_remote_deleted timeout) in `TopicDeleteCloudStorageTest`.`topic_delete_installed_snapshots_test` #8496

jcsp · 2023-01-30T11:29:12Z

This is a relatively recently added test, first time we've seen it fail.

https://buildkite.com/redpanda/redpanda/builds/22044#01860016-6553-4a49-b948-ebdd36604013

Module: rptest.tests.topic_delete_test
Class:  TopicDeleteCloudStorageTest
Method: topic_delete_installed_snapshots_test

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/topic_delete_test.py", line 269, in topic_delete_installed_snapshots_test
    wait_until(lambda: self._topic_remote_deleted(self.topic),
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

The text was updated successfully, but these errors were encountered:

jcsp · 2023-01-30T13:28:56Z

A leadership transfer is leading to double upload of a segment, and only the final one that ends up in the manifest is deleted. This is not a bug per-se as the tiered storage design allows for some garbage to be left behind after leadership movement, it is an unnecessary rough edge in the context of an orchestrated leadership transfer.

While we could add some logic to detect and accept this case in the test, maybe it's better to go ahead and amend leadership transfer code to do a gentle shutdown of ntp_archiver during the transfer.

jcsp · 2023-01-31T14:22:30Z

Reproduced with same failure mode:
https://buildkite.com/redpanda/redpanda/builds/22127#0186065d-4335-40ba-a32f-0912423c1958

This substantially reduces the probability of leaving orphaned objects in the object store when partitions change leadership under load, e.g. during upgrades or leader balancing. This fixes a test failure that indirectly detects orphan objects by checking that topic deletion clears all objects. Fixes redpanda-data#8496

rystsov · 2023-02-03T01:01:40Z

https://buildkite.com/redpanda/redpanda/builds/22392#01861451-1886-4533-b27f-93adfbbf8566

graphcareful · 2023-02-08T22:54:42Z

https://buildkite.com/redpanda/redpanda/builds/22802#01863309-7a71-4aa0-b3d9-d7bdc64cb652

This substantially reduces the probability of leaving orphaned objects in the object store when partitions change leadership under load, e.g. during upgrades or leader balancing. This fixes a test failure that indirectly detects orphan objects by checking that topic deletion clears all objects. Fixes redpanda-data#8496

mmaslankaprv · 2023-02-20T08:03:24Z

Similar error in 3 tests:

mmaslankaprv · 2023-02-20T08:03:30Z

FAIL test: TopicDeleteCloudStorageTest.topic_delete_unavailable_test (2/34 runs)
failure at 2023-02-17T18:33:51.439Z: TimeoutError('')
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/23485#01866065-c283-4087-9858-85c95a82d98a
FAIL test: TopicDeleteCloudStorageTest.topic_delete_unavailable_test.cloud_storage_type=CloudStorageType.ABS (1/35 runs)
failure at 2023-02-18T07:33:23.813Z: TimeoutError('')
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/23528#01866339-eb65-494f-bdec-1bfba8586722
FAIL test: TopicDeleteCloudStorageTest.topic_delete_unavailable_test.cloud_storage_type=CloudStorageType.S3 (2/36 runs)
failure at 2023-02-19T07:17:24.906Z: TimeoutError('')
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/23538#01866841-6238-478f-8d2d-65f449dfb8ef

Scandiravian · 2023-02-20T17:41:52Z

Also just happened to me https://buildkite.com/redpanda/redpanda/builds/23591#01866fa0-8cbf-47c7-b8e2-735317cc8777

Module: rptest.tests.topic_delete_test
Class:  TopicDeleteCloudStorageTest
Method: topic_delete_unavailable_test
Arguments:
{
  "cloud_storage_type": 1
}

====================================================================================================
test_id:    rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_unavailable_test.cloud_storage_type=CloudStorageType.S3
status:     FAIL
run time:   3 minutes 52.267 seconds


    TimeoutError('')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/topic_delete_test.py", line 332, in topic_delete_unavailable_test
    wait_until(lambda: self._topic_remote_deleted(next_topic),
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

All these tests rely on leadership transfers being done gracefully to avoid orphan objects, which is not reliable in debug mode. This is a more complete followup to 4f9bc2d Fixes redpanda-data#8496

twmb · 2023-03-10T00:46:50Z

Just happened in tip,
https://buildkite.com/redpanda/redpanda/builds/24747#0186c8a3-7da3-4d43-a900-adfecb5e35ef

====================================================================================================
test_id:    rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_installed_snapshots_test
status:     FAIL
run time:   1 minute 57.472 seconds


    TimeoutError('')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/utils/mode_checks.py", line 63, in f
    return func(*args, **kwargs)
  File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/topic_delete_test.py", line 272, in topic_delete_installed_snapshots_test
    wait_until(lambda: self._topic_remote_deleted(self.topic),
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

dlex · 2023-03-10T07:38:58Z

on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/24726#0186c7b0-34fe-4357-9603-fb2b470f67ec

BenPope · 2023-03-15T11:10:33Z

These tests ran before the merge above:

BenPope · 2023-03-15T12:31:32Z

https://buildkite.com/redpanda/redpanda/builds/25101#0186e3f5-0e89-4e53-9d00-fe1d7228afee

BenPope · 2023-05-08T16:47:19Z

https://buildkite.com/redpanda/redpanda/builds/28719#0187f4f0-29ea-4212-bf8e-6721cfb61596

BenPope · 2023-05-08T16:47:59Z

https://buildkite.com/redpanda/redpanda/builds/28702#0187f179-5bb3-4775-9714-22d98efa4a38

This code is giong to get revised ahead of 23.2 with tombstones, which will generally make it more reliable but also involvev test changes, so let's defer stabilizing these until then. Related: redpanda-data#9629 Related: redpanda-data#8496

Underlying issues have been addressed: - Adjacent segment merging is disabled. - We now have a scrubber that finishes deletion eventually if anything interrupts the initial best-effort deletion - A bug was fixed where the `replaced` list of the manifest was ignored during deletion. - A test bug is fixed where it was incorrectly asserting that objects should still exist at the end of the 'unavailable' variant of the test. Fixes: redpanda-data#8496 Fixes: redpanda-data#9629

dotnwat · 2023-06-03T18:40:17Z

This has been closed and re-opened several times, and I think it was triggered again here. The latest fix was merged on May 24 and yesterday a ci-repeat 5 triggered it.

The PR in question was the client fetch throttling. However, the code that is timing out in this test appears to be object storage APi (self.cloud_storage_client.list_objects). Of course that doesn't prove that the throttling code isn't to blame, but presumably we'd have seen a different failure earlier in the test if this were the culprit.

andijcr · 2023-06-05T12:20:20Z

https://buildkite.com/redpanda/redpanda/builds/30568#01888a22-5476-43b6-8d0f-1f46708efc40
and similar failure in topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_unavailable_test
https://buildkite.com/redpanda/vtools/builds/7910#0188804a-c980-4ca8-9797-6afa79dd52f5

andijcr · 2023-06-07T14:49:09Z

https://buildkite.com/redpanda/redpanda/builds/30615#01888c85-dbbe-434b-840a-8c947aeda966

andijcr · 2023-06-09T13:48:43Z

https://buildkite.com/redpanda/vtools/builds/7990#01889a09-9f19-41f6-8d6d-76bc4566bcc9

michael-redpanda · 2023-06-12T19:54:11Z

https://buildkite.com/redpanda/vtools/builds/8050#0188a6e9-85d8-48ed-81ef-8307498c7772

michael-redpanda · 2023-06-12T19:55:20Z

https://buildkite.com/redpanda/vtools/builds/8053#0188a97e-103b-4222-a146-9e5360524691

ztlpn · 2023-06-16T20:44:14Z

https://buildkite.com/redpanda/redpanda/builds/31440#0188c3d2-b4c3-4850-8982-ccb6b8bb871d

travisdowns · 2023-06-19T22:35:50Z

FAIL test: TopicDeleteCloudStorageTest.topic_delete_unavailable_test.cloud_storage_type=CloudStorageType.ABS (1/29 runs)
failure at 2023-06-19T07:31:50.725Z: TimeoutError('')
on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/31582#0188d25c-cfaf-4822-a594-91fb3371ebc3

travisdowns · 2023-06-20T22:04:39Z

FAIL test: TopicDeleteCloudStorageTest.topic_delete_installed_snapshots_test (2/13 runs)
failure at 2023-06-20T17:16:32.585Z: TimeoutError('')
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/31659#0188d99c-fed9-40a6-b43c-00b508233319
failure at 2023-06-20T10:26:17.109Z: TimeoutError('')
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/31637#0188d81e-727e-453f-a9a9-0fba8d86cfa8

As soon as a topic is deleted, raft is shut down. This prevents ntp_archiver from updating archival_metadata_stm after uploading a segment, resulting in an orphan segment after deletion. Fixes redpanda-data#8496 Fixes redpanda-data#10655

StephanDollberg · 2023-06-26T10:09:55Z

Think this happened again: https://buildkite.com/redpanda/redpanda/builds/31920#0188f66a-2a94-46a5-a587-37c2c09f0e81 ?

This fixes commit 8355e4e quiesce_uploads would fail silently if you called it for a topic that didn't exist: that's fixed in the previous commit. This commit fixes the underlying issue, that we were passing a string instead of a list of strings, so quiesce_upload was trying to wait for each character in the string as if it was a topic name. Fixes redpanda-data#8496 Fixes redpanda-data#10655 Fixes redpanda-data#9629

piyushredpanda · 2023-06-27T04:16:54Z

@jcsp should we remove the ci-disabled-test label now?

NyaliaLui · 2023-10-11T14:16:12Z

Pandatriage reports to re-open this because of

      {
          "title": "TimeoutError('')",
          "id": 8939,
          "ts": 1689901360.004222,
          "type": "pr-merged",
          "build": "release",
          "arch": "amd64",
          "link": "https://buildkite.com/redpanda/redpanda/builds/33679"
      }

which occurred in late july and we have not seen new failures since. So it seems OK to remain closed.

piyushredpanda · 2023-10-11T14:38:46Z

Hmm, why is pandatriage reporting a july build for reopening?

jcsp added kind/bug Something isn't working ci-failure area/cloud-storage Shadow indexing subsystem labels Jan 30, 2023

jcsp self-assigned this Jan 30, 2023

jcsp added the sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages label Jan 30, 2023

rystsov mentioned this issue Feb 3, 2023

Introduces new compacted verifier unaffected by the "ssh" errors #8413

Merged

6 tasks

jcsp closed this as completed in #8560 Feb 16, 2023

mmaslankaprv reopened this Feb 20, 2023

jcsp mentioned this issue Mar 1, 2023

tests: skip all cloud storage deletion in debug mode #9187

Merged

6 tasks

jcsp closed this as completed in #9187 Mar 1, 2023

twmb reopened this Mar 10, 2023

mmaslankaprv mentioned this issue Mar 14, 2023

Evenly distributing topic partitions when adding node to cluster #9344

Merged

7 tasks

jcsp mentioned this issue May 22, 2023

tests: reinstate cloud storage deletion tests #10925

Merged

7 tasks

jcsp closed this as completed in #10925 May 24, 2023

dotnwat reopened this Jun 3, 2023

dotnwat mentioned this issue Jun 3, 2023

Node-wide throughput exemptions for clients #10755

Merged

7 tasks

abhijat mentioned this issue Jun 12, 2023

ducktape: testing for chunk read path #11291

Merged

7 tasks

dlex mentioned this issue Jun 12, 2023

Avoid duplicate group names in the test in snc trhoughput config test #11339

Merged

7 tasks

jcsp mentioned this issue Jun 23, 2023

tests/topic_delete_test: wait for all uploads before deleting topics #11630

Merged

7 tasks

jcsp closed this as completed in #11630 Jun 23, 2023

jcsp mentioned this issue Jun 26, 2023

tests: fix call to quiesce_uploads in topic deletion test #11684

Merged

7 tasks

jcsp removed the ci-disabled-test label Jun 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI Failure (_topic_remote_deleted timeout) in `TopicDeleteCloudStorageTest`.`topic_delete_installed_snapshots_test` #8496

CI Failure (_topic_remote_deleted timeout) in `TopicDeleteCloudStorageTest`.`topic_delete_installed_snapshots_test` #8496

jcsp commented Jan 30, 2023 •

edited

Loading

jcsp commented Jan 30, 2023

jcsp commented Jan 31, 2023

rystsov commented Feb 3, 2023

graphcareful commented Feb 8, 2023

mmaslankaprv commented Feb 20, 2023

mmaslankaprv commented Feb 20, 2023

Scandiravian commented Feb 20, 2023

twmb commented Mar 10, 2023

dlex commented Mar 10, 2023

BenPope commented Mar 15, 2023 •

edited

Loading

BenPope commented Mar 15, 2023

BenPope commented May 8, 2023

BenPope commented May 8, 2023

dotnwat commented Jun 3, 2023

andijcr commented Jun 5, 2023 •

edited

Loading

andijcr commented Jun 7, 2023

andijcr commented Jun 9, 2023

michael-redpanda commented Jun 12, 2023

michael-redpanda commented Jun 12, 2023

ztlpn commented Jun 16, 2023

travisdowns commented Jun 19, 2023

travisdowns commented Jun 20, 2023

StephanDollberg commented Jun 26, 2023

piyushredpanda commented Jun 27, 2023

NyaliaLui commented Oct 11, 2023 •

edited

Loading

piyushredpanda commented Oct 11, 2023

CI Failure (_topic_remote_deleted timeout) in TopicDeleteCloudStorageTest.topic_delete_installed_snapshots_test #8496

CI Failure (_topic_remote_deleted timeout) in TopicDeleteCloudStorageTest.topic_delete_installed_snapshots_test #8496

Comments

jcsp commented Jan 30, 2023 • edited Loading

jcsp commented Jan 30, 2023

jcsp commented Jan 31, 2023

rystsov commented Feb 3, 2023

graphcareful commented Feb 8, 2023

mmaslankaprv commented Feb 20, 2023

mmaslankaprv commented Feb 20, 2023

Scandiravian commented Feb 20, 2023

twmb commented Mar 10, 2023

dlex commented Mar 10, 2023

BenPope commented Mar 15, 2023 • edited Loading

BenPope commented Mar 15, 2023

BenPope commented May 8, 2023

BenPope commented May 8, 2023

dotnwat commented Jun 3, 2023

andijcr commented Jun 5, 2023 • edited Loading

andijcr commented Jun 7, 2023

andijcr commented Jun 9, 2023

michael-redpanda commented Jun 12, 2023

michael-redpanda commented Jun 12, 2023

ztlpn commented Jun 16, 2023

travisdowns commented Jun 19, 2023

travisdowns commented Jun 20, 2023

StephanDollberg commented Jun 26, 2023

piyushredpanda commented Jun 27, 2023

NyaliaLui commented Oct 11, 2023 • edited Loading

piyushredpanda commented Oct 11, 2023

CI Failure (_topic_remote_deleted timeout) in `TopicDeleteCloudStorageTest`.`topic_delete_installed_snapshots_test` #8496

CI Failure (_topic_remote_deleted timeout) in `TopicDeleteCloudStorageTest`.`topic_delete_installed_snapshots_test` #8496

jcsp commented Jan 30, 2023 •

edited

Loading

BenPope commented Mar 15, 2023 •

edited

Loading

andijcr commented Jun 5, 2023 •

edited

Loading

NyaliaLui commented Oct 11, 2023 •

edited

Loading